Ask HN: Cheaper way to do model inference?

2 points by masterofall2612 2y ago ↗ HN

Does anyone know of any solutions for saving GPU compute during server downtime? Is there a managed solution to turn off a pod and turn it back on when I need it? I'm currently doing model inference and most of the time I'm just paying for compute without serving any user requests.

3 comments

[ 5.8 ms ] story [ 30.3 ms ] thread

PaulHoule 2y ago ↗

You are running inference on something like an EC2 instance?

masterofall2612 2y ago ↗

Yea exactly that's what I'm doing right now. I've looked into solutions like RunPod serverless, anyone have any experience with it?

brianjking 2y ago ↗

Huggingface Inference Endpoints can autoscale to 0 and cost nothing when not being used.