HN Explorer

Ask HN: What do you use for ML Hosting?

133 points by blululu ↗ HN
I'm trying to setup server to run ML inferences. I need to provision a somewhat beefy gpu with a decent amount of RAM (8-16 GB). Does anyone here have personal experience and recommendations about the various companies operating in this space?

68 comments

[ 3.2 ms ] story [ 138 ms ] thread
tikkun
For serverless: check the list I posted here https://news.ycombinator.com/item?id=34742087 (I ended up using Banana, it was fine)

For non-serverless, some to check out are these (though likely all overkill if you just need a single GPU)

https://www.coreweave.com/

vast.ai

Lambda labs

thundergolfer
How come you didn't end up using Modal, seeing at it was recommended in the only reply in the thread? [I'm a Modal person looking for insight :)]
smoldesu
I'm currently running a Discord bot with a 7B model off a free Oracle Ampere instance with their Pytorch Accelerated[0] image. It's not terribly fast, but totally usable for group chats that want to interrogate an AI. If you're doing some sort of offline processing or non-time-imperative operation, something like this might be worth looking into.

[0] https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

password4321
Does that use all 4 OCPUs / 24GB memory?
smoldesu
It can! I'm using 2 cores per request though, and I've got memory to spare.
aditya
what discord bot? :)
bootsmann
Oracles free forever tier is so underrated. They just throw half a startup at you at no cost.
mathewvp
They also shutdown your instances after a while and ask you to upgrade to paid tier.
tehsauce
Vast.ai Nobody has better prices.
mcsniff
Another vote for vast.ai, has been around quite a while and I've been using them for shell access to bare metal machines stuffed with GPUs, always had a decent experience.
jetml
Check out JetML.com (I'm the founder). Happy to help get you started with a demo if you want to reach out nick@jetml.com.
edunteman
Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use.

I'm at erik@banana.dev if you want any help with it :)

senko
Looking at your pricing, is that per seconds of GPU usage or per total seconds the app is running?

Eg. might have only a few minutes of usage in an hour and the rest of the time is spent waiting for requests. How's that billed?

edunteman
https://docs.banana.dev/banana-docs/core-concepts/billing You're only billed for active replica time. Call comes in, we start a replica, it handles the request, it waits around for a 10s (configurable) idle timeout to handle any additional calls, and shuts down if no other calls to serve. The idle timeout is to prevent cold boots when not necessary, but is billed, so you can get closer to pure pay-per-call pricing by reducing idle timeout.
jaflo
+1 on banana.dev, I used it for a side project and deployed some custom code and it was a good experience! I liked the pricing model (lack of minimums and pay for usage instead of a "plan") and how you can package up whatever code you want.
samstave
Kind of funny how cellphones went the opposite way - we all hated "paying for usage (minutes/txts))" and now we want just the Plan.
alpaca128
I still use that because I rarely use the phone without wifi and don't make a lot of calls. And because it's 2023 I can change to a paid plan for a month at any time in the app, so it's the best of both worlds.
samstave
I have the cheapest plan, 5GB, and an unlimited data Android tablet for a total of $30+$15=$45 for both...

I pay my cell bill a year in advance - then I never have to worry about a bill ever.

edunteman
Thanks for the +1!

Small note here: our billing is changing within the next month, to up-front payments that apply as a credit balance to your account. It still won't have minimums and you'll have the option to set up auto-refill on your balance, so it will functionally remain pay-as-you-go, but just wanted to add flavor to your comment on the pricing model.

Thanks for using us btw, you rock

aaronharnly
Looks great. It appears aimed at the inference use case rather than training, yes?
edunteman
yeah, we're optimizing the infra for realtime inference, though people definitely still do run training on us, with a weights upload implemented at the end of your handler.
silksowed
how were you able to tell this? still trying to understand what infra is better used for inference (say realtime image category matching) vs training (feeding a chatbot huge sums of data)
sjkoelle
if you want to host voice ML models, check out Uberduck.
outdoorblake
Banana.dev is what I use. The cold boots are fast
pj_mukh
If you’re using python, Modal (modal.com) was awesome to setup.

They’ll take a FastAPI setup too and just put it online to be used on demand.

psshank
Try www.salad.com. We've got 10k+ GPUs - from 8GB to 24GB. You get 10x more inferences per dollar compared to others. Our product team is pretty happy to help out on Discord. Some prices of interest. RTX 3060 - 12 GB - $0.08/hr RTX 3090 - 24 GB - $0.25/hr
thundergolfer
On Modal.com these 34 lines of code is all you need to serverlessly run BERT text generation inference on an A10G (which has 24GB of GPU memory). No Dockerfile, no YAML, no Terraform or AWS Cloudformation. Just these 34 lines.

  import modal

  def download_model():
      from transformers import pipeline
      pipeline("fill-mask", model="bert-base-uncased")

  CACHE_PATH = "/root/model_cache"  # model location in image
  ENV = modal.Secret({"TRANSFORMERS_CACHE": CACHE_PATH})

  image = (
      modal.Image.debian_slim()
      .pip_install("torch", "transformers")
      .run_function(download_model, secret=ENV)
  )
  stub = modal.Stub(name="hn-demo", image=image)


  class Model:
      def __enter__(self):
          from transformers import pipeline
          self.model = pipeline("fill-mask", model="bert-base-uncased", device=0)

      @stub.function(
          gpu="a10g",
          secret=ENV,
      )
      def handler(self, prompt: str):
          return self.model(prompt)


  if __name__ == "__main__":
      with stub.run():
          prompt = "Hello World! I am a [MASK] machine learning model."
          print(Model().handler.call(prompt)[0]["sequence"])
Running `python hn_demo.py` prints "Hello World! I am a simple machine learning model."

You can check out available GPUs at https://modal.com/docs/reference/modal.gpu.

There's also a bunch of easy-to-run examples in our docs :) https://modal.com/docs/guide/ex/stable_diffusion_cli

codeptualize
Love Modal. We use it for data processing, queues, apis, and all sorts of random things. Such a great product!
ArtWomb
Perfect sell. I'm signing-up just from this comment alone ;)
ronyfadel
How does Modal perform, latency-wise? Since it has to spin up an instance/GPU, does it take long to return results?
chpatrick
Modal's usability is amazing, I'm just a bit wary because they use AWS but somehow their prices are lower than that of AWS's GPU machines.
asadm
I have had good experience with Replicate and Runpod. Replicate seems to be nicer but has very bad cold boot issue. Runpod is great once you have an app set up!

I use mix of both for my side project: https://trainengine.ai

bfirsh
Founder of Replicate here. Looks like you're using DreamBooth for TrainEngine. We have a beta version of really fast cold boots for DreamBooth trainings. I'll drop you an email to get you set up with it.

We've got some big cold improvements rolling out across everything soon. We can also just keep models switched on to avoid cold boots entirely.

Rastonbury
Seconding the cold boot issue, hoping it's fixed I've had to look elsewhere because I can't have users waiting 3 min for a 5-15 second inference unexpectedly, I must say I do like the site and product!
sa-code
What about just using a cloud VM with an ansible script? I find ML deployment solutions to be very over engineered
version_five
My preference is not to have to change my code to use some special framework, and just get access to a gpu machine I can run my stuff on.

I'm assuming you know what you need for a GPU. If you're unsure, consider trying to run inferences on a CPU and see how long it takes and if it could work.

And then just look at price and reliability for a gpu machine with the different cloud providers. Ovh is cheap but the only thing worse than their reliability is their customer service. Various niche players offering V100s used to pop up that were pretty cheap. AWS is more expensive, more reliable, they may still have availability problems. Paperspace looks pretty good. Etc.

thundergolfer
> are worth avoiding so you don't get stuck with somebody else's framework.

Modal eng here. Modal is not setup as a framework. Think of more as Python-defined serverless infrastructure that has native support for the Python runtime. This is in some places called "Infrastructure from code", as opposed to "Infrastructure as code" which means just source-controlling K8s YAML and Cloudformation.

A major benefit of this approach is that the cloud becomes part of your dev loop, as opposed to doing `docker build`, `docker push`, `kubectl`, etc just to ship a change to a GPU.

In the script I posted Modal APIs are mixed in with standard Python code for brevity, but many customers just keep their code in their own modules and have a `modal_infra.py` module that defines the serverless infrastructure.

version_five
Understood, thanks for clarifying. I'll edit my post.
quickthrower2
Glad to see this concept being used. I was thinking the other day how odd it is to have IaC then keep the code siloed from the app code. Why doesn’t the app make it’s own infra? I think you have articulated how that would work.
naderkhalil
That makes sense, Brev.dev is a really simple way to run your code on a configured GPU without having to change your code. It'll also optimize your GPU to save money when possible.
jvanillaaaa
Brev.dev

This is exactly what you’re looking for

[deleted]
(comment deleted)
zitterbewegung
Have you tried self hosting? All you need is business internet with a static IP which is quite inexpensive and doing inference can be done on CPU depending what you want to perform inferfence. Also, wherever you are hosting a good rule of thumb is to have at least 1.5 times the amount of regular ram with your VRAM.
lordofgibbons
Do any of the "serverless"/saas model hosting services perform optimizations such as quantization or input micro-batching?
rgbrgb
Wow, looks like there's a ton of choices here I haven't looked at. For iterate.world we use replicate but just added kandinsky from runpod. Thinking about switching everything to runpod because it's 5-10x cheaper and we only use models that they have anyway.

There's one I won't share that's is now defunct but you could use any diffuser's compatible project on Hugging Face, which was such a cool feature. I wish someone (cheap) would implement this!

edit: just looked at banana.dev in this thread, their templates look closest to the HuggingFace integration though I don't think they have webhooks.

bfirsh
Founder of Replicate here. Also YC founder (W20). :)

It's also worth noting that we bill by the second for how long your prediction is running, and we don't bill for any idle time, so in practice Replicate works out cheaper for many workloads. We can give discounts if you're putting through a decent amount of traffic. We should be able to match Runpod's pricing.

Drop me an email: ben@replicate.com

edunteman
Hey! Banana founder here. Explicit webhook support coming out soon, though one could always add an http POST request to their webhook endpoint at the end of their handler to send the data that way rather than awaiting the results from the client. It'd take some customization, but you're into our templates, you can click the github icon in the UI to see the source repo, fork it, add the HTTP POST call at the end of the handler, and then deploy that to Banana as a custom repo.
_boffin_
I’m using a docker container on Ubuntu, which is on my home lab that’s an esxi 6.5 hypervisor. Going to be building a new machine with a few hundred GB of ram and then, at some point in the next 6 months, looking at getting a good GPU with a bunch of vRAM.

Wrapped the thing in a flask app so I can expose APIs I build out.