Looks interesting! How about models that require dictionaries - e.g. tf-idf to convert text into a feature vector? Does it allow for some preprocessing?
Hi! If don’t need any pre processing we have built in support for all the major librairies.
In addition we support custom pre and post processor via custom environments! Simply write your inference code into a predict.py and we take care of the rest.
Is this able to support more than 50 requests per second? Are there any benchmarks on performance overhead of the underlying web server/routing that is handling the requests?
I see that you accept models up to 1 GB. It seems the inference time might be high for models of this size on CPUs. Do you use GPUs to speed up inference for deep learning models ?
11 comments
[ 3.3 ms ] story [ 33.0 ms ] threadIn addition we support custom pre and post processor via custom environments! Simply write your inference code into a predict.py and we take care of the rest.
(Btw I am really into CML.dev, great idea)
If you have a more custom pipeline, we have a custom environment where you can deploy any custom code with specific package versions!
We stopped at 50 request/s in the pricing table because that seemed like a reasonable number for most use cases.