Super stoked to finally release this on HN. Looking forward to everyone's feedback!
You're totally right and we are doing a mixed batch. What we changed was the priority of performing prefills over decodes. When looking at a variety of workloads, we realized that prioritizing finishing a query…
If you deploy our solution on-prem, you would be able to handle 2x the workload on the same amount of hardware. This ensures you scale up your hardware 2x slower, giving you a ~50% reduction in your GPU Infrastructure…
1. Yes that's correct to some degree. Depending on the model details we might need to do some manual tweaking to get everything up and running, but generally we can get a model up within a day. There's always…
Thanks for the feedback and the specific model suggestion! Compared to the replicate/modal solutions our big focus is to ensure you don't experience rate limits. We want to ensure you get a good quality of service no…
Thanks for the suggestions! This is definitely something we'll be looking at. We're currently working on providing a more extensive interface to show users a variety of performance metrics of the models they're running.…
Thanks for letting us know, we've updated it now! Although we're currently only supporting text models, we do definitely have image and video generation models in our roadmap as these are very compute intensive models…
That's a great question, but its hard to get enough insight into how Groq is serving models to properly know what's missing. If I had to hazard a guess, it would be that their system architecture (# of chips and chip…
Hey, that links corresponds to the private models list that only works once you've created an account. If you'd like to see the public models page, please check it out here: https://console.ncompass.tech/public-models.…
Hey, I'm one of the co-founders, thanks for letting us know! I've just run it a few times and seems fine on my end. It does take a second to load the models, but feel free to let me know if this persists.
Super stoked to finally release this on HN. Looking forward to everyone's feedback!
You're totally right and we are doing a mixed batch. What we changed was the priority of performing prefills over decodes. When looking at a variety of workloads, we realized that prioritizing finishing a query…
If you deploy our solution on-prem, you would be able to handle 2x the workload on the same amount of hardware. This ensures you scale up your hardware 2x slower, giving you a ~50% reduction in your GPU Infrastructure…
1. Yes that's correct to some degree. Depending on the model details we might need to do some manual tweaking to get everything up and running, but generally we can get a model up within a day. There's always…
Thanks for the feedback and the specific model suggestion! Compared to the replicate/modal solutions our big focus is to ensure you don't experience rate limits. We want to ensure you get a good quality of service no…
Thanks for the suggestions! This is definitely something we'll be looking at. We're currently working on providing a more extensive interface to show users a variety of performance metrics of the models they're running.…
Thanks for letting us know, we've updated it now! Although we're currently only supporting text models, we do definitely have image and video generation models in our roadmap as these are very compute intensive models…
That's a great question, but its hard to get enough insight into how Groq is serving models to properly know what's missing. If I had to hazard a guess, it would be that their system architecture (# of chips and chip…
Hey, that links corresponds to the private models list that only works once you've created an account. If you'd like to see the public models page, please check it out here: https://console.ncompass.tech/public-models.…
Hey, I'm one of the co-founders, thanks for letting us know! I've just run it a few times and seems fine on my end. It does take a second to load the models, but feel free to let me know if this persists.