We (the Modelplane OSS maintainer team) wrote an open guide on inference engineering based on our experience running open-source frontier models at scale with references to some of the best reading materials shared in the industry. Check it out, ask questions, discuss and contribute!
The Case study for me was the most interesting part; it shows that you really have to check all the different options to get to your optimal serving stack, and that there's no single best-solution to your configuration - each workload, traffic and hardware require their own unique mix of engine+configuration.
2 comments
[ 4.5 ms ] story [ 13.8 ms ] threadThe Case study for me was the most interesting part; it shows that you really have to check all the different options to get to your optimal serving stack, and that there's no single best-solution to your configuration - each workload, traffic and hardware require their own unique mix of engine+configuration.
BTW, the link doesn't work - I think you submitted https://docs.modelplane.ai/main/guides/inference_guide instead of https://docs.modelplane.ai/main/guides/inference-guide.