Efficient LLM Scheduling by Learning to Rank (hao-ai-lab.github.io)

2 points by zhisbug 1y ago ↗ HN

1 comment

[ 3.1 ms ] story [ 14.3 ms ] thread

We study how to approximate the famous shortest-job-first scheduling in LLM inference!