I'm curious if the 16ms budget is actually driven by the token stream here. Even with the fastest providers right now, you are looking at maybe 100-150 tokens per second. That is orders of magnitude below the 30MB/s terminal parsing limit mentioned in the article.
Unless I am missing something about how they handle the diffs, the bottleneck is surely the inference latency and not the render loop. It seems like a lot of architectural complexity for a data stream that is inherently slow.
3 comments
[ 2.7 ms ] story [ 16.4 ms ] threadim not on twitter much these days, but damn people were not kind to anthropic
Unless I am missing something about how they handle the diffs, the bottleneck is surely the inference latency and not the render loop. It seems like a lot of architectural complexity for a data stream that is inherently slow.