Lots of emphasis on no kernels written and nothing recompiled but the intro and conclusion somehow suggest this is also the value of runinfra’s agents the rewrite software ?
Walk through the theory of peak decode throughout based on memory bandwidth with one weight load pass per token to provide a target for optimized decide but then doesn’t address low bandwidth utilization in that scenario at all and uses speculative decode where that theory doesn’t apply
1 comment
[ 0.30 ms ] story [ 18.0 ms ] threadLots of emphasis on no kernels written and nothing recompiled but the intro and conclusion somehow suggest this is also the value of runinfra’s agents the rewrite software ?
Walk through the theory of peak decode throughout based on memory bandwidth with one weight load pass per token to provide a target for optimized decide but then doesn’t address low bandwidth utilization in that scenario at all and uses speculative decode where that theory doesn’t apply