Do you think it can be done by adjusting GC aggressiveness (or even disabling it for short periods of time) and correlating it with execution time?
Are there plans to elucidate implicit GC costs as well?
I have a question about what "validation" means exactly. Does this process work by having the main model compute the "probability" that it would generate the draft sequence, then probabilistically accepting the draft?…
If the workload were perfectly parallelizable, your claim would be true. However, if it has serial dependency chains, it is absolutely worth it to compute it quickly and unreliably and verify in parallel
Does the lack of forward progress guarantees (ITS) on other architectures pose challenges for async/await?
Is this Nvidia-only or does it work on other architectures?
Do you think it can be done by adjusting GC aggressiveness (or even disabling it for short periods of time) and correlating it with execution time?
Are there plans to elucidate implicit GC costs as well?
I have a question about what "validation" means exactly. Does this process work by having the main model compute the "probability" that it would generate the draft sequence, then probabilistically accepting the draft?…
If the workload were perfectly parallelizable, your claim would be true. However, if it has serial dependency chains, it is absolutely worth it to compute it quickly and unreliably and verify in parallel
Does the lack of forward progress guarantees (ITS) on other architectures pose challenges for async/await?
Is this Nvidia-only or does it work on other architectures?