Using the free playground link, and it is in fact extremely fast. The "diffusion mode" toggle is also pretty neat as a visualization, although I'm not sure how accurate it is - it renders as line noise and then refines, while in reality presumably those are tokens from an imprecise vector in some state space that then become more precise until it's only a definite word, right?
Some text diffusion models use continuous latent space but they historically haven't done that well. Most the ones we're seeing now typically are trained to predict actual token output that's fed forward into the next time series. The diffusion property comes from their ability to modify previous timesteps to converge on the final output.
I'm kind of impressed by the speed of it. I told it to write a MQTT topic pattern matcher based on a Trie and it spat out something reasonable on first try. It hat a few compilation issues though, but fair enough.
A good chance to bring up something I've been flagging to colleagues for a while now: with LLM agents we are very quickly going to become even more CPU bottlenecked on testing performance than today, and every team I know of today was bottlenecked on CI speed even before LLMs. There's no point having an agent that can write code 100x faster than a human if every change takes an hour to test.
Maybe I've just got unlucky in the past, but in most projects I worked on a lot of developer time was wasted on waiting for PRs to go green. Many runs end up bottlenecked on I/O or availability of workers, and so changes can sit in queues for hours, or they flake out and everything has to start again.
As they get better coding agents are going to be assigned simple tickets that they turn into green PRs, with the model reacting to test failures and fixing them as they go. This will make the CI bottleneck even worse.
It feels like there's a lot of low hanging fruit in most project's testing setups, but for some reason I've seen nearly no progress here for years. It feels like we kinda collectively got used to the idea that CI services are slow and expensive, then stopped trying to improve things. If anything CI got a lot slower over time as people tried to make builds fully hermetic (so no inter-run caching), and move them from on-prem dedicated hardware to expensive cloud VMs with slow IO, which haven't got much faster over time.
Mercury is crazy fast and in a few quick tests I did, created good and correct code. How will we make test execution keep up with it?
Good God I hate CI. Just let me run the build automation myself dammit! If you're worried about reproducibility make it reproducible and hash the artifacts, make people include the hash in the PR comment if you want to enforce it.
The amount of time people waste futzing around in eg Groovy is INSANE and I'm honestly inclined to reject job offers from companies that have any serious CI code at this point.
In most companies the CI/Dev Tools team is a career dead end. There is no possibility to show a business impact, it's just a money pit that leadership can't/won't understand (and if they do start to understand it, then it becomes _their_ money pit, which is a career dead end for them) So no one who has their head on straight wants to spend time improving it.
And you can't even really say it's a short sighted attitude. It definitely is from a developer's perspective, and maybe it is for the company if dev time is what decides the success of the business overall.
Yet, now I have added a LLM workflow to my coding the value of my old and mostly useless workflows is now 10x'd.
Git checkpoints, code linting and my naive suite of unit and integration tests are now crucial to my LLM not wasting too much time generating total garbage.
It’s because people don’t know how to write tests. All of the “don’t do N select queries in a for loop” comments made in PRs are completely ignored in tests.
Each test can output many db queries. And then you create multiple cases.
People don’t even know how to write code that just deals with N things at a time.
I am confident that tests run slowly because the code that is tested completely sucks and is not written for batch mode.
Ignoring batch mode, tests are most of the time written in a a way where test cases are run sequentially. Yet attempts to run them concurrently result in flaky tests, because the way you write them and the way you design interfaces does not allow concurrent execution at all.
Another comment, code done by the best AI model still sucks. Anything simple, like a music player with a library of 10000 songs is something it can’t do. First attempt will be horrible. No understanding of concurrent metadata parsing, lists showing 10000 songs at once in UI being slow etc.
So AI is just another excuse for people writing horrible code and horrible tests. If it’s so smart , try to speed up your CI with it.
I agree. I think there are potentially multiple solutions to this since there are multiple bottlenecks. The most obvious is probably network overhead when talking to a database. Another might be storage overhead if storage is being used.
Frankly another one is language. I suspect type-safe, compiled, functional languages are going to see some big advantages here over dynamic interpreted languages. I think this is the sweet spot that grants you a ton of performance over dynamic languages, gives you more confidence in the models changes, and requires less testing.
Faster turn-around, even when you're leaning heavily on AI, is a competitive advantage IMO.
> If anything CI got a lot slower over time as people tried to make builds fully hermetic (so no inter-run caching), and move them from on-prem dedicated hardware to expensive cloud VMs with slow IO, which haven't got much faster over time.
I am guesstimating (based on previous experience self-hosting the runner for MacOS builds) that the project I am working on could get like 2-5x pipeline performance at 1/2 cost just by using self-hosted runners on bare metal rented machines like Hetzner. Maybe I am naive, and I am not the person that would be responsible for it - but having a few bare metal machines you can use in the off hours to run regression tests, for less than you are paying the existing CI runner just for build, that speed up everything massively seems like a pure win for relatively low effort. Like sure everyone already has stuff on their plate and would rather pay external service to do it - but TBH once you have this kind of compute handy you will find uses anyway and just doing things efficiently. And knowing how to deal with bare metal/utilize this kind of compute sounds generally useful skill - but I rarely encounter people enthusiastic about making this kind of move. Its usually - hey lets move to this other service that has slightly cheaper instances and a proprietary caching layer so that we can get locked into their CI crap.
Its not like these services have 0 downtime/bug free/do not require integration effort - I just don't see why going bare metal is always such a taboo topic even for simple stuff like builds.
Before cars people spent little on petroleum products or motor oil or gasoline or mechanics. Now they do. That's how systems work. You wanna go faster well you need better roads, traffic lights, on ramps, etc. you're still going faster.
Use AI to solve the IP bottlenecks or build more features that ear more revenue that buy more ci boxes. Same as if you added 10 devs which you are with AI so why wouldn't some of the dev support costs go up.
Are you not in a place where you can make an efficiency argument to get more ci or optimize? What's a ci box cost?
1. As implementation phase gets faster, the bottleneck could actually switch to PM. In which case, changes will be more serial, so a lot fewer conflicts to worry about.
2. I think we could see a resurrection of specs like TLA+. Most engineers don't bother with them, but I imagine code agents could quickly create them, verify the code is consistent with them, and then require fewer full integration tests.
3. When background agents are cleaning up redundant code, they can also clean up redundant tests.
4. Unlike human engineering teams, I expect AIs to work more efficiently on monoliths than with distributed microservices. This could lead to better coverage on locally runnable tests, reducing flakes and CI load.
5. It's interesting that even as AI increases efficiency, that increased velocity and sheer amount of code it'll write and execute for new use cases will create its own problems that we'll have to solve. I think we'll continue to have new problems for human engineers to solve for quite some time.
Wow, your story gives me flashbacks to the 1990s when I worked in a mainframe environment. Compile jobs submitted by developers were among the lowest priorities. I could make a change to a program, submit a compile job, and wait literally half a day for it to complete. Then I could run my testing, which again might have to wait for hours. I generally had other stuff I could work on during those delays but not always.
>There's no point having an agent that can write code 100x faster than a human if every change takes an hour to test.
Testing every change incrementally is a vestige of the code being done by humans (and thus of the current approach where AI helps and/or replaces one given human), in small increments at that, and of the failures being analyzed by individual humans who can keep in their head only limited number of things/dependencies at once.
> Maybe I've just got unlucky in the past, but in most projects I worked on a lot of developer time was wasted on waiting for PRs to go green. Many runs end up bottlenecked on I/O or availability of workers
No, this is common. The devs just haven't grokked dependency inversion. And I think the rate of new devs entering the workforce will keep it that way forever.
Here's how to make it slow:
* Always refer to "the database". You're not just storing and retrieving objects from anywhere - you're always using the database.
* Work with statements, not expressions. Instead of "the balance is the sum of the transactions", execute several transaction writes (to the database) and read back the resulting balance. This will force you to sequentialise the tests (simultaneous tests would otherwise race and cause flakiness) plus you get to write a bunch of setup and teardown and wipe state between tests.
* If you've done the above, you'll probably need to wait for state changes before running an assertion. Use a thread sleep, and if the test is ever flaky, bump up the sleep time and commit it if the test goes green again.
For Python apps, I've gotten good CI speedups by moving over to the astral.sh toolchain, using uv for the package installation with caching. Once I move to their type-checker instead of mypy, that'll speed the CI up even more. The playwright test running will then probably be the slowest part, and that's only in apps with frontends.
(Also, Hi Mike, pretty sure I worked with you at Google Maps back in early 2000s, you were my favorite SRE so I trust your opinion on this!)
The nice part about most CI workloads is that they can almost always be split up and executed in parallel. Make sure you're utilizing every core on every CI worker and your worker pools are appropriately sized for the workload. Use spot instances and add auto scaling where it makes sense. No one should be waiting more than a few minutes for a PR build. Exception being compile time which can vary significantly between languages. I have a couple projects that are stuck on ancient compilers because of CPU architecture and C variant, so those will always be a dog without effort to move to something better. Ymmv
If I am coding, I want to stay in the flow and get my PR green asap, so I can continue on the project.
If I am orchestrating agents, I might have 10 or 100 PRs in the oven. In that case I just look at the ones that finish CI.
It’s gonna be less, or at least different, kind of flow IMO. (Until you can just crank out design docs and whiteboard sessions and have the agents fully autonomously get their work green.)
Any modern MacBook can run those tests 100x faster than the crappy cloud runners most companies use. You can also configure runners that run locally and get the benefit of those speed gains. So all of this is really a business and technical problem that is solved for those who want to solve it. It can be solved very cheap, or it can be solved very expensive. Regardless, it's precisely those types of efficiency gains that motivate companies to finally do something about it.
And if not, then enjoy being paid waiting for CI to go green. Maybe it's a reminder to go take a break.
It will be worse when the process is super optimized and the expectation changes. So now instead of those 2 PRs that went to prod today because everyone knows CI takes forever, you'll be expected to push 8 because in our super optimized pipeline it only takes seconds. No excuses. Now the bottleneck is you.
- Just spin up more test instances. If the AI is as good as people claim then it's still way cheaper than extra programmers.
- Write fast code. At $WORK we can test roughly a trillion things per CPU physical core year for our primary workload, and that's in a domain where 20 microsecond processing time is unheard of. Orders of magnitude speed improvements pay dividends quickly.
- LLMs don't care hugely about the language. Avoid things like rust where compile times are always a drag.
- That's something of a strange human problem you're describing. Once the PR is reviewed, can't you just hit "auto-merge" and go to the next task, only circling back if the code was broken? Why is that a significant amount of developer time?
- The thing you're observing is something every growing team witnesses. You can get 90% of the way to what you want by giving the build system a greenfield re-write. If you really have to run 100x more tests, it's worth a day or ten sanity checking docker caching or whatever it is your CI/CD is using. Even hermetic builds have inter-run caching in some form; it's just more work to specify how the caches should work. Put your best engineer on the problem. It's important.
- Be as specific as possible in describing test dependencies. The fastest tests are the ones which don't run.
- Separate out unit tests from other forms of tests. It's hard to write software operating with many orders of magnitude of discrepancies, and tests are no exception. Your life is easier if conceptually they have a separate budget (e.g., continuous fuzz testing or load testing or whatever). Unit tests can then easily be fast enough for a developer to run all the changed ones on precommit. Slower tests are run locally when you think they might apply. The net effect is that you don't have the sort of back-and-forth with your CI that actually causes lost developer productivity because the PR shouldn't have a bunch of bullshit that's green locally and failing remotely.
CI should just run on each developer's machine. As in, each developer should have a local instance of the CI setup in a VM or a docker container. If tests pass, the result is reported to a central server.
We write and run tests to build trust in our code changes. But maybe tests aren’t the only way to achieve that trust.
When I was younger, I had a friend who was a senior software engineer. I remember he would make changes to production systems without even running the application locally or executing any tests, and yet his changes never failed. The team had a high level of trust in all his code changes.
We don’t. We switch to proven-correct code. Languages like Lean, Coq, and Idris allow proofs of correctness for code. The LLM can generate proofs for most of the correctness conditions.
CI is still needed for performance, UI testing, etc. but it can have a much smaller role than it does now.
One of the core premises of what we've been trying to do with our product (Testkube) is to decouple Testing from CI/CD's. Those were never built with testing in mind, let alone scaling to 100's or 1000's of efficient executions.
We have a light weight open-source agent, which lives inside a K8s cluster, tests are stored as CRD's cloned from your GIT, executed as K8's jobs. Create whatever heuristics or parallelization necessary, leverage the power of K8s to dynamically scale compute resources as needed, trigger executions by whatever means (GitHub Actions, K8s' events, schedule, etc.), do it on your existing infra.
Admittedly, we don't solve the test creation problem. If there are new tools out there which could automagically generate tests along with code, please share.
ICYMI, DeepMind also has a Gemini model that is diffusion-based[1]. I've tested it a bit and while (like with this model) the speed is indeed impressive, the quality of responses was much worse than other Gemini models in my testing.
The speed here is super impressive! I am curious - are there any qualitative ways in which modeling text using diffusion differs from that using autoregressive models? The kind of problems it works better on, creativity, and similar.
One works in the coarse-to-fine direction, another works start-to-end. Which means different directionality biases, at least. Difference in speed, generalization, etc. is less clear and needs to be proven in practice, as fundamentally they are closer than it seems. Diffusion models have some well-studied shortcuts to trade speed for quality, but nothing stops you from implementing the same for the other type.
> I strongly believe that this will be a really important technique in the near future.
I share the same belief, but regardless of cost. What excites me is the ability to "go both ways", edit previous tokens after others have been generated, using other signals as "guided generation", and so on. Next token prediction works for "stories", but diffusion matches better with "coding flows" (i.e. going back and forth, add something, come back, import something, edit something, and so on).
It would also be very interesting to see how applying this at different "abstraction layers" would work. Say you have one layer working on ctags, one working on files, and one working on "functions". And they all "talk" to each other, passing context and "re-diffusing" their respective layers after each change. No idea where the data for this would come, maybe from IDEs?
I, for one, am willing to trade accuracy for speed. I'd rather have 10 iterations of poor replies which forces me to ask the right question than 1 reply which takes 10 times as long and _maybe_ is good, since it tries to reason about my poor question.
I was curious to know the statistics on the mentions of various programming languages on HN over the years, so I got me a copy of all HN comments from a BigTable public source. But now I need to interpret each comment and so what I need is a semantic grep. The easiest would be to prompt an LLM.
Comments are pretty short, but there are many millions of them. So getting high throughput at minimum cost is key.
I'm hoping that Inception might be able to churn through this quickly.
If you folks have other ideas or suggestions, what might also work, I'd love to hear them!
The idea is having a semgrep command line tool. If latencies are dropping dramatically, it might be feasible.
Damn, that is fast. But it is faster than I can read, so hopefully they can use that speed and turn it into better quality of the output. Because otherwise, I honestly don't see the advantage, in practical terms, over existing LLMs. It's like having a TV with a 200Hz refresh rate, where 100Hz is just fine.
I've used mercury quite a bit in my commit message generator. I noticed it would always produce the exact same response if you ran it multiple times, and increasing temperature didn't affect it. To get some variability I added a $(uuidgen) to the prompt. Then I could run it again for a new response if I didn't like the first.
I wonder if diffusion llms solve the hallucination problem more effectively. In the same way that image models learned to create less absurd images, dllms can perhaps learn to create sensical responses more predictably
Having token embeddings with diffusion models, for 16x16 transformer encoding. Image is tokenized before transformers compile it. If decomposed virtualization modulates according to a diffusion model.
I guess this makes specific language patterns cheaper and more artistic language patterns more expensive. This could be a good way to limit pirated and masqueraded materials submitted by students.
I tried the playground and got a strange response. I asked for a regex pattern, and the model gave itself a little game-plan, then it wrote the pattern and started to write tests for it. But it never stopped writing tests. It continued to write tests of increasing size until I guess it reached a context limit and the answer was canceled. Also, for each test it wrote, it added a comment about if the test should pass or fail, but after about the 30th test, it started giving the wrong answer for those too, saying that a test should fail when actually it should pass if the pattern is correct. And after about the 120th test, the tests started to not even make sense anymore. They were just nonsense characters until the answer got cut off.
The pattern it made was also wrong, but I think the first issue is more interesting.
85 comments
[ 38.6 ms ] story [ 624 ms ] threadI have an explanation about one of these recent architectures that seems similar to what Mercury is doing under the hood here: https://pierce.dev/notes/how-text-diffusion-works/
Maybe I've just got unlucky in the past, but in most projects I worked on a lot of developer time was wasted on waiting for PRs to go green. Many runs end up bottlenecked on I/O or availability of workers, and so changes can sit in queues for hours, or they flake out and everything has to start again.
As they get better coding agents are going to be assigned simple tickets that they turn into green PRs, with the model reacting to test failures and fixing them as they go. This will make the CI bottleneck even worse.
It feels like there's a lot of low hanging fruit in most project's testing setups, but for some reason I've seen nearly no progress here for years. It feels like we kinda collectively got used to the idea that CI services are slow and expensive, then stopped trying to improve things. If anything CI got a lot slower over time as people tried to make builds fully hermetic (so no inter-run caching), and move them from on-prem dedicated hardware to expensive cloud VMs with slow IO, which haven't got much faster over time.
Mercury is crazy fast and in a few quick tests I did, created good and correct code. How will we make test execution keep up with it?
The amount of time people waste futzing around in eg Groovy is INSANE and I'm honestly inclined to reject job offers from companies that have any serious CI code at this point.
And you can't even really say it's a short sighted attitude. It definitely is from a developer's perspective, and maybe it is for the company if dev time is what decides the success of the business overall.
these redundant processes are for human interoperability
Git checkpoints, code linting and my naive suite of unit and integration tests are now crucial to my LLM not wasting too much time generating total garbage.
Each test can output many db queries. And then you create multiple cases.
People don’t even know how to write code that just deals with N things at a time.
I am confident that tests run slowly because the code that is tested completely sucks and is not written for batch mode.
Ignoring batch mode, tests are most of the time written in a a way where test cases are run sequentially. Yet attempts to run them concurrently result in flaky tests, because the way you write them and the way you design interfaces does not allow concurrent execution at all.
Another comment, code done by the best AI model still sucks. Anything simple, like a music player with a library of 10000 songs is something it can’t do. First attempt will be horrible. No understanding of concurrent metadata parsing, lists showing 10000 songs at once in UI being slow etc.
So AI is just another excuse for people writing horrible code and horrible tests. If it’s so smart , try to speed up your CI with it.
I agree. I think there are potentially multiple solutions to this since there are multiple bottlenecks. The most obvious is probably network overhead when talking to a database. Another might be storage overhead if storage is being used.
Frankly another one is language. I suspect type-safe, compiled, functional languages are going to see some big advantages here over dynamic interpreted languages. I think this is the sweet spot that grants you a ton of performance over dynamic languages, gives you more confidence in the models changes, and requires less testing.
Faster turn-around, even when you're leaning heavily on AI, is a competitive advantage IMO.
I am guesstimating (based on previous experience self-hosting the runner for MacOS builds) that the project I am working on could get like 2-5x pipeline performance at 1/2 cost just by using self-hosted runners on bare metal rented machines like Hetzner. Maybe I am naive, and I am not the person that would be responsible for it - but having a few bare metal machines you can use in the off hours to run regression tests, for less than you are paying the existing CI runner just for build, that speed up everything massively seems like a pure win for relatively low effort. Like sure everyone already has stuff on their plate and would rather pay external service to do it - but TBH once you have this kind of compute handy you will find uses anyway and just doing things efficiently. And knowing how to deal with bare metal/utilize this kind of compute sounds generally useful skill - but I rarely encounter people enthusiastic about making this kind of move. Its usually - hey lets move to this other service that has slightly cheaper instances and a proprietary caching layer so that we can get locked into their CI crap.
Its not like these services have 0 downtime/bug free/do not require integration effort - I just don't see why going bare metal is always such a taboo topic even for simple stuff like builds.
Use AI to solve the IP bottlenecks or build more features that ear more revenue that buy more ci boxes. Same as if you added 10 devs which you are with AI so why wouldn't some of the dev support costs go up.
Are you not in a place where you can make an efficiency argument to get more ci or optimize? What's a ci box cost?
1. As implementation phase gets faster, the bottleneck could actually switch to PM. In which case, changes will be more serial, so a lot fewer conflicts to worry about.
2. I think we could see a resurrection of specs like TLA+. Most engineers don't bother with them, but I imagine code agents could quickly create them, verify the code is consistent with them, and then require fewer full integration tests.
3. When background agents are cleaning up redundant code, they can also clean up redundant tests.
4. Unlike human engineering teams, I expect AIs to work more efficiently on monoliths than with distributed microservices. This could lead to better coverage on locally runnable tests, reducing flakes and CI load.
5. It's interesting that even as AI increases efficiency, that increased velocity and sheer amount of code it'll write and execute for new use cases will create its own problems that we'll have to solve. I think we'll continue to have new problems for human engineers to solve for quite some time.
Testing every change incrementally is a vestige of the code being done by humans (and thus of the current approach where AI helps and/or replaces one given human), in small increments at that, and of the failures being analyzed by individual humans who can keep in their head only limited number of things/dependencies at once.
However, improving CI performance is valuable regardless.
GPUs can do 1 million trillion instructions per second.
Are you saying it’s impossible to write a test that finishes in less than one second on that machine?
Is that a fundamental limitation or an incredibly inefficient test?
No, this is common. The devs just haven't grokked dependency inversion. And I think the rate of new devs entering the workforce will keep it that way forever.
Here's how to make it slow:
* Always refer to "the database". You're not just storing and retrieving objects from anywhere - you're always using the database.
* Work with statements, not expressions. Instead of "the balance is the sum of the transactions", execute several transaction writes (to the database) and read back the resulting balance. This will force you to sequentialise the tests (simultaneous tests would otherwise race and cause flakiness) plus you get to write a bunch of setup and teardown and wipe state between tests.
* If you've done the above, you'll probably need to wait for state changes before running an assertion. Use a thread sleep, and if the test is ever flaky, bump up the sleep time and commit it if the test goes green again.
(Also, Hi Mike, pretty sure I worked with you at Google Maps back in early 2000s, you were my favorite SRE so I trust your opinion on this!)
If I am coding, I want to stay in the flow and get my PR green asap, so I can continue on the project.
If I am orchestrating agents, I might have 10 or 100 PRs in the oven. In that case I just look at the ones that finish CI.
It’s gonna be less, or at least different, kind of flow IMO. (Until you can just crank out design docs and whiteboard sessions and have the agents fully autonomously get their work green.)
And if not, then enjoy being paid waiting for CI to go green. Maybe it's a reminder to go take a break.
It will be worse when the process is super optimized and the expectation changes. So now instead of those 2 PRs that went to prod today because everyone knows CI takes forever, you'll be expected to push 8 because in our super optimized pipeline it only takes seconds. No excuses. Now the bottleneck is you.
- Write fast code. At $WORK we can test roughly a trillion things per CPU physical core year for our primary workload, and that's in a domain where 20 microsecond processing time is unheard of. Orders of magnitude speed improvements pay dividends quickly.
- LLMs don't care hugely about the language. Avoid things like rust where compile times are always a drag.
- That's something of a strange human problem you're describing. Once the PR is reviewed, can't you just hit "auto-merge" and go to the next task, only circling back if the code was broken? Why is that a significant amount of developer time?
- The thing you're observing is something every growing team witnesses. You can get 90% of the way to what you want by giving the build system a greenfield re-write. If you really have to run 100x more tests, it's worth a day or ten sanity checking docker caching or whatever it is your CI/CD is using. Even hermetic builds have inter-run caching in some form; it's just more work to specify how the caches should work. Put your best engineer on the problem. It's important.
- Be as specific as possible in describing test dependencies. The fastest tests are the ones which don't run.
- Separate out unit tests from other forms of tests. It's hard to write software operating with many orders of magnitude of discrepancies, and tests are no exception. Your life is easier if conceptually they have a separate budget (e.g., continuous fuzz testing or load testing or whatever). Unit tests can then easily be fast enough for a developer to run all the changed ones on precommit. Slower tests are run locally when you think they might apply. The net effect is that you don't have the sort of back-and-forth with your CI that actually causes lost developer productivity because the PR shouldn't have a bunch of bullshit that's green locally and failing remotely.
When I was younger, I had a friend who was a senior software engineer. I remember he would make changes to production systems without even running the application locally or executing any tests, and yet his changes never failed. The team had a high level of trust in all his code changes.
CI is still needed for performance, UI testing, etc. but it can have a much smaller role than it does now.
One of the core premises of what we've been trying to do with our product (Testkube) is to decouple Testing from CI/CD's. Those were never built with testing in mind, let alone scaling to 100's or 1000's of efficient executions. We have a light weight open-source agent, which lives inside a K8s cluster, tests are stored as CRD's cloned from your GIT, executed as K8's jobs. Create whatever heuristics or parallelization necessary, leverage the power of K8s to dynamically scale compute resources as needed, trigger executions by whatever means (GitHub Actions, K8s' events, schedule, etc.), do it on your existing infra.
Admittedly, we don't solve the test creation problem. If there are new tools out there which could automagically generate tests along with code, please share.
[1] https://deepmind.google/models/gemini-diffusion/
US$0.000001 per output token ($1/M tokens)
US$0.00000025 per input token ($0.25/M tokens)
https://platform.inceptionlabs.ai/docs#models
I share the same belief, but regardless of cost. What excites me is the ability to "go both ways", edit previous tokens after others have been generated, using other signals as "guided generation", and so on. Next token prediction works for "stories", but diffusion matches better with "coding flows" (i.e. going back and forth, add something, come back, import something, edit something, and so on).
It would also be very interesting to see how applying this at different "abstraction layers" would work. Say you have one layer working on ctags, one working on files, and one working on "functions". And they all "talk" to each other, passing context and "re-diffusing" their respective layers after each change. No idea where the data for this would come, maybe from IDEs?
Comments are pretty short, but there are many millions of them. So getting high throughput at minimum cost is key.
I'm hoping that Inception might be able to churn through this quickly.
If you folks have other ideas or suggestions, what might also work, I'd love to hear them!
The idea is having a semgrep command line tool. If latencies are dropping dramatically, it might be feasible.
The pattern it made was also wrong, but I think the first issue is more interesting.