Launch HN: Sweep (YC S23) – A bot to create simple PRs in your codebase (github.com)
Kevin and I met while working at Roblox. We talked to our friends who were junior developers and noticed a lot of them doing grunt work. We wanted to let them focus on important work. Copilot is great, but we realized some tasks could be completely offloaded to an AI (e.g. adding a banner to your webpage https://github.com/sweepai/landing-page/issues/225).
Sweep does this with a code search engine. We use code chunking, ranking, and formatting tricks to represent your codebase in a token-efficient manner for LLMs. You might have seen our blog on code chunking here: https://news.ycombinator.com/item?id=36948403.
We take these fetched code snippets and come up with a plan to write the PR. We found that having the LLM provide structured information using XML tags is very robust, as it’s easy for us to parse with regex, has good support for multi-line answers and is hard for the LLM to mess up.
This is because XML is common in the LLM’s training data (the internet / HTML), and the opening and closing tags rarely appear naturally in text and code, unlike the quotations, brackets, backticks and newlines used by JSON’s and markdown’s delimiters. Further, XML lets you skip the preamble (“This question has to do with xyz. Here is my answer:”) and handles multi-line answers like PR plans and code really well. For example, we ask the LLM for the new code in <new_code> tags and a final boolean answer by writing <answer>True</answer>.
We use this XML format to get the LLM to create a plan, generating a list of files to create and modify from the retrieved relevant files. We iterate through the file changes and edit/create the necessary files. Finally, we push the commits to GitHub and create the PR.
We’ve been using Sweep to handle small issues in Sweep’s own repo (it recently passed 100 commits). We’ve become well acquainted with its limitations. For example, Sweep sometimes leave unimplemented functions with just “# rest of code” since it runs on GPT-4, a model tuned for chatting. Other times, there’s minor syntax errors or undefined variables. This is why we spend the other half of our time building self-recovery methods for Sweep to fix and test its PRs.
First, we invite the developer to review and add comments to Sweep’s pull request. This helps to a point, but Sweep’s code sometimes wouldn’t lint. This is table stakes. It’s frustrating to have to tell the bot to “add an import here” or “this variable is undefined”. To make this better, we used GitHub Actions, which automatically runs the flow of “check the code → tell sweep → sweep fixes the code → check the code again”. We like this flow because you might already have GitHub Actions, and it’s fully configurable. Check out this blog to learn more https://docs.sweep.dev/blogs/giving-dev-tools.
So far, Sweep isn’t that fast, can’t handle massive problems yet, and doesn’t write hundreds of lines of code. We’re excited to work towards that. In the meantime, a lot of our users have been able to get useful results. For example, a user reported that an app was not working correctly on Windows, and Sweep wrote the PR at https://github.com/swe...
116 comments
[ 2.3 ms ] story [ 204 ms ] threadThere's still a long way to go on automated testing, building, and running code, but I don't see any reason it's not possible!
(1) unless the PRs that they generate are garbage.
Definitely check out what we've been able to merge now though. The ceiling for tools like Sweep is incredibly high.
In my experience, junior developers become mid level developers by writing code, by practicing, by building small features, by doing grunt work. If they wanted to use an AI to do those tasks for them, I would tell them no - the whole point of having junior devs do simpler tasks is that's what level they're at. They don't get to the next level magically, it's by doing the work. If a high school football quarterback asked if he could skip practice and let his AI go to practice for him, I would wonder how he plans to get good at football.
I apologize that I don't have anything constructive to say here but you did ask for any of my thoughts.
Also as a dev, writing code is energizing and I love spending my day building a new feature. But when you get into maintenance mode, it's not that fun anymore. There's a good amount of code in the intersection of "easy to review" + "annoying to write", so Sweep is aiming to address that first.
Overall, it's not so much about not writing any more code and more about writing more interesting code. Similarly for junior devs, even in the space of "grunt work", there's more and less interesting options.
Also, I'm not sure if you'd enjoy writing code for those "grunt work". I'd love PRs that I can easily check correctness for and would get some small job done.
Also, here's another example of "grunt work". Sweep added a banner to our landing page, and I didn't touch my IDE at all. https://github.com/sweepai/landing-page/pull/226
I’m not sure. If these AI tools become sophisticated enough it might be better experience to learn how to use them instead of doing the underlying work. Career-wise anyway.
We're also trying to make it easy to use Sweep. One outcome is an entirely simulated teammate, which is part of what we're doing with allowing you to review Sweep's PR
The previous belief here would be the 10 person gets more work done, but that will change as AI developers like Sweep become more popular. There are a lot of additional benefits for small teams, like fewer meetings + faster decisions.
After all, if it doesn’t keep pace, the two-tier system I mentioned in my other comment will definitively be such that shops using these tools will not be as good as shops with a more traditional engineer skill development path.
Right now you could have some junior devs picking up work that Sweep can't handle in order to grow and learn, and eventually still become senior devs. Having a small team also helps with mentorship (more focused attention).
What if you have the opposite - a large number of relatively simple bugs waiting to get fixed and not enough junior devs do the work?
I think Sweep is a great idea and all of the additional developer capacity will be greedily soaked up by understaffed organisations.
How well it works will depend on how good those pull requests are. If it takes too much time of senior developers to review the pull requests then that is a problem.
Good luck!
Sweep can do this right now, you just have to label it yourself. We're doing this right now so you don't get flooded with PRs if you have a lot of open issues.
Why is increasing your code coverage to 100% matter? Would that reduce bugs or speed up development in any way?
Wouldn't it just add lots more code to maintain and make refactors more time consuming?
To put it another way, code coverage isn't a direct measure of how good your testing is, but it is still a useful metric to try and improve.
In most cases 100% is too hardcore a target, but you should probably aim for at least 80%.
(Shameless plug) I wrote a short post about this here: https://about.codecov.io/blog/the-case-against-100-code-cove...
This is sarcasm, but I did have a VP who tracked commit frequency for a while. And people heard about it if they weren't commiting enough.
Congrats on the launch!
We noticed that Sweep's search works way better if there are comments, because the comments match up really well with the search queries (language <-> language is easier than language <-> code)
It'd significantly increase the costs though.
Similar to the prompt engineering for previous era GPT completion models.
The logs from Sweep(which contain snippets of code) are logged for debugging purposes. We don't train on any of your code. These will only be stored for 30 days. We send this data to OpenAI to generate code. We're using the OpenAI api, and OpenAI has an agreement stating they will not train on this data and will persist it for 30 days to monitor trust and safety.
We index your codebase for search, but we use a system that only reads your repo at runtime in Modal. This runs as a serverless function which is torn down after your request completes. Here's a blog we wrote about it! https://docs.sweep.dev/blogs/search-infra
So for context, this is running in a ephemeral function from Modal https://modal.com/docs/reference/modal.Function#modalfunctio....
We need a way to store the computed embeddings, because the function doesn't persist any state by default, so we use Redis. But we don't want to store the actual code as the key, so we hash the code + add some versioning. Because it's a cache, it supports concurrent writes + reads, which a lot of vector dbs do poorly.
So the actual code is only accessed at runtime (using the GitHub app authentication to clone the repo), and we also build the vector db in memory at runtime. It's slow(redis call, embedding the misses, constructing the index), but 1-2s is negligible in the context of Sweep because a single openai call could be 7s+.
And one nice feature is that when you have Sweep running on 10+ branches (which probably share 95%+ of the code) we just use the cache hits/misses to automatically handle diffs in the vector db. It's super easy to setup, we don't need to manage different indices (imagine a new index per branch), and it's very cost efficient.
What kinds of videos would you like? We can make anything, the two repos we use the most are Sweep itself and our landing page
All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with.
Good luck!
Is Sweep also taking in suggestions and then incorporates them with follow-up commits to the PR?
This is useful in some ways. Thinking about situations like pre-release software testing, there are exploratory test cases that are simply too numerous to ever have a human perform economically. A lot of AI is going to do this kind of very low-valued grunt work where it doesn't matter if it's 90% or 99% correct, it's the fact that it can get done at all. A lot of this work is "additive" in the sense that, it's just too expensive to do today (with a human).
The work product of these systems is best seen as a "rough draft" or "suggestion". It's a first cut, not the last word.
On the other hand, a lot (most?) of the meat-and-potatoes coding done today, is situations where things have to WORK. Stuff where correctness absolutely matters--billing/money/settlement (calculating tax, handling returns, moving money between accounts), a lot of OS code for things like memory management / locking / resource management, drug dosing, reservation management, etc.
Granted, this stuff is a lot more complex and nuanced than the code of an average CRUD app, but then, I also don't spend my days implementing bcrypt, quicksort, or self-synchronizing Unicode parsing. We have libraries for that. The question is whether we're better off relying on agents to write a bunch of grunty code, or come up with better top-level organization / code structures, that doing it "by hand" is the better approach.
I'm actually optimistic that we can do better code-wise. But I'd love to see how things develop. Maybe we wouldn't need AI if we just had better programming languages.
Further, we're adding better test systems to Sweep. For now, you can just comment to get Sweep to cover the edge cases and write tests. Happy to take any other feedback.
We have this (to some degree) with issue templates, where you can pre-populate some text and fill in the rest. We're also thinking about good ways to offload that work to Sweep.
1. They refer to as an "AI junior developer" 2. It creates pull requests, not commits
From (2), your problem is with the person who commits this code without modification, not with the AI.
It's very cool that it inferred the right place to make the change and the steps of finding relevant code, making a plan, then doing it are things I wish all my junior developers would do! This is certainly moving in the right direction.
The self-review generally catches stuff like this since we tell it that this code is written by an inexperienced developer, so that Sweep becomes more critical.
A weird question: How do you feel about possibly ~wasted efforts of these techniques when gpt in a year or so is probably gonna be 100k+ in context length? I've felt this a bit. E.g. I really want to create a 'massive document' conversational agent but I'm doing around 90% of work just juggling and preempting token constraints with super hueristic indexing. I just feel it's all a bit.. wasted, in terms of effort. At some point the LLM apis (openai, claude, ..) will just accept massive zips of code and use them as entire prompts without need for these creative trickeries. Thoughts?
Oh! And have you tried out the function-calling APIs? I see you've found that XML is far more reliable as it's semantically enriched. I have found this to be the case as well, which is a shame because I really want the function-calling stuff to work equally well.
I'm loving stuff like this that starts to pseudo-expand the token limit.
My stance is with models ignoring long contexts(https://arxiv.org/pdf/2307.03172.pdf), we'll have this problem for a long time. I could be wrong though.
Also we did try function calling, but it doesn't allow for a chain of thought step. This made the plan/code way worse. Cool to see you found the same!
E.g. in the demo video, the code doesn't cover if splitName.length === 0. I would want to prompt it to cover that case as well
Do you have a strategy to supplement ChatGPT to handle post-2021 updates to languages and libraries? I tried it on a NextJS repo and it came up with something that looked like it would have been correct a few versions ago, but I had to make some manual changes. Certain fast-moving ecosystems might frequently have this issue.
You can give Sweep docs about a framework and it should help a lot.
And I'm curious to know more about your costs of deployment and running on Modal.
The developer experience is also great, so we highly recommend it :)
Now, a bit off topic. I've seen a proliferation of "open source" startups that just use "open source" as a way of promotion. The repo feels more like a landing page than a README. And the Installation shows anything BUT install instructions.
Yes, We made a small tool to help developer review their PR. Seems a great supplement for Sweep AI.
Build your own PR review bot in 3 minutes here: https://github.com/flows-network/github-pr-summary
One thing we also do is match any files mentioned in the issue. So if you mention sweepai/api.py, we'll find that and add it to the fetched files. There's still more work to be done here, so look out for those!
Likely file name based scoring, and other rules + finetuned retrieval models (opt-in of course)
There's also an interesting discussion here about how it works: https://news.ycombinator.com/reply?id=36990160
What’s your strategy around oss? Why wouldn’t I just clone and use your repo if it can talk directly to GPT4/Claude?
And, if I go through you, does Sweep retain my code in a DB or logs?
We have had users in our discord set it up themselves, so it's possible. If you go through us we retain your repo in a hashed redis cache (no plaintext code stored) and store some (~50 line) snippets in our logs for debugging. We don't train on any of your code. You can find our privacy policy here: https://docs.sweep.dev/privacy
Your post claims this is open source, but your license says otherwise: https://www.oshwa.org/2014/05/21/cc-oshw
Meanwhile you're soliciting feedback, improvements, reviews, and other free work from the community here...
I'm not a die-hard advocate of open source, but advertising software under a restrictive license as open source comes off as disingenuously trying to sow good will in the community
Great job.
I’m personally a fan of letting users insert their own instructions per repository, and Sweep can use that for each PR.