Show HN: GPT Repo Loader – load entire code repos into GPT prompts (github.com)
gpt-repository-loader as-is works pretty well in helping me achieve better responses. Eventually, I thought it would be cute to load itself into GPT-4 and have GPT-4 improve it. I was honestly surprised by PR#17. GPT-4 was able to write a valid an example repo and an expected output and throw in a small curveball by adjusting .gptignore. I did tell GPT the output file format in two places: 1.) in the preamble when I prompted it to make a PR for issue #16 and 2.) as a string in gpt_repository_loader.py, both of which are indirect ways to infer how to build a functional test. However, I don't think I explained to GPT in English anywhere on how .gptignore works at all!
I wonder how far GPT-4 can take this repo. Here is the process I'm following for developing:
- Open an issue describing the improvement to make
- Construct a prompt - start with using gpt_repository_loader.py on this repo to generate the repository context, then append the text of the opened issue after the --END-- line.
- Try not to edit any code GPT-4 generates. If there is something wrong, continue to prompt GPT to fix whatever it is.
- Create a feature branch on the issue and create a pull request based on GPT's response.
- Have a maintainer review, approve, and merge.
I am going to try to automate the steps above as much as possible. Really curious how tight the feedback loop will eventually get before something breaks!
161 comments
[ 0.31 ms ] story [ 210 ms ] threadDoes ChatGPT 4 now accept more tokens maybe?
Our codebase is 1 million lines of code.
Can we feed the documentation to it? What are the limits?
Is it possible to train it on our data without doing prompt engineering? How?
Otherwise are we supposed to use embeddings? Can someone explain how these all work and the tradeoffs?
Clearly this will break eventually, but I am playing around with some ideas to extend how much context I can give it. One is to do something like base64 encode file contents. I've seen some early success that GPT-4 knows how to decode it, so that'll allow me to stuff more characters into it. I'm also hoping that with the use of .gptignore, I can just selectively give the files I think are relevant for whatever prompt I'm writing.
I wonder if you could teach it to understand a binary encoding using the raw bytestream, feed it compressed text, and just tell it to decompress it first.
To teach me to understand a particular binary encoding and compressed text format, you should provide the following information:
The binary encoding used (e.g., ASCII, UTF-8, UTF-16, etc.). The compression algorithm employed (e.g., gzip, Lempel-Ziv-Welch (LZW), Huffman coding, etc.). Once you provide these details, I can help you process the raw bytestream and decompress the text. However, keep in mind that my primary focus is on natural language understanding and generation, and I might not be as efficient at handling compressed data as a dedicated compression/decompression tool."
Genuine question.
Probably doesn't work this way lol
Unfortunately GPT is not yet aware of what LangChain is or how it works, and the docs are too long to feed the whole thing to GPT.
But you can still ask it to figure something out for you.
For example: “write pseudo-code that can read documents in chunks of 800 tokens at a time, then for each chunk create a prompt for GPT to summarize the chunk, then save the responses per document and finally aggregate+summarize all the responses per document”
Basically a kind of recursive map/reduce process to get, process and aggregate GPT responses about the data.
LangChain provides tooling to do the above and even allow the model to use tools, like search or other actions.
You can use our repo (which we are currently updating to include QuickStart tutorials, coming in the next few days) to do embedding retrieval and query
www.GitHub.com/Jerpint/buster
Also, do we know what languages GPT-4 "understands" at a sufficient level? What knowledge does it have of post-2021 language features, like in C23?
(https://gist.github.com/darius/b463c7089358fe138a6c29286fe2d... paste in painful-to-read format if anyone's really curious. In three parts: intro to language; I ask it to code symbolic differentiation; then a metacircular interpreter.)
The initial prompt would be, "person wants to do x, here are the file list of this repo: ...., give me a list of files that you'd want to edit, create or delete" -> take the list, try to fit the contents of them into 32k tokens and re-prompt with "user is trying to achieve x, here's the most relevant files with their contents:..., give me a git commit in the style of git patch/diff output". From playing around with it today, I think this approach would work rather well and can be like a huge step up from AI line autocompletion.
I'm happy to wait even 30-60 seconds for this which I can easily evaluate, criticize (and the model will correct it) and then proceed to just patch and move on. I think the results from this will be much better with the 32k model, but remains to be seen.
You... might wanna consider a self hosted alternative for that use case, or at least do like, a `| wc` to get an idea of what you're potentially sending before calling the api.
[1] - https://help.openai.com/en/articles/7127956-how-much-does-gp...
I'm not exactly sure you define that scale, perhaps Minecraft bots or the like, where the damage after complete failure is self contained to perhaps a few dollars or a few hours of human annoyance. I'm sure there are many niches where a 50% success rate of mass generated programs can earn you big bucks.
But in my experience Codex does very limited reasoning about code paths. For the current state of the art, you are almost guaranteed to have catastrophic bugs in any non-trivial programs engineered by prompt.
I tried it with phone number formats and it came up with more than I could.
It's best illustrated by the old joke:
A good chunk of all bugs in software are down to the requirements being insufficiently well specified. Further, many bugs are the discovery of new requirements when informal specification encounters reality."Read from standard input into this byte array" doesn't specify what to do when the input exceeds the byte array.
When you overflow the buffer, you get a "well obviously you're supposed to not do that"... that's wasn't stated at all.
When the function keeps going after a newline or a null byte or whatever, there's another "well obviously you're supposed to stop at those points". That was also not specified.
and so on.
At the point you're specifying all these cases and what to do when, it's so specific and stilted, you might as well be using a programming language.
(We already program in English, in a sense, when we tell humans what we want, and they go code it. Now we'll just be telling machines.)
Originally as formulaic syllogisms and Aristotelian logic, but then onto other forms of codified language, formal logic etc.
Adding more words often makes things less clear, not more so. What you need is well-defined terms with no overloaded meaning.
> (We already program in English, in a sense, when we tell humans what we want, and they go code it. Now we'll just be telling machines.)
Humans get it wrong all the time though. A great many bugs arise from quite simply misinterpreting the requirements. Which leads to requirements becoming more formulaic and resembling a programming language.
Plus also most math and logic is still communicated and developed in a mix of human languages (English etc) and ad-hoc, not rigoursly defined notation; it's nowhere near the precision of a programming language.
Though you CAN of course grind it out at that level, if you want, but it's very unwieldy and not how people actually work.
If you're looking to deduce some sort of proof from well-defined principles and you wish to eliminate the possibility of error, then sure well-defined terms (a rigorous language) is useful.
If you're looking to produce a sofwtare artifact, just saying what you want in high-level terms and providing iterative natural language feedback is going to work great and be way nicer than trying to formalize everything.
(Maybe not true for low-level plumbing and things that need to be secure. But for like "build an app", "build a game", "make a shell script that does x", I think it will certainly end up being true.)
Given what you want is a singular specific behavior, not just sorta that behavior, but exactly that behavior, it makes sense to use a language that caters to specificity.
As an aside, we've experimented with more natural-language like programming languages before (COBOL, BASIC). They're a pain in the ass because you needed to type a novel to do even trivial things. The terseness of modern programming languages is a feature.
[1] https://ai.googleblog.com/2006/06/extra-extra-read-all-about...
Don't think of LLMs as akin to COBOL or Basic (or HyperTalk, is another great example) -- those are essentially rigid languages with a sprinkling of "Englishness" to try to make them seem less foreign to novice users. And yes, that just gets in the way once you become proficient in rigid languages.
But that's a completely different beast than programming in actual English.
So for example, the reason binary search (a simple algorithm) is hard to implement is because of the mismatch between easy to understand human desire and the pickiness of the rigid language designed for the machine.
Now, it is very easy to implement a correct binary search.
Tell ChatGPT:
> Write a binary search method in C#
It responds:
We can also ask it:> Prove it is correct
It responds:
> To prove the correctness of the binary search algorithm, we can use a technique called loop invariant. A loop invariant is a condition that remains true for every iteration of the loop. In this case, the loop invariant is that the target element, if it exists in the array, will always be within the range defined by the left and right pointers.
> Loop Invariant: At the start of every iteration, if the target element exists in the array, it must be within the subarray arr[left, right].
> Initialization: At the beginning, left = 0 and right = arr.Length - 1, which covers the entire array. So, if the target element is present in the array, it must be within the initial range.
> Maintenance: During each iteration, the algorithm examines the middle element (mid = left + (right - left) / 2). There are three cases:
> If arr[mid] == target, the target element is found and returned. > If arr[mid] < target, the target must be in the right half of the subarray. Therefore, we update left = mid + 1. Since the array is sorted, we are sure the target is in the range arr[mid + 1, right] if it exists in the array. > If arr[mid] > target, the target must be in the left half of the subarray. Therefore, we update right = mid - 1. Since the array is sorted, we are sure the target is in the range arr[left, mid - 1] if it exists in the array. > In cases 2 and 3, we have updated either the left or right pointer, maintaining the loop invariant for the next iteration.
> Termination: The loop terminates when left > right. Due to the loop invariant, if the target element exists in the array, it must be in the range arr[left, right]. Since left > right, this range is now empty, which means the target element is not in the array, so the algorithm returns -1.
> In conclusion, the binary search algorithm is correct, as it either returns the index of the target element if it exists in the sorted array, or -1 if it doesn't.
(OK, I'd quibble with that proof a little bit -- it was a little hand-wavy on why the loop must terminate.)
How about the broken binary search you linked to?
Ask ChatGPT:
> Can you find the bug in this java binary search implementation? [Code]
It responds:
> The given binary search i...
It's almost unique in that the problem has a corpus of literature about how difficult they are to implement correctly, which pitfalls are common, and how to solve them. ChatGPT being able to regurgitate this solution is not a good demonstration of it's ability to solve general programming problems.
And LLMs aren't just good at binary search, they're good at lots of things.
Imagine you are in a room with a programmer who is unquestionably better and more expert than you are.
Now let's say you need to write a program. Would you be better off trying to write it yourself, or describing what you want to the better programmer, and letting them write it?
Obviously the latter!
Given a sufficiently advanced compatriot, English is the preferred programming language.
Now, are LLMs good enough? Probably not yet, but getting there rapidly!
Edit: This, btw, is also the reason why I think that this here popped up on the hackernews frontpage a short while ago: https://github.com/pgvector/pgvector
When giving a prompt, the prompt causes the crawling of many APIs to build the response - the power of such activity/features, will be scary power-to-authoratarian goals.
Imagine if the prompt is "Select all users who have political beliefs, posts, comments, links from APIs A, B, C, etc where sentiment appears to dissent from [party line]"
Sam Altman takes comfort in the thought that their AI does nothing without a human prompting it, so it has a human in the loop, as a circuit breaker if you will.
This assumption is rapidly becoming a mere hope, as right now probably hundreds of developers are working on systems which, when put into production and connected to other systems, might just come down to: the AI is calling itself, and giving itself orders.
Imagine I want you to shoot me.'
Imagine I Want [whatever]GPT to make a cyberpunk Anime based on such...
https://github.com/jerryjliu/llama_index
and/or
https://github.com/hwchase17/langchain
also this is slick as hell
on the main topic, have you heard about RWKV ( rnn based gpt style network )? the project's actively working on implementing "infinite" context length support, which would probably pair very well with a project like yours
I wonder if the future will just be software hobbled together with shitty AI code that no one understands, with long loops and deep call stacks and abstractions on top of abstractions, while tech priests take a prompt and pray approach to eventually building something that does kind of what they want.
Or to hell with priests! Build some temple where users themselves can come leave prompts for the general AI to hear and maybe put out a fix for some app running on their tablets devices.
The future that I see, coding and AI are divided into two camps. The one is what we would call "script kiddies" today - people who don't understand how to write software, but know enough to ask the right questions and bodge what they get together into something that mostly works. The other camp would be programmers who are similar to programmers today, but use AI to write boilerplate for them, as well as replace Stack Overflow.
Except you can’t just blindly copy code snippets, you have to read the author’s explanation and perhaps adapt things to your own code sometimes, or reject their solution entirely. GPT-4 can’t do this because it doesn’t actually know what the hell it’s doing, it’s just putting stuff together in a form that is most probably correct based on what it has seen in training data for past examples.
I fear for the layman who sees a bunch of AI generated code and think it must be right. Who knows what bugs, security flaws, or performance issues they will run into, that they have no idea how to solve or even to begin asking a prompt for.
That’s kind of impressive.
I usually have zero luck with stack overflow except for super trivial things.
Some of the stuff I’ve done has no documentation and I had to find an example (like one other person in the entire history of mankind though this was a good idea) on some random repo on GitHub. Or I’m implementing code from a paper written 30 or 40 years ago and there’s no example code to look at. I could ask on stack overflow but who needs that abuse?
Admittedly I just do this to amuse myself and think that having a LLM digest a paper and spit out code is the bee’s knees.
This is what I have been doing also. Verification/Implementation of Software Engineering papers will be juicy especially with image ingestion.
e.g., if I type into google, "Build me a React component todo list that loads its data using fetch() to /api/v1/todos and is styled with Tailwind", I'm going to get a bunch of stuff that I can maybe use, if I invest 15 minutes wading through cancer-inducing blogs. Whereas at least for trivial problems, ChatGPT just tesseracts something that's extremely close and you copy-paste it in, change a few things, boom, component done. Something doesn't work? Often a follow up message fixes it.
LOC is a fuzzy proxy for the amount information in a code base. It should never be used for any kind of productivity metric however since that is obviously dumb and easily gameable (win by writing terrible code). Of course most productivity metrics are fall into the dumb/gameable category so lets not use them.
That sounds terrible. Who is going to read all that code when you need to understand and modify it? The LLMs, I guess?
I'd much rather be able to write 10,000 lines of code that can do what your million lines of code does. Better programming languages, libraries, and other abstractions are what we need.
Sure, who wouldn't. Unfortunately this is my hypothetical and I get to control what I mean by it. The million lines of code in my hypothetical is good quality, maintainable with reasonable density.
>> Better programming languages, libraries, and other abstractions are what we need.
In the entire history of languages, we've only managed about a 10X improvement via these mechanisms (that is being charitable probably). Several important things are still written in C which would mostly be recognizable to a programmer from 40 years ago. There are still problems to solve but I feel we are on the asymptotic section of the curve in this regard.
There's plenty of software out there that fits this description if you just remove "AI" from the statement. There's nothing new about bad codebases. Now it just costs pennies and is written in seconds instead of thousands paid to an outsourcing middleman firm that takes weeks or months to turn it around.
That’s my fear as well. Software may just get shittier overall (aka “good enough”), and in higher volumes, due to it taking less time to crank out using AI.
We're going to witness he greatest copy pasta in history and find out how that goes.
And once we get good tooling to inject in recent data, source code, etc so it can use those as well... gonna be great
Infact, it probably performs so well because of the heavy use of libraries we see today.
Also side note:
I doubt you're ever going to be able to upload code to an LLM my friend, not a publicly accessible one. It would be quite dangerous? Could you imagine the attacks, the implanting of back doors etc? Don't think so.
I'd say scraping the web is probably going to be an increasingly dangerous pursuit for ChatGPT now people are getting to better understand the attack vectors available.
For example, maybe we write some code now with the goal of helping a customer service person do their job. But we all know that plenty of people are trying to replace customer service people with LLMs, not use LLMs to write tools to help customer service people.
I see that the LLM still needs to know what's going on with the customer account, and maybe for a long time that takes the form of conventional APIs. But surely something is going to change here?
"The Age of Em" by Robin Hanson thinks through a lot of this in great depth
All programmers will become translators from product vision to architecture implementation via guided code review. Eventually this gap will also be closed. Product will say: Make a website that aggregates powerlifting meet dates and keeps them up to date. Deploy it. Use my card on file. Don't spend more than $100/month. The AI will execute the plan.
Programmers will come in when product can't figure out what's wrong with the system.
TOS and trust matter the most.
For an individual function I can totally believe GPT4 could strip creative expression from it today. For example you could ask it to give a detailed description of a function in English, and then feed that English description back in (in a new session) and ask it to generate a code based upon the description.
Somehow I feel the lawyers won't agree.
Copyright is a law agreed by a humans in a social contract created to protect humans and further their interests in a 'fair' manner. There is no inalienable right to copyright, no universal law that requires it, it's not an emergent property of intelligence that mechanically applies to artificial entities.
So while the current copyright laws could be interpreted in the way you suggest for the time being, they are clearly written without any notion of AI, and can and should be revised to incorporate the new state of the world; you can bet creators will push hard in that direction. It's pretty clear that the mechanical transformation of a human body of work for the sole purpose of stripping it of copyright is a violation of the spirit of copyright law *.
*( as long as that machine can't also generate a similar work from scratch, in which case the point becomes moot. But we are far, far, from that point)
Functional purposes where never meant to be covered by copyright.
[1] https://en.wikipedia.org/wiki/Clean_room_design
this repo currently has more HN upvotes than LOC.
very high leverage code!
He wrote the issue on https://github.com/mpoon/gpt-repository-loader/issues/16 and summarized https://github.com/mpoon/gpt-repository-loader/discussions/1...
"Open an issue describing the improvement to make Construct a prompt - start with using gpt_repository_loader.py on this repo to generate the repository context, then append the text of the opened issue after the --END-- line."
Feels like it needs to add a little Github client to be able to automatically append the text of issues at the end of the output. I'm sure ChatGPT can write a Github client in Python no problem.
Who’s to say GPT-4 isn’t some ploy to gather data to train private AGI?
What other 'relationships' does OpenAI have with [corp/gov] where the private AGI is shared/sold/service as product to NGO or GOV customers?
also, the entire thing is literally 60 lines of python. sometimes i don't get what gets upvoted on HN anymore
Which is a lot as you can do this in one line of bash. And have in the past for other reasons.
Something like;
To summarize the GPT inference process as I understood it, with GPT3 as example:
1) the input buffer is made of 4k token. There are about 50k token. So the input is a vector of token ids. We can see it as a point in a high dimensional space;
2) The core neural network is a pure function: for such an input point, it will return an output vector as large as there are token. So here, a 50k element vector, where each entry is the probability that the associated token is the next element.
The very important thing here is that the whole neural network is a pure function: same input, same output. With immensely large super fast memory this function could be implemented as a look-up table, from an input point (buffer) to an output probability vector. No memory, no side effect here.
3) The probability vector is fed into a "next token" function. It doesn't just take the highest probability token (boring result), but use a "temperature" to randomize a bit, while using the output probabilities;
4) The next token chosen is inserted into the input buffer, keeping the same total number of token. Go back to (1) until a "stop" token is selected at (3).
So in effect, the whole process is a function from a point to a point. "point" here is the buffer seen as a (high dimensional) vector, so a point in a high dimension space. The generation process is in effect a walk in this "buffer space". Prompting puts the model into some part of the state, with some semantic relation to the prompt semantic content (that's the magic part). Then generation is a walk in this space, with a purely deterministic part (2) and a bit of randomization (3) to make the walk trajectory (and its meaning, which is what we care about) more interesting to us.
So if this is correct, there is no point in injecting a lot of data into a GPT model: the output is defined by the input buffer size. Just input the last 4k token (for GPT3, more for GPT4) and you're done: everything else would have disappeared. So here, just input the last 4k token of a repo and save some money ;)
To avoid this limitation, one would have to summarize the previous input, and make this summary part of the current input buffer. This is what chaining is all about if I understood correctly. But I don't see chaining here.
Sooo... Am I missing something? Or is the author of this script the one missing something? I don't mind it either way, but I'd appreciate some clarification from knowledgeable people ;)
Thanks
Another path you can take is to fine tune a model on your business. Each training item has to fit within the token limit, but you can send hundreds of megs of these for training.
It's more expensive to run a FT model, but you don't have to include any prior context (assuming it's common to all prompts).
ChatGPT is just a big tech demo of what anyone could achieve. Granted, the training data is the hardest part. But, if you have a narrow domain and piles of existing data to work with, I don't see why you can't exceed the performance of these offerings for topics that actually matter to you.
If this was using the git library and https://langchain.readthedocs.io/en/latest/reference/modules... it would be a more complete solution.