I've been thinking about something similar. GPT-3 has dated knowledge on a lot of libraries. If you could feed it the latest source code and docs pages, would it be able to help you write code faster?
As others have mentioned, it seems down at the moment. But will be curious to give this a go and see how well it works.
I tried it for my (incomplete) compiler project[0], and the results are quite funny:
> This program prints a large string of text to the console. It does this by using a String variable called myLongString, which is initialized with a long string of text. The program begins with the line "program LargeString {" and ends with the line "}". Inside the program, there is a comment that reads "Now let's try a comment". This comment is not necessary for the program to work, but it helps to explain what the program does. The program then uses the write() function, which is a function that prints a given string to the console, to print the large string of text.
It seems as though its looking at the sample code files used for testing rather than the actual compiler codebase.
There’s no way I’m logging into rando apps using my GitHub id. Aside from their bad UX and the potential to accidentally over grant permissions, the app could abuse GitHub ToS with my credentials and get me banned.
Not worth the risk just to play around with a site.
The sign-up form doesn't seem to allow passwords with non-alphanumeric characters in them. Ironically, when it sees one, it complains that "password must have at least one number and at least one letter" (even though it already does) - I suspect that's just a catch-all message for invalid passwords?
In any case, the restriction makes it incompatible with pretty much all password generators, including the built-in ones in modern browsers. To improve matters, I would suggest dropping all restrictions on what cannot be included altogether.
> This appears to be a poem written in English. It is written in a form that is often used for teaching purposes, in which each letter of the alphabet is associated with a concept related to a particular field of study. In this case, the poem is about quantum field theory and particle physics. The poem is written in a rhyming couplet form and contains many scientific terms that are specific to the field.
I imagine that it might already know how to answer that question for non-Github related texts, which presumably are often not programs. I'd be more interested to see what it responds to "What programming language is this written in?" to see if it will question the premise of the question (although there's always that it might make an educated guess that the poem is valid Perl!)
How curious: I have a small children's book I picked up at the California Academy of Sciences entitled, "The ABCs of particle physics," made by a joint collaboration between Fermilab and SLAC [1], and here I find another poem in the same vein!
This appears to be quite nice assuming it works, but I wonder: how does this work?
I thought ChatGPT could only process a very limited context (a few thousand words or so) and this claims to work for repositories <100 MB, which is gigantic in comparison.
The site says: "we will download [the repo] to the server and create embeddings against the code". What kind of embeddings are these?
Can you elaborate on this? What is the input to the GPT in that case? I was under the impression that GPT is given an array of tokens and it produces one token as its output.
I don't know about embeddings, but it was able to decode a URL encoded string into a javascript animated SVG. So larger texts can be encoded and still "understood." You probably wouldn't even have to declare your encoding method. https://www.tiktok.com/@y.i.t.z.i/video/7184820436839042306
From what I can see, they download the repo to their server, generate per-function semantic embeddings (which will no doubt cost them at scale) [1], then use some combination of GPT-3 and Codex prompted with something like "Here's the source code of a git repo. Answer the following questions about it: {USER_QUESTION}."
it seems to have some issue. i asked about a repo (the hive / cortex) and it returned me:
It is difficult to answer this question without knowing more about the context provided. It appears to be a PGP public key block, but without additional information it is impossible to answer the question.
it seems it might not be able to read it?? (its got a normal readme and docs etc.)
No, macwatch is a program designed specifically for Mac computers and is not compatible with Windows systems. The program is designed to provide users with a comprehensive overview of their Mac's performance, including CPU usage, memory usage, disk usage, and more. It can also be used to monitor network activity and alert users when a process is consuming too much system resources. Additionally, the code provided is written in Objective-C and C, which are not supported on Windows systems.
---
The first sentence is correct, the rest is just completely wrong.
When it works, it’s flawless. When it doesn’t work, it just makes stuff up that kind of look good, just to make you happy. And there is really no way to tell from the output…
Pretty funny how it seems to just have invented something based on nothing more than the name macwatch. The last sentence is remarkably strange, like it's just riffing on the idea of "not supported on Windows" and decided to throw in some random facts about the repo, and wound up with a totally false statement.
"Something went wrong, please try again later. If the problem persists, please contact us."
Edit: looks like it is, I see it's making a POST request to https://gptduck-production.up.railway.app/query which responds with a 500 Internal Server Error when I use a URL that looks like a GitHub repository (whether it actually exists or not), and "invalid repo" otherwise.
I was building something on similar lines. It is a semantic search engine for codebases using Openai's embedding and FAISS https://github.com/rahuldan/codesearch
57 comments
[ 2.5 ms ] story [ 103 ms ] threadTried several times.
[1] https://gptduck-production.up.railway.app/query
As others have mentioned, it seems down at the moment. But will be curious to give this a go and see how well it works.
https://github.com/calebegg/website
It's very impressive!
> This program prints a large string of text to the console. It does this by using a String variable called myLongString, which is initialized with a long string of text. The program begins with the line "program LargeString {" and ends with the line "}". Inside the program, there is a comment that reads "Now let's try a comment". This comment is not necessary for the program to work, but it helps to explain what the program does. The program then uses the write() function, which is a function that prints a given string to the console, to print the large string of text.
It seems as though its looking at the sample code files used for testing rather than the actual compiler codebase.
[0] - https://github.com/JLCarveth/zpl-compiler
Logging with GitHub is still lowish but like 1k per hour or so. (IIRC)
Not worth the risk just to play around with a site.
In any case, the restriction makes it incompatible with pretty much all password generators, including the built-in ones in modern browsers. To improve matters, I would suggest dropping all restrictions on what cannot be included altogether.
https://github.com/chbarts/particle-abcs
And my question:
> What language is this written in?
The answer:
> This appears to be a poem written in English. It is written in a form that is often used for teaching purposes, in which each letter of the alphabet is associated with a concept related to a particular field of study. In this case, the poem is about quantum field theory and particle physics. The poem is written in a rhyming couplet form and contains many scientific terms that are specific to the field.
Which is entirely correct. I'm impressed.
I am very impressed with your test - and the result.
[1] https://www.symmetrymagazine.org/article/the-abcs-of-particl...
> Update: just shipped a fix for the rate limits - you will need to signup for an account and put your own personal access token in.
I thought ChatGPT could only process a very limited context (a few thousand words or so) and this claims to work for repositories <100 MB, which is gigantic in comparison.
The site says: "we will download [the repo] to the server and create embeddings against the code". What kind of embeddings are these?
[1] https://beta.openai.com/docs/guides/embeddings
I found this guide/example from OpenAI which was pretty clear:
https://github.com/openai/openai-cookbook/blob/main/examples...
I'm trying to ask ChatGPT what the core insight is to my repository
https://GitHub.com/samsquire/ideas4
It is difficult to answer this question without knowing more about the context provided. It appears to be a PGP public key block, but without additional information it is impossible to answer the question.
it seems it might not be able to read it?? (its got a normal readme and docs etc.)
For one of my repos: https://github.com/drpriver/macwatch
> Does this program work on windows?
No, macwatch is a program designed specifically for Mac computers and is not compatible with Windows systems. The program is designed to provide users with a comprehensive overview of their Mac's performance, including CPU usage, memory usage, disk usage, and more. It can also be used to monitor network activity and alert users when a process is consuming too much system resources. Additionally, the code provided is written in Objective-C and C, which are not supported on Windows systems.
---
The first sentence is correct, the rest is just completely wrong.
When it works, it’s flawless. When it doesn’t work, it just makes stuff up that kind of look good, just to make you happy. And there is really no way to tell from the output…
I tried adding https://github.com/torvalds/linux and asking it to write a simple kernel module for me, but it gave me this error:
"Something went wrong, please try again later. If the problem persists, please contact us."
Edit: looks like it is, I see it's making a POST request to https://gptduck-production.up.railway.app/query which responds with a 500 Internal Server Error when I use a URL that looks like a GitHub repository (whether it actually exists or not), and "invalid repo" otherwise.
"'NoneType' object has no attribute 'similarity_search'. Try logging in and entering your own Github access token."
I was building something on similar lines. It is a semantic search engine for codebases using Openai's embedding and FAISS https://github.com/rahuldan/codesearch
It fetches the functions in the code for a query