Why does every one of these example use pinecone when that costs money and Faiss[0] is free? If you’re going to let something run a bunch of fee-per-use API calls on its own, why double up on getting charged?
Using langchain I’ve been introduced to chroma which uses a local store and it’s perfect. Check out their question and answer demo, the VectorstoreIndexCreator utilizes chroma and duckdb it seems. https://python.langchain.com/en/latest/use_cases/question_an...
Unless you have several hundred million documents, just write a simple encoder that serializes the embedding vectors to a flat binary file.
Writing code from scratch to process and search 200k unstructured documents -- parsing, cleaning, chunking, OpenAI embedding API, serialization code, linear search with cosine similarity, and the actual time to debug, test and run all this -- took me less than 3 hours in Go. The flat binary representation of all vectors is under 500 MB. I even went ahead and made it mmap-friendly for the fun of it even though I could read it into all into memory.
Even the dumb linear search I wrote takes just 20-30ms per query on my Macbook for the 200k documents. The search results are fantastic.
I didn't bother cleaning it so it's just a code dump, but it's fairly straightforward. Not included are a Python script to parse and clean the raw documents into JSON files (used in `summarize` to output results), code to read these files and get the embeddings from OpenAI for use in `newEmbeddingJSON `, and a bunch of random parallelization shell scripts that I didn't save.
To use it, I call newDBFromJSON from a directory of JSON embedding vectors and serialize the binary representation. This takes a few minutes mostly because parsing JSON is slow, but you I only needed to do this once. When I need to search for the top 10 documents most similar to document X, I call `search` with the embedding vector for that doc. Alternatively if I need to do semantic search with natural language, I'll call the OpenAI API to get the embedding vector for the query and call `search` with that vector. It's pretty fast thanks to Go concurrency maxing out my CPU. It's super accurate with the search results thanks to OpenAI's embeddings.
It's nowhere close to production-ready (it's littered with panics), but it was good enough for me.
Hope this helps!
Edit: oh and don't use float64 (OpenAI's vectors are float16)
I had been thinking of ideas such as memory compaction (summerize things), or using the LLM to translate into an execution language for the automation.
I've been using AutoGPT, which is the other ChatGPT automation tool, and frankly I'm a bit disappointed. It spends most of its time figuring out why system commands are failing and why javascript is blocked, and/or why selenium can't start.
Yeah, tried it with trivial goals like "make a 5 second .wav file with a 440hz tone", and after running for an hour and burning tons of openai requests, it had not figured out how to even pip install a module to create audio files.
It mostly seems like a massively stoned teenager trapped in thought loops about how to use thought loops to achieve something, without ever hitting on the idea of actually doing something.
@dang might be worth linking to Yohei's original repo, which is definitely HN front page worthy: https://github.com/yoheinakajima/babyagi. This repo is not just a strict fork though, Oliveira seems to be working on making BabyAGI more autonomous.
So I’m a tech noob, but I recently finished the Lex Friedman podcast with Max Tegmark. This is a serious person with strong credentials ringing the warning bell about AI. Yet, a lot of people in my tech circle seem to swing between being unconcerned and unimpressed by AI.
Where exactly are we with AI as a legitimate threat if we continue down our current path? Are people like Max just jockeying for attention? Or is there merit to their concerns?
Worry about human beings being the one to kill us all because that's a threat that's existed for decades with nukes and biological warfare. If it isn't weapons, it'll be self-inflicted climate change, or an asteroid we haven't noticed. SkyNet isn't happening.
Six billion human beings of average intelligence are a greater threat than a single hypothetical superintelligence.
I'll pass. If it's worth knowing about then there'll be a paper on it that isn't generating someone money as a commercial podcast with sensationalist takes.
Hypothetical obsolescence is not the same as being killed by a hypothetical superintelligence. We would be no more hypothetically obsolete than the overwhelming majority of lifeforms.
Which video did you watch? Tegmark certainly is afraid that the first smarter-than-human AI will kill everyone: he dicsusses how that might happen at 2:08:40
> The argument for SkyNet happening is missing an important element: why would it happen? Because sci-fi says it will?
That element isn't missing. It's explained every time someone asks, and the answer never relies on sci-fi as evidence. The reason is because of (A) orthogonality and (B) instrumental convergence. If you aren't familiar with those terms, (A) refers to the thesis that an AI's degree of logical reasoning ability is independent of its goals. This stems from Hume's Law [1]. (B) refers to the observation that regardless of an agent's terminal goal, many instrumental goals will be predictably shared: the agent will be incentivized to increase its own capabilities, as a means toward nearly any possible end. This includes steps that would prevent humans from interfering with the AI's goals, such as a human attempting to turn it off or reprogram it.
If you have questions about either of those, there are great learning materials available, including a well-made youtube series by Robert Miles. [2]
> Because LLMs have reached a point of diminishing returns and there isn't enough compute globally to do produce a hypothetical AGI.
What makes you think LLMs have reached a point of diminishing returns? So far, the benchmarks are continuing to show accelerating returns, to the point that it's becoming difficult for benchmarks to keep up.[3]
A group of humans is a super-intelligence; a group-of-groups is even smarter. The 8 billion of us, together, have an unimaginable intelligence. AGI isn't going to be a problem.
I feel like this is missing the point. There is no amount of human effort that can do things like break modern encryption in any reasonable timescale. There is an upper bound on what can be comprehended, and especially in the timescale of decades, by any amount of humans. An ASI would have the ability to comprehend and problem solve on a level thousands of times higher than a human, and if we had the source code for an ASI today, without any previous training, it could likely both train itself on the entire world's body of knowledge and break AES-strength encryption in a matter of seconds.
Encryption is just one example. Its ability across the entirety of math and science would be equally powerful.
As my physics professor said, "the intelligence of committees adds like resistors in parallel".* There are 8 billion of us, but it's hard to get us all pulling in the same direction: we're limited by communication bandwidth, distrust, incentives to defect and freeload, ~20 years of education to get up to speed, duplication of efforts, and other organizational difficulties. Meanwhile, we're manufacturing about 50 million discrete GPUs per year, which is about the same as the number of humans we add per year (60 million and falling). Obviously it's not simply 1 GPU per AI agent, but just in terms of rough trends, there you are.
We've got a Kasparov because there are N billion of us: he is the Chess superintelligence for humanity. Similarly, my aunt is the person helping him eat soy beans, through the proper investment & maintenance of her combine.
Fair enough, but these N billion humans also comprise the "unimaginable super-intelligence" that fails to solve basic collective action problems like climate change and overfishing, and which fights immensely destructive wars with itself every generation. It's exactly what it says on the tin, the common greatness and stupidity of humanity that we are all familiar with from every day experience; there's no need to dress it up as a superintelligence to claim that AI will never be able to rival it. AI can certainly beat Kasparov at chess, and another N billion humans won't change that by producing a Magnus Carlsen. With our collective efforts we might produce a better Stockfish, but that rebuttal would be begging the question.
You could apply the same logic to anyone ringing the alarm bells about climate change (or alarm bells in general). Just because snake oil is usually unfalsifiable doesn't mean everything _currently_ unfalsified is snake oil.
To be fair a large amount of the doomsaying around climate change has been proven to be hyperbole and wrong. The data around climate change is very real, but the most vocal people about it have been saying stuff like cities being underwater and ice caps melting and extreme weather like never before seen, and all in the timescale of 10-20 years. And this is going back to the 60s. We're now 60 years in the future and things are basically the same.
Climate change doesn't impact every part of the world equally. If the temperature rises a couple of degrees in the US Midwest, it's probably not going to be catastrohpic. If it rises a couple of degrees in Bangladesh, it's going to make the place unlivable.
Personally, living in a hot tropical country, I've experienced weather patterns becoming weirder and weirder, and generally trending towards way too hot. Last year, the summer was so long and dry and hot that I really felt that I can't physically live here any longer. And that's when I had the luxury of AC - something the vast majority of my country can't afford.
So yeah, it's easy to dismiss climate change if you live in a cold climate. It's much more real in warm, dry countries.
Huh? In 1 he dies. In 2 and 3 he becomes irrelevant eventually and can only claim credit to the extent his predictions were true, while fighting off all the actual snake oil salesmen claiming that all of this was obvious.
Well, just like in every other case where acting to prevent something will avoid that something to happen. Accident prevention, quit smoking to avoid cancers, vaccines against a pandemic etc etc
Asking this openly, even on HN, is going to get a lot of unqualified answers. So I'll preface that I have a recent PhD in deep RL and pretty well versed on the cutting edge develops of ML.
I think your question has two angles. First, do LLMs have potential for AGI? I think that's an emphatic no. There is nothing special about this generative model vs. say something like sampling from a mixture of Gaussians. Much better generations and super impressive it works, sure, but there is no mechanism for it to improve itself let alone change its "prime directive". See Sam Altman's claim that RL with human feedback being where most are the gains are now.
At a higher level, there is the concern that the pace at which we are going that we can extrapolate that AGI is around the corner. My take on this is that basically everything made possible in the last 20 years is because of GPUs and better tooling. In specific, the recent hype we see is because of how democratized things are getting. We have kids using ChatGPT to do homework. This is very disruptive from a societal perspective but we are essentially at the height of the rate at which this technology is being adopted. The growth rate will decelerate and it will stop being news once society has learned to adapt to the new technology. However, from a technical perspective, these concerns are like looking at children playing legos and being concerned they will build the next nuclear bomb. Hypothetically feasible but there are so many fundamental gaps that clearly separate the two that the real concerns would be when we see Manhattan projects being successful. From an outsider's perspective, Deep Mind and AlphaZero or OpenAI and LLMs seem like project Manhattans but the fact these companies have spent billions with no returns yet should say a lot about the utility of these models.
I fall entirely in the unqualified category, but I will add that there are people as equally or more credentialed than you that disagree. I don't say this to be snotty, but just to add context for other people reading that there is a wide spectrum of beliefs among people qualified to make such statements
> My take on this is that basically everything made possible in the last 20 years is because of GPUs and better tooling.
I feel like part of the optimism on the part of people like Sam Altman is that it's true the recent advances are made possible due to computing power available, but the tools that we will now be able to create to develop AI systems better and faster is what will enable us to maintain the acceleration we're seeing right now. As a simple example, we've massively reduced the time and cost of training in only a couple-few years.
Am I wrong in thinking that if GPT-4 gets trained on all your organization-wide data, it will become much better at, say, coding for your organization-specific apps?
Like right now, you ask it to code and it give you a generic, contextless function. But if it were trained on your company’s data, it should be able to give you a function that’s relevant to your tech stack, existing codebase, organizatinal best practices, etc?
Has anyone got this to actually do anything at all yet?
I see all of the half demos where it doesn't complete anything, I've tried it myself and.. well, if we're being honest it was shite. I've seen a whole load of tweet threads saying what it could be used for..
Literally just looking for one example of a successful run. Anything at all.
I can definitely see that there may be potential (if not this then the ideas that come off the back of this) but even I don't have a real use case for it yet, I'm just tinkering.
I guess my XY question: Am I being suckered into the web3 of AI? Lots of buzz, no use case.
There is some magical thinking at play, that once you have what appears to be an intelligence of some kind, simply allowing it to recursively self-critique will lead to singularity, but the other possibility is the signal just degrades, like deep-fried jpgs.
My testing would definitely lean towards deep fried jpgs aye. Every time it seemed to hit a bump somewhere then got caught up in a loop of googling whatever problem it was having and not finding an answer it was looking for, then going on to look into that problem, etc, etc. I guess you could put a step in to allow the human to nudge the AI back on track but then it's not really autonomous.
Also it's really slow at doing anything, even before it goes off into the weeds. I understand it's new and wont be polished but man. Really slow.
At least it's a decent motivator! "Oooh this is a task AutoGPT/BabyAGI could do... ehh it'd be quicker to just do it myself"
I'll keep an open mind, still eager to see a successful attempt at something. That'd at least open up a chance I'm doing it wrong rather than it not doing anything useful.
This has been reinforcement learning for decades. People say "OMG! The agent is acting and learning and getting better all on its own!" And they're right, in the beginning. Eventually the agent plateaus or collapses.
Supervised learning is much more stable. GTP is supervised learning. Once you start letting the agent choose or modify its own training data, then you're moving towards the much less stable realms of reinforcement learning.
True. I've read a lot about reinforcement learning and the exploration vs exploitation tradeoff is both technically interesting and philosophically interesting. I've also seen algorithms I've implemented fall to all sorts of human-ish flaws, such as learned helplessness, etc.
GPT is not supervised learning, it is self-supervised learning (a form of unsupervised learning) since the learning objective is "fill in the missing training data".
I'm asking for actual uses, not theoretical. I can, and have, come up with theoretical uses myself but when I test them nothing has resulted in success.
This is not AGI by any stretch of imagination. It is does not even appear to be a step on the path to AGI.
Furthermore, due to autoregressive nature of GPT models, the more auto-gpt generates (the more it works, the more tasks it performs..) the chance of things going off the right path grow exponentially, and then it is 'doomed' to the end [1].
Thus, chance of this being actually useful for anything longer than what a simple prompt can already do with a tool like ChatGPT is very low.
The end result is an impressive concept but a practically unusable tool. And the problem, in general, is that as the auto-gpt improves (which it will at impressive pace), so will our ambition in using it, which will lead to constant disappointment and what we have today will be generally how we feel about it in the future. Always needing "just a bit more", but never really there.
We already have a "baby AGI" that has been deployed in production environment for a few years - it is called Tesla self driving. It was supposed to get us from point a to point b completely autonomously. And for 6 years now it has been almost "almost there", but never really there (and arguably never will be).
What this does though, is create and inflate a giant FOMO, and the best way of dealing with FOMOs (long term) is to stay on the firm ground, observe, wait for clarity and the right action.
I kind of agree but I really don't see Tesla self-driving as even aspirational to AGI. It seems like the poster child for a domain specific AI that nobody has an interest in making general.
Would be interesting to see data. My impression is that human crashes are predominately caused by errors like lack of attention, poor impulse control (speeding, tailgating, etc), and slow processing of surprises.
80 comments
[ 2.9 ms ] story [ 148 ms ] thread[0] https://github.com/facebookresearch/faiss
https://youtu.be/WosPGHPObx8
Writing code from scratch to process and search 200k unstructured documents -- parsing, cleaning, chunking, OpenAI embedding API, serialization code, linear search with cosine similarity, and the actual time to debug, test and run all this -- took me less than 3 hours in Go. The flat binary representation of all vectors is under 500 MB. I even went ahead and made it mmap-friendly for the fun of it even though I could read it into all into memory.
Even the dumb linear search I wrote takes just 20-30ms per query on my Macbook for the 200k documents. The search results are fantastic.
I didn't bother cleaning it so it's just a code dump, but it's fairly straightforward. Not included are a Python script to parse and clean the raw documents into JSON files (used in `summarize` to output results), code to read these files and get the embeddings from OpenAI for use in `newEmbeddingJSON `, and a bunch of random parallelization shell scripts that I didn't save.
To use it, I call newDBFromJSON from a directory of JSON embedding vectors and serialize the binary representation. This takes a few minutes mostly because parsing JSON is slow, but you I only needed to do this once. When I need to search for the top 10 documents most similar to document X, I call `search` with the embedding vector for that doc. Alternatively if I need to do semantic search with natural language, I'll call the OpenAI API to get the embedding vector for the query and call `search` with that vector. It's pretty fast thanks to Go concurrency maxing out my CPU. It's super accurate with the search results thanks to OpenAI's embeddings.
It's nowhere close to production-ready (it's littered with panics), but it was good enough for me.
Hope this helps!
Edit: oh and don't use float64 (OpenAI's vectors are float16)
It mostly seems like a massively stoned teenager trapped in thought loops about how to use thought loops to achieve something, without ever hitting on the idea of actually doing something.
Where exactly are we with AI as a legitimate threat if we continue down our current path? Are people like Max just jockeying for attention? Or is there merit to their concerns?
Worry about human beings being the one to kill us all because that's a threat that's existed for decades with nukes and biological warfare. If it isn't weapons, it'll be self-inflicted climate change, or an asteroid we haven't noticed. SkyNet isn't happening.
Six billion human beings of average intelligence are a greater threat than a single hypothetical superintelligence.
Hypothetical obsolescence is not the same as being killed by a hypothetical superintelligence. We would be no more hypothetically obsolete than the overwhelming majority of lifeforms.
The argument for SkyNet happening is missing an important element: why would it happen? Because sci-fi says it will?
That element isn't missing. It's explained every time someone asks, and the answer never relies on sci-fi as evidence. The reason is because of (A) orthogonality and (B) instrumental convergence. If you aren't familiar with those terms, (A) refers to the thesis that an AI's degree of logical reasoning ability is independent of its goals. This stems from Hume's Law [1]. (B) refers to the observation that regardless of an agent's terminal goal, many instrumental goals will be predictably shared: the agent will be incentivized to increase its own capabilities, as a means toward nearly any possible end. This includes steps that would prevent humans from interfering with the AI's goals, such as a human attempting to turn it off or reprogram it.
If you have questions about either of those, there are great learning materials available, including a well-made youtube series by Robert Miles. [2]
> Because LLMs have reached a point of diminishing returns and there isn't enough compute globally to do produce a hypothetical AGI.
What makes you think LLMs have reached a point of diminishing returns? So far, the benchmarks are continuing to show accelerating returns, to the point that it's becoming difficult for benchmarks to keep up.[3]
[1]: Hume's Law, AKA the Is-Ought problem: https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem
[2]: https://www.youtube.com/watch?v=hEUO6pjwFOo
[3]: "Foundation models and the next era of AI", Microsoft Research, https://youtu.be/HQI6O5DlyFc?t=958
Encryption is just one example. Its ability across the entirety of math and science would be equally powerful.
* See also: Kasparov Versus the World, in which grandmaster Garry Kasparov defeated the "superintelligence" of thousands of chess masters combined: https://en.wikipedia.org/wiki/Kasparov_versus_the_World
1. AI kills us all; Max: "See, I told you! ayeee!"
2. AI integrates smoothly into Society. Max: "Sure was good I got you worried about the risks, so we avoided them!"
3. AI keeps improving, little by little, causing hiccups along the way but nothing disastrous; Max: "Just wait, there will be doooom!"
He can't lose with this type of argument. Heads he wins, tails he wins.
Personally, living in a hot tropical country, I've experienced weather patterns becoming weirder and weirder, and generally trending towards way too hot. Last year, the summer was so long and dry and hot that I really felt that I can't physically live here any longer. And that's when I had the luxury of AC - something the vast majority of my country can't afford.
So yeah, it's easy to dismiss climate change if you live in a cold climate. It's much more real in warm, dry countries.
Can we scale up current architectures to beyond human intelligence without further advances in theory/methods? If I had to guess I’d say no.
Is another breakthrough of the order of transformers likely in the next decade? Who knows…
I think your question has two angles. First, do LLMs have potential for AGI? I think that's an emphatic no. There is nothing special about this generative model vs. say something like sampling from a mixture of Gaussians. Much better generations and super impressive it works, sure, but there is no mechanism for it to improve itself let alone change its "prime directive". See Sam Altman's claim that RL with human feedback being where most are the gains are now.
At a higher level, there is the concern that the pace at which we are going that we can extrapolate that AGI is around the corner. My take on this is that basically everything made possible in the last 20 years is because of GPUs and better tooling. In specific, the recent hype we see is because of how democratized things are getting. We have kids using ChatGPT to do homework. This is very disruptive from a societal perspective but we are essentially at the height of the rate at which this technology is being adopted. The growth rate will decelerate and it will stop being news once society has learned to adapt to the new technology. However, from a technical perspective, these concerns are like looking at children playing legos and being concerned they will build the next nuclear bomb. Hypothetically feasible but there are so many fundamental gaps that clearly separate the two that the real concerns would be when we see Manhattan projects being successful. From an outsider's perspective, Deep Mind and AlphaZero or OpenAI and LLMs seem like project Manhattans but the fact these companies have spent billions with no returns yet should say a lot about the utility of these models.
> My take on this is that basically everything made possible in the last 20 years is because of GPUs and better tooling.
I feel like part of the optimism on the part of people like Sam Altman is that it's true the recent advances are made possible due to computing power available, but the tools that we will now be able to create to develop AI systems better and faster is what will enable us to maintain the acceleration we're seeing right now. As a simple example, we've massively reduced the time and cost of training in only a couple-few years.
Like right now, you ask it to code and it give you a generic, contextless function. But if it were trained on your company’s data, it should be able to give you a function that’s relevant to your tech stack, existing codebase, organizatinal best practices, etc?
I see all of the half demos where it doesn't complete anything, I've tried it myself and.. well, if we're being honest it was shite. I've seen a whole load of tweet threads saying what it could be used for..
Literally just looking for one example of a successful run. Anything at all.
I can definitely see that there may be potential (if not this then the ideas that come off the back of this) but even I don't have a real use case for it yet, I'm just tinkering.
I guess my XY question: Am I being suckered into the web3 of AI? Lots of buzz, no use case.
Also it's really slow at doing anything, even before it goes off into the weeds. I understand it's new and wont be polished but man. Really slow.
At least it's a decent motivator! "Oooh this is a task AutoGPT/BabyAGI could do... ehh it'd be quicker to just do it myself"
I'll keep an open mind, still eager to see a successful attempt at something. That'd at least open up a chance I'm doing it wrong rather than it not doing anything useful.
Supervised learning is much more stable. GTP is supervised learning. Once you start letting the agent choose or modify its own training data, then you're moving towards the much less stable realms of reinforcement learning.
I don't think we're on the threshold of AGI, but it's interesting that the wanna-be AIs are running into human issues...
I'm asking for actual uses, not theoretical. I can, and have, come up with theoretical uses myself but when I test them nothing has resulted in success.
and follow along with the AI System patterns series I've just started: https://aistud.io/blog/ai-system-patterns-parallel-conversat...
Furthermore, due to autoregressive nature of GPT models, the more auto-gpt generates (the more it works, the more tasks it performs..) the chance of things going off the right path grow exponentially, and then it is 'doomed' to the end [1].
Thus, chance of this being actually useful for anything longer than what a simple prompt can already do with a tool like ChatGPT is very low.
The end result is an impressive concept but a practically unusable tool. And the problem, in general, is that as the auto-gpt improves (which it will at impressive pace), so will our ambition in using it, which will lead to constant disappointment and what we have today will be generally how we feel about it in the future. Always needing "just a bit more", but never really there.
We already have a "baby AGI" that has been deployed in production environment for a few years - it is called Tesla self driving. It was supposed to get us from point a to point b completely autonomously. And for 6 years now it has been almost "almost there", but never really there (and arguably never will be).
What this does though, is create and inflate a giant FOMO, and the best way of dealing with FOMOs (long term) is to stay on the firm ground, observe, wait for clarity and the right action.
[1] Watch in particular Yann LeCun's presentation at https://www.youtube.com/watch?v=x10964w00zk
AGI means it can do any task. I’m not seeing driving, as complicated as it is, as requiring truly general intelligence.
Lots of algorithms like nearest neighbor search are O(n^2) but algorithms for approximate results run in sublinear time.
<1> https://www.lesswrong.com/posts/566kBoPi76t8KAkoD/on-autogpt