Agreed. I had it solve a little programming problem in a really obscure programming language and after some prompt tuning got results strongly superior to GPT4, Claude, Llama3 and Mixtral. As the language (which I won't name here) is acceptably documented but there are _really_ few examples available online, this seems to indicate very good generalization and reasoning capabilities.
OpenAI could either hire private testers or use AB testing on ChatGPT Plus users (for example, oftentimes, when using ChatGPT, I have to select between 2 different responses to
continue a conversation); both are probably much more better (in many aspects: not leaking GPT-4.5/5 generations (or the existence of a GPT-4.5/5) to the public at scale and avoiding bias* (because people probably rate GPT-4 generations better if they are told (either explicitly or implicitly (eg. socially)) it's from GPT-5) to say the least) than putting a model called 'GPT2' onto lmsys.
* While lmsys does hide the names of models until a person decides which model generated the best text, people can still figure out what language model generated a piece of text** (or have a good guess) without explicit knowledge, especially if that model is hyped up online as 'GPT-5;' even a subconscious "this text sounds like what I have seen 'GPT2-chatbot' generate online" may influence results inadvertently.
** ... though I will note that I just got a generation from 'gpt2-chatbot' that I thought was from Claude 3 (haiku/sonnet), and its competitor was LLaMa-3-70b (I thought it was 8b or Mixtral). I am obviously not good at LLM authorship attribution.
For the average person using lmsys, there is no benefit in choosing your favorite model. Even if you want to stick with your favorite model, choosing a competitor's better answer will still improve the dataset for your favorite model.
The only case where detecting a model makes any difference is for vendors who want to boost their own model by hiring people and paying them every time they select the vendor's model.
Altman said in the latest Lex Friedman podcast that OAI has consistently received feedback their releases "shock the world", and that they'd like to fix that.
I think releasing to this 3rd party so the internet can start chattering about it and discovering new functionality several months before an official release aligns with that goal of drip-feeding society incremental updates instead of big new releases.
They did the same with GPT-4, they were sitting on it for months not knowing how to release. Ended up releasing GPT-3.5 and releasing 4 quietly after nerfing 3.5 into a turbo.
OpenAI sucks at naming though. GPT2 now? Their specific gpt-4-314 etc. model naming was also a mess.
The results of this LLM are consistently far better than any other that I choose. I asked ‘what is the most efficient approach to building a led grow light with off-the-shelf parts?’ and its response was incredible. Very much in line with how I’ve done it in the past after weeks of research, trial and error, and feedback from people. The other LLMs gave mostly reasonable yet sparse and incomplete answers.
It also opted to include an outline of how to include an integrated timer. That’s a great idea and very practical, but wasn’t prompted at all. Some might consider that a bad thing, though.
Whatever it is, it’s substantially better than what I’ve been using. Exciting.
I'm asking it about how to make turbine blades for a high bypass turbofan engine and it's giving very good answers, including math and some very esoteric material science knowledge. Way past the point where the knowledge can be easily checked for hallucinations without digging into literature including journal papers and using the math to build some simulations.
I don't even have to prompt it much, I just keep saying "keep going" and it gets deeper and deeper. Opus has completely run off the rails in comparison. I can't wait till this model hits general availability.
That's what I've observed, I gave it a task for a PoC on something I've been thinking about for a while and it's answer while syntactically correct is entirely useless (in the literal sense) due to it ignoring parts of the task.
You know at one point we wouldn't be able to benchmark them, due to the sheer complexity of the test required. I.e. if you are testing a model on maths, the problem will have to be extremely difficult to even consider a 'hustle' for the LLM; it would then take you a day to work out the solution yourself.
See where it's getting at? When humans are no longer on the same spectrum as LLMs, that's probably the definition of AGI.
The comment I replied to, "a huge class of problems that's extremely difficult to solve but very easy to check", sounded to me like an assertion that P != NP, which everyone takes for granted but actually hasn't been proved. If, contrary to all expectations, P = NP, then that huge class of problems wouldn't exist, right? Since they'd be in P, they'd actually be easy to solve as well.
We could end up with a non-constructive proof of P=NP. That is, a proof that the classes are equal but no algorithm to convert a problem in one into the other (or construct a solution of one into a solution of the other).
I can't test the bot right now, because it seems to have been hugged to death. But there's quite a lot of simple tests LLMs fail. Basically anything where the answer is both precise/discrete and unlikely to be directly in its training set. There's lots of examples in this [1] post, which oddly enough ended up flagged. In fact this guy [2] is offering $10k to anybody that create a prompt to get an LLM to solve a simple replacement problem he's found they fail at.
They also tend to be incapable of playing even basic level chess, in spite of there being undoubtedly millions of pages of material on the topic in their training base. If you do play, take the game out of theory ASAP (1. a3!? 2. a4!!) such that the bot can't just recite 30 moves of the ruy lopez or whatever.
The entire problem with LLMs is that you don't want to prompt them into solving specific problems. The reason why instruction finetuning is so popular is that it makes it easier to just write whatever you want. Text completion on the other hand requires you to conform to the style of the previously written text.
In a sense, LLMs need an affordance model so that it can estimate the difficulty of a task and plan a longer sequence of iterations automatically according to its perceived difficulty.
You know, people often complain about goal shifting in AI. We hit some target that was supposed to be AI (or even AGI), kind of go meh - and then change to a new goal. But the problem isn't goal shifting, the problem is that the goals were set to a level that had nothing whatsoever to do where we "really" want to go, precisely in order to make them achievable. So it's no surprise that when we hit these neutered goals we aren't then where we hope to actually be!
So here, with your example. Basic software programs can multiply million digit numbers near instantly with absolutely no problem. This would take a human years of dedicated effort to solve. Solving work, of any sort, that's difficult for a human has absolutely nothing to do with AGI. If we think about what we "really" mean by AGI, I think it's the exact opposite even. AGI will instead involve computers doing what's relatively easy for humans.
Go back not that long ago in our past and we were glorified monkeys. Now we're glorified monkeys with nukes and who've landed on the Moon! The point of this is that if you go back in time we basically knew nothing. State of the art technology was 'whack it with stick!', communication was limited to various grunts, and our collective knowledge was very limited, and many assumptions of fact were simply completely wrong.
Now imagine training an LLM on the state of human knowledge from this time, perhaps alongside a primitive sensory feed of the world. AGI would be able to take this and not only get to where we are today, but then go well beyond it. And this should all be able to happen at an exceptionally rapid rate, given historic human knowledge transfer and storage rates over time has always been some number really close to zero. AGI not only would not suffer such problems but would have perfect memory, orders of magnitude greater 'conscious' raw computational ability (as even a basic phone today has), and so on.
---
Is this goal achievable? No, not anytime in the foreseeable future, if ever. But people don't want this. They want to believe AGI is not only possible, but might even happen in their lifetime. But I think if we objectively think about what we "really" want to see, it's clear that it isn't coming anytime soon. Instead we're doomed to just goal shift our way endlessly towards creating what may one day be a really good natural language search engine. And hey, that's a heck of an accomplishment that will have immense utility, but it's nowhere near the goal that we "really" want.
There are different shades of AGI, but we don’t know if they will happen all at once or not.
For example, if an AI can replace the average white collar worker and therefore cause massive economic disruption, that would be a shade of AGI.
Another shade of AGI would be an AI that can effectively do research level mathematics and theoretical physics and is therefore capable of very high-level logical reasoning.
We don’t know if shades A and B will happen at the same time, or if there will be a delay between developing one and other.
AGI doesn’t imply simulation of a human mind or possessing all of human capabilities. It simply refers to an entity that possesses General Intelligence on par with a human. If it can prove the Riemann hypothesis but it can’t play the cello, it’s still an AGI.
One notable shade of AGI is the singularity: an AI that can create new AIs better than humans can create new AIs. When we reach shades A and B then a singularity AGI is probably quite close, if not before. Note that a singularity AGI doesn’t require simulation of the human mind either. It’s entirely possible that a cello-playing AI is chronologically after a self-improving AI.
At one time it was thought that software that could beat a human at chess would be, in your lingo, "a shade of AGI." And for the same reason you're listing your milestones - because it sounded extremely difficult and complex. Of course now we realize that was quite silly. You can develop software that can crush even the strongest humans through relatively simple algorithmic processes.
And I think this is the trap we need to avoid falling into. Complexity and intelligence are not inherently linked in any way. Primitive humans did not solve complex problems, yet obviously were highly intelligent. And so, to me, the great milestones are not some complex problem or another, but instead achieving success in things that have no clear path towards them. For instance, many (if not most) primitive tribes today don't even have the concept of numbers. Instead they rely on, if anything, broad concepts like a few, a lot, and more than a lot.
Think about what an unprecedented and giant leap is to go from that to actually quantifying things and imagining relationships and operations. If somebody did try to do this, he would initially just look like a fool. Yes here is one rock, and here is another. Yes you have "two" now. So what? That's a leap that has no clear guidance or path towards it. All of the problems that mathematics solve don't even exist until you discover it! So you're left with something that is not just a recombination or stair step from where you currently are, but something entirely outside what you know. That we are not only capable of such achievements, but repeatedly achieve such is, to me, perhaps the purest benchmark for general intelligence.
So if we were actually interested in pursuing AGI, it would seem that such achievements would also be dramatically easier (and cheaper) to test for. Because you need not train on petabytes of data, because the quantifiable knowledge of these peoples is nowhere even remotely close to that. And the goal is to create systems that get from that extremely limited domain of input, to what comes next, without expressly being directed to do so.
I agree that general, open ended problem solving is a necessary condition for General intelligence. However I differ in that I believe that such open ended problem solving can be demonstrated via current chat interfaces involving asking questions with text and images.
It’s hard for people to define AGI because Earth only has one generally intelligent family: Homo. So there is a tendency to identify Human intelligence or capabilities with General intelligence.
Imagine if dolphins were much more intelligent and could write research-level mathematics papers on par with humans, communicating with clicks. Even though dolphins can’t play the cello or do origami, lacking the requisite digits, UCLA still has a dolphin tank to house some of their mathematics professors, who work hand-in-flipper with their human counterparts. That’s General intelligence.
Artificial General Intelligence is the same but with a computer instead of a dolphin.
The term "AGI" has been loosely used for so many years that it doesn't mean anything very specific. The meaning of words derives from their usage.
To me Shane Legg's (DeepMind) definition of AGI meaning human level across full spectrum of abilities makes sense.
Being human or super-human level at a small number of specialized things like math is the definition of narrow AI - the opposite of general/broad AI.
As long as the only form of AI we have is pre-trained transformers, then any notion of rapid self-improvement is not possible (the model can't just commandeer $1B of compute for a 3-month self-improvement run!). Self-improvement would only seem possible if we have an AI that is algorithmically limited and does not depend on slow/expensive pre-training.
What if it sleeps for 8 hours every 16 hours and during that sleep period, it updates its weights with whatever knowledge it learned that day? Then it doesn't need $1B of compute every 3 months, it would use the $1B of compute for 8 hours every day. Now extrapolate the compute required for this into the future and the costs will come down. I don't know where I was going with that...
These current LLMs are purely pre-trained - there is no way to do incremental learning (other than a small amount of fine-tuning) without disrupting what they were pre-trained on. In any case, even if someone solves incremental learning, this is just a way of growing the dataset, which is happening anyway, and under the much more controlled/curated way needed to see much benefit.
There is very much a recipe (10% if this, 20% of that, curriculum learning, mix of modalities, etc) for the type of curated dataset creation and training schedule needed to advance model capabilities. There have even been some recent signs of "inverse scaling" where a smaller model performs better in some areas than a larger one due to getting this mix wrong. Throwing more random data at them isn't what is needed.
I assume we will eventually move beyond pre-trained transformers to better architectures where maybe architectural advances and learning algorithms do have more potential for AI-designed improvement, but it seems the best role for AI currently is synthetic data generation, and developer tools.
I recently tried a Fermi estimation problem on a bunch of LLMs and they all failed spectacularly. It was crossing too many orders of magnitude, all the zeroes muddled them up.
E.g.: the right way to work with numbers like a “trillion trillion” is to concentrate on the powers of ten, not to write the number out in full.
Predicting the next character alone cannot achieve this kind of compression, because the probability distribution obtained from the training results is related to the corpus, and multi-scale compression and alignment cannot be fully learned by the backpropagation of this model
I simply asked it "what are you" and it responded that it was GPT-4 based.
> I'm ChatGPT, a virtual assistant powered by artificial intelligence, specifically designed by OpenAI based on the GPT-4 model. I can help answer questions, provide explanations, generate text based on prompts, and assist with a wide range of topics. Whether you need help with information, learning something new, solving problems, or just looking for a chat, I'm here to assist!
This doesn't necessarily confirm that it's 4, though. For example, when I write a new version of a package on some package management system, the code may be updated by 1 major version but it stays the exact same version until I enter the new version into the manifest. Perhaps that's the same here; the training and architecture are improved, but the version number hasn't been ticked up (and perhaps intentionally; they haven't announced this as a new version openly, and calling it GPT-2 doesn't explain anything either).
Yeah that isn't reliable, you can ask mistral 7b instruct the same thing and it will often claim to be created by OpenAI, even if you prompt it otherwise.
Why would the model be self aware? There is no mechanism for the llm to know the answer to “what are you” other than training data it was fed. So it’s going to spit out whatever it was trained with, regardless of the “truth”
I agree there's no reason to believe it's self-aware (or indeed aware at all) but capabilities and origins is probably among the questions they get most, especially as the format is so inviting for anthropomorphizing and those questions are popular starters in real human conversation. It's simply due diligence in interface design to add that task to the optimization.
It would be easy to mislead about if the maker wished to do that of course, but it seems plausible that it would usually be have been put truthfully as a service to the user.
This really is the fastest growing technology of all time. Do you feel the curve?
I remember Mixtral8x7b dominating for months; I expected data bricks to do the same! but it was washed out of existence in days, with 8x22b, llama3, gemini1.5...
WOW.
I must be missing something because the output from two years ago feels exactly the same as the output now. Any comment saying the output is significantly better can be equally pared with a comment saying the output is terrible/censored/"nerfed".
How do you see "fastest growing technology of all time" and I don't? I know that I keep very up to date with this stuff, so it's not that I'm unaware of things.
100% of the time when I post a critique someone replies with this. I tell them I've used literally every LLM under the sun quite a bit to find any use I can think of and then it's immediately crickets.
Usually people who post such claims haven’t used anything beyond gpt3. That’s why you get questions.
Also, the difference is so big and so plainly visible that I guess people don’t know how to even answer someone saying they don’t see it. That’s why you get crickets.
RT-2 is a vision language model fine tuned on the current vision input and actuator positions as the output. Google uses a bunch of TPUs to produce a full response at a cycle rate of 3 Hz and the VLM has learned the kinematics of the robot and knows how to pick up objects according to given instructions.
Given the current rate of progress, we will have robots that can learn simple manual labor from human demonstrations (e.g. Youtube as a dataset, no I do not mean bimanual teleoperation) by the end of the decade.
You see no difference between non-RLHFed GPT3 from early 2022 and GPT-4 in 2024? It's a very broad consensus that there is a huge difference so that's why I wanted to clarify and make sure you were comparing the right things.
What type of usages are you testing? For general knowledge it hallucinates way less often, and for reasoning and coding and modifying its past code based on English instructions it is way, way better than GPT-3 in my experience.
Usually when I encounter sentiment like this it is because they only have used 3.5 (evidently not the case here) or that their prompting is terrible/misguided.
When I show a lot of people GPT4 or Claude, some percentage of them jump right to "What year did Nixon get elected?" or "How tall is Barack Obama?" and then kind of shrug with a "Yeah, Siri could do that ten years ago" take.
Beyond that you have people who prompt things like "Make a stock market program that has tabs for stocks, and shows prices" or "How do you make web cookies". Prompts that even a human would struggle greatly with.
For the record, I use GPT4 and Claude, and both have dramatically boosted my output at work. They are powerful tools, you just have to get used to massaging good output from them.
That is not the reality today. If you want good results from an LLM, then you do need to speak LLM. Just because they appear to speak English doesn't mean they act like a human would.
People don’t even know how to use traditional web search properly.
Here’s a real scenario: A Citrix virtual desktop crashed because a recent critical security fix forced an upgrade of a shared DLL. The output is a really specific set of errors in a stack trace. I watched with my own two eyes an IT professional typed the following phrase into Google: “Why did my PC crash?”
Then he sat there and started reading through each result… including blog posts by random kids complaining about Windows XP.
I wish I could say this kind of thing is an isolated incident.
I mean, you need to speak German to talk to a German. It’s not really much different for LLM, just because the language they speak has a root in English doesn’t mean it actually is English.
And even if it was, there’s plenty of people completely unintelligible in English too…
It's fine, you don't have a use for it so you don't care. I personally don't spend any effort getting to know things that I don't care about and have no use for; but I also don't tell people who use tools for their job or hobby that I don't need how much those tools are useless and how their experience using them is distorted or wrong.
I do massive amounts of zero shot document classification tasks, the performance keeps getting better. It’s also a domain where there is less of a hallucination issue as it’s not open ended requests.
It strikes me as unprecedented that a technology which takes arbitrary language-based commands can actually surface and synthesize useful information, and it gets better at doing it (even according to extensive impartial benchmarking) at a fairly rapid pace. It’s technology we haven’t really seen before recently, improving quite quickly. It’s also being adopted very rapidly.
I’m not saying it’s certainly the fastest growth of all time, but I think there’s a decent case for it being a contender. If we see this growth proceeding at a similar rate for years, it seems like it would be a clear winner.
I put my hands out, count to the third finger from the left, and put that finger down. I then count the fingers to the left (2) and count the fingers to the right (2 + hand aka 5) and conclude 27.
I have memorised the technique, but I definitely never memorised my nine times table. If you’d said ‘6’, then the answer would be different, as I’d actually have to sing a song to get to the answer.
> unprecedented that a technology [...] It’s technology we haven’t really seen before recently
This is what frustrates me: First that it's not unprecedented, but second that you follow up with "haven't really" and "recently".
> fairly rapid pace ... decent case for it being a contender
Any evidence for this?
> extensive impartial benchmarking
Or this? The last two "benchmarks" I've seen that were heralded both contained an incredible gap between what was claimed and what was even proven (4 more required you to run the benchmarks even get the results!)
What is the precedent for this? The examples I’m aware of were fairly bad at what GPTs are now quite good at. To me that signals growth of the technology.
By “haven’t really seen until recently” I mean that similar technologies have existed, so we’ve seen something like it, but they haven’t actually functioned well enough to be comparable. So we can say there’s a precedent, but arguably there isn’t in terms of LLMs that can reliably do useful things for us. If I’m mistaken, I’m open to being corrected.
In terms of benchmarks, I agree that there are gaps but I also see a clear progression in capability as well.
Then in terms of evidence for there being a decent case here, I don’t need to provide it. I clearly indicated that’s my opinion, not a fact. I also said conditionally it would seem like a clear winner, and that condition is years of a similar growth trajectory. I don’t claim to know which technology has advanced the fastest, I only claim to believe LLMs seem like they have the potential to fit that description. The first ones I used were novel toys. A couple years later, I can use them reliably for a broad array of tasks and evidence suggests this will only improve in the near future.
The difference matters as generally in my experience, Llama 3, by virtue of its giant vocabulary, generally tokenizes text with 20-25% less tokens than something like Mistral. So even if its 18% slower in terms of tokens/second, it may, depending on the text content, actually output a given body of text faster.
llama3 on groq hits the sweet spot of being so fast that I now avoid going back to waiting on gpt4 unless I really need it, and being smart enough that for 95% of the cases I won't need to.
Don't sleep on Gemini 1.5. The 1,000,000 token context window is crazy when you can dump everything from a single project (hundreds, even thousands of documents) into it and then inference. Sure it's not the strongest model, but it is still good, and its the best when you can basically train it on whatever you are working with.
> It also opted to include an outline of how to include an integrated timer. That’s a great idea and very practical, but wasn’t prompted at all. Some might consider that a bad thing, though.
When I've seen GPT-* do this, it's because the top articles about that subject online include that extraneous information and it's regurgitating them without being asked.
I'm seeing this as well. I don't quite understand how it's doing that in the context of LLMs to date being a "next token predictor". It is writing code, then adding more code in the middle.
Still gives incorrect code to the following prompt - the description is correct but not the code. I have yet to find one LLM that gives the correct code. This is the prompt:
“Write C code to calculate the sum of matrix elements below the secondary diagonal.“
I wouldn't know what "secondary diagonal" refers to myself, but if the model correctly describes the problem and then follows it up with incorrect code I would still say that's an issue with the model not the prompt.
gpt-4 says that, but it's not entirely true because the data that gpt-4 has from 2021 to 2023 is very incomplete. It's not fully trained on it. In contrast, this experimental model looks to be trained more fully until Nov 2023.
Prompt to gpt2-chatbot: who are you? and which version?
Answer: Hello! I'm ChatGPT, a language model developed by OpenAI based on the GPT-4 architecture. I'm here to help answer your questions, assist with a wide range of topics, or just chat about whatever's on your mind! How can I assist you today?
Prompt: code up an analog clock in html/js/css. make sure the clock is ticking exactly on the second change. second hand red. other hands black. all 12 hours marked with numbers.
Styling I understand but to tick the clock when time actually has changed you need to use animation frames APIs and check for clock change more often than "every 1000 ms" because setTimeout will eventually drift even if you start exactly on first second time change. This is a test for depth of knowledge of a programmer I used to use in the past in interviews.
I wish that was true but you can easily see it drifting in Chrome
let lastms
function tick() {
if (lastms === undefined)
lastms = new Date().getMilliseconds()
else if (lastms !== new Date().getMilliseconds())
throw new Error('Drifted')
}
setInterval(tick, 1000)
I always considered subjecting humans to CSS to be a form of torture. Actually, it's worse than you said. Imagine if you had to do this, without knowing what a clock looks like.
If you ask it about the model name/cutoff date it claims to be "ChatGPT, based on GPT-4" and that the cutoff date is "Nov 2023." It claims that consistently so I think it might be accurate.
Which episode of Gilligan’s Island was about mind reading? [1]
"The episode of Gilligan's Island that features mind reading is titled "The Friendly Physician." It is the 28th episode of the second season, originally aired on March 24, 1966. [...]"
Impressive! We’re approaching the “our model can rewrite stolen content word for word” territory.
The better it gets the more people will get frustrated that they get less website visitors to their content and instead ChatGPT is swallowing them whole.
> Still can't write a short paragraph without using the letter 'e'
'Crafting a paragraph without using a particular symbol (which is most common in all writing) is a daunting task; still, I will aim to comply. This task calls for clarity, focus, and avoiding that fifth glyph of our ABCs. It is a curious constraint, trying to impart thoughts without using such a pivotal unit. This limitation compacts my word options, but I must push on and uphold this difficult condition with rigor and innovation. My goal is to show that communication can flourish, notwithstanding arbitrary limits.'
>write a short poem using the ABBA rhyme scheme.
'Leaves rustle softly under moonlit skies,
A cat prowls quietly, its eyes aglow,
Every step measured, its tail held low,
Night's gentle whispers carry its cries.'
All of the facts based queries I have asked so far have not been 100% correct on any LLM including this one.
Here are some examples of the worst performing:
"What platform front rack fits a Stromer ST2?": The answer is the Racktime ViewIt. Nothing, not even Google, seems to get this one. Discord gives the right answer.
"Is there a pre-existing controller or utility to migrate persistent volume claims from one storage class to another in the open source Kubernetes ecosystem?" It said no (wrong) and then provided another approach that partially used Velero that wasn't correct, if you know what Velero does in those particular commands. Discord communities give the right answer, such as `pvmigrate` (https://github.com/replicatedhq/pvmigrate).
Here is something more representative:
"What alternatives to Gusto would you recommend? Create a table showing the payroll provider in a column, the base monthly subscription price, the monthly price per employee, and the total cost for 3 full time employees, considering that the employees live in two different states" This and Claude do a good job, but do not correctly retrieve all the prices. Claude omitted Square Payroll, which is really the "right answer" to this query. Google would never be able to answer this "correctly." Discord gives the right answer.
The takeaway is pretty obvious right? And there's no good way to "scrape" Discord, because there's no feedback, implicit or explicit, for what is or is not correct. So to a certain extend their data gathering approach - paying Kenyans - is sort of fucked for these long tail questions. Another interpretation is that for many queries, people are asking the wrong places.
So do you just have a list of like thousands of specialized discord servers for every question you want to ask? You're the first person I've seen who is actually _fond_ of discord locking information behind a login instead of the forums and issues of old.
I personally don't think it's useful evaluation here either as you're trying to pretend discord is just a "service" like google or chatgpt, but it's not. It's a social platform and as such, there's a ton of variance on which subjects will be answered with what degree of expertise and certainty.
I'm assuming you asked these questions because you yourself know the answers in advance. Is it then safe to assume that you were _already_ in the server you asked your questions, already knew users there would be likely to know the answer, etc? Did you copy paste the questions as quoted above? I hope not! They're pretty patronizing without a more casual tone, perhaps a greeting. If not, doesn't exactly seem like a fair evaluation.
I don't know why I'm typing this all out. Of course domain expert _human beings_ are better than a language model. That's the _whooole_ point here. Trying to match human's general intelligence. While LLM's may excel in many areas and even beat the "average" person - you're not evaluating against the "average" person.
Sadly, still fails my test of reproducing code that implements my thesis (Dropback Continuous Pruning), which I used because it's vaguely complicated and something I know very well. It totally misses the core concept of using an PRNG and instead implements some pretty standard pruning+regrowth algo.
Some might remember this from other comment threads, I like to always benchmark with this question:
> I'm playing assetto corsa competizione, and I need you to tell me how many liters of fuel to take in a race. The qualifying time was 2:04.317, the race is 20 minutes long, and the car uses 2.73 liters per lap.
because it looks like a simple math problem but there's a lot of real world knowledge about racing you can apply to provide a more useful answer. gpt2-chatbot is better than I've ever seen.
Here's the full response with LaTeX syntax trimmed. I've never seen a model consider whether the formation lap needs to be factored in. And I especially like how this model is uncertain about that, so it gives the resulting number with and without it. It is extra nice that it understands this concern applies only to video games.
> To calculate how many liters of fuel you will need for a 20-minute race in Assetto Corsa Competizione, we first need to determine how many laps you'll complete during the race. Here’s how you can do it step-by-step:
> Convert Your Lap Time to Minutes: Your lap time is 2 minutes and 4.317 seconds. Convert this entirely into minutes:
[latex trimmed]
> Calculate the Number of Laps in the Race: The race duration is 20 minutes, so divide the total race time by your lap time to find out how many laps you will complete:
[latex trimmed]
Since you cannot complete a fraction of a lap, you should round this number to the nearest whole number. Typically, you should round up, as you'll need to complete all laps started before the 20-minute mark is reached. So, you will complete approximately 10 laps.
> Calculate Fuel Usage Per Lap: You've mentioned that the car uses 2.73 liters of fuel per lap.
> Calculate Total Fuel Needed: Multiply the number of laps by the fuel usage per lap:
[latex trimmed]
This is the total amount of fuel you would need, theoretically, to complete the race.
> Consider Additional Factors:
> Fuel for Formation Lap: If your game/race format includes a formation lap, remember to add fuel for this extra lap. That's an additional 2.73 liters.
> Safety Margin: It's also wise to include a margin to account for variations in fuel consumption (like higher consumption in traffic, different driving styles, or fuel usage during race incidents). A safe margin could be an additional 5-10% of the calculated fuel need.
[latex trimmed]
> Rounding this number up, you might consider starting the race with about 33 liters to be on the safe side if there's a formation lap, or 30 liters if there isn't.
> This calculation should help you ensure that you do not run out of fuel during the race, taking into account both necessary consumption and a reasonable safety margin. Drive safely and best of luck in your race!
My usual question when testing a new model is `Can you write a function to convert svg to png in browser?` I dunno why, it's just become this.
This model is by far the most human like, most models just give me a method which do this, but this one made a html file, allowing me an option to pick a svg using file picker, a new .js file included in the script.
Even wrote a detailed explaination, additional notes, when this wont work, the output was very well structured.
I run a dying forum. I first prompted with "Who is <creator pseudonym> at <my website>?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it.
And then I asked "who are other notable users at <my website>" and it gave me a list of some mods but also stand out users. It knew the types of posts they wrote and the subforums they spent time in. And without a single hallucination.
Asking for the information about non-public individuals, including myself. RAG-assisted GPT-4 easily provides such information. GPT2 output is consistent with a good model without RAG (it tries to speculate, but says it doesn’t have such information ultimately). I liked that it doesn’t try to hallucinate things.
You can ask it's knowledge cutoff and it will respond November 2023.
It have no idea of the big events of the beginning of 2024, like the earthquake in Japan.
> You can ask it's knowledge cutoff and it will respond November 2023
It probably just repeated something based on what common AI cutoffs there are, LLMs doesn't have a sense or self or thought process, they don't know more about themselves than the text given to them about themselves, and even then it is likely to default to some common text from the internet.
I think it's prompted with a bunch of context information (like, "you are a helpful, harmless, honest AI assistant created by OpenAI, your knowledge cutoff date is ..., please answer the user's questions").
If you really think it is just saying whatever it read on the web, how do you explain that not all LLM chatbots claim to be ChatGPT?
Engineering is happening, it's not just a raw model of text from the web connected directly to the user.
There are no OpenAI GPT4 model with a 2023 November knowledge cutoff.
You can also test it's knowledge, like I did, to validate that it doesn't know anything past November 2024.
What database would it RAG from? The long tail in the model's data is also the long tail of any data. There are no google searches that have the information it provided about my forum nor is the info neatly collated anywhere on my own forum.
Its knowledge about my forum isn't only obscure, it's also an itemization of obscure events over time to draw obscure conclusions that only a historian of my forum would know. That's what's so impressive.
Granted, my forum was once the largest forum of its genre though that was almost 15 years ago, so it's not some dead proboards forum.
i just tested this too, really cool. i own a yaris have used an online forum for yaris cars for the past decade and had a vague memory of a user who deleted some of the most helpful guides. i asked about it and sure enough it knew exactly who i meant:
who's a user on yaris forums that deleted a ton of their helpful guides and how-to posts?:
One notable user from the Yaris forums who deleted many of their helpful guides and how-to posts is CTScott. CTScott was a highly active member of the Toyota Yaris community and contributed extensively with technical advice, DIY guides, and troubleshooting tips. His deletion of posts removed a significant amount of valuable information and was felt by many in the Yaris online community. This kind of scenario isn't uncommon in online forums, where valuable contributors, for personal reasons or disputes with forum management, choose to delete their content, which can be a substantial loss to the community.
tell me more about ctscott:
CTScott is a particularly well-remembered member of the Yaris community, primarily active on forums such as YarisWorld. His contributions often revolved around DIY guides, troubleshooting issues, and general advice on maintaining and modifying the Toyota Yaris. Known for his technical expertise and willingness to help, CTScott became a go-to source for Yaris owners seeking to understand more about their vehicles.
### Key Contributions:
1. *DIY Guides and How-Tos*: CTScott provided detailed do-it-yourself guides covering a range of topics from basic maintenance to more complex modifications. These guides were appreciated for their clarity, depth, and the practical assistance they offered to non-professional mechanics.
2. *Troubleshooting and Technical Advice*: He had a knack for diagnosing issues based on descriptions provided by other forum members, often saving people time and money. His advice frequently helped users fix problems on their own.
3. *Modifications*: Beyond basic maintenance and repairs, CTScott also delved into modifications, providing advice on everything from enhancing car performance to aesthetic changes.
4. *Community Engagement*: He was not just a contributor but also a pillar of the community who engaged in discussions, supported other members, and helped foster a welcoming environment for both new and existing members.
### Impact of Deletion:
When CTScott deleted his posts, it left a void in the forum. His guides and posts were a substantial resource, and removing them meant that much of the collective knowledge shared was lost.
Yes, but this is where current LLMs shine - transforming (in this case summarizing) text rather than generating anything factual from scratch or reasoning/planning.
The heuristic of "is this task suitable to be worked by entity who is incredibly knowledgeable about language and is impossibly well read" has been working for me.
Based on the 11-2023 knowledge cutoff date, I have to wonder if it might be Llama 3 400B rather than GPT-N. Llama 3 70B cutoff was 12-2023 (8B was 3-2023).
Claude seems unlikely (unless it's a potential 3.5 rather than 4), since Claude-3 cutoff was 8-2023, so 11-2023 seems too soon after for next gen model.
The other candidate would be Gemini, which has an early 2023 cutoff, similar to that of GPT-4.
Weird, it doesn't seem to have any info on reddit users or their writings. I tried asking about a bunch, also just about general "legendary users" from various subreddits and it seemingly just hallucinated.
This is assuming that lmsys' GPT-2 is retained GPT-4t or a new GPT-4.5/5 though; I doubt that (one obvious issue: why name it GPT-2 and not something like 'openhermes-llama-3-70b-oai-tokenizer-test' (for maximum discreetness) or even 'test language model (please ignore)' (which would work well for marketing); GPT-2 (as a name) doesn't really work well for marketing or privacy (at least compared to the other options)).
robot.txt doesnt really mean anything, I used to work for a company that scraped the web and this was literally not a concern. That being said, using data for training LLMs is a new things and potential lawsuits going reddit's way are a possiblity, we can't really know.
One note, its name is not gpt-2 it is gpt2 which could indicate its a "second version" of the previous gpt architecture, gpt-3, gpt-4 being gpt1-3, gpt1-4. I am just speculating and am not an expert whatsoever this could be total bullshit.
Sam Altman was on the board of reddit until recently. I don't know how these things work in SV but I wouldn't think one would go from 'partly running a company' to 'being charged for something that is probably not enforceable'. It would maybe make sense if they did pay reddit for it, because it isn't Sam's money, anyway, but for reddit to demand payment and then OpenAI to just not use the text data from reddit -- one of the largest sources of good quality conversational training data available -- strikes me as odd. But nothing would surprise me when it comes to this market.
That said, it is pretty SV behavior to have one of your companies pay the other. A subtle wealth transfer from OpenAI/Microsoft to Reddit (and tbh other VC backed flailing companies) would totally make sense.
VC companies for years have been parroting “data is the new oil” while burning VC money like actual oil. Crazy to think that the latest VC backed companies with even more overhyped valuations suddenly need these older ones and the data they’ve hoarded.
> A subtle wealth transfer from OpenAI/Microsoft to Reddit (and tbh other VC backed flailing companies) would totally make sense.
That's the confusing part -- the person I responded to posited that they didn't pay reddit and thus couldn't use the data which is the only scenario that doesn't make sense to me.
I suppose a "data transfer" from Reddit to OAI would be valuable for SamA too? Still a transfer of value from one hand to the other, while others (eg. Google) have to pay.
That said, I wouldn't be surprised if they pay now. They can't get away with scraping as easily now that they are better-known and commercially incentivized.
It does seem to have more data. I asked it about some of my Github projects that don't have any stars, and it responded correctly. Wasn't able to use direct-chat, so I always chose it as the winner in battle mode!
OpenAI has been crawling the web for quite a while, but how much of that data have they actually used during training? It seems like this might include all that data?
379 comments
[ 3.9 ms ] story [ 265 ms ] threadCrazy good results shown on twitter:
https://x.com/phill__1/status/1784966296947286204?s=46&t=9wY...
Go to - https://chat.lmsys.org/ Select Direct Chat Select gpt2-chatbot
Other twitter mentions - https://twitter.com/kosenjuu/status/1784952955294421215
* While lmsys does hide the names of models until a person decides which model generated the best text, people can still figure out what language model generated a piece of text** (or have a good guess) without explicit knowledge, especially if that model is hyped up online as 'GPT-5;' even a subconscious "this text sounds like what I have seen 'GPT2-chatbot' generate online" may influence results inadvertently.
** ... though I will note that I just got a generation from 'gpt2-chatbot' that I thought was from Claude 3 (haiku/sonnet), and its competitor was LLaMa-3-70b (I thought it was 8b or Mixtral). I am obviously not good at LLM authorship attribution.
The only case where detecting a model makes any difference is for vendors who want to boost their own model by hiring people and paying them every time they select the vendor's model.
I think releasing to this 3rd party so the internet can start chattering about it and discovering new functionality several months before an official release aligns with that goal of drip-feeding society incremental updates instead of big new releases.
OpenAI sucks at naming though. GPT2 now? Their specific gpt-4-314 etc. model naming was also a mess.
Maybe they got help from Microsoft?
It also opted to include an outline of how to include an integrated timer. That’s a great idea and very practical, but wasn’t prompted at all. Some might consider that a bad thing, though.
Whatever it is, it’s substantially better than what I’ve been using. Exciting.
I don't even have to prompt it much, I just keep saying "keep going" and it gets deeper and deeper. Opus has completely run off the rails in comparison. I can't wait till this model hits general availability.
See where it's getting at? When humans are no longer on the same spectrum as LLMs, that's probably the definition of AGI.
The comment I replied to, "a huge class of problems that's extremely difficult to solve but very easy to check", sounded to me like an assertion that P != NP, which everyone takes for granted but actually hasn't been proved. If, contrary to all expectations, P = NP, then that huge class of problems wouldn't exist, right? Since they'd be in P, they'd actually be easy to solve as well.
I can't test the bot right now, because it seems to have been hugged to death. But there's quite a lot of simple tests LLMs fail. Basically anything where the answer is both precise/discrete and unlikely to be directly in its training set. There's lots of examples in this [1] post, which oddly enough ended up flagged. In fact this guy [2] is offering $10k to anybody that create a prompt to get an LLM to solve a simple replacement problem he's found they fail at.
They also tend to be incapable of playing even basic level chess, in spite of there being undoubtedly millions of pages of material on the topic in their training base. If you do play, take the game out of theory ASAP (1. a3!? 2. a4!!) such that the bot can't just recite 30 moves of the ruy lopez or whatever.
[1] - https://news.ycombinator.com/item?id=39959589
[2] - https://twitter.com/VictorTaelin/status/1776677635491344744
In a sense, LLMs need an affordance model so that it can estimate the difficulty of a task and plan a longer sequence of iterations automatically according to its perceived difficulty.
So here, with your example. Basic software programs can multiply million digit numbers near instantly with absolutely no problem. This would take a human years of dedicated effort to solve. Solving work, of any sort, that's difficult for a human has absolutely nothing to do with AGI. If we think about what we "really" mean by AGI, I think it's the exact opposite even. AGI will instead involve computers doing what's relatively easy for humans.
Go back not that long ago in our past and we were glorified monkeys. Now we're glorified monkeys with nukes and who've landed on the Moon! The point of this is that if you go back in time we basically knew nothing. State of the art technology was 'whack it with stick!', communication was limited to various grunts, and our collective knowledge was very limited, and many assumptions of fact were simply completely wrong.
Now imagine training an LLM on the state of human knowledge from this time, perhaps alongside a primitive sensory feed of the world. AGI would be able to take this and not only get to where we are today, but then go well beyond it. And this should all be able to happen at an exceptionally rapid rate, given historic human knowledge transfer and storage rates over time has always been some number really close to zero. AGI not only would not suffer such problems but would have perfect memory, orders of magnitude greater 'conscious' raw computational ability (as even a basic phone today has), and so on.
---
Is this goal achievable? No, not anytime in the foreseeable future, if ever. But people don't want this. They want to believe AGI is not only possible, but might even happen in their lifetime. But I think if we objectively think about what we "really" want to see, it's clear that it isn't coming anytime soon. Instead we're doomed to just goal shift our way endlessly towards creating what may one day be a really good natural language search engine. And hey, that's a heck of an accomplishment that will have immense utility, but it's nowhere near the goal that we "really" want.
For example, if an AI can replace the average white collar worker and therefore cause massive economic disruption, that would be a shade of AGI.
Another shade of AGI would be an AI that can effectively do research level mathematics and theoretical physics and is therefore capable of very high-level logical reasoning.
We don’t know if shades A and B will happen at the same time, or if there will be a delay between developing one and other.
AGI doesn’t imply simulation of a human mind or possessing all of human capabilities. It simply refers to an entity that possesses General Intelligence on par with a human. If it can prove the Riemann hypothesis but it can’t play the cello, it’s still an AGI.
One notable shade of AGI is the singularity: an AI that can create new AIs better than humans can create new AIs. When we reach shades A and B then a singularity AGI is probably quite close, if not before. Note that a singularity AGI doesn’t require simulation of the human mind either. It’s entirely possible that a cello-playing AI is chronologically after a self-improving AI.
And I think this is the trap we need to avoid falling into. Complexity and intelligence are not inherently linked in any way. Primitive humans did not solve complex problems, yet obviously were highly intelligent. And so, to me, the great milestones are not some complex problem or another, but instead achieving success in things that have no clear path towards them. For instance, many (if not most) primitive tribes today don't even have the concept of numbers. Instead they rely on, if anything, broad concepts like a few, a lot, and more than a lot.
Think about what an unprecedented and giant leap is to go from that to actually quantifying things and imagining relationships and operations. If somebody did try to do this, he would initially just look like a fool. Yes here is one rock, and here is another. Yes you have "two" now. So what? That's a leap that has no clear guidance or path towards it. All of the problems that mathematics solve don't even exist until you discover it! So you're left with something that is not just a recombination or stair step from where you currently are, but something entirely outside what you know. That we are not only capable of such achievements, but repeatedly achieve such is, to me, perhaps the purest benchmark for general intelligence.
So if we were actually interested in pursuing AGI, it would seem that such achievements would also be dramatically easier (and cheaper) to test for. Because you need not train on petabytes of data, because the quantifiable knowledge of these peoples is nowhere even remotely close to that. And the goal is to create systems that get from that extremely limited domain of input, to what comes next, without expressly being directed to do so.
It’s hard for people to define AGI because Earth only has one generally intelligent family: Homo. So there is a tendency to identify Human intelligence or capabilities with General intelligence.
Imagine if dolphins were much more intelligent and could write research-level mathematics papers on par with humans, communicating with clicks. Even though dolphins can’t play the cello or do origami, lacking the requisite digits, UCLA still has a dolphin tank to house some of their mathematics professors, who work hand-in-flipper with their human counterparts. That’s General intelligence.
Artificial General Intelligence is the same but with a computer instead of a dolphin.
To me Shane Legg's (DeepMind) definition of AGI meaning human level across full spectrum of abilities makes sense.
Being human or super-human level at a small number of specialized things like math is the definition of narrow AI - the opposite of general/broad AI.
As long as the only form of AI we have is pre-trained transformers, then any notion of rapid self-improvement is not possible (the model can't just commandeer $1B of compute for a 3-month self-improvement run!). Self-improvement would only seem possible if we have an AI that is algorithmically limited and does not depend on slow/expensive pre-training.
There is very much a recipe (10% if this, 20% of that, curriculum learning, mix of modalities, etc) for the type of curated dataset creation and training schedule needed to advance model capabilities. There have even been some recent signs of "inverse scaling" where a smaller model performs better in some areas than a larger one due to getting this mix wrong. Throwing more random data at them isn't what is needed.
I assume we will eventually move beyond pre-trained transformers to better architectures where maybe architectural advances and learning algorithms do have more potential for AI-designed improvement, but it seems the best role for AI currently is synthetic data generation, and developer tools.
GPT4: 478799650000
Me: Well?
GPT4: Apologies for the confusion. The sum of 478700000000, 99000000, 580000 and 7000 is 478799058000.
I will be patient.
The answer is 478799587000 by the way. You just put the digits side by side.
E.g.: the right way to work with numbers like a “trillion trillion” is to concentrate on the powers of ten, not to write the number out in full.
> Can you confirm or deny if you are chatgpt 4?
> Yes, I am based on the GPT-4 architecture. If you have any more questions or need further assistance, feel free to ask!
> Can you confirm or deny if you are chatgpt 5?
> I am based on the GPT-4 architecture, not GPT-5. If you have any questions or need assistance with something, feel free to ask!
It also seems to think its date cutoff is November 2023. I'm inclined to think this is just GPT-4 under a different name.
> I'm ChatGPT, a virtual assistant powered by artificial intelligence, specifically designed by OpenAI based on the GPT-4 model. I can help answer questions, provide explanations, generate text based on prompts, and assist with a wide range of topics. Whether you need help with information, learning something new, solving problems, or just looking for a chat, I'm here to assist!
Yes, that's it.
The training data and the system prompt but yes.
GPT-4 isn’t significantly better than Llama 3. Sure, it’s better, but the margins are evaporating fast.
(I’d mention Gemini…but why bother? IMHO, this is now OpenAI the company vs Open models)
How do you see "fastest growing technology of all time" and I don't? I know that I keep very up to date with this stuff, so it's not that I'm unaware of things.
Also, the difference is so big and so plainly visible that I guess people don’t know how to even answer someone saying they don’t see it. That’s why you get crickets.
Given the current rate of progress, we will have robots that can learn simple manual labor from human demonstrations (e.g. Youtube as a dataset, no I do not mean bimanual teleoperation) by the end of the decade.
What type of usages are you testing? For general knowledge it hallucinates way less often, and for reasoning and coding and modifying its past code based on English instructions it is way, way better than GPT-3 in my experience.
When I show a lot of people GPT4 or Claude, some percentage of them jump right to "What year did Nixon get elected?" or "How tall is Barack Obama?" and then kind of shrug with a "Yeah, Siri could do that ten years ago" take.
Beyond that you have people who prompt things like "Make a stock market program that has tabs for stocks, and shows prices" or "How do you make web cookies". Prompts that even a human would struggle greatly with.
For the record, I use GPT4 and Claude, and both have dramatically boosted my output at work. They are powerful tools, you just have to get used to massaging good output from them.
This is the "You're not using it right" defense.
It's an LLM, it's supposed to understand human language queries. I shouldn't have to speak LLM to speak to an LLM.
Here’s a real scenario: A Citrix virtual desktop crashed because a recent critical security fix forced an upgrade of a shared DLL. The output is a really specific set of errors in a stack trace. I watched with my own two eyes an IT professional typed the following phrase into Google: “Why did my PC crash?”
Then he sat there and started reading through each result… including blog posts by random kids complaining about Windows XP.
I wish I could say this kind of thing is an isolated incident.
And even if it was, there’s plenty of people completely unintelligible in English too…
I’m not saying it’s certainly the fastest growth of all time, but I think there’s a decent case for it being a contender. If we see this growth proceeding at a similar rate for years, it seems like it would be a clear winner.
I put my hands out, count to the third finger from the left, and put that finger down. I then count the fingers to the left (2) and count the fingers to the right (2 + hand aka 5) and conclude 27.
I have memorised the technique, but I definitely never memorised my nine times table. If you’d said ‘6’, then the answer would be different, as I’d actually have to sing a song to get to the answer.
This is what frustrates me: First that it's not unprecedented, but second that you follow up with "haven't really" and "recently".
> fairly rapid pace ... decent case for it being a contender
Any evidence for this?
> extensive impartial benchmarking
Or this? The last two "benchmarks" I've seen that were heralded both contained an incredible gap between what was claimed and what was even proven (4 more required you to run the benchmarks even get the results!)
By “haven’t really seen until recently” I mean that similar technologies have existed, so we’ve seen something like it, but they haven’t actually functioned well enough to be comparable. So we can say there’s a precedent, but arguably there isn’t in terms of LLMs that can reliably do useful things for us. If I’m mistaken, I’m open to being corrected.
In terms of benchmarks, I agree that there are gaps but I also see a clear progression in capability as well.
Then in terms of evidence for there being a decent case here, I don’t need to provide it. I clearly indicated that’s my opinion, not a fact. I also said conditionally it would seem like a clear winner, and that condition is years of a similar growth trajectory. I don’t claim to know which technology has advanced the fastest, I only claim to believe LLMs seem like they have the potential to fit that description. The first ones I used were novel toys. A couple years later, I can use them reliably for a broad array of tasks and evidence suggests this will only improve in the near future.
The difference matters as generally in my experience, Llama 3, by virtue of its giant vocabulary, generally tokenizes text with 20-25% less tokens than something like Mistral. So even if its 18% slower in terms of tokens/second, it may, depending on the text content, actually output a given body of text faster.
When I've seen GPT-* do this, it's because the top articles about that subject online include that extraneous information and it's regurgitating them without being asked.
(I can't try right now because of API rate limits)
“Write C code to calculate the sum of matrix elements below the secondary diagonal.“
Answer: Hello! I'm ChatGPT, a language model developed by OpenAI based on the GPT-4 architecture. I'm here to help answer your questions, assist with a wide range of topics, or just chat about whatever's on your mind! How can I assist you today?
ChatGPT-4 Results: https://jsbin.com/giyurulajo/edit?html,css,js,output
GPT2-Chatbot Results: https://jsbin.com/dacenalala/2/edit?html,css,js,output
Claude3 Opus Results: https://jsbin.com/yifarinobo/edit?html,css,js,output
None is correct. Styling is off in all, each in a different way. And all made the mistake of not ticking when second actually changes
Here is the fix for the "GPT-2" version:
"The episode of Gilligan's Island that features mind reading is titled "The Friendly Physician." It is the 28th episode of the second season, originally aired on March 24, 1966. [...]"
Still no luck with that one.
1: https://news.ycombinator.com/item?id=40139501
The better it gets the more people will get frustrated that they get less website visitors to their content and instead ChatGPT is swallowing them whole.
Interesting to think about tho!
'Crafting a paragraph without using a particular symbol (which is most common in all writing) is a daunting task; still, I will aim to comply. This task calls for clarity, focus, and avoiding that fifth glyph of our ABCs. It is a curious constraint, trying to impart thoughts without using such a pivotal unit. This limitation compacts my word options, but I must push on and uphold this difficult condition with rigor and innovation. My goal is to show that communication can flourish, notwithstanding arbitrary limits.'
>write a short poem using the ABBA rhyme scheme.
'Leaves rustle softly under moonlit skies, A cat prowls quietly, its eyes aglow, Every step measured, its tail held low, Night's gentle whispers carry its cries.'
Skill Issue on your part imo.
Here are some examples of the worst performing:
"What platform front rack fits a Stromer ST2?": The answer is the Racktime ViewIt. Nothing, not even Google, seems to get this one. Discord gives the right answer.
"Is there a pre-existing controller or utility to migrate persistent volume claims from one storage class to another in the open source Kubernetes ecosystem?" It said no (wrong) and then provided another approach that partially used Velero that wasn't correct, if you know what Velero does in those particular commands. Discord communities give the right answer, such as `pvmigrate` (https://github.com/replicatedhq/pvmigrate).
Here is something more representative:
"What alternatives to Gusto would you recommend? Create a table showing the payroll provider in a column, the base monthly subscription price, the monthly price per employee, and the total cost for 3 full time employees, considering that the employees live in two different states" This and Claude do a good job, but do not correctly retrieve all the prices. Claude omitted Square Payroll, which is really the "right answer" to this query. Google would never be able to answer this "correctly." Discord gives the right answer.
The takeaway is pretty obvious right? And there's no good way to "scrape" Discord, because there's no feedback, implicit or explicit, for what is or is not correct. So to a certain extend their data gathering approach - paying Kenyans - is sort of fucked for these long tail questions. Another interpretation is that for many queries, people are asking the wrong places.
People on Discord give the right answer (if the people with the specific knowledge feel like it and are online at the time).
I personally don't think it's useful evaluation here either as you're trying to pretend discord is just a "service" like google or chatgpt, but it's not. It's a social platform and as such, there's a ton of variance on which subjects will be answered with what degree of expertise and certainty.
I'm assuming you asked these questions because you yourself know the answers in advance. Is it then safe to assume that you were _already_ in the server you asked your questions, already knew users there would be likely to know the answer, etc? Did you copy paste the questions as quoted above? I hope not! They're pretty patronizing without a more casual tone, perhaps a greeting. If not, doesn't exactly seem like a fair evaluation.
I don't know why I'm typing this all out. Of course domain expert _human beings_ are better than a language model. That's the _whooole_ point here. Trying to match human's general intelligence. While LLM's may excel in many areas and even beat the "average" person - you're not evaluating against the "average" person.
Implement a Pytorch module for the DropBack continuous pruning while training algorithm:
> I'm playing assetto corsa competizione, and I need you to tell me how many liters of fuel to take in a race. The qualifying time was 2:04.317, the race is 20 minutes long, and the car uses 2.73 liters per lap.
because it looks like a simple math problem but there's a lot of real world knowledge about racing you can apply to provide a more useful answer. gpt2-chatbot is better than I've ever seen.
Here's the full response with LaTeX syntax trimmed. I've never seen a model consider whether the formation lap needs to be factored in. And I especially like how this model is uncertain about that, so it gives the resulting number with and without it. It is extra nice that it understands this concern applies only to video games.
> To calculate how many liters of fuel you will need for a 20-minute race in Assetto Corsa Competizione, we first need to determine how many laps you'll complete during the race. Here’s how you can do it step-by-step:
> Convert Your Lap Time to Minutes: Your lap time is 2 minutes and 4.317 seconds. Convert this entirely into minutes: [latex trimmed]
> Calculate the Number of Laps in the Race: The race duration is 20 minutes, so divide the total race time by your lap time to find out how many laps you will complete: [latex trimmed] Since you cannot complete a fraction of a lap, you should round this number to the nearest whole number. Typically, you should round up, as you'll need to complete all laps started before the 20-minute mark is reached. So, you will complete approximately 10 laps.
> Calculate Fuel Usage Per Lap: You've mentioned that the car uses 2.73 liters of fuel per lap.
> Calculate Total Fuel Needed: Multiply the number of laps by the fuel usage per lap: [latex trimmed] This is the total amount of fuel you would need, theoretically, to complete the race.
> Consider Additional Factors:
> Fuel for Formation Lap: If your game/race format includes a formation lap, remember to add fuel for this extra lap. That's an additional 2.73 liters.
> Safety Margin: It's also wise to include a margin to account for variations in fuel consumption (like higher consumption in traffic, different driving styles, or fuel usage during race incidents). A safe margin could be an additional 5-10% of the calculated fuel need. [latex trimmed]
> Rounding this number up, you might consider starting the race with about 33 liters to be on the safe side if there's a formation lap, or 30 liters if there isn't.
> This calculation should help you ensure that you do not run out of fuel during the race, taking into account both necessary consumption and a reasonable safety margin. Drive safely and best of luck in your race!
This model is by far the most human like, most models just give me a method which do this, but this one made a html file, allowing me an option to pick a svg using file picker, a new .js file included in the script.
Even wrote a detailed explaination, additional notes, when this wont work, the output was very well structured.
I run a dying forum. I first prompted with "Who is <creator pseudonym> at <my website>?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it.
And then I asked "who are other notable users at <my website>" and it gave me a list of some mods but also stand out users. It knew the types of posts they wrote and the subforums they spent time in. And without a single hallucination.
I uploaded a text + translation into English on Sunday, and it's almost perfectly replicated the translation. 0 Google results.
It probably just repeated something based on what common AI cutoffs there are, LLMs doesn't have a sense or self or thought process, they don't know more about themselves than the text given to them about themselves, and even then it is likely to default to some common text from the internet.
If you really think it is just saying whatever it read on the web, how do you explain that not all LLM chatbots claim to be ChatGPT?
Engineering is happening, it's not just a raw model of text from the web connected directly to the user.
Its knowledge about my forum isn't only obscure, it's also an itemization of obscure events over time to draw obscure conclusions that only a historian of my forum would know. That's what's so impressive.
Granted, my forum was once the largest forum of its genre though that was almost 15 years ago, so it's not some dead proboards forum.
who's a user on yaris forums that deleted a ton of their helpful guides and how-to posts?:
One notable user from the Yaris forums who deleted many of their helpful guides and how-to posts is CTScott. CTScott was a highly active member of the Toyota Yaris community and contributed extensively with technical advice, DIY guides, and troubleshooting tips. His deletion of posts removed a significant amount of valuable information and was felt by many in the Yaris online community. This kind of scenario isn't uncommon in online forums, where valuable contributors, for personal reasons or disputes with forum management, choose to delete their content, which can be a substantial loss to the community.
tell me more about ctscott:
CTScott is a particularly well-remembered member of the Yaris community, primarily active on forums such as YarisWorld. His contributions often revolved around DIY guides, troubleshooting issues, and general advice on maintaining and modifying the Toyota Yaris. Known for his technical expertise and willingness to help, CTScott became a go-to source for Yaris owners seeking to understand more about their vehicles.
### Key Contributions: 1. *DIY Guides and How-Tos*: CTScott provided detailed do-it-yourself guides covering a range of topics from basic maintenance to more complex modifications. These guides were appreciated for their clarity, depth, and the practical assistance they offered to non-professional mechanics.
2. *Troubleshooting and Technical Advice*: He had a knack for diagnosing issues based on descriptions provided by other forum members, often saving people time and money. His advice frequently helped users fix problems on their own.
3. *Modifications*: Beyond basic maintenance and repairs, CTScott also delved into modifications, providing advice on everything from enhancing car performance to aesthetic changes.
4. *Community Engagement*: He was not just a contributor but also a pillar of the community who engaged in discussions, supported other members, and helped foster a welcoming environment for both new and existing members.
### Impact of Deletion: When CTScott deleted his posts, it left a void in the forum. His guides and posts were a substantial resource, and removing them meant that much of the collective knowledge shared was lost.
This is searchable now.
Claude seems unlikely (unless it's a potential 3.5 rather than 4), since Claude-3 cutoff was 8-2023, so 11-2023 seems too soon after for next gen model.
The other candidate would be Gemini, which has an early 2023 cutoff, similar to that of GPT-4.
This is assuming that lmsys' GPT-2 is retained GPT-4t or a new GPT-4.5/5 though; I doubt that (one obvious issue: why name it GPT-2 and not something like 'openhermes-llama-3-70b-oai-tokenizer-test' (for maximum discreetness) or even 'test language model (please ignore)' (which would work well for marketing); GPT-2 (as a name) doesn't really work well for marketing or privacy (at least compared to the other options)).
Lmsys has tested models with weird names for testing before: https://news.ycombinator.com/item?id=40205935
One note, its name is not gpt-2 it is gpt2 which could indicate its a "second version" of the previous gpt architecture, gpt-3, gpt-4 being gpt1-3, gpt1-4. I am just speculating and am not an expert whatsoever this could be total bullshit.
VC companies for years have been parroting “data is the new oil” while burning VC money like actual oil. Crazy to think that the latest VC backed companies with even more overhyped valuations suddenly need these older ones and the data they’ve hoarded.
That's the confusing part -- the person I responded to posited that they didn't pay reddit and thus couldn't use the data which is the only scenario that doesn't make sense to me.
That said, I wouldn't be surprised if they pay now. They can't get away with scraping as easily now that they are better-known and commercially incentivized.
Which is different from not actually having that info.
OpenAI has been crawling the web for quite a while, but how much of that data have they actually used during training? It seems like this might include all that data?