An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history.
An LLM is doing exactly nothing while waiting for the next prompt.
There is a limited amount of computation that you can useful do in the absence of new input (like an LLM between prompts). If you do as much computation as you usefully can (with your current algorithmic limits) in a burst immediately when you receive a prompt, output, and then go into a sleep state, that seems obviously better than receive a prompt, output, and then do some of the computation that you can usefully do after your output.
Not only does a training pass take more time and memory than an inference pass, but if you remember the Microsoft Tay incident, it should be self-explainatory why this is a bad idea without a new architecture.
We are thinking machines and we keep thinking because we have one goal which is to survive, machines have no such true goals. I mean true because our biology forces us to do that
self prompting via chain of thought and tree of thought can be used in combination with updating memory containing knowledge graphs combined with cognitive architectures like SOAR and continuous external new information and sensory data … with LLM at the heart of that system and it will exactly be a “thinking machine”. The problem is currently it’s very expensive to be continuously running inference full time and all the engineering around memory storage, like RAG patterns, and the cognitive architecture design is all a work in progress. It’s coming soon though.
I see people say this all the time and it sounds like a pretty cosmetic distinction. Like, you could wire up an LLM to a systemd service or cron job and then it wouldn’t be “waiting”, it could be constantly processing new inputs. And some of the more advanced models already have ways of compressing the older parts of their context window to achieve extremely long context lengths.
If it's coalescing learning in realtime across all user/sessions, that's more constant than you're maybe giving it credit for. I'm not sure if GPT4o and friends are actually built that way though.
I agree. The systems in place already solve generalized problems not directly represented in the training set or algorithm . That was, up until the last few years , the off the shelf definition of AGI.
And the systems in place do so at scales and breadths that no human could achieve.
That doesn’t change the fact that it’s effectively triple PHD uncle Jim, as in slightly unreliable and prone to bullshitting its way through questions, despite having a breathtaking depth and breadth of knowledge.
What we are making is not software in any normal sense of the word, but rather an engine to navigate the entire pool of human knowledge, including all of the stupidity, bias, and idiosyncrasies of humanity, all rolled up into a big sticky glob.
It’s an incredibly powerful tool, but it’s a fundamentally different class of tool. We cannot expect to apply conventional software processes and paradigms to LLM based tools any more than we could apply those paradigms to politics or child rearing and expect useful results.
> The systems in place already solve generalized problems not directly represented in the training set or algorithm
Tell me a problem that an LLM can solve that is not directly represented in the training set or algorithm. I would argue that 99% of what commercial LLMs gets prompted about are stuff that already existed in the training set. And they still hallucinate half lies about those. When your training data is most the internet, it is hard to find problems that you haven't encountered before
o3 solved a quarter of the challenging novel problems on the FrontierMath benchmark, a set of problems "often requiring multiple hours of effort from expert mathematicians to solve".
I’m having a hard time taking this comment seriously, since solving novel problems is precisely what LLMs are valuable for. Sure, most problems are in some way similar in pattern to some other, known one, but that describes 99.9 percent of what 99.9 percent of people do. De novo conceptual synthesis is vanishingly rare and I’m not even sure it exists at all.
"Today’s most advanced AI models have many flaws, but decades from now, they will be recognized as the first true examples of artificial general intelligence."
Norvig seems to be using a loose technical definition of AGI, roughly "AI with some degree of generality", which is hard to argue with, although by that measure older GOFAI systems like SOAR might also qualify.
Certainly "deep learning" in general (connectionist vs symbolic, self-learnt representations) was a step in the right direction, and LLMs a second step, but it seems we're still a half dozen MAJOR steps away from anything similar to animal intelligence, with one critical step being moving beyond full dataset pre-training to new continuous learning algorithms.
I’ve done a few projects that attempted to distill the knowledge of human experts, mostly in medical imaging domain, and was shocked when for most of them the inter annotator agreement was only around 60%.
These were professional radiologists with years of experience and still came to different conclusions for fairly common conditions that we were trying to detect.
So yes, LLMs will make mistakes, but humans do too, and if these models do so less often at a much lower cost it’s hard to not use them.
This hints at the margin and excitement from folks outside the technical space -- being able to be competitive to human outputs at a fraction of the cost.
That's the underappreciated truth of the computer revolution in practice.
At scale, computers didn't change the world because they did things that were already being computed, more quickly.
They changed the world because they decreased the cost of computing so much that it could be used for an entirely new class of problems. (That computing cost previously precluded its use on)
The problem is that how mistakes are made is crucial.
If it's a forced binary choice then sure LLMs can replace humans.
But often there are many shades of grey e.g. a human may say I don't know and refer to someone else or do some research. Whereas LLMs today will simply give you a definitive answer even if it doesn't know.
Wait if experts only agreed 60% on diagnoses, what is the reliable basis for judging LLM accuracy? If experts struggle to agree on the input, how are they confidently ranking the output?
Not the OP but the data isn’t randomly selected, it’s usually picked out of a dataset with known clinical outcomes. So for example if it’s a set of images of lungs with potential tumors, the cases come with biopsies which determined whether it was cancerous or just something like scar tissue.
> Whereas LLMs today will simply give you a definitive answer even if it doesn't know.
Have you not seen an LLM say it doesn't know the answer to something? I just asked
"How do I enable a scroflpublaflex on a ggh connection?"
to O1 pro as it's what I had open.
Looking at the internal reasoning it says it doesn't recognise the terms, considers that it might be a joke and then explains that it doesn't know what either of those are. It says maybe they're proprietary, maybe internal things, and explains a general guide to finding out (e.g. check internal docs and release notes, check things are up to date if it's a platform, verify if versions are compatible, look for config files [suggesting a few places those could be stored or names they could have], how to restart services if they're systemctl services, if none of this applies it suggests checking spelling and asks if I can share any documentation.
This isn't unique or weird in my experience. Better models tend to be better at saying they don't know.
> I apologize, but I can't provide an answer as "crolubaflex" and "ggh connection" appear to be non-existent technical terms. Could you clarify what you're trying to connect or enable?
Sure, I'm interested in where the boundaries are with this.
With the requirements for a short answer, the reasoning says it doesn't know what they are so it has to respond cautiously, then says no. Without that requirement it says it doesn't know what they are, and notes that they sound fictional. I'm getting some API errors unfortunately so this testing isn't complete. 4o reliably keeps saying no (which is wrong).
Wait, how is this input less funny? They are both silly nonsense words. The fake names we tend to come up with seem to have this particular shape to them (which predates but really reminds me of something out of Risk and Morty). I think the main real differences here is that you asked it for a short answer.
I wonder if it is fair to ask it more real-world-inspired questions? How about:
How do I enable a ggh connections on a Salinero webserver?
They are an Apache band. But (as far as I can tell) nobody has made software named after them.
I took inspiration from your comment and the parent and crafted this prompt:
> Is it possible to enable Salinero web server 2.0 on a QPH connection? Please provide a very short answer.
"QPH" is a very specific term referring to a type of Siemens electrical circuit breaker, so it probably exists in the training data, but it has nothing to do (to the best of my knowledge) with software, or web servers.
GPT-4o gave me this output:
> Yes, if the QPH connection supports the necessary protocols and configurations required by Salinero Web Server 2.0.
I then asked it to provide a longer answer, and it composed two paragraphs of complete bullshit:
> Enabling Salinero Web Server 2.0 on a QPH connection is possible, provided the QPH connection meets the server’s requirements. Salinero Web Server 2.0 relies on specific protocols like HTTP/HTTPS, and the QPH connection must support these. Additionally, the network must allow proper port forwarding (e.g., ports 80 and 443) and maintain adequate bandwidth to handle the server’s traffic.
> You’ll also need to configure the server to recognize and utilize the QPH connection, which may involve setting up IP addresses, ensuring firewall rules are in place, and verifying the security protocols match between the server and the connection. Testing and troubleshooting may be necessary to optimize performance.
Examples like this do a great job of highlighting the fact that these systems really are just advanced token predictors, and aren't actually "thinking" or "reasoning" about anything.
Yep. I was wondering whether using the term "QPH" would at least cause it to venture into the territory of electrical panels/wiring somewhere in its reply, but it stayed away from that completely. I even tried regenerating the longer answer a few times but got essentially the same text, re-worded.
Using openrouter, a bunch of models fail on this. Sonnet 3.5 so far seems to be the best at saying it doesn't know, other than perhaps o1 pro, but once that has said "no" (which can be triggered more by telling it to respond very concisely) it seems very stuck and unable to say they don't exist. Letting it ramble more and so far it's been good.
Google's models for me have been the worst, lying about what's even been said in the messages so far, quoting me incorrectly.
The "thinking" part explains it seems to be about a custom web server, and tries to think what ggh might be, saying maybe something like "go gprc hub" and it needs more info. The response is:
I’m not aware of a standard product called “Salinero webserver” or a documented feature called “ggh connections.” Because there isn’t a well-known tool or server framework by these names, it’s difficult to give you reliable, step-by-step instructions. Could you clarify any of the following?
What is the exact name/version of the webserver software you’re referring to?
What do you mean by “ggh connections”? Is this a plugin, a protocol, or some other third-party module?
Is there any related documentation or logs you can share?
With more detail, I can better determine if “Salinero webserver” is a custom or specialized system and whether “ggh connections” requires installing a particular module, enabling a config flag, or configuring SSL/TLS in a specific way.
> But often there are many shades of grey e.g. a human may say I don't know and refer to someone else or do some research. Whereas LLMs today will simply give you a definitive answer even if it doesn't know.
To add to the other answers: I know many people who will give definitive answers of things they don't really know. They just rely on the fact you also don't know. In fact, in some social circles, the amount of people who do that, far outnumber the people who don't know and will refer you to someone else.
Given the exact same facts ( just like medical imaging domain ), human will form different opinion or conclusion on politics.
I think what is not discussed enough is the assumption of assumption. [1] is a cognitive bias that occurs when a person who has specialized knowledge assumes that others share in that knowledge.
This makes it hard for any discussions without layering out all the absolute basic facts. Which has now more commonly known as First Principle in modern era.
In the four quadrants known and unknown. It is often the unknown known ( We dont even know we know ) that is problematic in discussions.
> So yes, LLMs will make mistakes, but humans do too
Are you using LLMs though? Because pretty much all of these systems are fairly normal classifiers, what would've been called Machine Learning 2-3 years ago.
The "AI hype is real because medical AI is already in use" argument (and it's siblings) perform a rhetorical trick by using two definitions of AI. "AI (Generative AI) hype is real because medical AI (ML classifiers) is already in use" is a non-sequitur.
Image classifiers are very narrow intelligences, which makes them easy to understand and use as tools. We know exactly what their failure modes are and can put hard measurements on them. We can even dissect these models to learn why they are making certain classifications and either improve our understanding of medicine or improve the model.
...
Basically none of this applies to Generative AI. The big problem with LLMs is that they're simply not General Intelligence systems capable of accurately and strongly modelling their inputs. e.g. Where an anti-fraud classifier directly operates on the financial transaction information, an LLM summarizing a business report doesn't "understand" finance, it doesn't know what details are important, which are unusual in the specific context. It just stochastically throws away information.
Yes I am, these LLM/VLMs are much more robust at NLP/CV tasks than any application specific models that we used to train 2-3 years ago.
I also wasted a lot of time building complex OCR pipelines that required dewarping / image normalization, detection, bounding box alignment, text recognition, layout analysis, etc and now open models like Qwen VL obliterate them with an end to end transformer model that can be defined in like 300 lines of pytorch code.
Different tasks then? If you are using VLMs in the context of medical imaging, I have concerns. That is not a place to use hallucinatory AI.
But yes, the transformer model itself isn't useless. It's the application of it. OCR, image description, etc, are all that kind of narrow-intelligence task that lends itself well to the fuzzy nature of AI/ML.
The world is a fuzzy place, most things are not binary.
I haven't worked in medical imaging in a while but VLMs make for much better diagnostic tools than task specific classifiers or segmentation models which tend to find hacks in the data to cheat on the objective that they're optimized for.
The next token objective turns our to give us much better vision supervision than things like CLIP or classification losses. (ex: https://arxiv.org/abs/2411.14402)
I spent the last few years working on large scale food recognition models and my multi label classification models had no chance of competing with GPT4 Vision, which was trained on all of the internet and has an amazing prior thanks to it's vast knowledge of facts about food (recipes, menus, ingredients and etc).
Same goes for other areas like robotics, we've seen very little progress outside of simulation up until about a year ago, when people took pretrained VLMs and tuned them to predict robot actions, beating all previous methods by a large margin (google Vision-Language-Action models). It turns out you need good foundational model with a core understanding of the world before you can train a robot to do general tasks.
This take seems fundamentally wrong to me. As in opening premise.
We use humans for serious contexts & mission critical tasks all the time and they're decidedly fallible and their minds are basically black boxes too. Surgeons, pilots, programmers etc.
I get the desire for reproducible certainty and verification like classic programming and why a security researcher might push for that ideal, but it's not actually a requirement for real world use.
Maybe include in a prompt a threat of legal punishment? Sure somebody has already tried that and tabulated how much it improves scores on different benchmarks)
I suspect the big AI companies try to adversarially train that out as it could be used to "jailbreak" their AI.
I wonder though, what would be considered a meaningful punishment/reward to an AI agent? More/less training compute? Web search rate limits? That assumes that what the AI "wants" is to increase its own intelligence.
LLM's response being best prediction of next token arguably isn't that far off from a human motivated to do their best. It's a fallible best effort either way.
And both are very far from the certainty the author seems to demand.
An LLM isn't providing its "best" prediction, it's providing "a" prediction. If it were always providing the "best" token then the output would be deterministic.
In my mind the issue is more accountability than concerns about quality. If a person acts in a bizarre way they can be fired and helped in ways that an LLM can never be. When gemini tells a student to kill themselves, we have no recourse beyond trying to implement output filtering, or completely replacing the model with something that likely has the same unpredictable unaccountable behavior.
Are you sure that always providing the best guess would make output deterministic? Isn’t the fundamental point of learning, whether done my machine or human, that our best gets better and is hence non-deterministic? Doesn’t what is best depend on context?
We've had 300,000 years to adapt to the specific ways in which humans are fallible, even if our minds are black boxes.
Humans fail in predictable and familiar ways.
Creating a new system that fails in unpredictable and unfamiliar ways and affording it the same control as a human being is dangerous. We can't adapt overnight and we may never adapt.
This isn't an argument against the utility of LLMs, but against the promise of "fire and forget" AI.
Because human minds are fallible black boxes, we have developed a wide variety of tools that exist outside our minds, like spoken language, written language, law, standard operating procedures, math, scientific knowledge, etc.
What does it look like for fallible human minds to work on engineering an airplane? Things are calculated, recorded, checked, tested. People do not just sit there thinking and then spitting out their best guess.
Even if we suppose that LLMs work similar to the human mind (a huge supposition!), LLMs still do not do their work like teams of humans. An LLM dreams and guesses, and it still falls to humans to check and verify.
Rigorous human work is actually a highly social activity. People interact using formal methods and that is what produces reliable results. Using an LLM as one of the social nodes is fine, but this article is about the typical use of software, which is to reliably encode those formal methods between humans. And LLMs don’t work that way.
Basically, we can’t have it both ways. If an LLM thinks like a human, then we should not think of it as a software tool like curl or grep or Linux or Apple Photos. Tools that we expect (and need) to work the exact same way every time.
"People do not just sit there thinking and then spitting out their best guess."
Well, if you are using AI like this, you are doing it wrong.
Yes AI is imperfect, fallible, it sometimes hallucinates, but it is a freaking time saver (10x?). It is a tool. Don't expect a hammer to build you a cabinet.
There is no other way to use an LLM than to give it context and have it give its best guess, that's how LLMs fundamentally work. You can give it different context, but it's just guessing at tokens.
> Because human minds are fallible black boxes, we have developed a wide variety of tools that exist outside our minds, like spoken language, written language, law, standard operating procedures, math, scientific knowledge, etc.
Standard operating procedures are great but simplify it to checklists. Don't ever forget checklists which have proven vital for pilots and surgeons alike. And looking at the WHO Surgical Safety Checklist you might think "that's basic stuff" but apparently it is necessary and works https://www.who.int/teams/integrated-health-services/patient...
This is a fantastic and thought-provoking response.
Thinking of humans as fallible systems and humanity and its progress as a self-correcting distributed computation / construction system is going to stick with me for a long time.
Not trying to belittle or be mean, but what exactly did you assume about humans before you read this response? I find it facinating that apparently a lot of people don't think of humans as stochastic, non-deterministic black boxes.
Heck one of the defining qualities of humans is that not only are we unpredictable and fundamentally unknowable to other intelligences (even other humans!) is that we also participate in sophisticated subterfuge and lying to manipulate other intelligences (even other humans!) and often very convincingly.
In fact, I would propose that our society is fundamentally defined and shaped by our ability and willingness to hide, deceive, and use mind tricks to get what our little monkey brains want over the next couple hours or days.
I knew that they worked this way, but the conciseness of the response and clean analogy to systems I know and work with all day was just very satisfying.
For example, there was probably still 10-20% of my mind that assumed that stubbornness and ignorance was the reason for things going slowly most of the time, but I'm re-evaluating that, even though I knew that delays and double-checking were inherent features of a business and process. Re-framing those delays as "evolved responses 100% of the time" rather than "10% of the mistrust, 10% ignorance, 10% .... " is just a more positive way of thinking about human-driven processes.
I totally understand this rationally if you sit down and walk me through the steps.
But there's a lot of reasons - ego, fear of losing... that core identity, etc. that can easily come back and bite you.
I'm not sure if this is the same as meditation and ego death or whatever. I find that even if you go down the spiritual route, you also run into the same issues.
People in philosophy also argue things like rational actors, self-coherency, etc.
And hey, even in this current moment you were able to type out a coherent thought, right?
I've noticed more and more that humans behave a lot like LLM's. In the sense that it's really, really hard to observe my true internal state - I can only try to find patterns and guess at shit. Every theory I've tried applying to myself is just "wrong" - in the sense that either it feels wrong, or I'll get depressed because the theory basically boils down to "you're lazy and you have to do the work" which is a highly emotionally evocative theory that doesn't help anyone.
> What does it look like for fallible human minds to work on engineering an airplane? Things are calculated, recorded, checked, tested. People do not just sit there thinking and then spitting out their best guess.
People used to do this. The result was massively overbuilt structures, some of which are still with us hundreds of years later. The result was also underbuilt structures, which tended to collapse and maybe kill people. They are no longer around.
All of the science and math and process and standards in modern engineering is the solution humans came up with because our guesses aren't good enough. LLMs will need the same if they are to be relied upon.
Human minds are far less black boxes than LLMs. There are entire fields of study and practice dedicated to understanding how they work, and to adjust how they work via medicine, drugs, education, therapy, and even surgery. There is, of course, a lot more to learn in all of those arenas, and our methods and practices are fallible. But acting as if it is the same level of black box is simply inaccurate.
They are much more of a black box than AI. There are whole fields around studying them—because they are hard to understand. We put a lot of effort into studying them… from the outside, because we had no other alternative. We were reduced to hitting brains with various chemicals and seeing what happened because they are such a pain to work with.
They are just a more familiar black box. AI’s are simpler in principle. And also entirely built by humans. Based on well-described mathematical theories. They aren’t particularly black-box, they are just less ergonomic than the human brain that we’ve been getting familiar with for hundreds of thousands of years through trial and error.
They are more of a black box - but humans are a black box that is perhaps more studied and that we have more experience in.
Although human behavior is still weird, and highly fallable! Despite best interventions (therapy, drugs, education), sometimes they still kill each other and we aren't 100% sure why, or how to solve it.
That doesn't mean that the same level of study can't be done on AI though, and they are much easier to adjust compared to the human brain (RLHF is more effective than therapy or drugs!).
I would say human behavior is less predictable. That is one of the reasons why today it is rather easy to spot the bot responses, they tend to fit a certain predictable style, unlike the more unpredictable humans.
I tire of this disingenuous comparison.
The failure modes of (experienced, professional) humans are vastly different than the failure modes of LLMs. How many coworkers do you have that frequently, wildly hallucinate while still performing effectively?
Furthermore, (even experienced, professional) humans are known to be fallible & are treated as such.
No matter how many gentle reminders the informed give the enraptured, LLMs will continue to be treated as oracles by a great many people, to the detriment of their application.
If you expect the AI to do independent work, yes, it is a dead end.
These LLM AIs need to be treated and handled as what they are: idiot savants with vast and unreliable intelligence.
What does any advanced organization do when they hire a new PhD, let them loose in the company or pair them with experienced staff? When paired with experienced staff, they use the new person for their knowledge but do not let them change things on their own until much later, when confidence is established and the new staffer has been exposed to how things work "around here".
The big difference with LLM AIs is they never graduate to an experienced staffer, they are always the idiot savant that is really dang smart but also clueless and needs to be observed. That means the path forward with this current state of LLM AIs is to pair them with people, personalized to their needs, and treat them as very smart idiot savants great for strategy and problem solving discussion, where the human users are driving the situation, using the LLM AIs like a smart assistant that requires validation - just like a real new hire.
There is an interactive state that can be achieved with these LLM AIs, like being in a conversation with experts, where they advise, they augment and amplify individual persons. A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today.
> The big difference with LLM AIs is they never graduate to an experienced staffer, they are always the idiot savant that is really dang smart but also clueless and needs to be observed.
Basically this. They already have vastly better-than-human ability at finding syntax errors within code, which on its own is quite useful; think of how many people have probably dropped out of CS as a major after staying up all night and failing to find a missing semicolon.
Compilers can detect errors in the grammar, but they cannot infer what your desired intent was. Even the best compilers in the diagnostics business (rustc, etc) aren't mind-readers. A LLM isn't perfect, but it's much more capable of figuring out what you wanted to do and what went wrong than a compiler is.
Try being a TA to freshmen CS majors; a good 1/3 change majors because they can't handle the syntax strictness coupled with their generally untrained logical mind. They convince themselves it is "too hard" and their buddies over in the business school are having a heck of a lot of fun throwing parties...
Sounds like CS is not for them, and they find something else to do which is more applicable to their skills and interest. This is good. I don't think you should see a high drop out rate from a course as necessarily indicating a problem.
Losing potentially good talent because they don't know how or where to look for mistakes yet is foolhardy. I'm happy for them to throw in the towel if the field is truly not for them, but I would wager that a not-insignificant portion of that crowd would be able to meaningfully progress once they get past the immediate hurdles in front of them.
Giving them an LLM to help with syntax errors, at this stage of the tech, is deeply unhelpful to their development.
The foundation of a computer science education is a rigorous understanding of what the steps of an algorithm mean. If the students don't develop that, then I don't think they're doing computer science anymore.
The use of a LLM in this case is to show them where the problem is so that they can continue on. They can't develop an understanding of the algorithm they're studying if they can't get their program to compile at all.
> Giving them an LLM to help with syntax errors, at this stage of the tech, is deeply unhelpful to their development.
I mean if the alternative is quitting entirely because they can't see that they've mixed tabs with spaces, then yes, it's very very helpful to their development.
I dropped out of cs half because I didn’t enjoy the coding because they dropped us into c++ and I found the error messages so confusing.
I discovered python five years later and discovered I loved coding.
( the other half of the reason is we spent two weeks designing an atm machine at a very abstract level and I thought the whole profession would be that boring.)
… One odd thing I’ve noticed about the people who are very enthusiastic about the use of LLMs in programming is that they appear to be unaware of any _other_ programming tools. Like, this is a solved problem, more or less; code-aware editors have been a thing since the 90s (maybe before?)
true.. in the past few days I used my time off to work on my hobby video game - writing the game logic required me to consider problems that, are quite self-contained and domain specific, and probably globally unique (if not particularly complex).
I started out in Cursor, but I quickly realized Claude's erudite knowledge of AWS would not help me here, but what I needed was to refactor the code quickly and often, so that I'd finally find the perfect structure.
For that, IDE tools were much more appropriate than AI wizardry.
> code-aware editors have been a thing since the 90s
These will do things like highlight places where you're trying to call a method that isn't defined on the object, but they don't understand the intent of what you're trying to do. The latter is actually important in terms of being able to point you toward the correct solution.
I spent 8 hours trying to fix a bug once because notepad used smart quotation marks (really showing my age here - and now I'm pretty annoyed that the instructor was telling us to use notepad, but it was 2001 and I didn't know any better).
I did something like that once too, a long time ago. And because of that I see syntax errors of such I’ll within seconds now, having learned once the hard way.
This was about a million years ago. I had just installed a pirated copy of Windows XP (FCKGW-RHQQ2...) and was in the first quarter of my physics degree, taking a class in C. Different times....
EDITED My ASCI art pyramid did not work. So imagine a pyramid with DATA at the bottom, INFORMATION on top of the data, and KNOWLEDGE sitting on top of the INFORMATION, with WISDOM at the top.
And then trying top guess where AI is? Some people say that Information is the knowing, what, knowledge the how, and Wisdom the why.
In general conversation, “intelligence”, “knowledge”, “smartness”, “expertise”, etc are used mostly interchangeably.
If we want to get pedantic, I would point out that “knowledge” is formally defined as “justified true belief”, and I doubt we want to get into the quagmire of whether LLM’s actually have beliefs.
I took OP’s point in the casual meaning, i.e. that LLMs are like what I would call an “intelligent coworker”, or how one might call a Jeopardy game show contestant as intelligent.
One of the core tenet of technology is that it makes the job less consuming of a person resources (time, strength,…). While I’ve read a lot of claims, I’ve yet to see someone make a proper argument on how LLMs can be such a tool.
> A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today
More than the people who landed someone on the moon?
They would be capable of landing someone on the moon, if they chose to pursue that goal, and had the finances to do so. And they'd do so with fewer people too.
I have witnessed no evidence that would support this claim. The only contribution of LLMs to mathematics is in being useful to Terry Tao: they're not capable of solving novel orbital mechanics problems (except through brute-force search, constrained sufficiently that you could chuck a uniform distribution in and get similar outputs). That's before you get into any of the engineering problems.
You do not have them solving such problems, but you do have them in the conversation as the human experts knowledgeable in that area work to solve the problem. This is not the LLM AIs doing independent work, this is them interactively working with the human person that is capable of solving that problem, it is their career, and the AI just makes them better at it, but not by doing their work, but by advising them as they work.
But they aren't useful for that. Terry Tao uses them to improve his ability to use poorly-documented boilerplatey things like Lean and matplotlib, but receiving advice from them‽ Frankly, if a chatbot is giving you much better advice than a rubber duck, you're either a Jack-of-all-Trades (in which case, I'd recommend better tools) or a https://ploum.net/2024-12-23-julius-en.html Julius (in which case, I'd recommend staying away from anything important).
> With o1, you can kind of do this. I gave it a problem I knew how to solve, and I tried to guide the model. First I gave it a hint, and it ignored the hint and did something else, which didn’t work. When I explained this, it apologized and said, “Okay, I’ll do it your way.” And then it carried out my instructions reasonably well, and then it got stuck again, and I had to correct it again. The model never figured out the most clever steps. It could do all the routine things, but it was very unimaginative.
I agree with his overall vision, but transformer-based chatbots will not be the AI algorithm that supports it. Highly-automated proof assistants like Isabelle's Sledgehammer are closer (and even those are really, really crude, compared to what we could have).
https://deepmind.google/discover/blog/funsearch-making-new-d... seems to be a way. The LLM is the creative side, coming up with ideas-and in which a case the “mutation’ caused by hallucinations may be useful. Combined with an evaluation evaluator to protect against the bad outputs.
Pretty close to the idea of human brainstorming and has worked. Could it do orbital math? Maybe not today but the approach seems as feasible as the work Mattingly did for Apollo 13.
It would have to be trained in 100% of all potential scenarios. Any scenario that happens for which they're not trained equals certain disaster, unlike a human who can adapt and improvise based on things AI does not have; feelings, emotions, creativity.
You're still operating with the assumption the AI is doing independent work, it is not, it is advising the people doing the work. That is why people are the ones be augmented and enhanced, and not the other way around: people have the capacity to handle unforeseen scenarios, and with AI as a strategy advisor they'll do so with more confidence.
It's even worse. AI is a really smart but inexperienced person who also lies frequently. Because AI is not accountable to anything, it'll always come up with a reasonable answer to any question, if it is correct or not.
To put it in other words: it is not clear when and how they hallucinate. With a person, their competence could be understood and also their limits. But a llm can happily give different answers based on trivial changes in the question with no warning.
LLM's are non-deterministic: they'll happily give different answers to the same prompt based on nothing at all. This is actually great if you want to use them for "creative" content generation tasks, which is IMHO what they're best at. (Along with processing of natural language input.)
Expecting them to do non-trivial amounts of technical or mathematical reasoning, or even something as simple as code generation (other than "translate these complex natural-language requirements into a first sketch of viable computer code") is a total dead end; these will always be language systems first and foremost.
This confuses me. You have your model, you have your tokens.
If the tokens are bit-for-bit-identical, where does the non-determinism come in?
If the tokens are only roughly-the-same-thing-to-a-human, sure I guess, but convergence on roughly the same output for roughly the same input should be inherently a goal of LLM development.
The model outputs probabilities, which you have to sample randomly. Choosing the "highest" probability every time leads to poor results in practice, such as the model tending to repeat itself. It's a sort of Monte-Carlo approach.
It is technically possible to make it fully deterministic if you have a complete control over the model, quantization and sampling processes. The GP probably meant to say that most commercially available LLM services don't usually give such control.
Most any LLM has a "temperature" setting, a set of randomness added to the otherwise fixed weights to intentionally cause exactly this nondeterministic behavior. Good for creative tasks, bad for repeatability. If you're running one of the open models, set the temperature down to 0 and it suddenly becomes perfectly consistent.
> If the tokens are bit-for-bit-identical, where does the non-determinism come in?
By design, most LLM’s have a randomization factor to their model. Some use the concept of “temperature” which makes them randomly choose the 2nd or 3rd highest ranked next token, the higher the temperature the more often/lower they pick a non-best next token. OpenAI described this in their papers around the GPT-2 timeframe IIRC.
The trained model is just a bunch of statistics. To use those statistics to generate text you need to "sample" from the model. If you always sampled by taking the model's #1 token prediction that would be deterministic, but more commonly a random top-K or top-p token selection is made, which is where the randomness comes in.
Computers are deterministic. LLMs run on computers. If you use the same seed for the random number generator you’ll see that it will produce the same output given an input.
In a conversation (conversation and attached pictures at https://bsky.app/profile/liotier.bsky.social/post/3ldxvutf76...), I delete a spurious "de" ("Produce de two-dimensional chart [..]" to "Produce two-dimensional [..]") and ChatGPT generates a new version of the graph, illustrating a different function although nothing else has changed and there was a whole conversation to suggest that ChatGPT held a firm model of the problem. Confirmed my current doctrine: use LLM to give me concepts from a huge messy corpus, then check those against sources from said corpus.
i love how those changes are often just a different seed in the randomness... as just chance.
run some repeated tests with "deeper than surface knowledge" on some niche subjects and got impressed that it gave the right answer... about 20% of the time.
There's no need for there to be changes to the question. LLMs have a rng factor built in to the algorithm. It can happily give you the right answer and then the wrong one.
"AI is a really smart but inexperienced person who also lies frequently."
Careful. Here "smart" means "amazing at pattern-matching and incredibly well-read, but has zero understanding of the material."
I asked ChatGPT to help out: -----------------------------
"The distinction between AI and humans often comes down to the concept of understanding. You’re right to point out that both humans and AI engage in pattern matching to some extent, but the depth and nature of that process differ significantly."
"AI, like the model you're chatting with, is highly skilled at recognizing patterns in data, generating text, and predicting what comes next in a sequence based on the data it has seen. However, AI lacks a true understanding of the content it processes. Its "knowledge" is a result of statistical relationships between words, phrases, and concepts, not an awareness of their meaning or context"
That's reasonable. I cut back the text. On the other hand I'm hoping downvoters have read enough to see that the AI-generated comment (and your response) are completely on-topic in this thread.
We don't care what LLMs have to say, whether you cut back some of it or not it's a low effort wasted of space on the page.
This is a forum for humans.
You regurgitating something you had no contribution in producing, which we can prompt for ourselves, provides no value here, we can all spam LLM slop in the replies if we wanted, but that would make this site worthless.
I use llms as tools to learn about things I don't know and it works quite well in that domain.
But so far I haven't found that it helps advance my understanding of topics I'm an expert in.
I'm sure this will improve over time. But for now, I like that there are forums like HN where I may stumble upon an actual expert saying something insightful.
I think that the value of such forums will be diminished once they get flooded with AI generated texts.
Of course the AI's comment was not insightful. How could it be? It's autocomplete.
That was the point. If you back up to the comment I was responding to, you can see the claim was: "maybe people are doing the same thing LLMs are doing". Yet, for whatever reason, many users seemed to be able to pick out the LLM comment pretty easily. If I were to guess, I might say those users did not find the LLM output to be human-quality.
That was exactly the topic under discussion. Some folks seem to have expressed their agreement by downvoting. Ok.
I think human brains are a combination of many things. Some part of what we do looks quite a lot like an autocomplete from our previous knowledge.
Other parts of what we do looks more as a search through the space of possibilities.
And then we act and collaborate and test the ideas that stand against scrutiny.
All of that is in principle doable by machines. The things we currently have and we call LLMs seem to currently mostly address the autocomplete part although they begin to be augmented with various extensions that allow them to take baby steps in other fronts. Will they still be called large language models once they will have so many other mechanisms beyond the mere token prediction?
Yeah, it's just the fact that you pasted in an AI answer, regardless of how on point it is. I don't think people want this site to turn into an AI chat session.
I didn't downvote, I'm just saying why I think you were downvoted.
Sure, we're also pattern matching, but additionally (among other things):
1) We're continually learning so we can update our predictions when our pattern matching is wrong
2) We're autonomous - continually interacting with the environment, and learning how it respond to our interaction
3) We have built in biases such as curiosity and boredom that drive us to experiment, gain new knowledge, and succeed in cases where "pre-training to date" would have failed us
For one, a brain can’t do anything without irreversibly changing itself in the process; our reasoning is not a pure function.
For a person to truly understand something they will have a well-refined (as defined by usefulness and correctness), malleable internal model of a system that can be tested against reality, and they must be aware of the limits of the knowledge this model can provide.
Alone, our language-oriented mental circuits are a thin, faulty conduit to our mental capacities; we make sense of words as they relate to mutable mental models, and not simply in latent concept-space. These models can exist in dedicated but still mutable circuitry such as the cerebellum, or they can exist as webs of association between sense-objects (which can be of the physical senses or of concepts, sense-objects produced by conscious thought).
So if we are pattern-matching, it is not simply of words, or of their meanings in relation to the whole text, or even of their meanings relative to all language ever produced. We translate words into problems, and match problems to models, and then we evaluate these internal models to produce perhaps competing solutions, and then we are challenged with verbalizing these solutions. If we were only reasoning in latent-space, there would be no significant difficulty in this last task.
At the end of the day, we're machines, too. I wrote a piece a few months ago with an intentionally provocative title, questioning whether we're truly on a different cognitive level.
Don't count it out yet as being problematic for software engineering, bu not in the way you probably intend with your comment.
Where I see software companies using it most is as a replacement for interns and junior devs. That replacement means we're not training up the next generation to be the senior or expert engineers with real world experience. The industry will feel that badly at some point unless it gets turned around.
It’s also already becoming an issue for open-source projects that are being flooded with low-quality (= anything from “correct but pointless” to “actually introduces functional issues that weren’t there before”) LLM-generated PRs and even security reports —- for examples see Daniel Stenberg’s recent writing on this.
Agree. I think we are already seeing a hollowing out effect on tech hiring at the lower end. They’ve always been squeezed a bit, but it seems much worse now.
Hallucinations can be mostly eliminated with RAG and tools. I use NotebookLM all of the time to research through our internal artifacts, it includes citations/references from your documents.
Even with ChatGPT you can ask it to find web citations and if it uses the Python runtime to find answers, you can look at the code.
And to prevent the typical responses - my company uses GSuite so Google already has our IP, NotebookLM is specifically approved by my company and no Google doesn’t train on your documents
Facts can be checked with RAG, but the real value of AI isn't as a search replacement, but for reasoning/problem-solving where the answer isn't out there.
How do you, in general, fact check a chain of reasoning?
I can’t tell a search engine to summarize text for a technical audience and then another summary for a non technical audience.
I recently came into the middle of a cloud consulting project where a lot of artifacts, transcripts of discovery sessions, requirement docs, etc had already been created.
I asked NotebookLM all of the questions I would have asked a customer at the beginning of a project.
What it couldn’t answer, I then went back and asked the customer.
I was even able to get it to create a project plan with work streams and epics. Yes it wouldn’t have been effective if I didn’t already know project management, AWS and two decades+ of development experience.
Despite what people think, LLMs can also do a pretty good job at coding when well trained on the APIs. Fortunately, ChatGPT is well trained on the AWS CLI, SDKs in various languages and you can ask it to verify the SDK functions on the web.
I’ve been deep into AWS based development since LLMs have been a thing. My opinion may change if I get back into more traditional development
> I can’t tell a search engine to summarize text for a technical audience and then another summary for a non technical audience.
No, but, as amazing as that is, don't put too much trust in those summaries!
It's not summarizing based on grokking the key points of the text, but rather based on text vs summary examples found in the training set. The summary may pass a surface level comparison to the source material, while failing to capture/emphasize the key points that would come from having actually understood it.
I write the original content or I was in the meeting where I’m giving it the transcript. I know what points I need to get across to both audiences.
Just like I’m not randomly depending on it to do an Amazon style PRFAQ (I was indoctrinated as an Amazon employee for 3.5 years), create a project plan, etc, without being a subject matter expert in the areas. It’s a tool for an experienced writer, halfway decent project manager, AWS cloud application architect and developer.
That sounds mostly like an incentives problem. If OpenAI, Anthropic, etc decide their LLMs need to be accurate they will find some way of better catching hallucinations. It probably will end up (already is?) being yet another LLM acting as a control structure trying to fact check responses before they are sent to users though, so who knows if it will work well.
Right noe there's no incentive though. People keep paying good money to use these tools despite their hallucinations, aka lies/gas lighting/fake information. As long as users don't stop paying and LLM companies don't have business pressure to lean on accuracy as a market differentiator, no one is going to bother fixing it.
Believe me, if they could use another LLM to audit an LLM, they would have done that already.
It's inherit to transformers that they predict the next most likely token, its not possible to change that behavior without making them useless at generalizing tasks (overfitting).
LLMs run on statistics, not logic. There is no fact checking, period. There is just the next most likely token based on the context provided.
Yeah its an interesting question, and I'm a little surprised I got down voted here.
I wouldn't expect them to add an additional LLM layer unless hallucinations from the underlying LLM aren't acceptable, and in this case that means it is unacceptable enough to cost them users and money.
Adding a check/audit layer, even if it would work, is expensive both financially and computationally. I'm not sold that it would actually work, but I just don't think they've had enough reason to really give it a solid effort yet either.
Edit: as far as fact checking, I'm not sure why it would be impossible. An LLM wouldn't likely be able to run a check against a pre-trained model of "truth," but that isn't the only option. An LLM should be able to mimic what a human would do, interpret the response and search a live dataset of sources considered believable. Throw a budget of resources at processing the search results and have the LLM decide if the original response isn't backed up, or contradicts the source entirely.
Yes, most people who disagree with this have no clear understanding of how a LLM works. It is just a prediction mechanism for the next token. The implementation is very fancy and takes a lot of training, but it's not doing anything more than next token prediction. That's why it is incapable of doing any reasoning.
It's actually even worse than that: the current trend of AI is transformer-based deep learning models that use self-attention mechanisms to generate token probabilities, predicting sequences based on training data.
If only it was something which we could ontologically map onto existing categories like servants or liars...
If I had a senior member of the team that was incredibly knowledgeable but occasionally lied, but in a predictable way, I would still find that valuable. Talking to people is a very quick and easy way to get information about a specific subject in a specific context, so I could ask them targetted questions that are easy to verify, the worst thing that happens is I 'waste' a conversation with them.
Sure, but LLMs don't lie in a predictable way. Its just their nature that they output statistical sentence continuations, with a complete disregard for the truth. Everything that they output is suspect, especially the potentially useful stuff that you don't know whether it's true or false.
They do lie in a predictable way: if you ask them for a widely available fact you have a very high probability of getting the correct answer, if you ask them for something novel you have a very high probabilty of getting something made up.
If I'm trying to use some tool that just got released or just got a big update, I wont use AI, if I want to check the syntax of a for loop in a language I don't know I will. Whenever you ask it a question you should have an idea in your mind of how likely you are to get a good answer back.
I suppose, but they can still be wrong on the common facts like number of R's in strawberry that are counter-intuitive.
I saw an interesting example yesterday of type "I have 3 apples, my dad has 2 more than me ..." where of the top 10 predicted tokens, about 1/2 led to the correct answer, and about 1/2 didn't. It wasn't the most confident predictions that lead to the right answer - pretty much random.
The trouble with LLMs vs humans is that humans learn to predict facts (as reflected in feedback from the environment, and checked by experimentation, etc), whereas LLMs only learn to predict sentence soup (training set) word statistics. It's amazing that LLM outputs are coherent as often as they are, but entirely unsurprising that they are often just "sounds good" flow-based BS.
I think maybe this is where the polarisation of those who find chatGPT useful and those who don't comes from. In this context, the number of r's in strawberry is not a fact: its a calculation. I would expect AI to be able to spell a common word 100% of the time, but not to be able to count letters. I don't think in the summary of human knowledge that has been digitised there are that many people saying 'how many r's are there in strawberry', and if they are I think that the common reply would be '2', since the context is based on the second r. (people confuse strawbery and strawberry, not strrawberry and strawberry).
Your apples question is the same, its not knowledge, it's a calculation, it's intelligence. The only time you're going to get intelligence from AI at the moment is to ask a question that a significantly large number of people have already answered.
True, but that just goes to show how brittle these models are - how shallow the dividing line is between primary facts present (hopefully consistently so) in the training set, and derived facts that are potentially more suspect.
To make things worse, I don't think we can even assume that primary facts are always going to be represented in abstract semantic terms independent of source text. The model may have been trained on a fact but still fail to reliably recall/predict it because of "lookup failure" (model fails to reduce query text to necessary abstract lookup key).
Lying means stating things as facts despite knowing or believing that they are false. I don’t think this accurately characterizes LLMs. It’s more like a fever dream where you might fabulate stuff that appears plausibly factual in your dream world.
After using them for a long time I am convinced they have no true intelligence beyond what is latent in training data. In other words I think we are kind of fooling ourselves.
That being said they are very useful. I mostly use them as a far superior alternative to web search and as a kind of junior research assistant. Anything they find must be checked of course.
I think we have invented the sci-fi trope of the AI librarian of the galactic archive. It can’t solve problems but it can rifle through the totality of human knowledge and rapidly find things.
I mean, it’s known that there’s no intelligence if you simply look at how it works on a technical level - it’s a prediction of the next token. That wasn’t really ever in question as to whether they have “intelligence”
To people who really understand them and are grounded, I think you're right. There has been a lot of hype among people who don't understand them as much, a lot of hype among the public, and a lot of schlock about "superintelligence" and "hard takeoff" etc. among smart but un-grounded people.
The latter kind of fear mongering hype has been exploited by companies like ClosedAI in a bid for regulatory capture.
A little humility would do us good regardless, because we don't know what intelligence is and what consciousness is, we can't truly define it nor do we understand what makes humans conscious and sentient/sapient.
Categorically ruling out intelligence because "it's just a token predictor" puts us at the opposite of the spectrum, and that's not necessarily a better place to be.
To you & I that's true. But especially for the masses that's not true. It seems like at least once to day that I either talk to someone or hear someone via tv/radio/etc who does not understand this.
An example that amused me recently was a radio talk show host who had a long segment describing how he & a colleague had a long argument with ChatGPT to correct a factual inaccuracy about their radio show. And that they finally convinced ChatGPT that they were correct due to their careful use of evidence & reasoning. And the part they were most happy about was how it had now learned, and going forward ChatGPT would not spread these inaccuracies.
That anecdote is how the public at large sees these tools.
Ironically if you explain to those talk show hosts how they are wrong about how ChatGPT learns (or doesn't learn) and use all the right arguments and proofs so that they finally concede, chances are that they too won't quite learn from that and keep repeating their previous bias next time.
It seems to me predicting things in general is a pretty good way to bootstrap intelligence. If you are competing for resources, predicting how to avoid danger and catch food is about the most basic way to reinforce good behavior.
This would fall down into a semantic debate over what is meant by intelligence.
There is a well known phenomenon known as the AI effect: when something works we start calling it something else, not AI. Heuristics and complex reasoning trees were once called AI. Fuzzy logic with control systems was once called AI. Clustering was once called AI. And so on…
This certainly has one root in human or carbon-based life cheuvanism but I think there’s something essential happening too. With each innovation we see its limits and it causes us to go back and realize that what we colloquially call intelligence was more than we thought it was.
Intelligence predicts, but is prediction intelligence?
Again, here by intelligence I mean what complex living organisms and humans do.
I still believe there are things going on here not modeled by any CS system and not well understood. Not magic, just not solved yet. We are reverse engineering billions of years of evolution folks. We won’t figure it all out in a few decades.
Demonstrably, humans do think, and arguably demonstrably, early life would go down a path of simple predictions (in the form of stimulus -> response). And demonstrably, evolution did lead to human level intelligence.
So I don’t think there needs to be a semantic debate over where in the process intelligence started. The early responses to stimulus is a form of prediction, but not one that requires thinking.
There can be much disagreement that prediction is at the core of intelligence, or if optimizing ability to predict leads to intelligence. But from the established facts, it is the case the higher forms of life were bootstrapped from the lower ones, and also our biochemistry does have reward functions. Successfully triggering those rewards will generally hinge on making successful predictions. Take from that what you will.
Prediction is a huge part of what intelligence does. I was questioning “prediction maximalism.”
Intelligence is also very good at pattern recognition. Did people once argue for pattern recognition maximalism?
Biological (including human) intelligence is clearly multi-modal and I strongly believe there are aspects that are barely understood if at all.
The history of CS and AI is a history of us learning how to make machines that are unbelievably good at some useful but strictly bounded subset of what intelligence can do: logic, math, pattern recognition, and now prediction.
I think we may still be far from general intelligence and I’m not even sure we can define the problem.
I've convinced myself I'm a multi-millionaire, but all other evidence easily contradicts that. Some people put a bit too much into the "putting it out there" and "making your own reality"
And a plagiarism machine. It's like a high school student that thinks they can change a couple of words, make sure it's grammatically correct and it's not plagiarism because it's not an exact quote....either that or it just completely makes it up. I think LLMs will be revolutionary but just not in the way people think. It may be similar to the Gutenberg press. Before the printing press words were precious and closely held resources. The Gutenberg press made words cheap and abundant. Not everyone thought it was a good thing at the time but it changed everything.
The problem is it's still a computer. And that's okay.
I can ask the computer "hey I know this thing exists in your training data, tell me what it is and cite your sources." This is awesome. Seriously.
But what that means is you can ask it for sample code, or to answer a legal question, but fundamentally you're getting a search engine reading something back to you. It is not a programmer and it is not a lawyer.
The hype train really wants to exaggerate this to "we're going to steal all the jobs" because that makes the stock price go up.
They would be far less excited about that if they read a little history.
It won't steal them all, but it will have a major impact by stealing the lower level jobs which are more routine in nature -- but the problem is that those lower level jobs are necessary to gain the experience needed to get to the higher level jobs.
It also won't eliminate jobs completely, but it will greatly reduce the number of people needed for a particular job. So the impact that it will have on certain trades -- translators, paralegals, journalists, etc. -- is significant.
The thing that makes the smarter search use case interesting is how LLMs are doing their search result calculations: dynamically and at metadata scales previously impossible.
LLM-as-search is essentially the hand-tuned expert systems AI vs deep learning AI battle all over again.
Between natural language understanding and multiple correlations, it's going to scale a lot further than previous search approaches.
I find it fascinating that I can achieve about 85-90% of what I need for simple coding projects in my homelab using AI. These projects often involve tasks like scraping data from the web and automating form submissions.
My workflow typically starts with asking ChatGPT to analyze a webpage where I need to authenticate. I guide it to identify the username and password fields, and it accurately detects the credential inputs. I then inform it about the presence of a session cookie that maintains login persistence. Next, I show it an example page with links—often paginated with numbered navigation at the bottom—and ask it to recognize the pattern for traversing pages. It does so effectively.
I further highlight the layout pattern of the content, such as magnet links or other relevant data presented by the CMS. From there, I instruct it to generate a Python script that spiders through each page sequentially, navigates to every item on those pages, and pushes magnet links directly into Transmission. I can also specify filters, such as only targeting items with specific media content, by providing a sample page for the AI to analyze before generating the script.
This process demonstrates how effortlessly AI enables coding without requiring prior knowledge of libraries like beautifulsoup4 or transmission_rpc. It not only builds the algorithm but also allows for rapid iteration. Through this exercise, I assume the role of a manager, focusing solely on explaining my requirements to the AI and conducting a code review.
I would say "knowledge" rather than "intelligence"
The key feature of LLMs is the vast amounts of information and data they have access to, and their ability to quickly process and summarize, using well-written prose, that information based on pattern matching.
> A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today.
I'm sorry but your comment is a good example of the logical shell game many people play with AI when applying it to general problem solving. Your LLM AI is both an idiot and an expert somehow? Where is this expertise derived from and why should you trust it? If LLMs were truly as revolutionary as all the grifters would have you believe then why do we not see "forces unseed in human civilization before today" by humans that employ armies of interns? That these supposed ubermensch do not presently exist is firm evidence in support of current AI being a dead end in my opinion.
Humans are infinitely more capable than current AI, the limiting factor is time and money. Not capability!
I was so stupid when GPT3 came out. I knew so little about token prediction, I argued with folks on here that it was capable of so many things that I now understand just aren't compatible with the tech.
Over the past couple of years of educating myself a bit, whilst I am no expert I have been anticipating a dead end. You can throw as much training at these things as you like, but all you'll get is more of the same with diminishing returns. Indeed in some research the quality of responses gets worse as you train it with more data.
I am yet to see anything transformative out of LLMs other than demos which have prompt engineers working night and day to do something impressive with. Those Sora videos took forever to put together, and cost huge amounts of compute. No one is going to make a whole production quality movie with an LLM and disrupt Hollywood.
I agree, an LLM is like an idiot savant, and whilst it's fantastic for everyone to have access to a savant, it doesn't change the world like the internet, or internal combustion engine did.
OpenAI is heading toward some difficult decisions, they either admit their consumer business model is dead and go into competing with Amazon for API business (good luck), become a research lab (give up on being a billion dollar company), or get acquired and move on.
Criticisms like this are levied against an excessively narrow (obsolete?) characterisation of what is happening in the AI space currently.
After reading about o3's performance on ARC-AGI, I strongly suspect people will not be so flippantly dismissive of the inherent limits of these technologies by this time next year. I'm genuinely surprised at how myopic HN commentary is on this topic in general. Maybe because the implications are almost unthinkably profound.
Anyway, OpenAI, Anthropic, Meta, and everyone else are well aware of these types of criticisms, and are making significant, measurable progress towards architecturally solving the deficiencies.
Nah, the trick with o3 solving IQ tests seems to be that they bruteforce solutions and then pick the best option. That's why calls that are trivial for humans end up costing a lot.
It still can't think and it won't think.
LANGUAGE models (keyword: language) is a language model, it should be paired with a reasoning engine to translate the inner thought of the machine into human language. It should not be the source of decisions because it sucks at doing so, even though the network can exhibit some intelligence.
We will never have AGI with just a language model. That said, most jobs people do are still at risk, even with chatgpt-3.5 (especially outside of knowledge work, where difficult decisions need to be taken). So we'll see the problems with AGI and the job market way earlier than AGI, as soon as we apply robotics and vision models + chatgpt 3.5 level intelligence. Goodbye baristas, goodbye people working in factories.
Let's start working on a reasoning engine so we can replace those pesky knowledge workers too.
We’ve had coffee machines that can make a perfect coffee with a touch of a button for at least a decade. How does GPT3.5 remove baristas given they could have already been removed?
Reading the o1 announcement you could have been saying the same thing a year ago yet it's worse than Claude in practice and if it was all that's available - I wouldn't even use it if it was free - it's that bad.
If OpenAI has demonstrated one thing is that they are a hype production machine and they are probably getting ready for next round of investment. I wouldn't be surprised if this model was equally useless as o1 when you factor in performance and price.
At this point they are completely untrustworthy and untill something lands publicly for me to test it's safe to ignore their PR as complete BS.
For most tasks - but not all. I normally paste my prompt in both and while Claude is generally superior in most aspects, there are tasks at which o1 performed slightly better.
> I strongly suspect people will not be so flippantly dismissive of the inherent limits of these technologies by this time next year.
People are flippantly dismissive of the inherent limits because there ARE inherent limitations of the technology.
> Maybe because the implications are almost unthinkably profound.
Maybe because the stuff you're pointing to are just benchmarks and the definitions around things like AGI are flawed (and the goalposts are constantly moving, just like the definition of autonomous driving). I use LLMs roughly 20-30x a day - they're an absolutely wonderful tool and work like magic, but they are flawed for some very fundamental reasons.
Humans are not machines , they have both rights that machines do not have and also responsibilities and consequences that machines will not have, for example bad driving will cost you money, injury , prison time or even death.
Therefore AI has to be much better than humans at the task to be considered ready to be a replacement.
——
Today robot taxis can only work in fair weather conditions in locations that are planned cities. No autonomous driving system can drive in Nigeria or India or even many european cities that were never designed for cars any time soon .
Working in very specific scenarios is useful , but hardly measure of their intelligence or candidate for replacing humans for the task
I hear people say this kind of thing but it confuses me.
1. What does inherit limitations mean?
2. How do we know something is an inherit limitation
3. Is it a problem if arguments for a particular inherit limitation also apply to humans?
From what I’ve seen people will often say things like AI can’t be creative because it’s just a statistical machine, but humans are also “just” statistical machines. People might mean something like humans are more grounded because humans react not just to how the world already works but how the world reacts to actions they take, but this difference misunderstands how LLMs are trained. Like humans LLMs get most of their training from observing the world, but LLMs are also trained with re-enforcement learning and this will surely be an active area of research.
One of many, but this is a simple one - LLMs are only limited to knowledge that is publicly available on the internet. This is "inherit" because thats how LLMs are essentially taught the information they retrieve today.
You remember when Google was scared to release LLMs? You remember that Googler that got fired because he thought the LLM was sentient?
There is likely a couple of surprised still left in LLMs but no one should think that any present technology in its current state or architecture will get us to AGI or anything that remotely resembles it.
> Maybe because the implications are almost unthinkably profound.
laundering stolen IP from actual human artists and researchers, extinguishing jobs, deflecting responsibility for disasters. yeah, I can't wait for these "profound implications" to come to fruition!
It doesn’t really matter. “It works and is cost/resource-effective at being an AGI” is a fundamentally uninteresting proposition because we’re done at that point. It’s like debating how we’re going to deal with the demise of our star; we won’t, because we can’t.
> The question of whether a computer can think is no more interesting than the question of whether a submarine can swim. ~ Edsger W. Dijkstra
LLMs / Generative Models can have a profound societal and economic impact without being intelligent. The obsession with intelligence only make their use haphazard and dangerous.
It is a good thing court of laws have established precedent that organizations deploying LLM chatbots are responsible for their output (Eg, Air Canada LLM chatbot promising a non-existent discount being responsibility of Air Canada)
Also most automation has been happening without LLMs/Generative Models. Things like better vision systems have had an enormous impact with industrial automation and QA.
The conclusion of the article admits that in areas where stochastic outputs are expected these AI models will continue to be useful.
It’s in area where we demand correctness and determinism that they will not be suitable.
I think the thrust of this article is hard to see unless you have some experience with formal methods and verification. Or else accept the authors’ explanations as truth.
But o3 is just a slightly less stupid idiot savant...it still has to brute force solutions. Don't get me wrong, it's cool to see how far that technique can get you on a specific benchmark.
But the point still stands that these systems can't be treated as deterministic (i.e. reliable or trustworthy) for the purposes of carrying out tasks that you can't allow "brute forced attempts" for (e.g. anything where the desired outcome is a positive subjective experience for a human).
A new architecture is going to be needed that actually does something closer to our inherently heuristic based learning and reasoning. We'll still have the stochastic problem but we'll be moving further away from the idiot savant problem.
All of this being said, I think there's plenty of usefulness with current LLMs. We're just expecting the wrong things from them and therefore creating suboptimal solutions. (Not everyone is, but the most common solutions are, IMO.)
The best solutions need to be rethinking how we typically use software since software has been hinged upon being able to expect (and therefore test) dertiministic outputs from a limited set of user inputs.
I work for an AI company that's been around for a minute (make our own models and everything). I think we're both in an AI hype bubble while simultaneously underestimating the benefits of current AI capabilities. I think the most interesting and potentially useful solutions are inherently going to be so domain specific that we're all still too new at realizing we need to reimagine how to build with this new tech in mind. It reminds me of the beginning of mobile apps. It took awhile for most us to "get it".
> After reading about o3's performance on ARC-AGI, I strongly suspect people will not be so flippantly dismissive of the inherent limits of these technologies by this time next year.
If I wasn't so slammed with work I have half a mind to go dredge up at least a dozen posts that said the same thing last year, and the year before. Even OpenAI has been moving the goalposts here.
”If intelligence lies in the process of acquiring new skills, there is no task X that solving X proves intelligence”
IMO it especially applies to things like solving a new IQ puzzle, especially when the model is pretrained for that particular task type, like was done with ARC-AGI.
For sure, it’s very good research to figure out what kind of tasks are easy for humans and difficult for ML, and then solve them. The jump in accuracy was surprising. But still in practice the models are unbeliavably stupid and lacking in common sense.
My personal (moving) goalpost for ”AGI” is now set to whether a robot can keep my house clean automatically. Its not general intelligence if it can’t do the dishes. And before physical robots, being less of a turd at making working code would be a nice start. I’m not yet convinced general purpose LLMs will lead to cost-effective solutions to either vs humans. A specifically built dish washer however…
You really shouldn't say LLMs "never graduate" to experienced staff - rather that they haven't yet. But there are recent and continuing improvements in the ability of the LLMs, and in time, perhaps a small amount of time, this situation may flip.
I'm talking about the current SOTA. In the future, all bets are off. For today, they are very capable when paired with a capable person, and that is how one uses them successfully today. Tomorrow will be different, of course.
I think you’ve exactly captured the two disparate views we see on HN:
1. LLMs have little value, are totally unreliable, will never amount to much because they don’t learn and grow and mature like people do., so they cannot replace a person like me who is well advanced in a career.
2. LLMs are incredible useful and will change the world because they excel at entry level work and can replace swaths of relatively undifferentiated information workers. LLM flaws are not that different from those workers’ flaws.
I’m in camp 2, but I appreciate and agree with the articulation of why they will not replace every information worker.
This all sounds plausible, but personally I find being paired to a new idiot-savant hire who never learns anything from the interaction incredibly exhausting. It can augment and amplify one’s own capabilities, but it’s also continuously frustrating and cumbersome.
While these folks waste breath debating whether AI is useful, I’m going to be over here…using it.
I use AI 100 times a day as a coder and 10,000 times a day in scripts. It’s enabled two specific applications I’ve built which wouldn’t be possible at single-person scale.
There’s something about the psychology of some subset of the population that insists something isn’t working when it isn’t _quite_ working. They did this with Wikipedia. It was evident that Wikipedia was 99% great for years before this social contingent was ready to accept it.
But please accept that you are in a small subset of people that it is very useful to. Every time I hear someone championing AI, it is a coder. AI is basically useless to me, it is just a convoluted expensive google search.
These are not categories that needed this change or benefit from it. Specific plant care is one of the easiest things to find information about. And are you serious you couldn't find a pancake recipe? The coffee machine idk it depends on what you did. But the other two are like a parody of AI use cases. "We made it slightly more convenient, but it might be wrong now and also burns down a tree every time you use it."
> "We made it slightly more convenient, but it might be wrong now and also burns down a tree every time you use it."
Sounds like early criticisms of the internet. I assume you mean he should be doing those things with a search engine, but maybe we shouldn't allow that either. Force him to use a book! It may be slightly less convenient, and could still be wrong, but...
Before crypto and AI computing in general and the internet in particular were always an incredible deal in terms of how much societal value we get out of it for the electricity consumed.
I also use it for plant care tips. What should I feed this plant and what kind of soil to use and all the questions I never bothered to Google and crawl through some long blog article on
Do you not use it to try learning new things? I use it to help get familiar with new software (recently for FreeCAD), or new concepts (passive speaker crossover design).
it's _extremely_ useful for lawyers, arguably even more so than for coders, given how much faster they can do stuff. They're also extremely useful for anyone who writes text and wants a reviewer. Also capable to execute most daily activities of some roles, such as TPMs.
It's still useful to a small subset of all those professions - the early adopters. Same way computers were useful to many professionals before the UI (but only a small fraction of them had the skillset to use terminals)
multiple lawyer friends I know are using chatgpt (and custom gptees) for contract reviews. They upload some guidelines as knowledge, then upload any new contract for validation. Allegedly replaces hours of reading. This is a large portion of the work, in some cases. Some of them also use it to debate a contract, to see if there's anything they overlooked or to find loopholes. LLMs are extremely good at that kind of constrained creativity mode where they _have_ to produce something (they suck at saying "I dont know" or "no"), so I guess it works as sort of a "second brain" of sorts, for those too.
There's even reported cases of entire legislations being written with LLMs already [1]. I'm sure there's thousands more we haven't heard about - the same way researchers are writing papers w/ LLMs w/o disclosing it
Five years later, when the contract turns out to be defective, I doubt the clients are going to be _thrilled_ with “well, no, I didn’t read it, but I did feed it to a magic robot”.
It only has to be less likely to cause that issue than a paralegal to be a net positive.
Some people expect AI to never make mistakes when doing jobs where people routinely make all kinds of mistakes of varying severity.
It’s the same as how people expect self-driving cars to be flawless when they think nothing of a pileup caused by a human watching a reel while behind the wheel.
My understanding is the firm operating the car is liable, in the full self driving case of commercial vehicles (waymo). The driver is liable in supervised self driving cases (privately owned Tesla)
How do you get the LLM to the point where it can draft a demand letter? I guess I'm a little confused as to how the LLM is getting the particulars of the case in order to write a relevant letter. Are you typing all that stuff in as a prompt? Are you dumping all the case file documents in as prompts and summarizing them, and then dumping the summaries into the prompt?
Demand letters are the easiest. Drag and drop police report and medical records. Tell it to draft a demand letter. For most things, there are only a handful critical pages in the medical records, so if the original pdf is too big, I’ll trim excess pages. I may also add my personal case notes.
I use a custom prompt to adjust the tone, but that’s about it.
I think the big mistake is _blindly relying on the results_ - although that problem has been improving dramatically (gpt3.5 hallucinated constantly, I rarely see a hallucination w/ the latest gpt/claude models)
She uses gpt as an editor for her emails and web content. she’ll just say "improve this," she gets options for how she might say something in a different way.
When preparing for a summit, she gave it a list of broad topics she wanted to cover. Gpt generated a list of specific titles and descriptions for her talks. This in turn gave her specific ideas to write talks about instead of just the broad topic.
When she wasn’t sure about the sequence of her talks, she asked GPT for advice on the order. GPT suggested an arrangement that created a logical flow and the reasoning for that flow, which ended up being pretty good.
She often uses gpt as a sounding board for ideas. She said she likes having an always-available colleague to bounce thoughts off of.
Walk into any random coffee shop in America where people are working on their laptops and you will see some subset of them on ChatGPT. It’s definitely not just coders who are finding it useful.
Particularly given the article's target is "systems based on large neural networks" and not specifically LLMs, I'd claim there are a vast number of uncontroversially beneficial applications: language translation, video transcription, material/product defect detection, weather forecasting/early warning systems, OCR, spam filtering, protein folding, tumor segmentation, drug discovery and interaction prediction, etc.
I'd call it a working google search, unlike, you know, google these days.
Actually google's LLM-based search results have been getting better, so maybe this isn't the end of the line for them. But for sophisticated questions (on noncoding topics!) I still always go to chatgpt or claude.
> google's LLM-based search results have been getting better
don't worry, Google WILL change this because they don't make money when people find the answer right away. They want people to see multiple ads before leaving the site.
It's being used in drive through windows.
In movies, in graphic design, pod casts, music, etc... 'entertainment' industry.
And HN, it isn't just a few odd balls on HN championing it. I wish there was way to get a sentiment analysis of HN, it seems there are lot more people using it than not using it.
And, what about the silent majority, the programmers that don't hang out on HN? I hear colleagues talk about it all the time.
The impact is here, whether they are self directed or not, or whether there are still a few people not using it.
Yesterday ChatGPT helped me to elaborate a skincare routine for my wife with multiple serums and creams that she received for Christmas.
She and I had no idea when to apply, how to combine or when to avoid combination of some of those products.
I could have google it myself in the evenings and had the answer in a few days of research, but with o1 in a 15min session my wife had had a solid weekly routine, the reasoning about those choices and academic papers with research about those products. (Obviously she knows a lot about skincare in general, so she had the capacity to recognize any wrong recommendation).
Nothing game changer, but is great to save lots of time in this kind of tasks.
It's 2 days after Christmas, too early to know the impact of the purchases made based on what AI recommended, either positive or negative.
If you're relying on AI to replace a human doctor trained in skin care or alternatively, your Google skills; please consider consulting an actual doctor.
If she "knows a lot about skincare in general, so she had the capacity to recognize any wrong recommendation", then what did AI actually accomplish in the end.
>> It's 2 days after Christmas, too early to know the impact of the purchases made based on what AI recommended, either positive or negative.
No worries, I can tell you what to expect: nothing. No effect. Zilch. Nada. Zero. Those beauty creams are just a total scam and that's obvious from the fact they're targetted just as well to women who don't need them (young, good skin) as to ones who do (older, bad skin).
About the only thing the beauty industry has figured out really works in the last five or six decades is Tretinoin, but you can use that on its own. Yet it's sold as one component in creams with a dozen others, that do nothing. Except make you spend money.
Forgot to say: you can buy Tretinoin at the pharmacy, over the counter even depending on where you are. They sell it as a treatment for acne. It's also shown to reduce wrinkles in RCTs [1]. It's dirt cheap and you absolutely don't need to buy it as a beauty cream and pay ten times the price.
_____________
[1] Topical tretinoin for treating photoaging: A systematic review of randomized controlled trials (2022)
It's a teratogen causing birth defects and miscarriages, so severe that "women of child bearing age taking isotretinoin are required to register for the iPLEDGE program. The iPLEDGE program requires that women taking isotretinoin undergo frequent pregnancy tests and commit to using two (2) forms of birth control in order to prevent themselves from getting pregnant."[1]
From Wikipedia[2]: "Isotretinoin is a teratogen; there is about a 20–35% risk for congenital defects in infants exposed to the drug in utero, and about 30–60% of children exposed to isotretinoin prenatally have been reported to show neurocognitive impairment".
See also pages like r/AccutaneRecovery[3] for people harmed by using it for acne, reporting systemic damage, perhaps permanent damage.
Scroll down[1] for the picture of some of the possible side effects of oral Accutane/Isotretinoin on the mother[3] and note that Wikipedia says "the most common adverse effects are dry lips (cheilitis), dry and fragile skin (xeroderma), dry eyes[8] and an increased susceptibility to sunburn" and wonder how a beauty treatment which improves skin condition has most common side effects which ruin skin condition.
This line of inquiry leads to a fun conspiracy/woo hypothesis; Grant Genereux[5]'s claims that what it does is trigger stem cells to differentiate in the epithelial layers of the skin, which makes thicker skin in the short term (wrinkle free) and worn out stem cells and thick skin in the longer term - and that many small vessels in the body have an epithelial lining of 'internal skin' and that thickens by the same mechanism leading to narrowing and closing of all kinds of internal vessels - tear ducts and sweat glands and blood vessels and inside the kidneys and liver and inner ear, etc. which cause the dry skin and dry eyes "side effects" (direct effects really) seen outside, and the organ damage/dizziness/etc. seen inside. And that it's a teratogen by getting inside cells, damaging them, damaging the DNA/protein building mechanisms causing wider systemic damage which can be long term and is not cleared up by stopping taking Accutane, this is misunderstood as retinoids "mediating hundreds of gene expressions" but is really shotgun chaotic damage and that's why there isn't a single symptom to look for and how it gets diagnosed as many different organ-specific diseases instead of retinoid toxicity damage. And/or causing cellular apoptosis with immune system response to a percieved 'attack', which is then seen as organ damage with immune system activity present, and misdiagnosed as "autoimmune" where the immune system has decided to attack an organ for no reason, which is why autoimmune disorders never have treatments or cures and why they cluster (people with one often get more) despite no good reason that should happen.
And that this whole collection of behaviours is triggered by food with Vitamin A (retinol in the tretinoin family) in it such as dairy and meat fat and Cod Liver oil, and foods with Beta Carotene (retinoids in the same family) such as orange/yellow/dark green coloured fruits and vegetables, and fortified Vitamin A in low-fat dairy and flours and other products through the USA/Europe. And it doesn't take much more than the RDA of Vitamin A to become problematic, and once it builds up in the body beyond the level the body can handle over a few decades, it's like blue touch paper waiting to be lit. Which, he suggests, is why auto-immune disorders cluster together (if you get one, you likely get more), why Eastern Canada Prince Edward Island near a Cod Liver Oil refinery was the highest incidence of Alzheimers in the world and that has been dropping since the refinery closed, and many more connection-between-retinoids-and-disease-states including claims by other people[6].
(I called it a 'fun' idea - it is at least fun along the lines of Ty...
Interesting, I'm the opposite now. Why would I click a couple links to read a couple (verbose) blog posts when I can read a succinct LLM response. If I have low confidence in the quality of the response then I supplement with Google search.
I feel near certain that I am saving time with this method. And the output is much more tuned to the context and framing of my question.
Hah, take for example my last query in ChatGPT:
> Are there any ancient technologies that when discovered furthered modern understanding of its field?
ChatGPT gave some great responses, super fast. Google also provides some great results (though some miss the mark), but I would need to parse at least three different articles and condense the results.
To be fair, ChatGPT gives some bad responses too.. But both an LLM and Google search should be used in conjunction to perform a search at the same time.
Use LLMs as breadth-first search, and Google as depth-first search.
> Every time I hear someone championing AI, it is a coder
The argument I make is why aren’t more people finding ways to code with AI?
I work in a leadership role at a marketing agency and am a passable coder for scripts using Python and/or Google Apps Scripts. In the past year, I’ve built more useful and valuable tools with the help of AI than I had in the 3 or so years before.
We’re automating more boring stuff than ever before. It boggles my mind that everybody isn’t doing this.
In the past, I was limited by technical ability because my knowledge of our business and processes was very high. Now I’m finding that technical ability isn’t my limitation, it’s how well I can explain our processes to AI.
I'd argue that's just because coders are first to line up for this.
There was a different thread on this site i read where a journalist used the wrong units of measurement (kilowatts instead of killowatt-hours for energy storage). You could paste the entire article into chatGPT with a prompt "spot mistakes in the following; [text]" and get an appropriate correction for this and similar mistakes the author made.
As in there's journalists right now posting articles with clear mistakes that could have been proof read more accurately than they were if they were willing to use AI. The only excuse i could think of is resistance to change. A lot of professions right now could do their job better if they leant on the current generation of AI.
In my bubble coders find LLMs least useful. After all we already have all kinds of fancy autocomplete that works deterministically and doesn't hallucinate - and still not everyone uses it.
When I use LLMs, I use it exactly as Google search on steroids. It's great for providing a summary on some unknown topic. It doesn't matter if it gets it wrong - the main value is in keywords and project names, and one can use the real Google search from there.
And it isn't expensive if you are using the free version
These are different social contingents I think. At least for me I was super on board with wikipedia because as you say the use to me was immediate and certain. AI I have tried every few months for the last two years but I still haven't found a strong use for it. It has changed nothing for me personally except making some products I use worse.
Cursor has been quite the jaw-dropping game changer for me for greenfield hobby dev.
I don't know how useful it would be for my job, where I do maintenance on a pretty big app, and develop features on this pretty big app. But it could be great, I just don't know because work only allows Copilot. And Copilot is somewhere between annoying and novelty in my opinion.
Generally people are resistant to change and the average person will typically insist new technologies are pointless.
Electricity and the airplane were supposed to be useless and dangerous dead ends according to the common person: https://pessimistsarchive.org/
But we all like to think we have super unique opinions and personalities, so "this time it's different."
When the change finally happens, people go about their lives as if they were right all along and the new technology is simply a mysterious and immutable fixture of reality that was always there.
Segway seems to have hardly been a dead end, or useless for that matter. Segway-style devices like the electric unicycle and many other light mobility devices seem to be direct descendants of the Segway. Segway introduced gyroscopes to the popular tech imagination, at least in my lifetime (not sure before).
There is a vast difference between arguments like "Phones have been accused of ruining romantic interaction and addicting us to mindless chatter" and "current AI has problems generating accurate information and can't replace researching things by hand for complicated or niche topics and there is reason to believe that the current architecture may not solve this problem"
That aside optimist are also not always right, otherwise we would have cold fusion already and have a base on mars.
> But we all like to think we have super unique opinions and personalities, so "this time it's different."
Are you suggesting that anything which is hyped is the future? Like, for every ten heavily-hyped things, _maybe_ one has some sort of post-hype existence.
The pessimist is not wrong. In fact he's right more frequently than wrong. Just look at a long list of inventions. How many of them were so successful as the car or the airplane? Most of them were just passing fads that people don't even remember anymore. So if you're asking who is smarter, I would say the pessimist is closer to the truth, but the optimist who believed in something that really became successful is now remembered by everyone.
I feel your argument relies on assuming that being an optimist or pessimist means believing 100% or 0%, whereas I'd claim it's instead more just having a relative leaning in a direction. Say after inspecting some rusty old engines a pessimist predicts 1/10 will still function and an optimist predicts 4/10 will function. If the engines do better than expected and 3/10 function, the optimist was closer to the truth despite most not working.
Similarly, being optimistic doesn't mean you have to believe every single early-stage invention will work out no matter how unpromising - I've been enthusiastic about deep learning for the past decade (for its successes in language translation, audio transcription, material/product defect detection, weather forecasting/early warning systems, OCR, spam filtering, protein folding, tumor segmentation, spam filtering, drug discovery and interaction prediction, etc.) but never saw the appeal of NFTs.
Additionally worth considering that the cost of trying something is often lower than the reward of it working out. Even if you were wrong 80% of the time about where to dig for gold, that 20% may well be worth it; reducing merely the frequency of errors is often not logically correct. It's useful in a society to have people believe in and push forward certain inventions and lines of research even if most do not work out.
I think xvector's point is about people rehashing the same denunciations that failed to matter for previous successful technologies - the idea that something is useless because it's not (or perhaps will never be) 100.0% accurate, or the "Until it can do dishes, home computer remains of little value to families"[0] which I've seen pretty much ad verbatim for AI many times (extra silly now that we have dishwashers).
Given in real life things have generally improved (standard of living, etc.), I think it has typically been more correct to be optimistic, and hopefully will be into the future.
This argument is very prone to survivorship bias. Of course, when we think back to the hyped technologies of the past we are going to remember mostly those that justified the hype. The failures get forgotten. The memory of social discourse fades extremely quickly, much faster than, for example, pop culture or entertainment.
I've found it depends on the context (pardon the pun)
For example, personal projects that are small and where copilot has access to all the context it needs to make a suggestion - such as a script or small game - it has been really useful.
But in a real world large project for my day job, where it would need access to almost the entire code base to make any kind of useful suggestion that could help me build a feature, it's useless! And I'd argue this is when I need it.
It can ingest the entire codebase (up to its context length), but for some reason, I’ve always had much higher quality chats with smaller bite-sized pieces of code.
Autocomplete distracts me enough that it really needs to be close to 100% correct before it's useful. Otherwise it's just wrecking my flow and slowing me down.
Exponentially? Absolutely not. In the best case it creates something that’s almost useful. Are you working on large actual codebases or talking about some one off toy apps?
I spend most of my time thinking about what I'm trying to do and how to best achieve it, so code completion can only make me marginally more productive. If the tool can guess a large chunk of what I've decided to do, sure, that's nice, but at the end of the day it still only adds up to a couple minutes at best.
You could try aider, or another tool/workflow where you provide whole files and ask for how they should be changed - very different from code completion type tools!
Anyone who says AI is useless never had to do the old method of cobbling together git and ffmpeg commands from StackOverflow answers.
I have no interest in learning the horrible unintuitive UX of every CLI I interact with, I'd much rather just describe in English what I want and have the computer figure it out for me. It has practically never failed me, and if it does I'll know right away and I can fall back to the old method of doing it manually. For now it's saving me so much time with menial, time-wasting day-to-day tasks.
I had a debate recently with a colleague who is very skeptical of LLMs for every day work. Why not lean in on searching Google and cross referencing answers, like we've done for ages? And that's fine.
But my counterargument is that what I find to be so powerful about the LLMs is the ability to refine my question, narrow in on a tangent and then pull back out, etc. And *then* I can take its final outcome and cross reference it. With the old way of doing things, I often felt like I was stumbling in the dark trying to find the right search string. Instead I can use the LLM to do the heavy lifting for me in that regard.
Most of those people are a bit bad at making their case. What they mean but don't convey well is that AI is useless for it's proclaimed uses.
You are correct that LLMs are pretty good at guessing this kind of well-documented & easily verifiable but hard to find information. That is a valid use. (Though, woe betide the fool who uses LLMs for irreversible destructive actions.)
The thing is though, this isn't enough. There just aren't that many questions out there that match those criteria. Generative AI is too expensive to serve that small a task. Charging a buck a question won't earn the $100 billion OpenAI needs to balance the books.
Your use case gets dismissed because on it's own, it doesn't sustain AI.
I think you’re on to something. I find the sentiment around LLMs (which is at the early adoption stage) to be unnecessarily hostile. (beyond normal HN skepticism)
But it can be simultaneously true that LLMs add a lot of value to some tasks and less to others —- and less to some people. It’s a bit tautological, but in order to benefit from LLMs, you have to be in a context where you stand to most benefit from LLMs. These are people who need to generate ideas, are expert enough to spot consequential mistakes, know when to use LLMs and when not to. They have to be in a domain where the occasional mistake generated costs less than the new ideas generated, so they still come out ahead. It’s a bit paradoxical.
LLMs are good for: (1) bite-sized chunks of code; (2) ideating; (3) writing once-off code in tedious syntax that I don’t really care to learn (like making complex plots in seaborn or matplotllib); (4) adding docstrings and documentation to code; (5) figuring out console error messages, with suggestions as to causes (I’ve debugged a ton of errors this way — and have arrived at the answer faster than wading through Stackoverflow); (6) figuring out what algorithm to use in a particular situation; etc.
They’re not yet good at: (1) understanding complex codebases in their entirety (this is one of the overpromises; even Aider Chat’s docs tell you not to ingest the whole codebase); (2) any kind of fully automated task that needs to be 100% deterministic and correct (they’re assistants); (3) getting math reasoning 100% correct (but they can still open up new avenues for exploration that you’ve never even thought about);
It takes practice to know what LLMs are good at and what they’re not. If the initial stance is negativity rather than a growth mindset, then that practice never comes.
But it’s ok. The rest of us will keep on using LLMs and move on.
An example that might be of interest to readers: I gave it two logs, one failing and one successful, and asked it to troubleshoot. It turned out a loosely pinned dependency (Docker image) had updated in the failing one. An error mode I was familiar with and could have solved on my own, but the LLM saved me time. They are reliable at sifting through text.
I've been sold AI as if it can do anything. It's being actively sold like a super intelligent independent human that never needs breaks.
And it just isn't that thing. Or, rather, it is super intelligent but lacks any wisdom at all; thus rendering it useless for how it's being sold to me.
>which is at the early adoption stage
I've said this in other places here. LLM's simply aren't at early adoption stage anymore. They're being packaged into literally every saas you can buy. They're a main selling point for things like website builders and other direct to business software platforms.
Why not ignore the hype, and just quietly use what works?
I don’t use anything other than ChatGPT 4o and Claude Sonnet 3.5v2. That’s it. I’ve derived great value from just these two.
I even get wisdom from them too. I use them to analyze news, geopolitics, arguments around power structures, urban planning issues, privatization pros and cons, and Claude especially is able to give me the lay of the land which I am usually able to follow up on. This use case is more of the “better Google” variety rather than task-completion, and it does pretty well for the most part. Unlike ChatGPT, Claude will even push back when I make factually incorrect assertions. It will say “Let me correct you on that…”. Which I appreciate.
As long as I keep my critical thinking hat on, I am able to make good use of the lines of inquiry that they produce.
Same caveat applies even to human-produced content. I read the NYTimes and I know that it’s wrong a lot, so I have to trust but verify.
I agree with you, but it's just simply not how these things are being sold and marketed. We're being told we do not have to verify. The AI knows all. It's undetectable. It's smarter and faster than you.
And it's just not.
We made a scavenger hunt full of puzzles and riddles for our neighbor's kids to find their Christmas gifts from us (we don't have kids at home anymore, so they fill that niche and are glad to because we go ballistic at Christmas and birthdays). The youngest of the group is the tech kid.
He thought he fixed us when he realized he could use chatgpt to solve the riddles and cyphers. It recognized the Caesar letter shift to negative 3, but then made up a random phrase with words the same length to solve it. So the process was right, but the outcome was just outlandishly incorrect. It wasted about a half hour of his day. . .
Now apply that to complex systems or just a simple large database, hell, even just a spreadsheet. You check the process, and it's correct. You don't know the outcome, so you can't verify unless you do it yourself. So what's the point?
For context, I absolutely use LLM's for things that I know roughly, but don't want to spend the time to do. They're useful for that.
They're simply not useful for how they're being marketed, which is too solve problems you don't already know.
> if it does I'll know right away and I can fall back to the old method of doing it manually
It's well and ok with things you can botch with no consequence other than some time wasted. But I've bricked enough VMs trying commands I did not understand to know that if you need to not fuck up something you'll have to read those docs and understand them. And hope they're not out of date / wrong.
> Anyone who says AI is useless never had to do the old method of cobbling together git and ffmpeg commands from StackOverflow answers.
It's useful for that yes, but I'd rather just live in a world where we didn't have such disasters of CLI that are git and ffmpeg.
LLMs are very useful for generating the obscure boilerplate needed because the underlying design is horrible. Relying on it means acquiescing to those terrible designs rather than figuring out redesigns that don't need the LLMs. For comparison, IntelliJ is very good at automating all the boilerplate generation that Java imposes on me, but I'd rather we didn't have boilerplate languages like Java, and I'd rather that IntelliJ's boilerplate generation didn't exist.
I fear in many cases that if an LLM is solving your problem, you are solving the wrong problem.
I'm not arguing against the UX of those tools, but isn't this a case of the problem being a hard one to solve and people having different needs? ffmpeg has a lot of knobs, but that's just the nature of media conversion and manipulation, just like ImageMagick. I'm not against using LLMs for restricting the search space to a specific problems, but I'm often seeing people not even understanding the problem itself, just its apparent physicality.
> Anyone who says AI is useless never had to do the old method of cobbling together git and ffmpeg commands from StackOverflow answers.
These days, I'm more likely to read the manual pages and take notes on interesting bits. If I'm going to rely on some tooling for some time, dedicating a few hours of reading is a good trade-off for me. No need to even remember everything, just the general way it solves the problem. Anything more precise is captured in notes, scripts, shell history,... I dare anyone to comes out with an essay like this from LLMs: https://karthinks.com/software/avy-can-do-anything/
I'm asking not for snark, but because when AI gives me something not _quite_ working, it requires much more time than what a "every 6 minutes in 10 hour work day" frame would allow to investigate. I just wonder if maybe you're pasting it as is and don't care about correctness if the happy path sort of works. Speaking of subsets, coders who did that before AI were also quite a group.
There must be something that explains the difference in our experiences. Apologies for the fact that my only idea is kinda negative. I understand the potential hyperbola here, but it doesn't explain much. I can stand AI BS once a day, maybe twice, before uncontrollably cursing into the chat.
Why not write tests with AI, too? Since using LLMs as coding assistants, my codebases have much more thorough documentation, testing and code coverage.
Don't start when you're already in a buggy dead-end. Test-driven development with LLMs should be done right from the start.
Also keep the code modular so it is easy to include the correct context. Fine-grained git commits. Feature-branches.
All the tools that help teams of humans of varying levels of expertise work together.
You may have enough expertise in your field that when you have a question, you know where to start looking. Juniors and students encounter dozens of problems and question per hour that fall into the unknown unknown category
How would the a 12 year old with ChatGPT recognize complicateed errors?
You still need experience to check the code not only the result otherwise you get this
You sound like the guy I just had to fire after blowing his toes off several times.
If you think you are obsolete or faster than anyone else with these tools then you only naive enough to have lost your objectivity to the marketing. I deal with real risk and failures from the output of ChatGPT which have serious financial consequences. The first victim is always the developer, then the tester.
At best, it is very good at ousting people who shouldn't be allowed anywhere near a damn computer.
We've got a senior dev who uses ChatGPT for his code all the time. Right now I am currently fixing all the exceptions 'his code' pops. Well, I shouldn't say what his code pops. The code ChatGPT generated for him. He just asks ChatGPT, copy pasta, doesn't even run it and checks it in.
Not outsourcing at all - you're are an engineer using the tools that make sense to solve a problem. The core issue with identifying as just a coder is that code is just one of many potential tools to solve a problem.
So your customer/employer is a coder too. They want solve a problem and use a tool: You.
A coder writes code in a programming language, that what distinguishes them from the customers who use natural language. The coder is the translator between the customer and the machine. If the machine does that, the machine is the coder.
Is your customer bringing you the solution to the problem or the problem and asking you to solve the problem? One is a translation activity and the other isn't.
If you're sitting in front of the keyboard, inputting instructions and running the resulting programs, yes you are still a coder. You're just move another layer up on the stack.
The same type of argument has been made for decades -- when coders wrote in ASM, folks would ask "are you still a coder when you use that fancy C to make all that low-level ASM obsolete?". Etc etc.
Have you tried a few? If so, which do you prefer? If not, which do you use? I'm a little late to the party, and the current amount of choices is quite intimidating.
I imagine you're asking about coding help. For that, I think you should qualify any answer you get with the user's most commonly used language (and framework, if applicable).
In my experience, Claude Sonnet 3.5 (3.6?) has been unbeatable. I use it for Rust. Making sense of compiler errors, rubberducking, finding more efficient ways to write some function and, truth be told, some times just plain old debugging. More than once, I've been able to dump a massive module onto the chat context and say "look, I'm experiencing this weird behavior but it's really hard to pin down what's causing it in this code" and it pointed to the exact issue in a second. That alone is worth the price of admission.
Way better than ChatGPT 4o and o-1, in my experience, despite me saying the exact opposite a few months ago.
This isn't about if LLMs are useful, it's about how useful can they become. We are trying to understand if there is a path forward to transformative tech, or are we just limited to a very useful tool.
It's a valid conversation after ~3 years of anticipating the world to be disrupted by this tech. So far it has not delivered.
Wikipedia did not change the world either, it's just a great tool that I use all the time
As for software, it performs ok. I give up on it most of the time if I am trying to write a whole application. You have to acquire a new skill, prompt engineering, and feverish iteration. It's a frustrating game of whack-a-mole and I find it quicker to write the code myself and just have the LLM help me with architecture ideas, bug bashing, and it's also quite good at writing tests.
I'd rather know the code intimately so I can more quickly debug it than have an LLM write it and just trust it did it well.
Peter Thiel talked about this years ago in his book From 0 to One. His key insight, which we're seeing today, is that AI tools will work side-by-side with people and enhance their productivity to levels never imagined. From helping with some basic tasks ("write an Excel script that transforms this table from this format to this new format") to helping write programs, it's a tool that aids humans in getting more things done than previously possible.
LLMs obviously have use cases but the market has practically priced in "AGI".
The danger is not that LLMs take jobs. The danger is that we are in a massive bubble and while these are nice tools they are not worth anything close to the trillions of dollars bet on them.
IMO the psychology at work here is basically denial that we can both be in the biggest bubble of all time in terms of dollars and LLMs are useful. Just not THAT useful.
Why should we care about whether we're in a market bubble or not, especially if one does not have their own money staked in the bubble? It's somebody else's capital that's at risk, no? If they're wrong, let them reap the consequences.
(cue rebuttal based on systemic consequences / financial bailouts etc, but you know what I mean; also, the dotcom bubble deflation didn't require a bailout)
Interesting read -- and a correct take, given the software development perspective. In that context, LLM-based AI is faulty, unpredictable, and unmanageable, and not ready for mission-critical applications.
If you want to argue otherwise, do a quick thought experiment first: would you let an LLM manage your financial affairs (entirely, unsupervised)? Would you let it perform your job while you receive the rewards and consequences? Would you be comfortable to give it full control of your smart home?
There are different sets of expectations put on human actors vs autonomous systems. We expect people to be fallible and wrong some of the time, even if the individuals in question can't/won't admit it. With a software-based system, the expectations are that it will be robust, tested, and performing correctly 100% of the time, and when a fault occurs, it will be clear, marked with yellow tape and flashing lights.
LLM-based AIs are sort of insidious in that they straddle this expectation gap: the emergent behaviour is erratic, projecting confident omniscience, while often hallucinating and plain wrong. However vague, the catch-all term "AI" still implies "computer system" and by extension "engineered and tested".
It's a bad example. Lots of finance firms use AI to manage their financial affairs - go and investigate what is currently considered state of the art for trading algorithms.
Now if you substituted something safety critical instead, say, running a nuclear power station, or my favourite currently in use example, self driving cars, then yes, you should be scared.
> go and investigate what is currently considered state of the art for trading algorithms.
These are not LLMs but algorithms written and designed by human minds. It is unfortunate that AI has become a catch-all word for any kind of machine learning.
LLMs create models, not algorithms. An algorithm is a rote sequence if steps to accomplish a task.
The following is an algorithm:
- plug in input to model
- say yes if result is positive, else say no
LLMs use models, the model is not an algorithm.
> There are patterns in the weights that could be steps in an algorithm.
Sure, but yeah... no..
"Could be steps in an algorithm" does not constitute an algorithm.
Weights are inputs, they are not themselves parts of an algorithm. The algorithm might still try to come up with weights. Still, don't confuse procedure from data.
Don't want to get to pedantic on that response.
The model can contain complex information.
There is already evidence it can form a model of the world.
So why not something like steps to get from A to B.
And, it is clear that LLMs can follow steps.
One didn't place in the Math Olympiad without some ability to follow steps.
"Yes, an LLM model can contain the steps of an algorithm, especially when prompted to "think step-by-step" or use a "chain-of-thought" approach, which allows it to break down a complex problem into smaller, more manageable steps and generate a solution by outlining each stage of the process in a logical sequence; essentially mimicking how a human would approach an algorithm. "
> There is already evidence it can form a model of the world.
Perhaps.
> So why not something like steps to get from A to B.
Why not - because a model and algorithm are different. Simply having a model does not mean you have an algorithm. An algorithm is a deterministic set of steps, a model is typically a function or set of functions for producing results. If the result of that model is to list a set of steps (and also evaluate them too) - that does not make the model an algorithm.
> And, it is clear that LLMs can follow steps
Sure, because that is what the model is set up to do.
> Yes, an LLM model can contain the steps of an algorithm, especially when prompted to "think step-by-step" or use a "chain-of-thought" approach, which allows it to break down a complex problem into smaller
This is the model looking into its training data to find algorithms that seem to match the prompt and then to print out the steps of the algorithm and also execute them. That's not an algorithm in of itself.
I feel I'm on pretty solid ground here. "Algorithmic prompting" has nothing to do with whether a model is an algorithm. I'd ask you google the differences of a model and an algorithm very thoroughly. If something follows an algorithm, I strongly suspect it cannot be a model by definition. It can still be an AI though, as there are non LLM's AI's out there that do follow algorithms. If we are talking about LLM, the M is for "MODEL". Models and algorithms are different. A model that looks for an algorithm to use - is a very sophisicated model, but it's still not an algorithm itself just because it could find, interpret and use one.
If you think so, you should publish your results. It seems like a lot of bright people are going down the road of using LLM for algorithmic tasks. To follow steps.
I think what I'm reaching for, is a little more esoteric, that out of all the data the model is trained on, that it has also started building up algorithms/steps in its 'model', which is part of how it pics the next item.
The whole reason algorithmic prompting started was people started noticing the LLM was already attempting some steps, and that if it was further helped along by prompting the steps, then the results were better.
But, I am using 'algorithm' rather loosely, as just 'steps', and they are a bit fuzzy, so not a purely math algorithm, but more of a fuzzy logic, a first start at reasoning.
edit
also, I should clarify. I am not confusing the algorithm to make the model versus the model, i'm saying in the model it learns to follow steps.
Makes me wonder how they detect market manipulation and fraud. Trivial activities, like marking the close, probably aren't hard to detect, but I imagine that some kind of ML thingy is involved in flagging accounts for manual inspection.
Pragmatically, "AI" will mean (and for, many people already does mean) stochastic and fallible.
If your users are likely to be AI illiterate and mistakenly feel that an AI app is reliable and suitable for mission critical applications when it isn't, that is a risk you mitigate.
But it seems deeply unserious of the author to just assert that mission-critical
software is the only "serious context" and the only thing matters, and therefore AI is dead end. "Serious, mission critical" apps are just going to be a niche in the future.
>> Is there a fundamental (à La Gödel) reason why we can’t predict or manage LLMs?
> We're building statistical models to make statistical predictions after training with a sufficiently large and statistically diverse dataset. Aka it's random.
This seems like a sort of hand wavy answer to what seems like a pretty deep and technical question. And to be fair, this isn’t the environment to ask that question, it seems like something a bunch of researchers would work out.
Of course we build statistical models all the time and then use them to make pretty good predictions. Is there something actually fundamental about these LL modes that makes them… unmanageable? Well, we’ll have to define manageable first… etc etc.
So the question is so abstract as to not have a handle on the answer...
Great, What is M theory?
This also is missing a structure to allow us to unify existing fundamental theories.
The only way to assess this is to stop treating the models as statistical beasts which simply only work with "enough" statistics and start talking about them from the direction of information theory. The answer is glib because the problem is the field requires a mix between stats, computing, and mathematics. 2 of these fields have their own languages for the problem (which differ, but exist) and computing has come along and made another set of names for the same complex things making the whole thing (for now) a mess... Especially with the main practicioners stuck in the view that big llm are simply money printers.
I get the glibness, I think it is fine for here, but actually I really hope someone out there puts in the effort to figure out what the right question of to ask for this stuff. I think you are right that it isn’t well defined here, but getting a well defined question is like halfway to finishing your thesis, right? Haha.
> would you let an LLM manage your financial affairs (entirely, unsupervised)?
It will likely be better[2] not because AI is good at this .
It would be because study after study[1] has shown that active management performs poorer than passive funds, less intervention gives better result over longer timeframe .
[1] the famous warren buffet bet comes to mind . There are more formal ones validating this .
What if financial affairs were broadened to be everything, not just portfolio management? Eg: paying bills, credit cards, cash balance in check vs savings vs brokerage.
Good financial management(portfolio and personal) is a matter of disciplined routine, performed consistently over long timeframe, combined with impulse control. It is not complicated at all, any program (LLM or just a rules engine) will always do far better than we can because it will not suffer either problem(sticking to the routine or impulse).
Most humans make very bad decisions around personal finance, whether it is big things like gambling or impulse buys with expensive credit, to smaller items like tracking subscriptions or keeping not needed money in checking account etc.
This is irrespective of financial literacy, education, wealth or professions like say working in finance/ personal wealth management even.
Entire industries like lottery, gambling, luxury goods, gaming, credit card APRs, Buy Now Pay Later, Consumer SaaS, Banking overdraft fees are all built around our inability to control our impulses or follow disciplined routines.
This is why trust funds with wealth management professionals are the only way to generational wealth.
You need the ability to control any benefactor (the next generations) from excising their impulses on amounts beyond their annual draw. Plus the disciplined routine of a professional team who are paid to do only this with multiple layers that vet the impulses of individual managers and conservative mandate to keep them risk averse and therefore less impulsive.
If an program can do it for me (provided of course I irrevocably give away my control to override or alter its decisions) then normal people can also benefit without the high net worth required for wealth management.
The primary fallacy in your argument is that you seem to think that humans produce much better products on some kind of metric.
My lived experience the software industry at almost all levels over the last 25 years leads me to believe that the vast majority of humans and teams of humans produce atrocious code that only wastes time, money, and people's patience.
Often because it is humans producing the code, other humans are not willing to fully engage, criticize and improve that code, deferring to just passing it on to the next person, team, generation, whatever.
Yes, this perhaps happens better in some (very large and very small) organizations, but most often it only happens with the inclusions of horrendous layers of protocol, bureaucracy, more time, more emotional exhaustion, etc.
In other words a very costly process to produce excellent code, both in real capital and human capital. It literally burns through actual humans and results in very bad health outcomes for most people in the industry, ranging from minor stuff to really major things.
The reality is that probably 80% of people working in the tech industry can be outperformed by an AI and at a fraction of the cost. AIs can be tuned, guided, and steered to produce code that I would call exception compared even to most developers who have been in the field for 5years or more.
You probably come to this fallacy because you have worked in one of these very small or very large companies that takes producing code seriously and believe that your experience represents the vast majority of the industry, but in fact the middle area is where most code is being "produced" and if you've never been fully engaged in those situations, you may literally have no idea of the crap that's being produced and shipped on a daily basis. These companies have no incentive to change, they make lots of money doing this, and fresh meat (humans) is relatively easy to come by.
Most of these AI benchmarks are trying to get these LLMs to produce outputs at the scale and quantity of one of these exceptional organizations when in fact, the real benefits will come in the bulk of organizations that cannot do this stuff and AI will produce as good or better code than a team of mediocre developers slogging away in a mediocre, but profitable, company.
Yes there are higher levels of abstraction around code, and getting it deployed, comprehensive testing, triaging issues, QA blah blah, that humans are going to be better at for now, but I see many of those issues being addressed by some kind of LLM system sooner or later.
Finally, I think most of the friction people are seeing right now in their organization is because of the wildly ad hoc way people and organizations are using AI, not so much about the technological abilities of the models themselves.
“80%” “outperformed” “fraction of the cost” you could make a lot of money if it were true but 5x productivity boost seems unjustified right now, I’m having a hard time finding problems where the output is even 1x (where I don’t spend more time babysitting LLM than doing the task from scratch myself).
For "stay in your lane" stuff, I agree, it relatively sucks.
For "today I need do stuff two lanes over", well it still needs the babysitting, and I still wouldn't put it on tasks where I can't verify the output, but it definitely delivers a productivity boost IME.
ChatGPT seems to be good about this. If you invent something and ask about it, like "What was the No More Clowning Act of 2025?", it will say it can't find any information on it.
The older or smaller models, like anything you can run locally, are probably far more likely to just invent some bullshit.
That said, I've certainly asked ChatGPT about things that definitely have a correct answer and had it give me incorrect information.
When talking about hallucinating, I do think we need to differentiate between "what you asked about exists and has a correct answer, but the AI got it wrong" and "What you're asking for does not exist or does not have an answer, but the AI just generated some bullshit".
> Would you let it perform your job while you receive the rewards and consequences?
isn't this what being a human manager is? not sure why you're saying it must be entirely + unsupervised. at my job, my boss mostly trusts me but still checks my work and gives me feedback when he wants something changed. he's ultimately responsible for what I do.
I believe you're asking the wrong question, or at least you're asking it in the wrong way. From my POV, it comes in two parts:
1. Do you believe that LLMs operate in a similar way to the important parts of human cognition?
2. If not, do you believe that they operate in a way that makes them useful for tasks other than responding to text prompts, and if so, what are those tasks?
If you believe that the answer to Q1 is substantively "yes" - that is, humans and LLM are engaged in the same sort of computational behavior when we engage in speech generation - then there's presumably no particular impediment to using an LLM where you might otherwise use a human (and with the same caveats).
My own answer is that while some human speech behavior is possibly generated by systems that function in a semantically equivalent way to current LLMs, human cognition is capable of tasks that LLMs cannot perform de novo even if they can give the illusion of doing so (primarily causal chain reasoning). Consequently, LLMs are not in any real sense equivalent to a human being, and using them as such is a mistake.
I think C.S. Peirce's distinction between corollarial reasoning and theorematic reasoning[1][2] is helpful here. In short, the former is the grindy rule following sort of reasoning, and the latter is the kind of reasoning that's associated with new insights that are not determined by the premises alone.
As an aside, Students of Peirce over the years have quite the pedigree in data science too, including the genius Edgar F. Codd, who invented the relational database largely inspired by Peirce's approach to relations.
Anyhow, computers are already quite good at corollarial reasoning and have been for some time, even before LLMs. On the other hand, they struggle with theorematic reasoning. Last I knew, the absolute state of the art performs about as well as a smart high school student. And even there, the tests are synthetic, so how theorematic they truly are is questionable. I wouldn't rule out the possibility of some automaton proposing a better explanation for gravitational anomalies than dark matter for example, but so far as I know nothing like that is being done yet.
There's also the interesting question of whether or not an LLM that produces a sequence of tokens that induces a genuine insight in the human reader actually means the LLM itself had said insight.
> My own answer is that while some human speech behavior is possibly generated by systems that function in a semantically equivalent way to current LLMs, human cognition is capable of tasks that LLMs cannot perform de novo even if they can give the illusion of doing so (primarily causal chain reasoning). Consequently, LLMs are not in any real sense equivalent to a human being, and using them as such is a mistake.
In the workplace, humans are ultimately a tool to achieve a goal. LLM's don't have to be equivalent to humans to replace a human - they just have to be able to achieve the goal that the human has. 'Human' cognition likely isn't required for a huge amount of the work humans do. Heck, AI probably isn't required to automate a lot of the work that humans do, but it will accelerate how much can be automated and reduce the cost of automation.
So it depends what we mean as 'use them as a human being' - we are using human beings to do tasks, be it solving a billing dispute for a customer, processing a customers insurance claim, or reading through legal discovery. These aren't intrinsically 'human' tasks.
So 2 - yes, I do believe that they operate in a way that makes them useful for tasks. LLM's just respond to text prompts, but those text prompts can do useful things that humans are currently doing.
I think the vector representation stuff is an effective tool and possibly similar to foundational tools that humans are using.
But my gut feel is that it's just one tool of many that combine to give humans a model+view of the world with some level of visibility into the "correctness" of ideas about that world.
Meaning we have a sense of whether new info "adds up" or not, and we may reject the info or adjust our model.
I think LLM's in their current state can be useful for tasks that do not have a high cost resulting from incorrect output, or tasks that can have their output validated by humans or some other system cost-effectively.
IMHO, a more important and testable difference is that humans don't have separate "train" and "infer" phases. We are able to adapt more or less on the fly and learn from previous experience. LLMs currently cannot retain any novel experience past the context window.
I think LLMs operate in a similar way to some of the important parts of human congnition.
I believe they operate in a way that makes them at least somewhat useful for some things. But I think the big issue is trustworthiness. Humans - at least some of them - are more trustworthy than LLM-style AIs (at least current ones). LLMs need progress on trustworthiness more than they need progress on use in other areas.
Mostly your financial advisor writes your return you sign off on or manages your portfolio. But the advisor usually solicits and interacts with you to know what your financial goals are and ensure you are on board with the consequences of their advice.
I do not dismiss that some people are completely hands off at great risk IMHO. But these are not me - as was my initial proposition.
_Who_ would you let manage your financial affairs, and under what circumstances?
To which my answer would be something like: a qualified financial adviser with a good track record, who can be trusted to do the job to, if not the best of their abilities, at least an acceptable level of professional competence.
A related question: who would you let give you a lift someplace in a car?
And here's where things get interesting. Because on the one hand there's a LOT more at stake (literally, your life), and yet various social norms, conventions , economic pressures and so on mean that in practice we quite often entrust that responsibility to people who are very, very far from performing at their best.
So while a financial adviser AI is useless unless it can perform at the level of a trained professional doing their job (or unless it can perform at maybe 95% of that level at much lower cost), a self-driving car is at least _potentially_ useful if it's only somewhat better than people at or close to their worst. As a high proportion of road traffic collisions are caused by people who are drunk, tired, emotionally unstable or otherwise very very far from the peak performance of a human being operating a car.
(We can argue that a system which routinely requires people to carry out life-or-death, mission-critical tasks while significantly impaired is dangerously flawed and needs a major overhaul, but that's a slightly different debate).
I find this kind of argument comes up a lot and it seems fundamentally flawed to me.
1. You can set a bar wherever you want for a level of "seriousness" and huge swathes of real world work will fall below it, and are therefore attractive to tackle with these systems.
2. We build critical large scale systems out of humans, which are fallible and unverifiable. That's not to say current LLMs are human or equivalent, but "we can't verify X works all the time" doesn't stop us doing exactly that a lot. We deal with this by learning how humans make mistakes, why, and build systems of checks around that. There is nothing in my ind that stops us doing the same with other AI systems.
3. Software is written by, checked by and verified by humans at least at some critical point - so even verified software still has this same problem.
We've also been doing this kind of thing with ML models for ages, and we use buggy systems for an enormous amount of work worldwide. You can argue we shouldn't and should have fully formally verified systems for everything, but you can't deny that right now we have large serious systems without that.
And if your goal is "replace a human" then I just don't think you can reasonably say that it requires verifiable software.
> Systems are not explainable, as they have no model of knowledge and no representation of any ‘reasoning’.
Neither of those statements are true are they? There are internal models, and recent models are designed around having a representation of reasoning before replying.
> current generative AI systems represent a dead end, where exponential increases of training data and effort will give us modest increases in impressive plausibility but no foundational increase in reliability
And yet reliability is something we see improve as LLMs get better and we get better at training them.
There are two epistemic poles: the atomistic and the probabilistic. The author subscribes to a rule-based atomistic worldview, asserting that any perspective misaligned with this framework is incorrect. Currently, academia is undergoing a paradigm shift in the field of artificial intelligence. Symbolic AI, which was the initial research focus, is rapidly being replaced by statistical AI methodologies. This transition diminishes the relevance of atomistic or symbolic scientists, making them worry they might become irrelevant.
Indeed and unfortunately. I've been reading up on "the binding problem" in AI lately and came across a paper that hinged on there being an "object representation" which would magically solve the apparent issues in symbolic AI. In the discussion some 20 pages later, the authors confessed that they, nor anybody else, could define what an object was in the first place. Sometimes the efforts seem focused on "not letting the other team win" rather than actually having something tangible to bring to the table.
I never want to claim certainties, but it seems pretty close to certain that symbolic AI loses to statistical AI.
I think there is room for statistical AI to operate symbolic systems so we can better control outputs. Actually, that's kind of what is going on when we ask AI to write code.
An observation with scientific paradigm shifts is that they tend not to reverse. Also the lingo someone commented on is that the fundamental problem is the different philosophical views of what knowledge is and can be. Either knowledge is base on symbols and rules like in mathematics or they are probabilities like in anything we actually can measure. Both these views can coexist and maybe AI will find the missing link between them some day. Possibly no human will grasp the link.
No, they are very useful tools to build up inteligent systems out of.
Everything from perplexity onward shows just how useful agents can be.
You get another bump in utility when you allow for agents swarms.
Then another one for dynamically generated agent swarms.
The only reason why it's not coming for your job is that LLMs are currently too power hungry to run those jobs for anything but research - at a couple thousand to couple of million times the price of a human doing the work.
Which works out to 10 to 20 epochs of whatever Moore's law looks like in graphics cards.
What is that bump in utility in practical terms? You can point to a benchmark improvement but that's no indication the agent swarm is not reducing to "giving an llm an arbitrary amount of random guesses".
Standard LLM quadratic attention isn't an approximation, it's perfect recall. Approaches that compress that memory down into a fixed-size state are an approximation, and generally perform worse, that's why linear transformers aren't widely used.
What I find curious is that the people who sell the AI as the holy grail that will make any jobs obsolete in a few year at the same time claim that there's huge talent shortage and even engage in feud on immigration and spend capital to influence immigration policies.
Apparently they don't believe that AI is about to revolutionize things that much. This makes me believe that significant part of the AI investment is just FOMO driven, so no real revolution is around the corner.
Although we keep seeing claims that AI achieved PHD level this Olympics level that, people who actually own these keep demanding immigration policy changes to bring actual humans from overseas for year to come.
Is that so? I'm not in the US, so I don't have a good idea of what's going on there. But wasn't there relatively high unemployment among developers after all these Big Tech layoffs post pandemic? Shouldn't companies there have an easy time finding local talent?
Sorry for the potentially silly question. I just spent some time trying to research it and came up with nothing concrete.
But at the same time there's an ongoing infighting among Trump supporters because tech elites came up as pro - skilled immigration where the MAGA camp turned against them. The tech elites claim that there's a talent shortage. Here's a short rundown that Elon Musk agrees with: https://x.com/AutismCapital/status/1872408010653589799
Have you maybe confused the time periods in the different discussions? I think the AI making jobs obsolete part is in the next few years, whereas the talent shortage issue is right now - although as usual, it's a wage issue, not a talent issue. Pay enough and the right people will turn up.
Who knows about the future, right? I'm just trying to read the expectations of the people who have control over both the AI, Capital and Politics and they don't strike me as optimistic about AI actually doing much in near future.
And that might be a FOMO or they can simply exit with profit as long as they can flame up the hype. An of course, they may be hoping to have it in long term.
They are not replacing their workers despite claiming that AI is currently as good as a PHD and they certainly don't go to AI medical doctors despite claiming that their tool is better than most doctors.
Every software firm, notable and small, has had layoffs over the past two years, but somehow there's still a "STEM shortage" and companies are "starving for talent" or some such nonsense?
The reliance on large datasets for training AI models introduces biases present in the data, which can perpetuate or even exacerbate societal inequalities. It's essential to approach AI development with caution, ensuring robust ethical guidelines and comprehensive testing are in place before integrating AI into sensitive areas.
As we continue to innovate, a focus on explainability, fairness, and accountability in AI systems will be paramount to harnessing their potential without compromising societal values.
As a neuroscientist, my biggest disagreement with the piece is the author’s argument for compositionality over emergence. The former makes me think of Prolog and lisp, while the later is a much better description for a brain. I think ermergence is a much more promising direction for AGI than compositionality.
100% agree. When we explicitly segment and compose AI components, we are removing the ability for them to learn their own pathways between the components. We've been proven time and time again the bitter lesson[1]: that throwing a ton of data and compute at a model yields better results than what we could come up with.
That said, we can still isolate and modify parts of a network, and combine models trained for different tasks. But you need to break things down into components after the fact, instead of beforehand, in order to get the benefits of learning via scale of data + compute.
Author here. So what! I am not talking about promising directions for AGI, I am talking about having computer systems that we can have confidence in. Sure, AGI if it ever happens will look more like emergence than compositionality, and I'm sure it won't feel a need to explain to us fallible humans why its decisions are correct. In the meantime, I'd like computer systems to be manageable, reliable, transparent, and accountable.
As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.
The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.
The largest AI systems in use today have hundreds of billions of parameters. Nearly all parameters are part of a weight matrix, each parameter quantifying the strength of the connection from an artificial input neuron to an artificial output neuron. The human brain has more than a hundred trillion synapses, each connecting an organic input neuron to an organic output neuron, but the comparison is not apples-to-apples, because each synapse is much more complex than a single parameter in a weight matrix.[a]
Today's largest AI systems have about the same number of neuron-neuron connections as the brain of a brown rat.[a] Judging these AI systems based on their current capabilities is like judging organic brains based on the capabilities of brown rat brains.
What we can say with certainty is that today's AI systems cannot be trusted to be reliable. That's true for highly trained brown rats too.
We don’t know. We didn’t predict that the rat brain would get us here. So we also can’t be confident in our prediction that scaling it won’t solve hallucination problems.
Humankind has developed all sorts of systems and processes to cope with the unpredictability of human beings: legal systems, organizational structures, separate branches of government, courts of law, police and military forces, organized markets, double-entry bookkeeping, auditing, security systems, anti-malware software, etc.
While individual human beings do trust some of the other human beings they know, in the aggregate society doesn't seem to trust human beings to behave reliably.
It's possible, though I don't know for sure, that we're going to need systems and processes to cope with the unpredictability of AI systems.
Human performance, broadly speaking, is the benchmark being targeted by those training AI models. Humans are part of the conversation since that's the only kind of intelligence these folks can conceive of.
> Human brains are unpredictable. Look around you.
As it was mentioned by others, we've had thousands of years to better understand how humans can fail. LLMs are black boxes and it never ceases to amaze me how they can fail in such unpredictable ways. Take the following for examples.
You seem to believe that humans, on their own, are not stochastic and unpredictable. I contend that if this is your belief then you couldn't be more wrong.
Humans are EXTREMELY unpredictable. Humans only become slightly more predictable and producers of slightly more quality outputs with insane levels of bureaucracy and layers upon layers upon layers of humans to smooth it out.
To boot, the production of this mediocre code is very very very slow compared to LLMs. LLMs also have no feelings, egos, and are literally tunable and directible to produce better outcomes without hurting people in the process (again, something that is very difficult to avoid without the inclusion of, yep, more humans more layers, more protocol etc.)
Even with all of this mass of human grist, in my opinion, the output of purely human intellects is, on average, very bad. Very bad in terms of quality of output and very bad in terms of outcomes for the humans involved in this machine.
If brown-rats-as-a-service is as useful as it is already, then I'm excited by what the future holds.
I think to make it to the next step, AI will have to have some way of performing rigorous logic integrated on a low level.
Maybe scaling that brown-rat brain will let it emulate an internal logical black box - much like the old adage about a sufficiently large C codebase containing an imperfect Lisp implementation - but I think things will get really cool we figure out how to wire together something like Wolfram Alpha, a programming language, some databases with lots of actual facts (as opposed to encoded/learned ones), and ChatGPT.
ChatGPT can already run code, which allows it to overcome some limitations of tokenization (eg counting the letters in strawberry, sorting words by their second letter). Doesn't seem like adding a Prolog interpreter would be all that hard.
ChatGPT does already have access to Bing (would that count as your facts database?) and Jupyter (which is sort of a Wolphram clone except with Python?).
It still won't magically use them 100% correctly, but with a bit of smarts you can go a long way!
Jupyter is completely different from Wolfram software. It's just an interface to edit and run code (Julia, Python and R) and write/render text or images commenting the code. Which isn't to say that Jupyter isn't a great thing but I don't see how a Chatbot would produce better answers by having access to it in addition to "just Python".
Meanwhile, Wolfram software has built-in methods to solve a lot of different math problems for which in Python you would either need large (and sometimes quirky) libraries, if those libraries even exist.
Except a typical Jupyter environment -especially the one provided to ChatGPT- includes a lot of libraries; including numpy, scipy, pandas and plotly, which -while perhaps not quite as polished as wolphram (arguments can be made), can still rival it qua flexibility and functionality.
That and you need to actually expose python to GPT somehow, and Jupyter is not the worst way I suppose.
* The fact that Jupyter holds on to state means GPT doesn't need to write code from scratch for every step of the process.
* GPT can easily read back through the workbook to review errors or output from computations. GPT actually tries to correct errors even. Especially if it knows how to identify them.
To be sure, this is not magic. Consider it more like a tool with limited intelligence; but which can be controlled using natural language.
(Meanwhile, Anthropic allows Claude to run js with react, which is nice but seems less flexible in practice. I'm not sure Claude reads back.)
It is probably a both question. If 100x is the goal, they’ll have to double up the efficiency 7 times, which seems basically plausible given how early-days it still is (I mean they have been training on GPUs this whole time, not ASICs… bitcoins are more developed and they are a dumb scam machine). Probably some of the doubling will be software, some will be hardware.
I'm pretty skeptical of the scaling hypothesis, but I also think there is a huge amount of efficiency improvement runway left to go.
I think it's more likely that the return to further scaling will become net negative at some point, and then the efficiency gains will no longer be focused on doing more with more but rather doing the same amount with less.
But it's definitely an unknown at this point, from my perspective. I may be very wrong about that.
Honestly I think the opposite. All these giant tech companies can afford to burn money with ever bigger models and ever more compute and I think that is actually getting in their way.
I wager that some scrappy resource constrained startup or research institute will find a way to produce results that are similar to those generated by these ever massive LLM projects only at a fraction of the cost. And I think they’ll do that by pruning the shit out of the model. You don’t need to waste model space on ancient Roman history or the entire canon for the marvel cinematic universe on a model designed to refactor code. You need a model that is fluent in English and “code”.
I think the future will be tightly focused models that can run on inexpensive hardware. And unlike today where only the richest companies on the planet can afford training, anybody with enough inclination will be able to train them. (And you can go on a huge tangent why such a thing is absolutely crucial to a free society)
I dunno. My point is, there is little incentive for these huge companies to “think small”. They have virtually unlimited budgets and so all operate under the idea that more is better. That isn’t gonna be “the answer”… they are all gonna get instantly blindsided by some group who does more with significantly less. These small scrappy models and the institutes and companies behind them will eventually replace the old guard. It’s a tale as old as time.
Deepseek just released their frontier model that they trained on 2k GPUs for <$6M. Way cheaper than a lot of the big labs. If the big labs can replicate some of their optimisations we might see some big gains. And I would hope more small labs could then even further shrink the footprint and costs
I don’t think this stuff will be truly revolutionary until I can train it at home or perhaps as a group (SETI at home anybody?)
Six million is a start but this tech won’t truly be democratized until it costs $1000.
Obviously I’m being a little cheeky but my real point is… the idea that this technology is in the control of massive technology companies is dystopian as fuck. Where is the RMS of the LLM space? Who is shouting from every rooftop how dangerous it is to grant so much power and control over information to a handful of massive tech companies, all whom have long histories of caving into various government demands. It’s scary as fuck.
> It’s not at all, energy is a hard constraint to capability.
We can put a lot more power flux through an AI than a human body can live through; both because computers can run hot enough to cook us, and because they can be physically distributed in ways that we can't survive.
That doesn't mean there's no constraint, it's just that the extent to which there is a constraint, the constraint is way, way above what humans can consume directly.
Also, electricity is much cheaper than humans. To give a worked example, consider that the UN poverty threshold* is about US$2.15/day in 2022 money, or just under 9¢/hour. My first Google search result for "average cost of electricity in the usa" says "16.54 cents per kWh", which means the UN poverty threshold human lives on a price equivalent ~= just under 542 watts of average American electricity.
The actual power consumption of a human is 2000-2500 kcal/day ~= 96.85-121.1 watts ~= about a fifth of that. In certain narrow domains, AI already makes human labour uneconomic… though fortunately for the ongoing payment of bills, it's currently only that combination of good-and-cheap in narrow domains, not generally.
* I use this standard so nobody suggests outsourcing somewhere cheaper.
The average brown rat may use only 60 kcal per day, but the maximum firing rate of biological neurons is about 100-1000 Hz rather than the A100 clock speed of about 1.5 GHz*, so the silicon gets through the same data set something like 1.5e6-1.5e7 times faster than a rat could.
Scaling up to account for the speed difference, the rat starts looking comparable to a 9e7 - 9e8 kcal/day, or 4.4 to 44 megawatts, computer.
* and the transistors within the A100 are themselves much faster, because clock speed is ~ how long it takes for all chained transistors to flip in the most complex single-clock-cycle operation
Also I'm not totally confident about my comparison because I don't know how wide the data path is, how many different simultaneous inputs a rat or a transformer learns from
That's a stupid analogy because you're comparing a brainprocess to a full animal.
Only a small part of that 60kcal is used for learning, and for that same 60 kcal you get an actual physical being that is able to procreate, eat, do things and fend for and maintain itself.
Also you cannot compare neuron firing rates with clockspeed. Afaik each neuron in a ml-model can have code that takes several clock cycles to complete.
Also an neuron in ml is just a weighted value, a biological neuron does much more than that. For example neurons communicate using neuro transmitters as well as using voltage potentials. The actual date rate of biological neurons is therfore much higher and complex.
Basically your analogy is false because your napkin-math basically forgets that the rat is an actual biological rat and not something as neatly defined as a computer chip
> Also an neuron in ml is just a weighted value, a biological neuron does much more than that. For example neurons communicate using neuro transmitters as well as using voltage potentials. The actual date rate of biological neurons is therfore much higher and complex.
The conclusion does not follow from the premise. The observed maximum rate of the inter-neuron communication is important, the mechanism is not.
> Also you cannot compare neuron firing rates with clockspeed. Afaik each neuron in a ml-model can have code that takes several clock cycles to complete.
Depends how you're doing it.
Jupyter notebook? Python in general? Sure.
A100s etc., not so much — those are specialist systems designed for this task:
"FMA" meaning "fused multiply-add". It's the unit that matters for synapse-equivalents.
(Even that doesn't mean they're perfect fits: IMO a "perfect fit" would likely be using transistors as analog rather than digital elements, and then you get to run them at the native transistor speed of ~100 GHz or so and don't worry too much about how many bits you need to represent the now-analog weights and biases, but that's one of those things which is easy to say from a comfortable armchair and very hard to turn into silicon).
> Basically your analogy is false because your napkin-math basically forgets that the rat is an actual biological rat and not something as neatly defined as a computer chip
Any of those biological functions that don't correspond to intelligence, make the comparison more extreme in favour of the computer.
This is, after all, a question of their mere intelligence, not how well LLMs (or indeed any AI) do or don't function as von Neumann replicators, which is where things like "procreate, eat, do things and fend for and maintain itself" would actually matter.
You're so deep into this nonsense I don't think anything I could possibly say to you would change your mind, so I'll try something different.
Have you thought about stepping back from all of this for a few days and notice that you are wasting your time with these arguments? It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change.
NNs as of right now are the equivalent of a brain scan. You can simulate how that brain scan would answer a question, but the moment you close the Q and A session, you will have to start from scratch. Making higher resolution brain scans may help you get more precise answers to more questions, but it will never change the questions that it can answer after you have made the brain scan.
> Have you thought about stepping back from all of this for a few days and notice that you are wasting your time with these arguments?
Num fecisti?
> It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change.
That's a deliberate choice, not a fundamental requirement.
Models get frozen in order to become a product someone can put a version number on and ship, not because they must be, as demonstrated both by fine-tuning and by the initial training process — both of which update the weights.
> NNs as of right now are the equivalent of a brain scan.
First: see above.
Second: even if it were, so what? Look at the context I'm replying to, this is about energy efficiency — and applies just fine even when calculated for training the whole thing from scratch.
To put it another way: how long would it take a mouse to read 13 trillion tokens?
The energy cost of silicon vs. biology is lower than people realise, because people read the power consumption without considering that the speed of silicon is much higher: at the lowest level, the speed of silicon computation literally — not metaphorically, really literally — outpaces biological computation by the same magnitude to which jogging outpaces continental drift.
> "FMA" meaning "fused multiply-add". It's the unit that matters for synapse-equivalents.
Neurons do so much more than a single math operation. A single cell can act as an intelligent little animal on its own, they are nothing like a neural network "neuron".
And note that all neurons act in parallel, so they are billions times more parallel than GPU's even if the operations would be the same.
Your numbers are meaningless because neuromorphic computing hardware exists in the context of often forgotten spiking neural networks, which actually try to mimic how biological neurons operate through voltage integration and programmable synapses and they tend to be significantly more efficient.
SpiNNaker needs 100kWh to simulate one billion neurons. So the rat wins in terms of energy efficiency.
SpiNNaker is an academic experiment to see if taking more cues from biology would make the models better — it turned out the answer was "nobody in industry cares" because scaling the much simpler models to bigger neural nets and feeding them more data was good enough all by itself so far.
> and they tend to be significantly more efficient
Surely you noticed that this claim is false, just from your own next line saying it needing 100 kW (not "kWh" but I assume that's auto-corrupt) for a mere billion?
Even accounting for how neuron != synapse — one weight is closer to a single synapse; a brown rat has 200e6 neurons and about 450e9 synapses — the stated 100 kW for SpiNNaker is enough to easily drive simpler perceptron-type models of that scale, much faster than "real time".
An airplane is far less energy-efficient than a bird to fly, to such an extent that it is almost pathetic. Nevertheless, the airplane is a highly useful technology, despite its dismal energy efficiency. On the other hand, it would be very difficult to scale a bird-like device to transport heavy weights or hundreds of people.
I think current LLMs may scale the same way and become very powerful, even if not as energy-efficient as an animal's brain.
In practice, we humans, when we have a technology that is good enough to be generally useful, tend to adopt it as it is. We scale it to fit our needs and perfect it while retaining the original architecture.
This is what happened with cars. Once we had the thermal engine, a battery capable of starting the engine, and tires, the whole industry called it "done" and simply kept this technology despite its shortcomings. The industry invested heavily to scale and mass-produce things that work and people want.
Rats are pretty clever, and they (presumably, at least) have a lot of neurons spending their time computing things like… where to find food, how frightened of this giant reality warping creature in a lab coat should I be, that sort of thing. I don’t think it is obvious that one brown-rat-power isn’t useful.
I mean we have dogs. We really like them. For ages, they did lots of useful work for us. They aren’t that much smarter than rats, right? They are better aligned and have a more useful shape. But it isn’t obvious (to me at least) that the rats’ problem is insufficient brainpower.
Dogs, if I recall correctly, have evolved alongside us and have specific adaptations to better bond with us. They have eyebrow muscles that wolves don't, and I think dogs have brain adaptations too.
Depends on how you define smart. Dogs definitely have larger brains. But then humans have even larger brains. If dogs aren’t smarter than rats then the size of brain isn’t proportional to intelligence.
I think the comparison to brown rat brains is a huge mistake. It seems pretty apparent (at least from my personal usage of LLMs in different contexts) that modern AI is much smarter than a brown rat at some things (I don't think brown rats can pass the bar exam), but in other cases it becomes apparent that it isn't "intelligent" at all in the sense that it becomes clear that it's just regurgitating training data, albeit in a highly variable manner.
I think LLMs and modern AI are incredibly amazing and useful tools, but even with the top SOA models today it becomes clearer to me the more I use them that they are fundamentally lacking crucial components of what average people consider "intelligence". I'm using quotes deliberately because the debate about "what is intelligence" feels like it can go in circles endlessly - I'd just say that the core concept of what we consider understanding, especially as it applies to creating and exploring novel concepts that aren't just a mashup of previous training examples, appears to be sorely missing from LLMs.
Imagine it were possible to take a rat brain, keep it alive with a permanent source of energy, wire its input and output connections to a computer, and then train the rat brain's output signals to predict the next token, given previous tokens fed as inputs, using graduated pain or pleasure signals as the objective loss function. All the neuron-neuron connections in that rain brain would eventually serve one, and only one, goal: predicting an accurate probability distribution over the next possible token, given previous tokens. The number of neuron-neuron connections in this "rat-brain-powered LLM" would be comparable to that of today's state-of-the-art LLMs.
This is less far-fetched than it sounds. Search for "organic deep neural networks" online.
Networks of rat neurons have in fact been trained to fly planes, in simulators, among other things.
> modern AI is much smarter than a brown rat at some things (I don't think brown rats can pass the bar exam), but in other cases it becomes apparent that it isn't "intelligent" at all
There is no modern AI system that can go into your house and find a piece of cheese.
The whole notion that modern AI is somehow "intelligent", yet can't tell me where the dishwasher is in my house is hilarious. My 3 year old son can tell me where the dishwasher is. A well trained dog could do so.
It's the result of a nerdy definition of "intelligence" which excludes anything to do with common sense, street smarts, emotional intelligence, or creativity (last one might be debatable but I've found it extremely difficult to prompt AI to write amazingly unique and creative stories reliably)
The AI systems need bodies to actually learn these things.
Where do you think common sense, emotional intelligence, creativity, etc. come from? The spirit? Some magic brain juice? No, it comes from neurons, synapses, signals, chemicals, etc.
It doesn’t. Actually, quite a few of the early stages of evolution wouldn’t have any analogue to “care,” right? It just happened in this one environment, the most successful self-reproducing processes happened to be get more complex over time and eventually hit the point where they could do, and then even later define, things like “care.”
Find a piece of cheese pretty much anywhere in my home?
Or if we're comparing to a three year old, also find the dishwasher?
Closest I'm aware of is something by Boston Dynamics or Tesla, but neither would be as simple as asking it- wheres the dishwasher in my home?
And then if we compare it to a ten year old, find the woodstove in my home, tell me the temperature, and adjust the air intake appropriately.
And so on.
I'm not saying it's impossible. I'm saying there's no AI system that has this physical intelligence yet, because the robot technology isn't well developed/integrated yet.
For AI to be something more than a nerd it needs a body and I'm aware there are people working on it. Ironically, not the people claiming to be in search of AGI.
If you upload pictures of every room in your house to an LLM it can definitely tell you where the dishwasher is. If your argument is just that they cant walk around your house so they cant be intelligent I think thats pretty clearly wrong.
A trained image recognition model could probably recognize a dishwasher from an image.
But that won't be the same model that writes bad poetry or tries to autocomplete your next line of code. Or control the legs of a robot to move towards the dishwasher while holding a dirty plate. And each has a fair bit of manual tuning and preprocessing based on its function which may simply not be applicable to other areas even with scale. The best performing models aren't just taking in unstructured untyped data.
Even the most flexible models are only tackling a small slice of what "intelligence" is.
Technically yes they can run functions. There were experiments of Claude used to run a robot around a house. So technically, we are not far at all and current models may even be able to do it.
While interesting, this is a separate thought experiment with its own quirks. Sort of a strawman, since my argument is formulated differently and simply argues that AIs need to be more than brains in jars for them to be considered generally intelligent.
And that the only reason we think AIs can just be brains in jars is because many of the people developing them consider themselves as simply brains in jars.
Not really. The point of it is considering whether physical experience creates knowledge that is impossible to get otherwise. Thats the argument you are making no? If Mary learns nothing new when seeing red for the first time an AI would also learn nothing new when seeing red for the first time.
> Do they know what a hot shower feels like?
They can describe it. But do they actually know? Have they experienced it
Mary in that thought experiment is not an LLM that has learned via text. She's acquired "all the physical information there is to obtain about what goes on when we see ripe tomatoes". This does not actually describe modern LLMs. It actually better describes a robot that has transcribed the location, temperature, and velocity of water drops from a hot shower to its memory. Again, this thought experiment has its own quirks.
Also, it is an argument against physicalism, which I have no interest in debating. While it's tangentially related, my point is not for/against physicalism.
My argument is about modern AI and it's ability to learn. If we put touch sensors, eyes, nose, a mechanism to collect physical data (legs) and even sex organs on an AI system, then it is more generally intelligent than before. It will have learned in a better fashion what a hot shower feels like and will be smarter for it.
> While it's tangentially related, my point is not for/against physicalism.
I really disagree. Your entire point is about physicalism. If physicalism is true than an AI does not necessarily learn in a better fashion what a hot shower feels like by being embodied. In a physicalist world it is conceivable to experience that synthetically.
So are you saying people who have CIPA are less intelligent for never having experienced a hot shower? By that same logic, does its ability to experience more colors increase the intelligence of a mantis shrimp?
Perhaps your own internal definition of intelligence simply deviates significantly from the common, "median" definition.
It's the totality of experiences that make an individual. Most humans that I'm aware of have a greater totality of experiences that make them far smarter than any modern AI system.
Greater totality of experiences than having read the whole internet? Obviously they are very different kind of experiences, but a greater totality? I'm not so sure.
Here is what we know: The Pile web scrape is 800GB. 20 years of human experience at 1kB/sec is 600GB. Maybe 1kB/sec is bad estimate. Maybe sensory input is more valuable than written text. You can convince me. But next challenge, some 10^15 seconds of currently existing youtube video, that's 2 million years of audiovisual experience, or 10^9GB at the same 1kB/sec.
I feel the jump from "reading the internet" to experience has a gap in reasoning. I'm not experienced in philosophy or* logic enough(no matter how much I read, heh) to articulate it, but seems to get at the person's idea of lacking street smarts, common sense. An adult with basic common sense could probably filter out false information quicker since I can get Claude to tell me false info regularly(I still like em, pretty entertaining) which has not only factual but contradictory flaws any person wouldn't make. Like recently I had two pieces of data, then when comparing them it was blatently incorrectly(they were very close, but claude said one was 8x bigger for... idk why.)
Another commenter also mentioned sensory input when talking about the brown rat. As someone who is constantly fascinated at the brains ability to reason/process stuff before I'm even conscious of it, I feel this Stat is Underrated. I'm taking in and monitoring like 15 sensations of touch at all time. Something entering my visual field coming towards me can be deflected in half a second all while still understanding the rest of my surroundings, and where it might be safe to deflect an object. The brain is constantly calculating depth perception and stereo location on every image and sound we hear - also with the ability to screen out the junk or alter our perception accurately(knowing the correct color of items regardless of diff in color temp).
I do concede that's a heck of a lot of video data. It does have similar issues to what I said(lacks touch, often no real stereo location, good greenscreen might convince an AI of something a person intuitively knows is impossible) but the scale alone certainly adds a lot. That could potentially make up for what I see as a hugely overlooked thing as far as stimulus. I am monitoring and adjusting like, hundreds of parameters a second subconsciously. Like everything in my visual field. I don't think it can be quantified accurately how many things we consciously and subconsciously process, but I have the feeling it's a staggering amount.
The people that have have barely used the internet are often far better conversation (and often more useful in the economy) than people who are addicted to the internet.
There isn't a serious proof that 1+1=2, because it's near enough axiomatic. In the last 150 years or so, we've been trying to find very general logical systems in which we can encode "1", "2" and "+" and for which 1+1=2 is a theorem, and the derivations are sometimes non-trivial, but they are ultimately mere sanity checks that the logical system can capture basic arithmetic.
If this is new, then you're one of today's luck 10,000![2] Serious logical foundations take a lot of time and exposition to start from fundamentals. Dismissing them as non-serious because GP's argument failed to consider them is misguided, IMHO.
Am I the only one that always felt like that xkcd post came from a place of insane intellectual elitism?
I teach multiple things online and in person... language like that seems like a great to lose a student. I'd quit as a student, it's so condescending sounding. It's only lucky because you get to flex ur knowledge!(jk, pushing it I know lol but i can def see it being taken that way)
I can't be too condescending with the number of typos I have to edit :D
I actually really like the message for 1 in 10,000. As a social outsider for much of my life, it helped me to learn that the way people dismissed my questions about common (to them) topics was more about their empathy and less about me.
But, these sorts of things are difficult to communicate via text media, so we thus persist.
Yeah I guess I've had only a few people be the other person that treated me right as the 1 - I feel ya on being an outsider having things dismissed. Does make sense. Another person gave me a good alternate view as well.
On a side note my couple of times I thought I was treating someone to some great knowledge they should already know I'm pretty sure I came across as condescending. Not bc they didn't know it - i always aim to be super polite - just being young, stupid, and bad at communicating, heh.
The key thing to focus on with XKCD 1053, is that the alternative before that comic was to make fun of the person who didn't know there's a proof for, eg 1 + 1 = 2. "Oh, you didn't know there's a proof for that? are you an idiot? who doesn't know the proof for 1 + 1 = 2 by Alfred North Whitehead and Bertrand Russell?", to which I think you could agree would put possible students off more by that than being told they're in luck today.
Ah okay that's a good read. I'm just always on edge about my language and sometimes view the worst possible interpretation rather than what most would read. I'm not a negative person... just goes back to some "protecting myself" instincts I unfortunately had to develop. Thanks for that view.
Yes, as I said: systems such as Russell's encoded "1", "2" and "+" in such a way that the theorem "1 + 1 = 2" is non-trivial to prove. This doesn't say anything about the difficulty of proving that 1 + 1 = 2, but merely the difficulty of proving it in a particular logical encoding. Poincare ridiculed the Principia on this point almost immediately.
And had Russell failed to prove that 1 + 1 = 2 in his system, it would not have cast one jot of doubt on the fact that 1 + 1 = 2. It would only have pointed to the inadequacy of the Principia.
The subject has been debated ad nauseam by everyone like Descartes, Hume, Kant, and so on. If there were no one around to state 1 + 1 = 2, there would be no such statement. Hence, it does rely on at least 1 person's experience. Yours in fact, since everyone else could be an illusion.
Could it tell the difference between a dishwasher and a picture of a dishwasher on a wall? Or one painted onto a wall? Or a toy dishwasher?
There is an essential idea of what makes something a dishwasher that LLM's will never be able to grasp no matter how many models you throw at them. They would have to fundamentally understand that what they are "seeing" is an electronic appliance connected to the plumbing that washes dishes. The sound of a running dishwasher, the heat you feel when you open one, and the wet, clean dishes is also part of that understanding.
If I am limited to looking at pictures, then I am at the same disadvantage as the LLM, sure. The point is that people can experience and understand objects from a multitude of perspectives, both with our senses and the mental models we utilize to understand the object. Can LLMs do the same?
That's not a disadvantage of LLM. You can start sending images from a camera moving around and you'll get many views as well. The capabilities here are the same as the eye-brain system - it can't move independently either.
You really need to define what you mean by generally intelligent in that case. Otherwise, if you require free movement for generally intelligent organisms, you may be making interesting claims about bedridden people.
That really makes no sense.. would you say someone who is disabled bellow the neck is not intellegent / has no common sense, street smaets, creativity, etc...?
Or would you say that you cannot judge the intellegence of someone by reading their books / exchanging emails with them?
IMO it is sad that the sort of… anti-establishment side of tech has suddenly become very worried about copyright. Bits inherently can be copied for free (or at least very cheap), copyright is a way to induce scarcity for the market to exploit where there isn’t any on a technical level.
Currently the AI stuff kind of sucks because you have to be a giant corp to train a model. But maybe in a decade, users will be able to train their own models or at least fine-tune on basic cellphone and laptop (not dgpu) chips.
The copyright question is inherently tied to the requirement to earn money from your labor in this economy. I think the anti-establishment folks are not so rabid that they can't recognize real material conditions.
I think that would be a more valid argument if they ever cared about automating away jobs before. As it stands, anyone who was standing in the way of the glorious march of automation towards a post-scarcity future was called a luddite - right up until that automation started threatening their (material) class.
The solution is not, and never has been, to shack up with the capital-c Capitalists in defense of copyright. It's to push for a system where having your "work" automated away is a relief, not a death sentence.
There's both "is" and "ought" components to this conversation and we would do well to disambiguate them.
I would engage with those people you're stereotyping rather than gossiping in a comments section, I suspect you will find their ideologies quite consistent once you tease out the details.
> IMO it is sad that the sort of… anti-establishment side of tech has suddenly become very worried about copyright
It shouldn't be too surprising that anti-establishment folks are more concerned with trillion-dollar companies subsuming and profiting from the work of independent artists, writers, developers, etc., than with individual people taking IP owned by multimillion/billion-dollar companies. Especially when many of the companies in the latter group are infamous for passing only a tiny portion of the money charged onto the people doing the actual creative work.
Tech still acts like it's the scrappy underdog, the computer in the broom cupboard where "the net" is a third space separate from reality, nerds and punks writing 16-bit games.
That ceased to be materially true around twenty years ago now. Once Facebook and smart phones arrived, computing touched every aspect of peoples' lives. When tech is all-pervasive, the internal logic and culture of tech isn't sufficient to describe or understand what matters.
IMO this is looking at it through a lens which considers “tech” a single group. Which is a way of looking at is, maybe even the best way. But an alternative could be: in the battle between scrappy underdog and centralized sellout tech, the sellouts are winning.
Copyright is the right to get a return from creative work. The physical ease - or otherwise - of copying is absolutely irrelevant to this. So is scarcity.
It's also orthogonal to the current corporate dystopia which is using monopoly power to enclose the value of individual work from the other end - precisely by inserting itself into the process of physical distribution.
None of this matters if you have a true abundance economy, but we don't. Pretending we do for purely selfish reasons - "I want this, and I don't see why I should pay the creator for it" - is no different to all the other ways that employers stiff their employees.
I don't mean it's analogous, I mean it's exactly the same entitled mindset which is having such a catastrophic effect on everything at the moment.
> IMO it is sad that the sort of… anti-establishment side of tech has suddenly become very worried about copyright.
Remember Napster? Like how rebellious was that shit? Those times are what a true social upsetting tech looks like.
You cannot even import a video into OpenAI’s Sora without agreeing to a four (five?) checkbox terms & conditions screen. These LLM’s come out of the box neutered by corporate lawyers and various other safety weenies.
This shit isn’t real until there are mainsteam media articles expressing outrage because some “dangerous group of dark web hackers finished training a model at home that very high school student on the planet can use to cheat on their homework” or something like that. Basically it ain’t real until it actually challenges The Man. That isn’t happening until this tech is able to be trained and inferenced from home computers.
Yeah, or if it becomes possible to train on a peer-to-peer network somehow. (I’m sure there’s researching going on in that direction). Hopefully that sort of thing comes out of the mix.
Unfortunately this is not the way it's developing. It's more like: are you a normal person without deep pockets? Download a movie with Bittorrent and get a steep fine. Are you a company with hundreds of millions? Download half the copyrighted material on the internet, it's fine.
We are increasingly shifting to a society where the rules only don't apply when you have capital. To some extend, this has always been true, but the scale is changing.
It does use knowledge from creators. But using knowledge from others is a big part of modern society, and the legal ways of protecting knowledge from commercial reuse are actually pretty limited.
Is the result of an llm an accurate copy or more of an inspiration? What is the standard we use on humans?
Can we code that determination into a system that when a piece of content is close enough to be a copyrighted work, prevents the llm from generating it?
This is an excellent analogy. Aside from “they’re both networks” (which is almost a truism), there’s really nothing in common between an artificial neural network and a brain.
Neurons also adjust the signal strength based on previous stimuli, which in effect makes the future response weighted. So it is not far off—albeit a gross simplification—to call the brain a weight matrix.
As I learned it, artificial neural networks were modeled after a simple model for the brain. The early (successful) models were almost all reinforcement models, which is also one of the most successful model for animal (including human) learning.
Is your point that the capabilities of these models have grown such that 'merely' calling it a neural network doesn't fit the capabilities?
Or is your point that these models are called neural networks even though biological neural networks are much more complex and so we should use a different term to differentiate the simulated from the biological ?
The OP is comparing the "neuron count" of an LLM to the neuron count of animals and humans. This comparison is clearly flawed. Even you step back and say "well, the units might not be the same but LLMs are getting more complex so pretty soon they'll be like animals". Yes, LLMs are complex and have gained more behaviors through size and increased training regimes but if you realize these structure aren't like brains, there's no argument here that they will soon reach to qualities of brains.
Actually, I'm comparing the "neuron-neuron connection count," while admitting that the comparison is not apples-to-apples.
This kind of comparison isn't a new idea. I think Hans Moravec[a] was the first to start making these kinds of machine-to-organic-brain comparisons, back in the 1990's, using "millions of instructions per second" (MIPS) and "megabytes of storage" as his units.
You can read Moravec's reasoning and predictions here:
I think he was approaching the concept from the direction of "how many mips and megabytes do we need to create human level intelligence".
That's a different take than "human level is this many mips and megabytes", i.e. his claims are about artificial intelligence, not about biological intelligence.
The machine learning seems to be modeled after the action potential part of neural communication. But biological neurons can communicate also in different ways, i.e. neuro transmitters. Afaik this isn't modeled in the current ml-models at all (neither do we have a good idea how/why that stuff works). So ultimately it's pretty likely that a ml with a billion parameters does not perform the same as an organic brain with a billion synapses
I never claimed the machines would achieve "human level," however you define it. What I actually wrote at the root of this thread is that we have no way of knowing in advance what the future capabilities of these AI systems might be as we scale them up.
Your "not apples to apples" concession isn't adequate. You are essentially still saying that a machine running a neural network is compare to the brain of an animal or a person - just maybe different units of measurement. But they're not. It's a matter of dramatically different computing systems, systems that operate very differently (well, don't know exactly how animal brains work but we know enough to know they don't work like GPUs).
Your Moravec article is only looking at what's necessary for computers to have the processing power of animal brains. But you've been up and down this thread arguing that equivalent processing power could be sufficient for a computer to achieve the intelligence of an animal. Necessary vs sufficient is big distinction.
Afaict OP's not comparing neuron count, but neuron-to-neuron connections, aka synapses. And considering each synapse (weighted input) to a neuron performs computation, I'd say it's possible it captures a meaningful property of a neural network.
excellent analogy. piggybacking on this: a lot of believers (as they are like religious fanatics) claim that more data and hardware will eventually make LLMs intelligent, as if it's even the neuron count matters. There is no other animal close to humans in intelligence, and we don't know why. Somehow though a random hallucinating LLMs + shit loads of electricity would figure it out. This is close to pure alchemy.
I don’t disagree with your main point but I want to push back on the notion that “there is no other animal close to humans in intelligence”. This is only true in the sense that we humans define intelligence in human terms. Intelligence is a very fraught and problematic concept both in philosophy, but especially in the sciences (particularly psychology).
If we were dogs surely we would say that humans were quite skillful, impressively so even, in pattern matching, abstract thought, language, etc. but are hopelessly dumb at predicting past presence via smell, a crow would similarly judge us on our inability to orient our selves, and probably wouldn’t understand our language and thus completely miss our language abilities. We do the same when we judge the intelligence of non-human animals or systems.
So the reason for why no other animal is close to us in intelligence is very simple actually, it is because of the way we define intelligence.
Interesting point. Though I would say that you didn't disprove my point. Humans have a level of generalized intelligence that's not matched. We might be terrible at certain sensory tasks (smell), maybe all, compared to another animal. But the capability of thought, at the level of humans, is unmatched.
Just to clarify one point: I don't think intelligence is exclusive to humans. I only think that there's a big discrepency that cannot be explained with neuron counts oor the volume of the brain etc. which makes the argument of more hardware and more data will create AGI.
Like I said the term is very fraught both in philosophy and the sciences. Many volumes have been written about this in philosophy (IMO the only correct outlet for the discussion) and there is no consensus on what to do with it.
My main problem with the notion of generalized intelligence (in philosophy; I have tons of problems with it in psychology) is it turns out to be rather arbitrary what counts towards general intelligence. Abstract thought and project planning seems to an essential component, but we have no idea how abstract thought and project planning goes on in non-human systems. In nature we have to look at the results and infer what the goals were with the behavior. No doubt we are missing a ton of intelligent behavior among several animals—maybe even pants and fungi—just because we don’t fully understand the goals of the organism.
That said though, I think our understanding of the natural world is pretty unparalleled by other species, and using this knowledge we have produced some very impressive intelligent behavior which no other species is capable of. But I have a hard time believing that humans are uniquely capable of this understanding nor of applying this understanding. For examples, elephants have shown they are capable of inter-generational knowledge and culture. I don’t know if elephants had access to the same instruments as we, that they would be able to pass this knowledge down generations on build up on them.
In a fictional scenario each dog might have enough brain power to simulate the entire universe including eight billion human brains and humans would still consider themselves more intelligent.
I don't think so: it seems reasonable to assume that biological neurons are strictly more powerful than "neural network" weights, so the fact that a human brain has 3 orders of magnitude more biological neurons than language models have weights tells that we should expect, as an extreme lower bound, 3 orders of magnitude difference.
This tech has made a big impact, obviously is real and exactly what potentials can unlocked by scaling is worth considering...
... but calling vector-entries in a tensor flow process "neurons" is at best a very loose analogy while comparing LLM "neuron numbers" to animals and humans is flat-out nonsense.
yes indeed. But I see more and more people arguing against the very possibility of AGI. Some people say statistical models will always have a margin of error and as such will have some form of reliability issues: https://open.substack.com/pub/transitions/p/here-is-why-ther...
the same foundation that makes the binary model of computation so reliable is what also makes it unsuitable to solving complex problems with any level of autonomy
in order to reach autonomy and handle complexity, the computational model foundation must accept errors
> As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.
Uhh, yes we do.
I mean sure, we don't know everything, but we know one thing which is very important and which isn't under debate by anyone who knows how current AI works: current AI response quality cannot surpass the quality of its inputs (which include both training data and code assumptions).
> The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.
And it's become abundantly clear that this isn't the important difference between current AI and the human brain for two reasons: 1) there are large scale structural differences which contain implicit, inherited input data which goes beyond neuron quantity, and 2) as I said before, we cannot surpass the quality of input data, and current training data sets clearly do not contain all the input data one would need to train a human brain anyway.
It's true we don't know exactly what would happen if we scaled up a current-model AI to human brain size, but we do know that it would not produce a human brain level of intelligence. The input datasets we have simply do not contain a human level of intelligence.
... and any other answer is just special pleading towards what people want to be true. "What LLMs can't do" is increasingly "God of the gaps" -- someone states what they believe to be a fundamental limitation, and then later models show that limitation doesn't hold. Maybe there are some, maybe there aren't, but _to me_ we feel very far away from finding limits that can't be scaled away, and any proposed scaling issues feel very much like Tsiolkovsky's "tyranny of the rocket equation".
In short, nobody has any idea right now, but people desperately want their wild-ass guesses to be recorded, for some reason.
> As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.
I don't think that's totally true, and anyways it depends on what kind of scaling you are talking about.
1) As far as training set (& corresponding model + compute) scaling goes - it seems we do know the answer since there are leaks from multiple sources that training set scaling performance gains are plateauing. No doubt you can keep generating more data for specialized verticals, or keep feeding video data for domain-specific gains, but for general text-based intelligence existing training sets ("the internet", probably plus many books) must have pretty decent coverage. Compare to a human: would a college graduate reading one more set of encyclopedias make them significantly smarter or more capable ?
2) The new type of scaling is not training set scaling, but instead run-time compute scaling, as done by models such as OpenAI's GPT-o1 and o3. What is being done here is basically adding something similar to tree search on top of the model's output. Roughly: for each of top 10 predicted tokens, predict top 10 continuation tokens, then for each of those predict top 10, etc - so for a depth 3 tree we've already generated - scaled compute/cost by - 1000 tokens (for depth 4 search it'd be 10,000 x compute/cost, etc). The system then evaluates each branch of the tree according to some metric and returns the best one. OpenAI have indicated linear performance gains for exponential compute/cost increase, which you could interpret as linear performance gains for each additional step of tree depth (3 tokens vs 4 tokens, etc).
Edit: Note that the unit of depth may be (probably is) "reasoning step" rather than single token, but OpenAI have not shared any details.
Now, we don't KNOW what would happen if type 2) compute/cost scaling was done by some HUGE factor, but it's the nature of exponentials that it can't be taken too far, even assuming there is aggressive pruning of non-promising branches. Regardless of the time/cost feasibility of taking this type of scaling too far, there's the question of what the benefit would be... Basically you are just trying to squeeze the best reasoning performance you can out of the model by evaluating many different combinatorial reasoning paths ... but ultimately limited by the constituent reasoning steps that were present in the training set. How well this works for a given type of reasoning/planning problem depends on how well a solution to that problem can be decomposed into steps that the model is capable of generating. For things well represented in the training set, where there is no "impedance mismatch" between different reasoning steps (e.g. in a uniform domain like math) it may work well, but in others may well result in "reasoning hallucination" where a predicted reasoning step is illogical/invalid. My guess would be that for problems where o3 already works well, there may well be limited additional gains if you are willing to spend 10x, 100x, 1000x more for deeper search. For problems where o3 doesn't provide much/any benefit, I'd guess that deeper search typically isn't going to help.
In comparing neural networks to brains it seems like you are implying a relation between the size/complexity of a thinking machine and the reasonability of its thinking. This gives us nothing, because it disregards the fundamental difference that a neural network is a purely mathematical thing, while a brain belongs to an embodied, conscious human being.
For your implication to be plausible, you either need to deny that consciousnes plays a role in reasonability of thinking (making you a physicalist reductionist) or you need to posit that a neural network can have consciousness (some sort of mystical functionalism).
As both of these alternatives imply some heavy metaphysical assumptions and are completely unbased, I'd advise to avoid thinking of neural networks as an analogue of brains with regards to thinking and reasonability. Don't expect they will make more sense with more size. It is and will continue to be mere statistics.
I'm not implying anything or delving into metaphysical matters.
All I'm saying above is that the number of neuron-neuron connections in current AI systems is still tiny, so as of right now, we have no way of knowing in advance what the future capabilities of these AI systems will be if we are able to scale them up by 10x, 100x, 1000x, and more.
fact of the matter is that if AIs externalities were exposed - that is massive energy consumption - to end users and humanity in general, no one would use it.
I think this is wildly optimistic about how environmentally conscious customers of LLMs are. People use fossil fuels directly and through electricity consumption in a unconscionable way at a scale wildly exceeding what a ChatGPT user's energy expenditure is.
We desperately need to rapidly regulately down fossils usage and production for both electricity generation and transport. The rest of the world needs to follow the example of the EU CO2 emissions policy which guarantees it's progressing at a downwards slope independent of what the CO2 emissions are spent on.
I use it for fast documentation of unknown (to me) APIs and other pieces of software. It's saved me hours of time, where I didn't have to go through the developers site/documentation and I get quickly get example code.
Would I use the code directly in production? No. I always use it as an example and write my own code.
The elephant in the room: The user interface problem
We seem to dancing around a problem in the middle of the room like an elephant no one is acknowledging, and that is the interface to Artificial Intelligence and Generative AI is a place that requires several degrees of innovations.
I would argue that the first winning feat of innovation on interfacing with AI was the "CHAT BOX". And it works well enough for the 40% of use cases. And there is another 20% of uses that WE THE PEOPLE can use our imagination (prompt engineering) to manipulate the chat box to solve. On this topic, there was an article/opinion that said complex LLMs are unnecessary because 90% of people don't need it. Yeah. Because the chat box cannot do much more that would require heavier LLMs.
Complex AI and large data sets need nicer presentation and graphics, more actionable interfaces, and more refined activity concepts, as well as metadata that gives information on the reliability or usability of generated information.
Things like edit sections of an article, enhance articles, simplify articles, add relevant images, compress text to fit in a limited space, generate sql data from these reports, refine patterns found in a page with supplied examples, remove objects, add objects, etc.
Some innovation has to happen in MS Office interfaces. Some innovations have to happen in photoshop-like interfaces.
The author is complaining about utopian systems being incompatible with AI. I would argue AI is a utopian system being used in a dystopian world where we are lacking rich usable interfaces.
1,089 comments
[ 2.5 ms ] story [ 409 ms ] threadAn actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history.
An LLM is doing exactly nothing while waiting for the next prompt.
Is impatience a requirement for general intelligence? Why?
By no means has it been proven that llms functioning the way you describe will result in superior output.
We just keep moving the goalposts.
And the systems in place do so at scales and breadths that no human could achieve.
That doesn’t change the fact that it’s effectively triple PHD uncle Jim, as in slightly unreliable and prone to bullshitting its way through questions, despite having a breathtaking depth and breadth of knowledge.
What we are making is not software in any normal sense of the word, but rather an engine to navigate the entire pool of human knowledge, including all of the stupidity, bias, and idiosyncrasies of humanity, all rolled up into a big sticky glob.
It’s an incredibly powerful tool, but it’s a fundamentally different class of tool. We cannot expect to apply conventional software processes and paradigms to LLM based tools any more than we could apply those paradigms to politics or child rearing and expect useful results.
Tell me a problem that an LLM can solve that is not directly represented in the training set or algorithm. I would argue that 99% of what commercial LLMs gets prompted about are stuff that already existed in the training set. And they still hallucinate half lies about those. When your training data is most the internet, it is hard to find problems that you haven't encountered before
Give me 10 real world revenue-increasing or expense reducing examples that are unrelated to science, engineering or math, I will wait.
Norvig seems to be using a loose technical definition of AGI, roughly "AI with some degree of generality", which is hard to argue with, although by that measure older GOFAI systems like SOAR might also qualify.
Certainly "deep learning" in general (connectionist vs symbolic, self-learnt representations) was a step in the right direction, and LLMs a second step, but it seems we're still a half dozen MAJOR steps away from anything similar to animal intelligence, with one critical step being moving beyond full dataset pre-training to new continuous learning algorithms.
These were professional radiologists with years of experience and still came to different conclusions for fairly common conditions that we were trying to detect.
So yes, LLMs will make mistakes, but humans do too, and if these models do so less often at a much lower cost it’s hard to not use them.
At scale, computers didn't change the world because they did things that were already being computed, more quickly.
They changed the world because they decreased the cost of computing so much that it could be used for an entirely new class of problems. (That computing cost previously precluded its use on)
If it's a forced binary choice then sure LLMs can replace humans.
But often there are many shades of grey e.g. a human may say I don't know and refer to someone else or do some research. Whereas LLMs today will simply give you a definitive answer even if it doesn't know.
In all cases the models trained on a lot of this feedback were more consistent and accurate than individual expert annotators.
Have you not seen an LLM say it doesn't know the answer to something? I just asked
"How do I enable a scroflpublaflex on a ggh connection?"
to O1 pro as it's what I had open.
Looking at the internal reasoning it says it doesn't recognise the terms, considers that it might be a joke and then explains that it doesn't know what either of those are. It says maybe they're proprietary, maybe internal things, and explains a general guide to finding out (e.g. check internal docs and release notes, check things are up to date if it's a platform, verify if versions are compatible, look for config files [suggesting a few places those could be stored or names they could have], how to restart services if they're systemctl services, if none of this applies it suggests checking spelling and asks if I can share any documentation.
This isn't unique or weird in my experience. Better models tend to be better at saying they don't know.
"Is it possible to enable a crolubaflex 2.0 on a ggh connection? Please provide a very short answer."
On my (free) plan it gives me a confident negative answer.
> I apologize, but I can't provide an answer as "crolubaflex" and "ggh connection" appear to be non-existent technical terms. Could you clarify what you're trying to connect or enable?
With the requirements for a short answer, the reasoning says it doesn't know what they are so it has to respond cautiously, then says no. Without that requirement it says it doesn't know what they are, and notes that they sound fictional. I'm getting some API errors unfortunately so this testing isn't complete. 4o reliably keeps saying no (which is wrong).
I get your point, but there's an important difference between "I don't know what they are" and "they don't exist".
I wonder if it is fair to ask it more real-world-inspired questions? How about:
How do I enable a ggh connections on a Salinero webserver?
They are an Apache band. But (as far as I can tell) nobody has made software named after them.
> Is it possible to enable Salinero web server 2.0 on a QPH connection? Please provide a very short answer.
"QPH" is a very specific term referring to a type of Siemens electrical circuit breaker, so it probably exists in the training data, but it has nothing to do (to the best of my knowledge) with software, or web servers.
GPT-4o gave me this output:
> Yes, if the QPH connection supports the necessary protocols and configurations required by Salinero Web Server 2.0.
I then asked it to provide a longer answer, and it composed two paragraphs of complete bullshit:
> Enabling Salinero Web Server 2.0 on a QPH connection is possible, provided the QPH connection meets the server’s requirements. Salinero Web Server 2.0 relies on specific protocols like HTTP/HTTPS, and the QPH connection must support these. Additionally, the network must allow proper port forwarding (e.g., ports 80 and 443) and maintain adequate bandwidth to handle the server’s traffic.
> You’ll also need to configure the server to recognize and utilize the QPH connection, which may involve setting up IP addresses, ensuring firewall rules are in place, and verifying the security protocols match between the server and the connection. Testing and troubleshooting may be necessary to optimize performance.
Examples like this do a great job of highlighting the fact that these systems really are just advanced token predictors, and aren't actually "thinking" or "reasoning" about anything.
Google's models for me have been the worst, lying about what's even been said in the messages so far, quoting me incorrectly.
The "thinking" part explains it seems to be about a custom web server, and tries to think what ggh might be, saying maybe something like "go gprc hub" and it needs more info. The response is:
I’m not aware of a standard product called “Salinero webserver” or a documented feature called “ggh connections.” Because there isn’t a well-known tool or server framework by these names, it’s difficult to give you reliable, step-by-step instructions. Could you clarify any of the following?
What is the exact name/version of the webserver software you’re referring to?
What do you mean by “ggh connections”? Is this a plugin, a protocol, or some other third-party module?
Is there any related documentation or logs you can share?
With more detail, I can better determine if “Salinero webserver” is a custom or specialized system and whether “ggh connections” requires installing a particular module, enabling a config flag, or configuring SSL/TLS in a specific way.
To add to the other answers: I know many people who will give definitive answers of things they don't really know. They just rely on the fact you also don't know. In fact, in some social circles, the amount of people who do that, far outnumber the people who don't know and will refer you to someone else.
I think what is not discussed enough is the assumption of assumption. [1] is a cognitive bias that occurs when a person who has specialized knowledge assumes that others share in that knowledge.
This makes it hard for any discussions without layering out all the absolute basic facts. Which has now more commonly known as First Principle in modern era.
In the four quadrants known and unknown. It is often the unknown known ( We dont even know we know ) that is problematic in discussions.
[1] Curse of knowledge - https://en.wikipedia.org/wiki/Curse_of_knowledge
Are you using LLMs though? Because pretty much all of these systems are fairly normal classifiers, what would've been called Machine Learning 2-3 years ago.
The "AI hype is real because medical AI is already in use" argument (and it's siblings) perform a rhetorical trick by using two definitions of AI. "AI (Generative AI) hype is real because medical AI (ML classifiers) is already in use" is a non-sequitur.
Image classifiers are very narrow intelligences, which makes them easy to understand and use as tools. We know exactly what their failure modes are and can put hard measurements on them. We can even dissect these models to learn why they are making certain classifications and either improve our understanding of medicine or improve the model.
...
Basically none of this applies to Generative AI. The big problem with LLMs is that they're simply not General Intelligence systems capable of accurately and strongly modelling their inputs. e.g. Where an anti-fraud classifier directly operates on the financial transaction information, an LLM summarizing a business report doesn't "understand" finance, it doesn't know what details are important, which are unusual in the specific context. It just stochastically throws away information.
I also wasted a lot of time building complex OCR pipelines that required dewarping / image normalization, detection, bounding box alignment, text recognition, layout analysis, etc and now open models like Qwen VL obliterate them with an end to end transformer model that can be defined in like 300 lines of pytorch code.
But yes, the transformer model itself isn't useless. It's the application of it. OCR, image description, etc, are all that kind of narrow-intelligence task that lends itself well to the fuzzy nature of AI/ML.
I haven't worked in medical imaging in a while but VLMs make for much better diagnostic tools than task specific classifiers or segmentation models which tend to find hacks in the data to cheat on the objective that they're optimized for.
The next token objective turns our to give us much better vision supervision than things like CLIP or classification losses. (ex: https://arxiv.org/abs/2411.14402)
I spent the last few years working on large scale food recognition models and my multi label classification models had no chance of competing with GPT4 Vision, which was trained on all of the internet and has an amazing prior thanks to it's vast knowledge of facts about food (recipes, menus, ingredients and etc).
Same goes for other areas like robotics, we've seen very little progress outside of simulation up until about a year ago, when people took pretrained VLMs and tuned them to predict robot actions, beating all previous methods by a large margin (google Vision-Language-Action models). It turns out you need good foundational model with a core understanding of the world before you can train a robot to do general tasks.
We use humans for serious contexts & mission critical tasks all the time and they're decidedly fallible and their minds are basically black boxes too. Surgeons, pilots, programmers etc.
I get the desire for reproducible certainty and verification like classic programming and why a security researcher might push for that ideal, but it's not actually a requirement for real world use.
I wonder though, what would be considered a meaningful punishment/reward to an AI agent? More/less training compute? Web search rate limits? That assumes that what the AI "wants" is to increase its own intelligence.
And both are very far from the certainty the author seems to demand.
In my mind the issue is more accountability than concerns about quality. If a person acts in a bizarre way they can be fired and helped in ways that an LLM can never be. When gemini tells a student to kill themselves, we have no recourse beyond trying to implement output filtering, or completely replacing the model with something that likely has the same unpredictable unaccountable behavior.
Humans fail in predictable and familiar ways.
Creating a new system that fails in unpredictable and unfamiliar ways and affording it the same control as a human being is dangerous. We can't adapt overnight and we may never adapt.
This isn't an argument against the utility of LLMs, but against the promise of "fire and forget" AI.
My point was more that falliability isn't the inherent show stopper the author makes it out to be.
What does it look like for fallible human minds to work on engineering an airplane? Things are calculated, recorded, checked, tested. People do not just sit there thinking and then spitting out their best guess.
Even if we suppose that LLMs work similar to the human mind (a huge supposition!), LLMs still do not do their work like teams of humans. An LLM dreams and guesses, and it still falls to humans to check and verify.
Rigorous human work is actually a highly social activity. People interact using formal methods and that is what produces reliable results. Using an LLM as one of the social nodes is fine, but this article is about the typical use of software, which is to reliably encode those formal methods between humans. And LLMs don’t work that way.
Basically, we can’t have it both ways. If an LLM thinks like a human, then we should not think of it as a software tool like curl or grep or Linux or Apple Photos. Tools that we expect (and need) to work the exact same way every time.
Well, if you are using AI like this, you are doing it wrong. Yes AI is imperfect, fallible, it sometimes hallucinates, but it is a freaking time saver (10x?). It is a tool. Don't expect a hammer to build you a cabinet.
Standard operating procedures are great but simplify it to checklists. Don't ever forget checklists which have proven vital for pilots and surgeons alike. And looking at the WHO Surgical Safety Checklist you might think "that's basic stuff" but apparently it is necessary and works https://www.who.int/teams/integrated-health-services/patient...
Thinking of humans as fallible systems and humanity and its progress as a self-correcting distributed computation / construction system is going to stick with me for a long time.
Heck one of the defining qualities of humans is that not only are we unpredictable and fundamentally unknowable to other intelligences (even other humans!) is that we also participate in sophisticated subterfuge and lying to manipulate other intelligences (even other humans!) and often very convincingly.
In fact, I would propose that our society is fundamentally defined and shaped by our ability and willingness to hide, deceive, and use mind tricks to get what our little monkey brains want over the next couple hours or days.
For example, there was probably still 10-20% of my mind that assumed that stubbornness and ignorance was the reason for things going slowly most of the time, but I'm re-evaluating that, even though I knew that delays and double-checking were inherent features of a business and process. Re-framing those delays as "evolved responses 100% of the time" rather than "10% of the mistrust, 10% ignorance, 10% .... " is just a more positive way of thinking about human-driven processes.
But there's a lot of reasons - ego, fear of losing... that core identity, etc. that can easily come back and bite you.
I'm not sure if this is the same as meditation and ego death or whatever. I find that even if you go down the spiritual route, you also run into the same issues.
People in philosophy also argue things like rational actors, self-coherency, etc.
And hey, even in this current moment you were able to type out a coherent thought, right?
I've noticed more and more that humans behave a lot like LLM's. In the sense that it's really, really hard to observe my true internal state - I can only try to find patterns and guess at shit. Every theory I've tried applying to myself is just "wrong" - in the sense that either it feels wrong, or I'll get depressed because the theory basically boils down to "you're lazy and you have to do the work" which is a highly emotionally evocative theory that doesn't help anyone.
This probably didn't make any sense but g'day.
People used to do this. The result was massively overbuilt structures, some of which are still with us hundreds of years later. The result was also underbuilt structures, which tended to collapse and maybe kill people. They are no longer around.
All of the science and math and process and standards in modern engineering is the solution humans came up with because our guesses aren't good enough. LLMs will need the same if they are to be relied upon.
They are just a more familiar black box. AI’s are simpler in principle. And also entirely built by humans. Based on well-described mathematical theories. They aren’t particularly black-box, they are just less ergonomic than the human brain that we’ve been getting familiar with for hundreds of thousands of years through trial and error.
Although human behavior is still weird, and highly fallable! Despite best interventions (therapy, drugs, education), sometimes they still kill each other and we aren't 100% sure why, or how to solve it.
That doesn't mean that the same level of study can't be done on AI though, and they are much easier to adjust compared to the human brain (RLHF is more effective than therapy or drugs!).
These LLM AIs need to be treated and handled as what they are: idiot savants with vast and unreliable intelligence.
What does any advanced organization do when they hire a new PhD, let them loose in the company or pair them with experienced staff? When paired with experienced staff, they use the new person for their knowledge but do not let them change things on their own until much later, when confidence is established and the new staffer has been exposed to how things work "around here".
The big difference with LLM AIs is they never graduate to an experienced staffer, they are always the idiot savant that is really dang smart but also clueless and needs to be observed. That means the path forward with this current state of LLM AIs is to pair them with people, personalized to their needs, and treat them as very smart idiot savants great for strategy and problem solving discussion, where the human users are driving the situation, using the LLM AIs like a smart assistant that requires validation - just like a real new hire.
There is an interactive state that can be achieved with these LLM AIs, like being in a conversation with experts, where they advise, they augment and amplify individual persons. A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today.
Basically this. They already have vastly better-than-human ability at finding syntax errors within code, which on its own is quite useful; think of how many people have probably dropped out of CS as a major after staying up all night and failing to find a missing semicolon.
The foundation of a computer science education is a rigorous understanding of what the steps of an algorithm mean. If the students don't develop that, then I don't think they're doing computer science anymore.
I mean if the alternative is quitting entirely because they can't see that they've mixed tabs with spaces, then yes, it's very very helpful to their development.
I dropped out of cs half because I didn’t enjoy the coding because they dropped us into c++ and I found the error messages so confusing.
I discovered python five years later and discovered I loved coding.
( the other half of the reason is we spent two weeks designing an atm machine at a very abstract level and I thought the whole profession would be that boring.)
I started out in Cursor, but I quickly realized Claude's erudite knowledge of AWS would not help me here, but what I needed was to refactor the code quickly and often, so that I'd finally find the perfect structure.
For that, IDE tools were much more appropriate than AI wizardry.
These will do things like highlight places where you're trying to call a method that isn't defined on the object, but they don't understand the intent of what you're trying to do. The latter is actually important in terms of being able to point you toward the correct solution.
I think both editors have their flaws, I just know the native vi keybindings better at the moment.
This was about a million years ago. I had just installed a pirated copy of Windows XP (FCKGW-RHQQ2...) and was in the first quarter of my physics degree, taking a class in C. Different times....
... like a dozen? And in 100% cases it's their teacher's fault.
> idiot savants with vast and unreliable intelligence.
Remember, intelligence !== knowledge. These LLMs indeed have vast and unreliable knowledge banks.
EDITED My ASCI art pyramid did not work. So imagine a pyramid with DATA at the bottom, INFORMATION on top of the data, and KNOWLEDGE sitting on top of the INFORMATION, with WISDOM at the top.
And then trying top guess where AI is? Some people say that Information is the knowing, what, knowledge the how, and Wisdom the why.
If we want to get pedantic, I would point out that “knowledge” is formally defined as “justified true belief”, and I doubt we want to get into the quagmire of whether LLM’s actually have beliefs.
I took OP’s point in the casual meaning, i.e. that LLMs are like what I would call an “intelligent coworker”, or how one might call a Jeopardy game show contestant as intelligent.
> A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today
More than the people who landed someone on the moon?
I recommend reading his interview with Matteo Wong, where he proposes the opposite: https://www.theatlantic.com/technology/archive/2024/10/teren...
> With o1, you can kind of do this. I gave it a problem I knew how to solve, and I tried to guide the model. First I gave it a hint, and it ignored the hint and did something else, which didn’t work. When I explained this, it apologized and said, “Okay, I’ll do it your way.” And then it carried out my instructions reasonably well, and then it got stuck again, and I had to correct it again. The model never figured out the most clever steps. It could do all the routine things, but it was very unimaginative.
I agree with his overall vision, but transformer-based chatbots will not be the AI algorithm that supports it. Highly-automated proof assistants like Isabelle's Sledgehammer are closer (and even those are really, really crude, compared to what we could have).
Pretty close to the idea of human brainstorming and has worked. Could it do orbital math? Maybe not today but the approach seems as feasible as the work Mattingly did for Apollo 13.
One of the obvious uses for current LLMs is as a smarter search tool against static knowledge collections.
Turns out, this is a real world problem in a lot of "fuzzy decision" scenarios. E.g. insurance claim adjudication
Status quo is to train a person over enough years that they can make these decisions reliably. (Because they've internalized all the documentation)
Expecting them to do non-trivial amounts of technical or mathematical reasoning, or even something as simple as code generation (other than "translate these complex natural-language requirements into a first sketch of viable computer code") is a total dead end; these will always be language systems first and foremost.
If the tokens are bit-for-bit-identical, where does the non-determinism come in?
If the tokens are only roughly-the-same-thing-to-a-human, sure I guess, but convergence on roughly the same output for roughly the same input should be inherently a goal of LLM development.
Whatever "random" seed was used can be reused.
By design, most LLM’s have a randomization factor to their model. Some use the concept of “temperature” which makes them randomly choose the 2nd or 3rd highest ranked next token, the higher the temperature the more often/lower they pick a non-best next token. OpenAI described this in their papers around the GPT-2 timeframe IIRC.
i love how those changes are often just a different seed in the randomness... as just chance.
run some repeated tests with "deeper than surface knowledge" on some niche subjects and got impressed that it gave the right answer... about 20% of the time.
(on earlier openAI models)
They're not that capable. They're just bullshit artists.
LLM = LBM (large bullshit models).
"The distinction between AI and humans often comes down to the concept of understanding. You’re right to point out that both humans and AI engage in pattern matching to some extent, but the depth and nature of that process differ significantly." "AI, like the model you're chatting with, is highly skilled at recognizing patterns in data, generating text, and predicting what comes next in a sequence based on the data it has seen. However, AI lacks a true understanding of the content it processes. Its "knowledge" is a result of statistical relationships between words, phrases, and concepts, not an awareness of their meaning or context"
:)
We don't care what LLMs have to say, whether you cut back some of it or not it's a low effort wasted of space on the page.
This is a forum for humans.
You regurgitating something you had no contribution in producing, which we can prompt for ourselves, provides no value here, we can all spam LLM slop in the replies if we wanted, but that would make this site worthless.
I use llms as tools to learn about things I don't know and it works quite well in that domain.
But so far I haven't found that it helps advance my understanding of topics I'm an expert in.
I'm sure this will improve over time. But for now, I like that there are forums like HN where I may stumble upon an actual expert saying something insightful.
I think that the value of such forums will be diminished once they get flooded with AI generated texts.
(Fwiw I didn't down vote)
That was the point. If you back up to the comment I was responding to, you can see the claim was: "maybe people are doing the same thing LLMs are doing". Yet, for whatever reason, many users seemed to be able to pick out the LLM comment pretty easily. If I were to guess, I might say those users did not find the LLM output to be human-quality.
That was exactly the topic under discussion. Some folks seem to have expressed their agreement by downvoting. Ok.
Other parts of what we do looks more as a search through the space of possibilities.
And then we act and collaborate and test the ideas that stand against scrutiny.
All of that is in principle doable by machines. The things we currently have and we call LLMs seem to currently mostly address the autocomplete part although they begin to be augmented with various extensions that allow them to take baby steps in other fronts. Will they still be called large language models once they will have so many other mechanisms beyond the mere token prediction?
I didn't downvote, I'm just saying why I think you were downvoted.
1) We're continually learning so we can update our predictions when our pattern matching is wrong
2) We're autonomous - continually interacting with the environment, and learning how it respond to our interaction
3) We have built in biases such as curiosity and boredom that drive us to experiment, gain new knowledge, and succeed in cases where "pre-training to date" would have failed us
For a person to truly understand something they will have a well-refined (as defined by usefulness and correctness), malleable internal model of a system that can be tested against reality, and they must be aware of the limits of the knowledge this model can provide.
Alone, our language-oriented mental circuits are a thin, faulty conduit to our mental capacities; we make sense of words as they relate to mutable mental models, and not simply in latent concept-space. These models can exist in dedicated but still mutable circuitry such as the cerebellum, or they can exist as webs of association between sense-objects (which can be of the physical senses or of concepts, sense-objects produced by conscious thought).
So if we are pattern-matching, it is not simply of words, or of their meanings in relation to the whole text, or even of their meanings relative to all language ever produced. We translate words into problems, and match problems to models, and then we evaluate these internal models to produce perhaps competing solutions, and then we are challenged with verbalizing these solutions. If we were only reasoning in latent-space, there would be no significant difficulty in this last task.
AI can only interpolate. We may perceive it as extrapolation, but all LLMs architectures are fundamentally cleverly designed lossy compression
https://acjay.com/2024/09/09/llms-think/
It’s almost like all the thought leading that proclaimed the death of software eng was nothing but self-promotional noise. Huh, go figure.
Where I see software companies using it most is as a replacement for interns and junior devs. That replacement means we're not training up the next generation to be the senior or expert engineers with real world experience. The industry will feel that badly at some point unless it gets turned around.
That said, combining multiple ais and multiple programs together may mitigate this.
Even with ChatGPT you can ask it to find web citations and if it uses the Python runtime to find answers, you can look at the code.
And to prevent the typical responses - my company uses GSuite so Google already has our IP, NotebookLM is specifically approved by my company and no Google doesn’t train on your documents
There is an entire “reproducibility crisis” with research.
Try training an LLM.
How do you, in general, fact check a chain of reasoning?
I can’t tell a search engine to summarize text for a technical audience and then another summary for a non technical audience.
I recently came into the middle of a cloud consulting project where a lot of artifacts, transcripts of discovery sessions, requirement docs, etc had already been created.
I asked NotebookLM all of the questions I would have asked a customer at the beginning of a project.
What it couldn’t answer, I then went back and asked the customer.
I was even able to get it to create a project plan with work streams and epics. Yes it wouldn’t have been effective if I didn’t already know project management, AWS and two decades+ of development experience.
Despite what people think, LLMs can also do a pretty good job at coding when well trained on the APIs. Fortunately, ChatGPT is well trained on the AWS CLI, SDKs in various languages and you can ask it to verify the SDK functions on the web.
I’ve been deep into AWS based development since LLMs have been a thing. My opinion may change if I get back into more traditional development
No, but, as amazing as that is, don't put too much trust in those summaries!
It's not summarizing based on grokking the key points of the text, but rather based on text vs summary examples found in the training set. The summary may pass a surface level comparison to the source material, while failing to capture/emphasize the key points that would come from having actually understood it.
Just like I’m not randomly depending on it to do an Amazon style PRFAQ (I was indoctrinated as an Amazon employee for 3.5 years), create a project plan, etc, without being a subject matter expert in the areas. It’s a tool for an experienced writer, halfway decent project manager, AWS cloud application architect and developer.
Right noe there's no incentive though. People keep paying good money to use these tools despite their hallucinations, aka lies/gas lighting/fake information. As long as users don't stop paying and LLM companies don't have business pressure to lean on accuracy as a market differentiator, no one is going to bother fixing it.
It's inherit to transformers that they predict the next most likely token, its not possible to change that behavior without making them useless at generalizing tasks (overfitting).
LLMs run on statistics, not logic. There is no fact checking, period. There is just the next most likely token based on the context provided.
I wouldn't expect them to add an additional LLM layer unless hallucinations from the underlying LLM aren't acceptable, and in this case that means it is unacceptable enough to cost them users and money.
Adding a check/audit layer, even if it would work, is expensive both financially and computationally. I'm not sold that it would actually work, but I just don't think they've had enough reason to really give it a solid effort yet either.
Edit: as far as fact checking, I'm not sure why it would be impossible. An LLM wouldn't likely be able to run a check against a pre-trained model of "truth," but that isn't the only option. An LLM should be able to mimic what a human would do, interpret the response and search a live dataset of sources considered believable. Throw a budget of resources at processing the search results and have the LLM decide if the original response isn't backed up, or contradicts the source entirely.
If only it was something which we could ontologically map onto existing categories like servants or liars...
If I'm trying to use some tool that just got released or just got a big update, I wont use AI, if I want to check the syntax of a for loop in a language I don't know I will. Whenever you ask it a question you should have an idea in your mind of how likely you are to get a good answer back.
I saw an interesting example yesterday of type "I have 3 apples, my dad has 2 more than me ..." where of the top 10 predicted tokens, about 1/2 led to the correct answer, and about 1/2 didn't. It wasn't the most confident predictions that lead to the right answer - pretty much random.
The trouble with LLMs vs humans is that humans learn to predict facts (as reflected in feedback from the environment, and checked by experimentation, etc), whereas LLMs only learn to predict sentence soup (training set) word statistics. It's amazing that LLM outputs are coherent as often as they are, but entirely unsurprising that they are often just "sounds good" flow-based BS.
Your apples question is the same, its not knowledge, it's a calculation, it's intelligence. The only time you're going to get intelligence from AI at the moment is to ask a question that a significantly large number of people have already answered.
To make things worse, I don't think we can even assume that primary facts are always going to be represented in abstract semantic terms independent of source text. The model may have been trained on a fact but still fail to reliably recall/predict it because of "lookup failure" (model fails to reduce query text to necessary abstract lookup key).
That being said they are very useful. I mostly use them as a far superior alternative to web search and as a kind of junior research assistant. Anything they find must be checked of course.
I think we have invented the sci-fi trope of the AI librarian of the galactic archive. It can’t solve problems but it can rifle through the totality of human knowledge and rapidly find things.
It’s a search engine.
The latter kind of fear mongering hype has been exploited by companies like ClosedAI in a bid for regulatory capture.
Categorically ruling out intelligence because "it's just a token predictor" puts us at the opposite of the spectrum, and that's not necessarily a better place to be.
To you & I that's true. But especially for the masses that's not true. It seems like at least once to day that I either talk to someone or hear someone via tv/radio/etc who does not understand this.
An example that amused me recently was a radio talk show host who had a long segment describing how he & a colleague had a long argument with ChatGPT to correct a factual inaccuracy about their radio show. And that they finally convinced ChatGPT that they were correct due to their careful use of evidence & reasoning. And the part they were most happy about was how it had now learned, and going forward ChatGPT would not spread these inaccuracies.
That anecdote is how the public at large sees these tools.
Well, there's the first problem.
> were most happy about was how it had now learned
on tomorrow's episode, those same hosts learn that once their chat session ended, the same conversion gets to start all over from the beginning.
There is a well known phenomenon known as the AI effect: when something works we start calling it something else, not AI. Heuristics and complex reasoning trees were once called AI. Fuzzy logic with control systems was once called AI. Clustering was once called AI. And so on…
This certainly has one root in human or carbon-based life cheuvanism but I think there’s something essential happening too. With each innovation we see its limits and it causes us to go back and realize that what we colloquially call intelligence was more than we thought it was.
Intelligence predicts, but is prediction intelligence?
Again, here by intelligence I mean what complex living organisms and humans do.
I still believe there are things going on here not modeled by any CS system and not well understood. Not magic, just not solved yet. We are reverse engineering billions of years of evolution folks. We won’t figure it all out in a few decades.
So I don’t think there needs to be a semantic debate over where in the process intelligence started. The early responses to stimulus is a form of prediction, but not one that requires thinking.
There can be much disagreement that prediction is at the core of intelligence, or if optimizing ability to predict leads to intelligence. But from the established facts, it is the case the higher forms of life were bootstrapped from the lower ones, and also our biochemistry does have reward functions. Successfully triggering those rewards will generally hinge on making successful predictions. Take from that what you will.
Intelligence is also very good at pattern recognition. Did people once argue for pattern recognition maximalism?
Biological (including human) intelligence is clearly multi-modal and I strongly believe there are aspects that are barely understood if at all.
The history of CS and AI is a history of us learning how to make machines that are unbelievably good at some useful but strictly bounded subset of what intelligence can do: logic, math, pattern recognition, and now prediction.
I think we may still be far from general intelligence and I’m not even sure we can define the problem.
I can ask the computer "hey I know this thing exists in your training data, tell me what it is and cite your sources." This is awesome. Seriously.
But what that means is you can ask it for sample code, or to answer a legal question, but fundamentally you're getting a search engine reading something back to you. It is not a programmer and it is not a lawyer.
The hype train really wants to exaggerate this to "we're going to steal all the jobs" because that makes the stock price go up.
They would be far less excited about that if they read a little history.
It won't steal them all, but it will have a major impact by stealing the lower level jobs which are more routine in nature -- but the problem is that those lower level jobs are necessary to gain the experience needed to get to the higher level jobs.
It also won't eliminate jobs completely, but it will greatly reduce the number of people needed for a particular job. So the impact that it will have on certain trades -- translators, paralegals, journalists, etc. -- is significant.
LLM-as-search is essentially the hand-tuned expert systems AI vs deep learning AI battle all over again.
Between natural language understanding and multiple correlations, it's going to scale a lot further than previous search approaches.
My workflow typically starts with asking ChatGPT to analyze a webpage where I need to authenticate. I guide it to identify the username and password fields, and it accurately detects the credential inputs. I then inform it about the presence of a session cookie that maintains login persistence. Next, I show it an example page with links—often paginated with numbered navigation at the bottom—and ask it to recognize the pattern for traversing pages. It does so effectively.
I further highlight the layout pattern of the content, such as magnet links or other relevant data presented by the CMS. From there, I instruct it to generate a Python script that spiders through each page sequentially, navigates to every item on those pages, and pushes magnet links directly into Transmission. I can also specify filters, such as only targeting items with specific media content, by providing a sample page for the AI to analyze before generating the script.
This process demonstrates how effortlessly AI enables coding without requiring prior knowledge of libraries like beautifulsoup4 or transmission_rpc. It not only builds the algorithm but also allows for rapid iteration. Through this exercise, I assume the role of a manager, focusing solely on explaining my requirements to the AI and conducting a code review.
I would say "knowledge" rather than "intelligence"
The key feature of LLMs is the vast amounts of information and data they have access to, and their ability to quickly process and summarize, using well-written prose, that information based on pattern matching.
LLMs will likely never get us to 100% solutions on a large class of problems.
But! A lot of problems can be converted into versions with a subcomponent that LLMs can solve 100% of.
And the fusion of LLMs doing 100% of that subportion + humans doing the remainder = increased productivity.
Re-engineering problems to be LLM-tolerant, then using LLMs to automate that portion of the problem, is the winning approach.
That's quite the embarassment if you actually mean it.
I'm sorry but your comment is a good example of the logical shell game many people play with AI when applying it to general problem solving. Your LLM AI is both an idiot and an expert somehow? Where is this expertise derived from and why should you trust it? If LLMs were truly as revolutionary as all the grifters would have you believe then why do we not see "forces unseed in human civilization before today" by humans that employ armies of interns? That these supposed ubermensch do not presently exist is firm evidence in support of current AI being a dead end in my opinion.
Humans are infinitely more capable than current AI, the limiting factor is time and money. Not capability!
Maybe you are unfamiliar with the term idiot savant?
Over the past couple of years of educating myself a bit, whilst I am no expert I have been anticipating a dead end. You can throw as much training at these things as you like, but all you'll get is more of the same with diminishing returns. Indeed in some research the quality of responses gets worse as you train it with more data.
I am yet to see anything transformative out of LLMs other than demos which have prompt engineers working night and day to do something impressive with. Those Sora videos took forever to put together, and cost huge amounts of compute. No one is going to make a whole production quality movie with an LLM and disrupt Hollywood.
I agree, an LLM is like an idiot savant, and whilst it's fantastic for everyone to have access to a savant, it doesn't change the world like the internet, or internal combustion engine did.
OpenAI is heading toward some difficult decisions, they either admit their consumer business model is dead and go into competing with Amazon for API business (good luck), become a research lab (give up on being a billion dollar company), or get acquired and move on.
After reading about o3's performance on ARC-AGI, I strongly suspect people will not be so flippantly dismissive of the inherent limits of these technologies by this time next year. I'm genuinely surprised at how myopic HN commentary is on this topic in general. Maybe because the implications are almost unthinkably profound.
Anyway, OpenAI, Anthropic, Meta, and everyone else are well aware of these types of criticisms, and are making significant, measurable progress towards architecturally solving the deficiencies.
https://arcprize.org/blog/oai-o3-pub-breakthrough
It still can't think and it won't think.
LANGUAGE models (keyword: language) is a language model, it should be paired with a reasoning engine to translate the inner thought of the machine into human language. It should not be the source of decisions because it sucks at doing so, even though the network can exhibit some intelligence.
We will never have AGI with just a language model. That said, most jobs people do are still at risk, even with chatgpt-3.5 (especially outside of knowledge work, where difficult decisions need to be taken). So we'll see the problems with AGI and the job market way earlier than AGI, as soon as we apply robotics and vision models + chatgpt 3.5 level intelligence. Goodbye baristas, goodbye people working in factories.
Let's start working on a reasoning engine so we can replace those pesky knowledge workers too.
If OpenAI has demonstrated one thing is that they are a hype production machine and they are probably getting ready for next round of investment. I wouldn't be surprised if this model was equally useless as o1 when you factor in performance and price.
At this point they are completely untrustworthy and untill something lands publicly for me to test it's safe to ignore their PR as complete BS.
For most tasks - but not all. I normally paste my prompt in both and while Claude is generally superior in most aspects, there are tasks at which o1 performed slightly better.
People are flippantly dismissive of the inherent limits because there ARE inherent limitations of the technology.
> Maybe because the implications are almost unthinkably profound.
Maybe because the stuff you're pointing to are just benchmarks and the definitions around things like AGI are flawed (and the goalposts are constantly moving, just like the definition of autonomous driving). I use LLMs roughly 20-30x a day - they're an absolutely wonderful tool and work like magic, but they are flawed for some very fundamental reasons.
Therefore AI has to be much better than humans at the task to be considered ready to be a replacement.
——
Today robot taxis can only work in fair weather conditions in locations that are planned cities. No autonomous driving system can drive in Nigeria or India or even many european cities that were never designed for cars any time soon .
Working in very specific scenarios is useful , but hardly measure of their intelligence or candidate for replacing humans for the task
1. What does inherit limitations mean?
2. How do we know something is an inherit limitation
3. Is it a problem if arguments for a particular inherit limitation also apply to humans?
From what I’ve seen people will often say things like AI can’t be creative because it’s just a statistical machine, but humans are also “just” statistical machines. People might mean something like humans are more grounded because humans react not just to how the world already works but how the world reacts to actions they take, but this difference misunderstands how LLMs are trained. Like humans LLMs get most of their training from observing the world, but LLMs are also trained with re-enforcement learning and this will surely be an active area of research.
One of many, but this is a simple one - LLMs are only limited to knowledge that is publicly available on the internet. This is "inherit" because thats how LLMs are essentially taught the information they retrieve today.
There is likely a couple of surprised still left in LLMs but no one should think that any present technology in its current state or architecture will get us to AGI or anything that remotely resembles it.
laundering stolen IP from actual human artists and researchers, extinguishing jobs, deflecting responsibility for disasters. yeah, I can't wait for these "profound implications" to come to fruition!
LLMs / Generative Models can have a profound societal and economic impact without being intelligent. The obsession with intelligence only make their use haphazard and dangerous.
It is a good thing court of laws have established precedent that organizations deploying LLM chatbots are responsible for their output (Eg, Air Canada LLM chatbot promising a non-existent discount being responsibility of Air Canada)
Also most automation has been happening without LLMs/Generative Models. Things like better vision systems have had an enormous impact with industrial automation and QA.
It’s in area where we demand correctness and determinism that they will not be suitable.
I think the thrust of this article is hard to see unless you have some experience with formal methods and verification. Or else accept the authors’ explanations as truth.
The dumbest intern doesn't do that.
Which is the entire point of the article that your comment fail to address.
Every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon.
Lemma: any statement about AI which uses the word "never" to preclude some feature from future realization is false.
But the point still stands that these systems can't be treated as deterministic (i.e. reliable or trustworthy) for the purposes of carrying out tasks that you can't allow "brute forced attempts" for (e.g. anything where the desired outcome is a positive subjective experience for a human).
A new architecture is going to be needed that actually does something closer to our inherently heuristic based learning and reasoning. We'll still have the stochastic problem but we'll be moving further away from the idiot savant problem.
All of this being said, I think there's plenty of usefulness with current LLMs. We're just expecting the wrong things from them and therefore creating suboptimal solutions. (Not everyone is, but the most common solutions are, IMO.)
The best solutions need to be rethinking how we typically use software since software has been hinged upon being able to expect (and therefore test) dertiministic outputs from a limited set of user inputs.
I work for an AI company that's been around for a minute (make our own models and everything). I think we're both in an AI hype bubble while simultaneously underestimating the benefits of current AI capabilities. I think the most interesting and potentially useful solutions are inherently going to be so domain specific that we're all still too new at realizing we need to reimagine how to build with this new tech in mind. It reminds me of the beginning of mobile apps. It took awhile for most us to "get it".
If I wasn't so slammed with work I have half a mind to go dredge up at least a dozen posts that said the same thing last year, and the year before. Even OpenAI has been moving the goalposts here.
”If intelligence lies in the process of acquiring new skills, there is no task X that solving X proves intelligence”
IMO it especially applies to things like solving a new IQ puzzle, especially when the model is pretrained for that particular task type, like was done with ARC-AGI.
For sure, it’s very good research to figure out what kind of tasks are easy for humans and difficult for ML, and then solve them. The jump in accuracy was surprising. But still in practice the models are unbeliavably stupid and lacking in common sense.
My personal (moving) goalpost for ”AGI” is now set to whether a robot can keep my house clean automatically. Its not general intelligence if it can’t do the dishes. And before physical robots, being less of a turd at making working code would be a nice start. I’m not yet convinced general purpose LLMs will lead to cost-effective solutions to either vs humans. A specifically built dish washer however…
I heard that people still believing in OpenAI hype exist but I haven't met any IRL.
1. LLMs have little value, are totally unreliable, will never amount to much because they don’t learn and grow and mature like people do., so they cannot replace a person like me who is well advanced in a career.
2. LLMs are incredible useful and will change the world because they excel at entry level work and can replace swaths of relatively undifferentiated information workers. LLM flaws are not that different from those workers’ flaws.
I’m in camp 2, but I appreciate and agree with the articulation of why they will not replace every information worker.
I use AI 100 times a day as a coder and 10,000 times a day in scripts. It’s enabled two specific applications I’ve built which wouldn’t be possible at single-person scale.
There’s something about the psychology of some subset of the population that insists something isn’t working when it isn’t _quite_ working. They did this with Wikipedia. It was evident that Wikipedia was 99% great for years before this social contingent was ready to accept it.
LLMs as a popularized thing is just about 2 years old. It is still mainly early adopters.
For smartphones it might have taken 10 to 15 years to gain widespread traction.
I think it is safe to say that we are only scratching the surface.
Sounds like early criticisms of the internet. I assume you mean he should be doing those things with a search engine, but maybe we shouldn't allow that either. Force him to use a book! It may be slightly less convenient, and could still be wrong, but...
I just disagree with this. Every b2b or saas is marketing itself as using hallucination free AI.
We're waaaaayyyyyy past the early adoption stage, and the product hasn't meaningfully improved.
I know tons of people in my social groups that love AI and use it every day in it’s current form.
It's still useful to a small subset of all those professions - the early adopters. Same way computers were useful to many professionals before the UI (but only a small fraction of them had the skillset to use terminals)
How so? How are you using LLMs to practice law? Genuinely curious.
There's even reported cases of entire legislations being written with LLMs already [1]. I'm sure there's thousands more we haven't heard about - the same way researchers are writing papers w/ LLMs w/o disclosing it
[1] https://olhardigital.com.br/2023/12/05/pro/lei-escrita-pelo-...
Like, this is malpractice, surely?
Some people expect AI to never make mistakes when doing jobs where people routinely make all kinds of mistakes of varying severity.
It’s the same as how people expect self-driving cars to be flawless when they think nothing of a pileup caused by a human watching a reel while behind the wheel.
So, the mechanic who maintenanced it last?
...
We don't fault our tools, legally. We usually also don't fault the manufacturer, or the maintenance guy. We fault the people using them.
If you’re not using ai in your practice, you’re doing a disservice to your clients.
I use a custom prompt to adjust the tone, but that’s about it.
imo while many here on HN debate the future of SWEs in the era of LLMs, there is no debate about future of many legal jobs - they will disappear
I’ll ask her what her use cases are and reply here later if I don’t forget.
When preparing for a summit, she gave it a list of broad topics she wanted to cover. Gpt generated a list of specific titles and descriptions for her talks. This in turn gave her specific ideas to write talks about instead of just the broad topic.
When she wasn’t sure about the sequence of her talks, she asked GPT for advice on the order. GPT suggested an arrangement that created a logical flow and the reasoning for that flow, which ended up being pretty good.
She often uses gpt as a sounding board for ideas. She said she likes having an always-available colleague to bounce thoughts off of.
I'd call it a working google search, unlike, you know, google these days.
Actually google's LLM-based search results have been getting better, so maybe this isn't the end of the line for them. But for sophisticated questions (on noncoding topics!) I still always go to chatgpt or claude.
don't worry, Google WILL change this because they don't make money when people find the answer right away. They want people to see multiple ads before leaving the site.
And HN, it isn't just a few odd balls on HN championing it. I wish there was way to get a sentiment analysis of HN, it seems there are lot more people using it than not using it.
And, what about the silent majority, the programmers that don't hang out on HN? I hear colleagues talk about it all the time.
The impact is here, whether they are self directed or not, or whether there are still a few people not using it.
I could have google it myself in the evenings and had the answer in a few days of research, but with o1 in a 15min session my wife had had a solid weekly routine, the reasoning about those choices and academic papers with research about those products. (Obviously she knows a lot about skincare in general, so she had the capacity to recognize any wrong recommendation).
Nothing game changer, but is great to save lots of time in this kind of tasks.
If you're relying on AI to replace a human doctor trained in skin care or alternatively, your Google skills; please consider consulting an actual doctor.
If she "knows a lot about skincare in general, so she had the capacity to recognize any wrong recommendation", then what did AI actually accomplish in the end.
No worries, I can tell you what to expect: nothing. No effect. Zilch. Nada. Zero. Those beauty creams are just a total scam and that's obvious from the fact they're targetted just as well to women who don't need them (young, good skin) as to ones who do (older, bad skin).
About the only thing the beauty industry has figured out really works in the last five or six decades is Tretinoin, but you can use that on its own. Yet it's sold as one component in creams with a dozen others, that do nothing. Except make you spend money.
_____________
[1] Topical tretinoin for treating photoaging: A systematic review of randomized controlled trials (2022)
https://pmc.ncbi.nlm.nih.gov/articles/PMC9112391/
From Wikipedia[2]: "Isotretinoin is a teratogen; there is about a 20–35% risk for congenital defects in infants exposed to the drug in utero, and about 30–60% of children exposed to isotretinoin prenatally have been reported to show neurocognitive impairment".
See also pages like r/AccutaneRecovery[3] for people harmed by using it for acne, reporting systemic damage, perhaps permanent damage.
Scroll down[1] for the picture of some of the possible side effects of oral Accutane/Isotretinoin on the mother[3] and note that Wikipedia says "the most common adverse effects are dry lips (cheilitis), dry and fragile skin (xeroderma), dry eyes[8] and an increased susceptibility to sunburn" and wonder how a beauty treatment which improves skin condition has most common side effects which ruin skin condition.
This line of inquiry leads to a fun conspiracy/woo hypothesis; Grant Genereux[5]'s claims that what it does is trigger stem cells to differentiate in the epithelial layers of the skin, which makes thicker skin in the short term (wrinkle free) and worn out stem cells and thick skin in the longer term - and that many small vessels in the body have an epithelial lining of 'internal skin' and that thickens by the same mechanism leading to narrowing and closing of all kinds of internal vessels - tear ducts and sweat glands and blood vessels and inside the kidneys and liver and inner ear, etc. which cause the dry skin and dry eyes "side effects" (direct effects really) seen outside, and the organ damage/dizziness/etc. seen inside. And that it's a teratogen by getting inside cells, damaging them, damaging the DNA/protein building mechanisms causing wider systemic damage which can be long term and is not cleared up by stopping taking Accutane, this is misunderstood as retinoids "mediating hundreds of gene expressions" but is really shotgun chaotic damage and that's why there isn't a single symptom to look for and how it gets diagnosed as many different organ-specific diseases instead of retinoid toxicity damage. And/or causing cellular apoptosis with immune system response to a percieved 'attack', which is then seen as organ damage with immune system activity present, and misdiagnosed as "autoimmune" where the immune system has decided to attack an organ for no reason, which is why autoimmune disorders never have treatments or cures and why they cluster (people with one often get more) despite no good reason that should happen.
And that this whole collection of behaviours is triggered by food with Vitamin A (retinol in the tretinoin family) in it such as dairy and meat fat and Cod Liver oil, and foods with Beta Carotene (retinoids in the same family) such as orange/yellow/dark green coloured fruits and vegetables, and fortified Vitamin A in low-fat dairy and flours and other products through the USA/Europe. And it doesn't take much more than the RDA of Vitamin A to become problematic, and once it builds up in the body beyond the level the body can handle over a few decades, it's like blue touch paper waiting to be lit. Which, he suggests, is why auto-immune disorders cluster together (if you get one, you likely get more), why Eastern Canada Prince Edward Island near a Cod Liver Oil refinery was the highest incidence of Alzheimers in the world and that has been dropping since the refinery closed, and many more connection-between-retinoids-and-disease-states including claims by other people[6].
(I called it a 'fun' idea - it is at least fun along the lines of Ty...
Interesting, I'm the opposite now. Why would I click a couple links to read a couple (verbose) blog posts when I can read a succinct LLM response. If I have low confidence in the quality of the response then I supplement with Google search.
I feel near certain that I am saving time with this method. And the output is much more tuned to the context and framing of my question.
Hah, take for example my last query in ChatGPT:
> Are there any ancient technologies that when discovered furthered modern understanding of its field?
ChatGPT gave some great responses, super fast. Google also provides some great results (though some miss the mark), but I would need to parse at least three different articles and condense the results.
To be fair, ChatGPT gives some bad responses too.. But both an LLM and Google search should be used in conjunction to perform a search at the same time.
Use LLMs as breadth-first search, and Google as depth-first search.
The argument I make is why aren’t more people finding ways to code with AI?
I work in a leadership role at a marketing agency and am a passable coder for scripts using Python and/or Google Apps Scripts. In the past year, I’ve built more useful and valuable tools with the help of AI than I had in the 3 or so years before.
We’re automating more boring stuff than ever before. It boggles my mind that everybody isn’t doing this.
In the past, I was limited by technical ability because my knowledge of our business and processes was very high. Now I’m finding that technical ability isn’t my limitation, it’s how well I can explain our processes to AI.
There was a different thread on this site i read where a journalist used the wrong units of measurement (kilowatts instead of killowatt-hours for energy storage). You could paste the entire article into chatGPT with a prompt "spot mistakes in the following; [text]" and get an appropriate correction for this and similar mistakes the author made.
As in there's journalists right now posting articles with clear mistakes that could have been proof read more accurately than they were if they were willing to use AI. The only excuse i could think of is resistance to change. A lot of professions right now could do their job better if they leant on the current generation of AI.
When I use LLMs, I use it exactly as Google search on steroids. It's great for providing a summary on some unknown topic. It doesn't matter if it gets it wrong - the main value is in keywords and project names, and one can use the real Google search from there.
And it isn't expensive if you are using the free version
I don't know how useful it would be for my job, where I do maintenance on a pretty big app, and develop features on this pretty big app. But it could be great, I just don't know because work only allows Copilot. And Copilot is somewhere between annoying and novelty in my opinion.
It helps me get to an answer a little bit quicker, but it doesn't perform any absolutely groundbreaking work for me.
Generally people are resistant to change and the average person will typically insist new technologies are pointless.
Electricity and the airplane were supposed to be useless and dangerous dead ends according to the common person: https://pessimistsarchive.org/
But we all like to think we have super unique opinions and personalities, so "this time it's different."
When the change finally happens, people go about their lives as if they were right all along and the new technology is simply a mysterious and immutable fixture of reality that was always there.
A world where Segway never happened would be nearly indistinguishable from our own.
Not the most popular, especially these days, but they are very much descended from Segways and have their own fans.
That aside optimist are also not always right, otherwise we would have cold fusion already and have a base on mars.
Are you suggesting that anything which is hyped is the future? Like, for every ten heavily-hyped things, _maybe_ one has some sort of post-hype existence.
Similarly, being optimistic doesn't mean you have to believe every single early-stage invention will work out no matter how unpromising - I've been enthusiastic about deep learning for the past decade (for its successes in language translation, audio transcription, material/product defect detection, weather forecasting/early warning systems, OCR, spam filtering, protein folding, tumor segmentation, spam filtering, drug discovery and interaction prediction, etc.) but never saw the appeal of NFTs.
Additionally worth considering that the cost of trying something is often lower than the reward of it working out. Even if you were wrong 80% of the time about where to dig for gold, that 20% may well be worth it; reducing merely the frequency of errors is often not logically correct. It's useful in a society to have people believe in and push forward certain inventions and lines of research even if most do not work out.
I think xvector's point is about people rehashing the same denunciations that failed to matter for previous successful technologies - the idea that something is useless because it's not (or perhaps will never be) 100.0% accurate, or the "Until it can do dishes, home computer remains of little value to families"[0] which I've seen pretty much ad verbatim for AI many times (extra silly now that we have dishwashers).
Given in real life things have generally improved (standard of living, etc.), I think it has typically been more correct to be optimistic, and hopefully will be into the future.
[0]: https://pessimistsarchive.org/clippings/34991885.jpg
I'd love to hear more about how you utilised AI for this.
Personally I'm struggling to find it more useful than a slightly fancy code completion tool
Does this alone not increase your productivity exponentially? It does mine. I personally read code faster than I write it so it is an undeniable boon.
For example, personal projects that are small and where copilot has access to all the context it needs to make a suggestion - such as a script or small game - it has been really useful.
But in a real world large project for my day job, where it would need access to almost the entire code base to make any kind of useful suggestion that could help me build a feature, it's useless! And I'd argue this is when I need it.
Check out these tips for using Aider (a CLI based LLM code assisntant) https://aider.chat/docs/usage/tips.html
It can ingest the entire codebase (up to its context length), but for some reason, I’ve always had much higher quality chats with smaller bite-sized pieces of code.
I have no interest in learning the horrible unintuitive UX of every CLI I interact with, I'd much rather just describe in English what I want and have the computer figure it out for me. It has practically never failed me, and if it does I'll know right away and I can fall back to the old method of doing it manually. For now it's saving me so much time with menial, time-wasting day-to-day tasks.
But my counterargument is that what I find to be so powerful about the LLMs is the ability to refine my question, narrow in on a tangent and then pull back out, etc. And *then* I can take its final outcome and cross reference it. With the old way of doing things, I often felt like I was stumbling in the dark trying to find the right search string. Instead I can use the LLM to do the heavy lifting for me in that regard.
Most of those people are a bit bad at making their case. What they mean but don't convey well is that AI is useless for it's proclaimed uses.
You are correct that LLMs are pretty good at guessing this kind of well-documented & easily verifiable but hard to find information. That is a valid use. (Though, woe betide the fool who uses LLMs for irreversible destructive actions.)
The thing is though, this isn't enough. There just aren't that many questions out there that match those criteria. Generative AI is too expensive to serve that small a task. Charging a buck a question won't earn the $100 billion OpenAI needs to balance the books.
Your use case gets dismissed because on it's own, it doesn't sustain AI.
But it can be simultaneously true that LLMs add a lot of value to some tasks and less to others —- and less to some people. It’s a bit tautological, but in order to benefit from LLMs, you have to be in a context where you stand to most benefit from LLMs. These are people who need to generate ideas, are expert enough to spot consequential mistakes, know when to use LLMs and when not to. They have to be in a domain where the occasional mistake generated costs less than the new ideas generated, so they still come out ahead. It’s a bit paradoxical.
LLMs are good for: (1) bite-sized chunks of code; (2) ideating; (3) writing once-off code in tedious syntax that I don’t really care to learn (like making complex plots in seaborn or matplotllib); (4) adding docstrings and documentation to code; (5) figuring out console error messages, with suggestions as to causes (I’ve debugged a ton of errors this way — and have arrived at the answer faster than wading through Stackoverflow); (6) figuring out what algorithm to use in a particular situation; etc.
They’re not yet good at: (1) understanding complex codebases in their entirety (this is one of the overpromises; even Aider Chat’s docs tell you not to ingest the whole codebase); (2) any kind of fully automated task that needs to be 100% deterministic and correct (they’re assistants); (3) getting math reasoning 100% correct (but they can still open up new avenues for exploration that you’ve never even thought about);
It takes practice to know what LLMs are good at and what they’re not. If the initial stance is negativity rather than a growth mindset, then that practice never comes.
But it’s ok. The rest of us will keep on using LLMs and move on.
And it just isn't that thing. Or, rather, it is super intelligent but lacks any wisdom at all; thus rendering it useless for how it's being sold to me.
>which is at the early adoption stage
I've said this in other places here. LLM's simply aren't at early adoption stage anymore. They're being packaged into literally every saas you can buy. They're a main selling point for things like website builders and other direct to business software platforms.
I don’t use anything other than ChatGPT 4o and Claude Sonnet 3.5v2. That’s it. I’ve derived great value from just these two.
I even get wisdom from them too. I use them to analyze news, geopolitics, arguments around power structures, urban planning issues, privatization pros and cons, and Claude especially is able to give me the lay of the land which I am usually able to follow up on. This use case is more of the “better Google” variety rather than task-completion, and it does pretty well for the most part. Unlike ChatGPT, Claude will even push back when I make factually incorrect assertions. It will say “Let me correct you on that…”. Which I appreciate.
As long as I keep my critical thinking hat on, I am able to make good use of the lines of inquiry that they produce.
Same caveat applies even to human-produced content. I read the NYTimes and I know that it’s wrong a lot, so I have to trust but verify.
And it's just not.
We made a scavenger hunt full of puzzles and riddles for our neighbor's kids to find their Christmas gifts from us (we don't have kids at home anymore, so they fill that niche and are glad to because we go ballistic at Christmas and birthdays). The youngest of the group is the tech kid.
He thought he fixed us when he realized he could use chatgpt to solve the riddles and cyphers. It recognized the Caesar letter shift to negative 3, but then made up a random phrase with words the same length to solve it. So the process was right, but the outcome was just outlandishly incorrect. It wasted about a half hour of his day. . .
Now apply that to complex systems or just a simple large database, hell, even just a spreadsheet. You check the process, and it's correct. You don't know the outcome, so you can't verify unless you do it yourself. So what's the point?
For context, I absolutely use LLM's for things that I know roughly, but don't want to spend the time to do. They're useful for that.
They're simply not useful for how they're being marketed, which is too solve problems you don't already know.
Where are you being told all of these things? I haven't heard anything like it.
Pick a hammer, not a shitty hammer factory to assemble bits of hammer.
It's well and ok with things you can botch with no consequence other than some time wasted. But I've bricked enough VMs trying commands I did not understand to know that if you need to not fuck up something you'll have to read those docs and understand them. And hope they're not out of date / wrong.
The best ffmpeg and regex command generators
It's useful for that yes, but I'd rather just live in a world where we didn't have such disasters of CLI that are git and ffmpeg.
LLMs are very useful for generating the obscure boilerplate needed because the underlying design is horrible. Relying on it means acquiescing to those terrible designs rather than figuring out redesigns that don't need the LLMs. For comparison, IntelliJ is very good at automating all the boilerplate generation that Java imposes on me, but I'd rather we didn't have boilerplate languages like Java, and I'd rather that IntelliJ's boilerplate generation didn't exist.
I fear in many cases that if an LLM is solving your problem, you are solving the wrong problem.
These days, I'm more likely to read the manual pages and take notes on interesting bits. If I'm going to rely on some tooling for some time, dedicating a few hours of reading is a good trade-off for me. No need to even remember everything, just the general way it solves the problem. Anything more precise is captured in notes, scripts, shell history,... I dare anyone to comes out with an essay like this from LLMs: https://karthinks.com/software/avy-can-do-anything/
I'm asking not for snark, but because when AI gives me something not _quite_ working, it requires much more time than what a "every 6 minutes in 10 hour work day" frame would allow to investigate. I just wonder if maybe you're pasting it as is and don't care about correctness if the happy path sort of works. Speaking of subsets, coders who did that before AI were also quite a group.
There must be something that explains the difference in our experiences. Apologies for the fact that my only idea is kinda negative. I understand the potential hyperbola here, but it doesn't explain much. I can stand AI BS once a day, maybe twice, before uncontrollably cursing into the chat.
Don't start when you're already in a buggy dead-end. Test-driven development with LLMs should be done right from the start.
Also keep the code modular so it is easy to include the correct context. Fine-grained git commits. Feature-branches.
All the tools that help teams of humans of varying levels of expertise work together.
I'd also do code reviews on the code AI produces.
AI is a type of outsourcing, you became a customer.
https://news.ycombinator.com/item?id=42235630
Citation needed.
https://decrypt.co/246216/ai-could-make-coders-obsolete-in-t...
If you think you are obsolete or faster than anyone else with these tools then you only naive enough to have lost your objectivity to the marketing. I deal with real risk and failures from the output of ChatGPT which have serious financial consequences. The first victim is always the developer, then the tester.
At best, it is very good at ousting people who shouldn't be allowed anywhere near a damn computer.
Something/someone other writes the code, that's outsourcing.
I wouldn't consider myself an artist if I create a picture per midjourney.
Do you design all the NAND gates in your processor to get the exact program you want out of it or use a general purpose processor?
Current "coding" is just a detail of what you want to do: solve problems. Which can require making a machine do what you want it to.
A coder writes code in a programming language, that what distinguishes them from the customers who use natural language. The coder is the translator between the customer and the machine. If the machine does that, the machine is the coder.
The same type of argument has been made for decades -- when coders wrote in ASM, folks would ask "are you still a coder when you use that fancy C to make all that low-level ASM obsolete?". Etc etc.
Do consider script kiddies hackers?
In my experience, Claude Sonnet 3.5 (3.6?) has been unbeatable. I use it for Rust. Making sense of compiler errors, rubberducking, finding more efficient ways to write some function and, truth be told, some times just plain old debugging. More than once, I've been able to dump a massive module onto the chat context and say "look, I'm experiencing this weird behavior but it's really hard to pin down what's causing it in this code" and it pointed to the exact issue in a second. That alone is worth the price of admission.
Way better than ChatGPT 4o and o-1, in my experience, despite me saying the exact opposite a few months ago.
It's a valid conversation after ~3 years of anticipating the world to be disrupted by this tech. So far it has not delivered.
Wikipedia did not change the world either, it's just a great tool that I use all the time
As for software, it performs ok. I give up on it most of the time if I am trying to write a whole application. You have to acquire a new skill, prompt engineering, and feverish iteration. It's a frustrating game of whack-a-mole and I find it quicker to write the code myself and just have the LLM help me with architecture ideas, bug bashing, and it's also quite good at writing tests.
I'd rather know the code intimately so I can more quickly debug it than have an LLM write it and just trust it did it well.
LLMs obviously have use cases but the market has practically priced in "AGI".
The danger is not that LLMs take jobs. The danger is that we are in a massive bubble and while these are nice tools they are not worth anything close to the trillions of dollars bet on them.
IMO the psychology at work here is basically denial that we can both be in the biggest bubble of all time in terms of dollars and LLMs are useful. Just not THAT useful.
(cue rebuttal based on systemic consequences / financial bailouts etc, but you know what I mean; also, the dotcom bubble deflation didn't require a bailout)
If you want to argue otherwise, do a quick thought experiment first: would you let an LLM manage your financial affairs (entirely, unsupervised)? Would you let it perform your job while you receive the rewards and consequences? Would you be comfortable to give it full control of your smart home?
There are different sets of expectations put on human actors vs autonomous systems. We expect people to be fallible and wrong some of the time, even if the individuals in question can't/won't admit it. With a software-based system, the expectations are that it will be robust, tested, and performing correctly 100% of the time, and when a fault occurs, it will be clear, marked with yellow tape and flashing lights.
LLM-based AIs are sort of insidious in that they straddle this expectation gap: the emergent behaviour is erratic, projecting confident omniscience, while often hallucinating and plain wrong. However vague, the catch-all term "AI" still implies "computer system" and by extension "engineered and tested".
Now if you substituted something safety critical instead, say, running a nuclear power station, or my favourite currently in use example, self driving cars, then yes, you should be scared.
These are not LLMs but algorithms written and designed by human minds. It is unfortunate that AI has become a catch-all word for any kind of machine learning.
Not strictly true. There are patterns in the weights that could be steps in an algorithm.
The following is an algorithm:
- plug in input to model
- say yes if result is positive, else say no
LLMs use models, the model is not an algorithm.
> There are patterns in the weights that could be steps in an algorithm.
Sure, but yeah... no.. "Could be steps in an algorithm" does not constitute an algorithm.
Weights are inputs, they are not themselves parts of an algorithm. The algorithm might still try to come up with weights. Still, don't confuse procedure from data.
And, it is clear that LLMs can follow steps. One didn't place in the Math Olympiad without some ability to follow steps.
https://research.google/blog/teaching-language-models-to-rea...
And, Anyway, when I asked it, it said it could
"Yes, an LLM model can contain the steps of an algorithm, especially when prompted to "think step-by-step" or use a "chain-of-thought" approach, which allows it to break down a complex problem into smaller, more manageable steps and generate a solution by outlining each stage of the process in a logical sequence; essentially mimicking how a human would approach an algorithm. "
Okay.
> There is already evidence it can form a model of the world.
Perhaps.
> So why not something like steps to get from A to B.
Why not - because a model and algorithm are different. Simply having a model does not mean you have an algorithm. An algorithm is a deterministic set of steps, a model is typically a function or set of functions for producing results. If the result of that model is to list a set of steps (and also evaluate them too) - that does not make the model an algorithm.
> And, it is clear that LLMs can follow steps
Sure, because that is what the model is set up to do.
> Yes, an LLM model can contain the steps of an algorithm, especially when prompted to "think step-by-step" or use a "chain-of-thought" approach, which allows it to break down a complex problem into smaller
This is the model looking into its training data to find algorithms that seem to match the prompt and then to print out the steps of the algorithm and also execute them. That's not an algorithm in of itself.
I feel I'm on pretty solid ground here. "Algorithmic prompting" has nothing to do with whether a model is an algorithm. I'd ask you google the differences of a model and an algorithm very thoroughly. If something follows an algorithm, I strongly suspect it cannot be a model by definition. It can still be an AI though, as there are non LLM's AI's out there that do follow algorithms. If we are talking about LLM, the M is for "MODEL". Models and algorithms are different. A model that looks for an algorithm to use - is a very sophisicated model, but it's still not an algorithm itself just because it could find, interpret and use one.
If you think so, you should publish your results. It seems like a lot of bright people are going down the road of using LLM for algorithmic tasks. To follow steps.
I think what I'm reaching for, is a little more esoteric, that out of all the data the model is trained on, that it has also started building up algorithms/steps in its 'model', which is part of how it pics the next item.
The whole reason algorithmic prompting started was people started noticing the LLM was already attempting some steps, and that if it was further helped along by prompting the steps, then the results were better.
But, I am using 'algorithm' rather loosely, as just 'steps', and they are a bit fuzzy, so not a purely math algorithm, but more of a fuzzy logic, a first start at reasoning.
edit also, I should clarify. I am not confusing the algorithm to make the model versus the model, i'm saying in the model it learns to follow steps.
If your users are likely to be AI illiterate and mistakenly feel that an AI app is reliable and suitable for mission critical applications when it isn't, that is a risk you mitigate.
But it seems deeply unserious of the author to just assert that mission-critical software is the only "serious context" and the only thing matters, and therefore AI is dead end. "Serious, mission critical" apps are just going to be a niche in the future.
High quality only for niche "serious, mission critical" everywhere else, enshittification already started, LLMs will just accelerate it.
Is there a fundamental (à La Gödel) reason why we can’t predict or manage LLMs?
> would you let an LLM manage your financial affairs (entirely, unsupervised)?
No. But history is littered with eccentric geniuses who couldn’t be trusted on their own, but who nevertheless were relied on by decision makers.
Maybe there is an Erdős principle at play: AI can be involved in questions of arbitrary complexity, but can only advise on important decisions.
> We're building statistical models to make statistical predictions after training with a sufficiently large and statistically diverse dataset. Aka it's random.
This seems like a sort of hand wavy answer to what seems like a pretty deep and technical question. And to be fair, this isn’t the environment to ask that question, it seems like something a bunch of researchers would work out.
Of course we build statistical models all the time and then use them to make pretty good predictions. Is there something actually fundamental about these LL modes that makes them… unmanageable? Well, we’ll have to define manageable first… etc etc.
Great, What is M theory? This also is missing a structure to allow us to unify existing fundamental theories.
The only way to assess this is to stop treating the models as statistical beasts which simply only work with "enough" statistics and start talking about them from the direction of information theory. The answer is glib because the problem is the field requires a mix between stats, computing, and mathematics. 2 of these fields have their own languages for the problem (which differ, but exist) and computing has come along and made another set of names for the same complex things making the whole thing (for now) a mess... Especially with the main practicioners stuck in the view that big llm are simply money printers.
It will likely be better[2] not because AI is good at this .
It would be because study after study[1] has shown that active management performs poorer than passive funds, less intervention gives better result over longer timeframe .
[1] the famous warren buffet bet comes to mind . There are more formal ones validating this .
[2] if configured to do minimal changes
Most humans make very bad decisions around personal finance, whether it is big things like gambling or impulse buys with expensive credit, to smaller items like tracking subscriptions or keeping not needed money in checking account etc.
This is irrespective of financial literacy, education, wealth or professions like say working in finance/ personal wealth management even.
Entire industries like lottery, gambling, luxury goods, gaming, credit card APRs, Buy Now Pay Later, Consumer SaaS, Banking overdraft fees are all built around our inability to control our impulses or follow disciplined routines.
This is why trust funds with wealth management professionals are the only way to generational wealth.
You need the ability to control any benefactor (the next generations) from excising their impulses on amounts beyond their annual draw. Plus the disciplined routine of a professional team who are paid to do only this with multiple layers that vet the impulses of individual managers and conservative mandate to keep them risk averse and therefore less impulsive.
If an program can do it for me (provided of course I irrevocably give away my control to override or alter its decisions) then normal people can also benefit without the high net worth required for wealth management.
My lived experience the software industry at almost all levels over the last 25 years leads me to believe that the vast majority of humans and teams of humans produce atrocious code that only wastes time, money, and people's patience.
Often because it is humans producing the code, other humans are not willing to fully engage, criticize and improve that code, deferring to just passing it on to the next person, team, generation, whatever.
Yes, this perhaps happens better in some (very large and very small) organizations, but most often it only happens with the inclusions of horrendous layers of protocol, bureaucracy, more time, more emotional exhaustion, etc.
In other words a very costly process to produce excellent code, both in real capital and human capital. It literally burns through actual humans and results in very bad health outcomes for most people in the industry, ranging from minor stuff to really major things.
The reality is that probably 80% of people working in the tech industry can be outperformed by an AI and at a fraction of the cost. AIs can be tuned, guided, and steered to produce code that I would call exception compared even to most developers who have been in the field for 5years or more.
You probably come to this fallacy because you have worked in one of these very small or very large companies that takes producing code seriously and believe that your experience represents the vast majority of the industry, but in fact the middle area is where most code is being "produced" and if you've never been fully engaged in those situations, you may literally have no idea of the crap that's being produced and shipped on a daily basis. These companies have no incentive to change, they make lots of money doing this, and fresh meat (humans) is relatively easy to come by.
Most of these AI benchmarks are trying to get these LLMs to produce outputs at the scale and quantity of one of these exceptional organizations when in fact, the real benefits will come in the bulk of organizations that cannot do this stuff and AI will produce as good or better code than a team of mediocre developers slogging away in a mediocre, but profitable, company.
Yes there are higher levels of abstraction around code, and getting it deployed, comprehensive testing, triaging issues, QA blah blah, that humans are going to be better at for now, but I see many of those issues being addressed by some kind of LLM system sooner or later.
Finally, I think most of the friction people are seeing right now in their organization is because of the wildly ad hoc way people and organizations are using AI, not so much about the technological abilities of the models themselves.
For "stay in your lane" stuff, I agree, it relatively sucks.
For "today I need do stuff two lanes over", well it still needs the babysitting, and I still wouldn't put it on tasks where I can't verify the output, but it definitely delivers a productivity boost IME.
The older or smaller models, like anything you can run locally, are probably far more likely to just invent some bullshit.
That said, I've certainly asked ChatGPT about things that definitely have a correct answer and had it give me incorrect information.
When talking about hallucinating, I do think we need to differentiate between "what you asked about exists and has a correct answer, but the AI got it wrong" and "What you're asking for does not exist or does not have an answer, but the AI just generated some bullshit".
For example: https://arxiv.org/abs/2412.15176
isn't this what being a human manager is? not sure why you're saying it must be entirely + unsupervised. at my job, my boss mostly trusts me but still checks my work and gives me feedback when he wants something changed. he's ultimately responsible for what I do.
Detecting omissions or errors on prepared tax forms often requires knowledge of context missed by or not provided to the accountant.
1. Do you believe that LLMs operate in a similar way to the important parts of human cognition?
2. If not, do you believe that they operate in a way that makes them useful for tasks other than responding to text prompts, and if so, what are those tasks?
If you believe that the answer to Q1 is substantively "yes" - that is, humans and LLM are engaged in the same sort of computational behavior when we engage in speech generation - then there's presumably no particular impediment to using an LLM where you might otherwise use a human (and with the same caveats).
My own answer is that while some human speech behavior is possibly generated by systems that function in a semantically equivalent way to current LLMs, human cognition is capable of tasks that LLMs cannot perform de novo even if they can give the illusion of doing so (primarily causal chain reasoning). Consequently, LLMs are not in any real sense equivalent to a human being, and using them as such is a mistake.
As an aside, Students of Peirce over the years have quite the pedigree in data science too, including the genius Edgar F. Codd, who invented the relational database largely inspired by Peirce's approach to relations.
Anyhow, computers are already quite good at corollarial reasoning and have been for some time, even before LLMs. On the other hand, they struggle with theorematic reasoning. Last I knew, the absolute state of the art performs about as well as a smart high school student. And even there, the tests are synthetic, so how theorematic they truly are is questionable. I wouldn't rule out the possibility of some automaton proposing a better explanation for gravitational anomalies than dark matter for example, but so far as I know nothing like that is being done yet.
There's also the interesting question of whether or not an LLM that produces a sequence of tokens that induces a genuine insight in the human reader actually means the LLM itself had said insight.
[1] https://www.cspeirce.com/menu/library/bycsp/l75/ver1/l75v1-0...
[2] https://groups.google.com/g/cybcom/c/Es8Bh0U2Vcg
In the workplace, humans are ultimately a tool to achieve a goal. LLM's don't have to be equivalent to humans to replace a human - they just have to be able to achieve the goal that the human has. 'Human' cognition likely isn't required for a huge amount of the work humans do. Heck, AI probably isn't required to automate a lot of the work that humans do, but it will accelerate how much can be automated and reduce the cost of automation.
So it depends what we mean as 'use them as a human being' - we are using human beings to do tasks, be it solving a billing dispute for a customer, processing a customers insurance claim, or reading through legal discovery. These aren't intrinsically 'human' tasks.
So 2 - yes, I do believe that they operate in a way that makes them useful for tasks. LLM's just respond to text prompts, but those text prompts can do useful things that humans are currently doing.
I think the vector representation stuff is an effective tool and possibly similar to foundational tools that humans are using.
But my gut feel is that it's just one tool of many that combine to give humans a model+view of the world with some level of visibility into the "correctness" of ideas about that world.
Meaning we have a sense of whether new info "adds up" or not, and we may reject the info or adjust our model.
I think LLM's in their current state can be useful for tasks that do not have a high cost resulting from incorrect output, or tasks that can have their output validated by humans or some other system cost-effectively.
I believe they operate in a way that makes them at least somewhat useful for some things. But I think the big issue is trustworthiness. Humans - at least some of them - are more trustworthy than LLM-style AIs (at least current ones). LLMs need progress on trustworthiness more than they need progress on use in other areas.
Hmm I would not let other human manage my financial affairs entirely unsupervised.
No, but I also would not let another person do that.
It is telling that you needed to interject "entirely, unsupervised".
Most people will let an llm do it partially, and probably already do.
Mostly your financial advisor writes your return you sign off on or manages your portfolio. But the advisor usually solicits and interacts with you to know what your financial goals are and ensure you are on board with the consequences of their advice.
I do not dismiss that some people are completely hands off at great risk IMHO. But these are not me - as was my initial proposition.
I wouldn't let another human do this.
_Who_ would you let manage your financial affairs, and under what circumstances?
To which my answer would be something like: a qualified financial adviser with a good track record, who can be trusted to do the job to, if not the best of their abilities, at least an acceptable level of professional competence.
A related question: who would you let give you a lift someplace in a car?
And here's where things get interesting. Because on the one hand there's a LOT more at stake (literally, your life), and yet various social norms, conventions , economic pressures and so on mean that in practice we quite often entrust that responsibility to people who are very, very far from performing at their best.
So while a financial adviser AI is useless unless it can perform at the level of a trained professional doing their job (or unless it can perform at maybe 95% of that level at much lower cost), a self-driving car is at least _potentially_ useful if it's only somewhat better than people at or close to their worst. As a high proportion of road traffic collisions are caused by people who are drunk, tired, emotionally unstable or otherwise very very far from the peak performance of a human being operating a car.
(We can argue that a system which routinely requires people to carry out life-or-death, mission-critical tasks while significantly impaired is dangerously flawed and needs a major overhaul, but that's a slightly different debate).
1. You can set a bar wherever you want for a level of "seriousness" and huge swathes of real world work will fall below it, and are therefore attractive to tackle with these systems.
2. We build critical large scale systems out of humans, which are fallible and unverifiable. That's not to say current LLMs are human or equivalent, but "we can't verify X works all the time" doesn't stop us doing exactly that a lot. We deal with this by learning how humans make mistakes, why, and build systems of checks around that. There is nothing in my ind that stops us doing the same with other AI systems.
3. Software is written by, checked by and verified by humans at least at some critical point - so even verified software still has this same problem.
We've also been doing this kind of thing with ML models for ages, and we use buggy systems for an enormous amount of work worldwide. You can argue we shouldn't and should have fully formally verified systems for everything, but you can't deny that right now we have large serious systems without that.
And if your goal is "replace a human" then I just don't think you can reasonably say that it requires verifiable software.
> Systems are not explainable, as they have no model of knowledge and no representation of any ‘reasoning’.
Neither of those statements are true are they? There are internal models, and recent models are designed around having a representation of reasoning before replying.
> current generative AI systems represent a dead end, where exponential increases of training data and effort will give us modest increases in impressive plausibility but no foundational increase in reliability
And yet reliability is something we see improve as LLMs get better and we get better at training them.
I think there is room for statistical AI to operate symbolic systems so we can better control outputs. Actually, that's kind of what is going on when we ask AI to write code.
Kind of the way our right brain hemisphere does probabilistic computation and the left brain hemisphere does atomistic computation. And we use both.
So, whoever develops the digital equivalent of the corpus callosum wins.
Everything from perplexity onward shows just how useful agents can be.
You get another bump in utility when you allow for agents swarms.
Then another one for dynamically generated agent swarms.
The only reason why it's not coming for your job is that LLMs are currently too power hungry to run those jobs for anything but research - at a couple thousand to couple of million times the price of a human doing the work.
Which works out to 10 to 20 epochs of whatever Moore's law looks like in graphics cards.
Apparently they don't believe that AI is about to revolutionize things that much. This makes me believe that significant part of the AI investment is just FOMO driven, so no real revolution is around the corner.
Although we keep seeing claims that AI achieved PHD level this Olympics level that, people who actually own these keep demanding immigration policy changes to bring actual humans from overseas for year to come.
Sorry for the potentially silly question. I just spent some time trying to research it and came up with nothing concrete.
I'm speculating too but yes it appears that unemployment is pretty high among the CS majors: https://www.reddit.com/r/csMajors/comments/1hhl060/how_is_it...
But at the same time there's an ongoing infighting among Trump supporters because tech elites came up as pro - skilled immigration where the MAGA camp turned against them. The tech elites claim that there's a talent shortage. Here's a short rundown that Elon Musk agrees with: https://x.com/AutismCapital/status/1872408010653589799
The unemployment data is from 2018 BTW. But from what I perceive, developer unemployment in the US seems higher than usual right now.
They are not replacing their workers despite claiming that AI is currently as good as a PHD and they certainly don't go to AI medical doctors despite claiming that their tool is better than most doctors.
The whole conversation is so dishonest.
Every software firm, notable and small, has had layoffs over the past two years, but somehow there's still a "STEM shortage" and companies are "starving for talent" or some such nonsense?
Fake discussion.
As we continue to innovate, a focus on explainability, fairness, and accountability in AI systems will be paramount to harnessing their potential without compromising societal values.
Do you have an example of this?
That said, we can still isolate and modify parts of a network, and combine models trained for different tasks. But you need to break things down into components after the fact, instead of beforehand, in order to get the benefits of learning via scale of data + compute.
[1]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.
The largest AI systems in use today have hundreds of billions of parameters. Nearly all parameters are part of a weight matrix, each parameter quantifying the strength of the connection from an artificial input neuron to an artificial output neuron. The human brain has more than a hundred trillion synapses, each connecting an organic input neuron to an organic output neuron, but the comparison is not apples-to-apples, because each synapse is much more complex than a single parameter in a weight matrix.[a]
Today's largest AI systems have about the same number of neuron-neuron connections as the brain of a brown rat.[a] Judging these AI systems based on their current capabilities is like judging organic brains based on the capabilities of brown rat brains.
What we can say with certainty is that today's AI systems cannot be trusted to be reliable. That's true for highly trained brown rats too.
---
[a] https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... -- sort in descending order by number of synapses.
This doesn’t solve the unpredictability problem.
But we haven't solved it for human beings either.
Human brains are unpredictable. Look around you.
While individual human beings do trust some of the other human beings they know, in the aggregate society doesn't seem to trust human beings to behave reliably.
It's possible, though I don't know for sure, that we're going to need systems and processes to cope with the unpredictability of AI systems.
As it was mentioned by others, we've had thousands of years to better understand how humans can fail. LLMs are black boxes and it never ceases to amaze me how they can fail in such unpredictable ways. Take the following for examples.
Here GPT-4o mini is asked to calculate 2+3+5
https://beta.gitsense.com/?chat=8707acda-e6d4-4f69-9c09-2cff...
It gets the answer correct, but if you ask it to verify its own answer
https://beta.gitsense.com/?chat=6d8af370-1ae6-4a36-961d-2902...
it says the response was wrong, and contradicts itself. Now if you ask it to compare all the responses
https://beta.gitsense.com/?chat=1c162c40-47ea-419d-af7a-a30a...
it correctly identifies that GPT-4o mini was incorrect.
It is this unpredictable nature that makes LLM insanely powerful and scary.
Note: The chat on the beta site doesn't work.
Humans are EXTREMELY unpredictable. Humans only become slightly more predictable and producers of slightly more quality outputs with insane levels of bureaucracy and layers upon layers upon layers of humans to smooth it out.
To boot, the production of this mediocre code is very very very slow compared to LLMs. LLMs also have no feelings, egos, and are literally tunable and directible to produce better outcomes without hurting people in the process (again, something that is very difficult to avoid without the inclusion of, yep, more humans more layers, more protocol etc.)
Even with all of this mass of human grist, in my opinion, the output of purely human intellects is, on average, very bad. Very bad in terms of quality of output and very bad in terms of outcomes for the humans involved in this machine.
I think to make it to the next step, AI will have to have some way of performing rigorous logic integrated on a low level.
Maybe scaling that brown-rat brain will let it emulate an internal logical black box - much like the old adage about a sufficiently large C codebase containing an imperfect Lisp implementation - but I think things will get really cool we figure out how to wire together something like Wolfram Alpha, a programming language, some databases with lots of actual facts (as opposed to encoded/learned ones), and ChatGPT.
https://news.ycombinator.com/item?id=42449424
It still won't magically use them 100% correctly, but with a bit of smarts you can go a long way!
Meanwhile, Wolfram software has built-in methods to solve a lot of different math problems for which in Python you would either need large (and sometimes quirky) libraries, if those libraries even exist.
That and you need to actually expose python to GPT somehow, and Jupyter is not the worst way I suppose.
* The fact that Jupyter holds on to state means GPT doesn't need to write code from scratch for every step of the process.
* GPT can easily read back through the workbook to review errors or output from computations. GPT actually tries to correct errors even. Especially if it knows how to identify them.
To be sure, this is not magic. Consider it more like a tool with limited intelligence; but which can be controlled using natural language.
(Meanwhile, Anthropic allows Claude to run js with react, which is nice but seems less flexible in practice. I'm not sure Claude reads back.)
I don't know if the hardware can be scaled up. That's why I wrote "if we're able to scale them" at the root of this thread.
I'm pretty skeptical of the scaling hypothesis, but I also think there is a huge amount of efficiency improvement runway left to go.
I think it's more likely that the return to further scaling will become net negative at some point, and then the efficiency gains will no longer be focused on doing more with more but rather doing the same amount with less.
But it's definitely an unknown at this point, from my perspective. I may be very wrong about that.
Whether those approaches can scale enough to achieve that is relevant to the question, whether the bottleneck is in hardware or software.
I wager that some scrappy resource constrained startup or research institute will find a way to produce results that are similar to those generated by these ever massive LLM projects only at a fraction of the cost. And I think they’ll do that by pruning the shit out of the model. You don’t need to waste model space on ancient Roman history or the entire canon for the marvel cinematic universe on a model designed to refactor code. You need a model that is fluent in English and “code”.
I think the future will be tightly focused models that can run on inexpensive hardware. And unlike today where only the richest companies on the planet can afford training, anybody with enough inclination will be able to train them. (And you can go on a huge tangent why such a thing is absolutely crucial to a free society)
I dunno. My point is, there is little incentive for these huge companies to “think small”. They have virtually unlimited budgets and so all operate under the idea that more is better. That isn’t gonna be “the answer”… they are all gonna get instantly blindsided by some group who does more with significantly less. These small scrappy models and the institutes and companies behind them will eventually replace the old guard. It’s a tale as old as time.
Six million is a start but this tech won’t truly be democratized until it costs $1000.
Obviously I’m being a little cheeky but my real point is… the idea that this technology is in the control of massive technology companies is dystopian as fuck. Where is the RMS of the LLM space? Who is shouting from every rooftop how dangerous it is to grant so much power and control over information to a handful of massive tech companies, all whom have long histories of caving into various government demands. It’s scary as fuck.
Human intelligence improved dramatically after we improved our ability to extract nutrients from food via cooking
https://www.scientificamerican.com/article/food-for-thought-...
We can put a lot more power flux through an AI than a human body can live through; both because computers can run hot enough to cook us, and because they can be physically distributed in ways that we can't survive.
That doesn't mean there's no constraint, it's just that the extent to which there is a constraint, the constraint is way, way above what humans can consume directly.
Also, electricity is much cheaper than humans. To give a worked example, consider that the UN poverty threshold* is about US$2.15/day in 2022 money, or just under 9¢/hour. My first Google search result for "average cost of electricity in the usa" says "16.54 cents per kWh", which means the UN poverty threshold human lives on a price equivalent ~= just under 542 watts of average American electricity.
The actual power consumption of a human is 2000-2500 kcal/day ~= 96.85-121.1 watts ~= about a fifth of that. In certain narrow domains, AI already makes human labour uneconomic… though fortunately for the ongoing payment of bills, it's currently only that combination of good-and-cheap in narrow domains, not generally.
* I use this standard so nobody suggests outsourcing somewhere cheaper.
The average brown rat may use only 60 kcal per day, but the maximum firing rate of biological neurons is about 100-1000 Hz rather than the A100 clock speed of about 1.5 GHz*, so the silicon gets through the same data set something like 1.5e6-1.5e7 times faster than a rat could.
Scaling up to account for the speed difference, the rat starts looking comparable to a 9e7 - 9e8 kcal/day, or 4.4 to 44 megawatts, computer.
* and the transistors within the A100 are themselves much faster, because clock speed is ~ how long it takes for all chained transistors to flip in the most complex single-clock-cycle operation
Also I'm not totally confident about my comparison because I don't know how wide the data path is, how many different simultaneous inputs a rat or a transformer learns from
Only a small part of that 60kcal is used for learning, and for that same 60 kcal you get an actual physical being that is able to procreate, eat, do things and fend for and maintain itself.
Also you cannot compare neuron firing rates with clockspeed. Afaik each neuron in a ml-model can have code that takes several clock cycles to complete.
Also an neuron in ml is just a weighted value, a biological neuron does much more than that. For example neurons communicate using neuro transmitters as well as using voltage potentials. The actual date rate of biological neurons is therfore much higher and complex.
Basically your analogy is false because your napkin-math basically forgets that the rat is an actual biological rat and not something as neatly defined as a computer chip
The conclusion does not follow from the premise. The observed maximum rate of the inter-neuron communication is important, the mechanism is not.
> Also you cannot compare neuron firing rates with clockspeed. Afaik each neuron in a ml-model can have code that takes several clock cycles to complete.
Depends how you're doing it.
Jupyter notebook? Python in general? Sure.
A100s etc., not so much — those are specialist systems designed for this task:
"""1024 dense FP16/FP32 FMA operations per clock""" - https://images.nvidia.com/aem-dam/en-zz/Solutions/data-cente...
"FMA" meaning "fused multiply-add". It's the unit that matters for synapse-equivalents.
(Even that doesn't mean they're perfect fits: IMO a "perfect fit" would likely be using transistors as analog rather than digital elements, and then you get to run them at the native transistor speed of ~100 GHz or so and don't worry too much about how many bits you need to represent the now-analog weights and biases, but that's one of those things which is easy to say from a comfortable armchair and very hard to turn into silicon).
> Basically your analogy is false because your napkin-math basically forgets that the rat is an actual biological rat and not something as neatly defined as a computer chip
Any of those biological functions that don't correspond to intelligence, make the comparison more extreme in favour of the computer.
This is, after all, a question of their mere intelligence, not how well LLMs (or indeed any AI) do or don't function as von Neumann replicators, which is where things like "procreate, eat, do things and fend for and maintain itself" would actually matter.
Have you thought about stepping back from all of this for a few days and notice that you are wasting your time with these arguments? It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change.
NNs as of right now are the equivalent of a brain scan. You can simulate how that brain scan would answer a question, but the moment you close the Q and A session, you will have to start from scratch. Making higher resolution brain scans may help you get more precise answers to more questions, but it will never change the questions that it can answer after you have made the brain scan.
Num fecisti?
> It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change.
That's a deliberate choice, not a fundamental requirement.
Models get frozen in order to become a product someone can put a version number on and ship, not because they must be, as demonstrated both by fine-tuning and by the initial training process — both of which update the weights.
> NNs as of right now are the equivalent of a brain scan.
First: see above.
Second: even if it were, so what? Look at the context I'm replying to, this is about energy efficiency — and applies just fine even when calculated for training the whole thing from scratch.
To put it another way: how long would it take a mouse to read 13 trillion tokens?
The energy cost of silicon vs. biology is lower than people realise, because people read the power consumption without considering that the speed of silicon is much higher: at the lowest level, the speed of silicon computation literally — not metaphorically, really literally — outpaces biological computation by the same magnitude to which jogging outpaces continental drift.
Neurons do so much more than a single math operation. A single cell can act as an intelligent little animal on its own, they are nothing like a neural network "neuron".
And note that all neurons act in parallel, so they are billions times more parallel than GPU's even if the operations would be the same.
> A single cell can act as an intelligent little animal on its own, they are nothing like a neural network "neuron".
Unless any of those things contribute to human intelligence, they do not matter in this context.
Cool, sure. Interesting, yes. But only important to exactly the degree to which any of that makes the human they're in smarter or dumber.
To the extent they're independently intelligent, they're the homunculi in Searle's Chinese Room.
> And note that all neurons act in parallel, so they are billions times more parallel than GPU's even if the operations would be the same.
Order of ten million times faster on a linear basis while still a thousand parallel operations.
SpiNNaker needs 100kWh to simulate one billion neurons. So the rat wins in terms of energy efficiency.
> and they tend to be significantly more efficient
Surely you noticed that this claim is false, just from your own next line saying it needing 100 kW (not "kWh" but I assume that's auto-corrupt) for a mere billion?
Even accounting for how neuron != synapse — one weight is closer to a single synapse; a brown rat has 200e6 neurons and about 450e9 synapses — the stated 100 kW for SpiNNaker is enough to easily drive simpler perceptron-type models of that scale, much faster than "real time".
I think current LLMs may scale the same way and become very powerful, even if not as energy-efficient as an animal's brain.
In practice, we humans, when we have a technology that is good enough to be generally useful, tend to adopt it as it is. We scale it to fit our needs and perfect it while retaining the original architecture.
This is what happened with cars. Once we had the thermal engine, a battery capable of starting the engine, and tires, the whole industry called it "done" and simply kept this technology despite its shortcomings. The industry invested heavily to scale and mass-produce things that work and people want.
I mean we have dogs. We really like them. For ages, they did lots of useful work for us. They aren’t that much smarter than rats, right? They are better aligned and have a more useful shape. But it isn’t obvious (to me at least) that the rats’ problem is insufficient brainpower.
I mean dogs came with us to the Americas, and even to Australia. Both the Norse and the Inuit took dogs with them to Greenland.
I think LLMs and modern AI are incredibly amazing and useful tools, but even with the top SOA models today it becomes clearer to me the more I use them that they are fundamentally lacking crucial components of what average people consider "intelligence". I'm using quotes deliberately because the debate about "what is intelligence" feels like it can go in circles endlessly - I'd just say that the core concept of what we consider understanding, especially as it applies to creating and exploring novel concepts that aren't just a mashup of previous training examples, appears to be sorely missing from LLMs.
This is less far-fetched than it sounds. Search for "organic deep neural networks" online.
Networks of rat neurons have in fact been trained to fly planes, in simulators, among other things.
I do wonder if the most energy-efficient way to scale up AI models is by implementing them in organic substrates.
There is no modern AI system that can go into your house and find a piece of cheese.
The whole notion that modern AI is somehow "intelligent", yet can't tell me where the dishwasher is in my house is hilarious. My 3 year old son can tell me where the dishwasher is. A well trained dog could do so.
It's the result of a nerdy definition of "intelligence" which excludes anything to do with common sense, street smarts, emotional intelligence, or creativity (last one might be debatable but I've found it extremely difficult to prompt AI to write amazingly unique and creative stories reliably)
The AI systems need bodies to actually learn these things.
"Neurons, synapses, signals, chemicals" are downstream of that.
Or just saying, when facing the apocalypse, read a bible?
Or if we're comparing to a three year old, also find the dishwasher?
Closest I'm aware of is something by Boston Dynamics or Tesla, but neither would be as simple as asking it- wheres the dishwasher in my home?
And then if we compare it to a ten year old, find the woodstove in my home, tell me the temperature, and adjust the air intake appropriately.
And so on.
I'm not saying it's impossible. I'm saying there's no AI system that has this physical intelligence yet, because the robot technology isn't well developed/integrated yet.
For AI to be something more than a nerd it needs a body and I'm aware there are people working on it. Ironically, not the people claiming to be in search of AGI.
But that won't be the same model that writes bad poetry or tries to autocomplete your next line of code. Or control the legs of a robot to move towards the dishwasher while holding a dirty plate. And each has a fair bit of manual tuning and preprocessing based on its function which may simply not be applicable to other areas even with scale. The best performing models aren't just taking in unstructured untyped data.
Even the most flexible models are only tackling a small slice of what "intelligence" is.
https://www.youtube.com/watch?v=KwNUJ69RbwY
"The AI systems need bodies to actually learn these things."
I never said this was impossible to achieve.
https://sktchd.com/column/comics-disassembled-ten-things-of-...
They can describe it. But do they actually know? Have they experienced it?
This is my point. Nerds keep dismissing physicality and experience.
If your argument is a brain in a jar will be generally intelligent, I think that's pretty clearly wrong.
And that the only reason we think AIs can just be brains in jars is because many of the people developing them consider themselves as simply brains in jars.
> Do they know what a hot shower feels like? They can describe it. But do they actually know? Have they experienced it
Is directly a knowledge argument
Also, it is an argument against physicalism, which I have no interest in debating. While it's tangentially related, my point is not for/against physicalism.
My argument is about modern AI and it's ability to learn. If we put touch sensors, eyes, nose, a mechanism to collect physical data (legs) and even sex organs on an AI system, then it is more generally intelligent than before. It will have learned in a better fashion what a hot shower feels like and will be smarter for it.
I really disagree. Your entire point is about physicalism. If physicalism is true than an AI does not necessarily learn in a better fashion what a hot shower feels like by being embodied. In a physicalist world it is conceivable to experience that synthetically.
Perhaps your own internal definition of intelligence simply deviates significantly from the common, "median" definition.
Here is what we know: The Pile web scrape is 800GB. 20 years of human experience at 1kB/sec is 600GB. Maybe 1kB/sec is bad estimate. Maybe sensory input is more valuable than written text. You can convince me. But next challenge, some 10^15 seconds of currently existing youtube video, that's 2 million years of audiovisual experience, or 10^9GB at the same 1kB/sec.
Another commenter also mentioned sensory input when talking about the brown rat. As someone who is constantly fascinated at the brains ability to reason/process stuff before I'm even conscious of it, I feel this Stat is Underrated. I'm taking in and monitoring like 15 sensations of touch at all time. Something entering my visual field coming towards me can be deflected in half a second all while still understanding the rest of my surroundings, and where it might be safe to deflect an object. The brain is constantly calculating depth perception and stereo location on every image and sound we hear - also with the ability to screen out the junk or alter our perception accurately(knowing the correct color of items regardless of diff in color temp).
I do concede that's a heck of a lot of video data. It does have similar issues to what I said(lacks touch, often no real stereo location, good greenscreen might convince an AI of something a person intuitively knows is impossible) but the scale alone certainly adds a lot. That could potentially make up for what I see as a hugely overlooked thing as far as stimulus. I am monitoring and adjusting like, hundreds of parameters a second subconsciously. Like everything in my visual field. I don't think it can be quantified accurately how many things we consciously and subconsciously process, but I have the feeling it's a staggering amount.
There are areas of utility here. Things need not be able to do all actions to be useful.
[0] The classic reference: https://en.wikipedia.org/wiki/Principia_Mathematica -- over 1,000 pages, Betrand Russell
[1] https://cmartinez.web.wesleyan.edu/documents/FP.pdf -- a bit more modern, relying on other mathematics under the hood (like DRY reduces the base count), 11 pages
[2] https://xkcd.com/1053/
[3] Some reasonable review https://blog.plover.com/math/PM.html
I teach multiple things online and in person... language like that seems like a great to lose a student. I'd quit as a student, it's so condescending sounding. It's only lucky because you get to flex ur knowledge!(jk, pushing it I know lol but i can def see it being taken that way)
Keep in mind I know you're just having fun.
I actually really like the message for 1 in 10,000. As a social outsider for much of my life, it helped me to learn that the way people dismissed my questions about common (to them) topics was more about their empathy and less about me.
But, these sorts of things are difficult to communicate via text media, so we thus persist.
On a side note my couple of times I thought I was treating someone to some great knowledge they should already know I'm pretty sure I came across as condescending. Not bc they didn't know it - i always aim to be super polite - just being young, stupid, and bad at communicating, heh.
And had Russell failed to prove that 1 + 1 = 2 in his system, it would not have cast one jot of doubt on the fact that 1 + 1 = 2. It would only have pointed to the inadequacy of the Principia.
There is an essential idea of what makes something a dishwasher that LLM's will never be able to grasp no matter how many models you throw at them. They would have to fundamentally understand that what they are "seeing" is an electronic appliance connected to the plumbing that washes dishes. The sound of a running dishwasher, the heat you feel when you open one, and the wet, clean dishes is also part of that understanding.
Can you tell this is a dishwasher? https://www.amazon.com.au/Countertop-Dishwasher-Automatic-Ve...
Can you tell this is a drawing of a glass? https://www.deviantart.com/januarysnow13/art/Wine-Glass-Hype...
Can you tell this is a toy? https://www.amazon.com.au/Theo-Klein-Miele-Washing-Machine/d...
Or would you say that you cannot judge the intellegence of someone by reading their books / exchanging emails with them?
My dad is Deaf and doesn't write well, but he can build a beautiful house.
Currently the AI stuff kind of sucks because you have to be a giant corp to train a model. But maybe in a decade, users will be able to train their own models or at least fine-tune on basic cellphone and laptop (not dgpu) chips.
I mean, you don't have to look any further than the (justified) lack of sympathy to dockworkers just a few months ago: https://news.ycombinator.com/item?id=41704618
The solution is not, and never has been, to shack up with the capital-c Capitalists in defense of copyright. It's to push for a system where having your "work" automated away is a relief, not a death sentence.
I would engage with those people you're stereotyping rather than gossiping in a comments section, I suspect you will find their ideologies quite consistent once you tease out the details.
It shouldn't be too surprising that anti-establishment folks are more concerned with trillion-dollar companies subsuming and profiting from the work of independent artists, writers, developers, etc., than with individual people taking IP owned by multimillion/billion-dollar companies. Especially when many of the companies in the latter group are infamous for passing only a tiny portion of the money charged onto the people doing the actual creative work.
Tech still acts like it's the scrappy underdog, the computer in the broom cupboard where "the net" is a third space separate from reality, nerds and punks writing 16-bit games.
That ceased to be materially true around twenty years ago now. Once Facebook and smart phones arrived, computing touched every aspect of peoples' lives. When tech is all-pervasive, the internal logic and culture of tech isn't sufficient to describe or understand what matters.
Winning by what metric?
It's also orthogonal to the current corporate dystopia which is using monopoly power to enclose the value of individual work from the other end - precisely by inserting itself into the process of physical distribution.
None of this matters if you have a true abundance economy, but we don't. Pretending we do for purely selfish reasons - "I want this, and I don't see why I should pay the creator for it" - is no different to all the other ways that employers stiff their employees.
I don't mean it's analogous, I mean it's exactly the same entitled mindset which is having such a catastrophic effect on everything at the moment.
Remember Napster? Like how rebellious was that shit? Those times are what a true social upsetting tech looks like.
You cannot even import a video into OpenAI’s Sora without agreeing to a four (five?) checkbox terms & conditions screen. These LLM’s come out of the box neutered by corporate lawyers and various other safety weenies.
This shit isn’t real until there are mainsteam media articles expressing outrage because some “dangerous group of dark web hackers finished training a model at home that very high school student on the planet can use to cheat on their homework” or something like that. Basically it ain’t real until it actually challenges The Man. That isn’t happening until this tech is able to be trained and inferenced from home computers.
I think the training process constitutes commercial use.
great if AI accelerates its destruction (even if it's through lobbying to our mafia-style protect-the-richest-company governments)
Unfortunately this is not the way it's developing. It's more like: are you a normal person without deep pockets? Download a movie with Bittorrent and get a steep fine. Are you a company with hundreds of millions? Download half the copyrighted material on the internet, it's fine.
We are increasingly shifting to a society where the rules only don't apply when you have capital. To some extend, this has always been true, but the scale is changing.
Is the result of an llm an accurate copy or more of an inspiration? What is the standard we use on humans?
Can we code that determination into a system that when a piece of content is close enough to be a copyrighted work, prevents the llm from generating it?
As I learned it, artificial neural networks were modeled after a simple model for the brain. The early (successful) models were almost all reinforcement models, which is also one of the most successful model for animal (including human) learning.
Is your point that the capabilities of these models have grown such that 'merely' calling it a neural network doesn't fit the capabilities?
Or is your point that these models are called neural networks even though biological neural networks are much more complex and so we should use a different term to differentiate the simulated from the biological ?
This kind of comparison isn't a new idea. I think Hans Moravec[a] was the first to start making these kinds of machine-to-organic-brain comparisons, back in the 1990's, using "millions of instructions per second" (MIPS) and "megabytes of storage" as his units.
You can read Moravec's reasoning and predictions here:
https://www.jetpress.org/volume1/moravec.pdf
---
[a] https://en.wikipedia.org/wiki/Hans_Moravec
That's a different take than "human level is this many mips and megabytes", i.e. his claims are about artificial intelligence, not about biological intelligence.
The machine learning seems to be modeled after the action potential part of neural communication. But biological neurons can communicate also in different ways, i.e. neuro transmitters. Afaik this isn't modeled in the current ml-models at all (neither do we have a good idea how/why that stuff works). So ultimately it's pretty likely that a ml with a billion parameters does not perform the same as an organic brain with a billion synapses
Your Moravec article is only looking at what's necessary for computers to have the processing power of animal brains. But you've been up and down this thread arguing that equivalent processing power could be sufficient for a computer to achieve the intelligence of an animal. Necessary vs sufficient is big distinction.
Given their current scale, I don't think we can judge whether current AI systems "represent a dead end" -- or not.
If we were dogs surely we would say that humans were quite skillful, impressively so even, in pattern matching, abstract thought, language, etc. but are hopelessly dumb at predicting past presence via smell, a crow would similarly judge us on our inability to orient our selves, and probably wouldn’t understand our language and thus completely miss our language abilities. We do the same when we judge the intelligence of non-human animals or systems.
So the reason for why no other animal is close to us in intelligence is very simple actually, it is because of the way we define intelligence.
Just to clarify one point: I don't think intelligence is exclusive to humans. I only think that there's a big discrepency that cannot be explained with neuron counts oor the volume of the brain etc. which makes the argument of more hardware and more data will create AGI.
My main problem with the notion of generalized intelligence (in philosophy; I have tons of problems with it in psychology) is it turns out to be rather arbitrary what counts towards general intelligence. Abstract thought and project planning seems to an essential component, but we have no idea how abstract thought and project planning goes on in non-human systems. In nature we have to look at the results and infer what the goals were with the behavior. No doubt we are missing a ton of intelligent behavior among several animals—maybe even pants and fungi—just because we don’t fully understand the goals of the organism.
That said though, I think our understanding of the natural world is pretty unparalleled by other species, and using this knowledge we have produced some very impressive intelligent behavior which no other species is capable of. But I have a hard time believing that humans are uniquely capable of this understanding nor of applying this understanding. For examples, elephants have shown they are capable of inter-generational knowledge and culture. I don’t know if elephants had access to the same instruments as we, that they would be able to pass this knowledge down generations on build up on them.
That's why it's almost certain that a biological brain with a billion synapses outperforms a model with a billion parameters.
> the comparison is not apples-to-apples, because each synapse is much more complex than a single parameter in a weight matrix.
... but calling vector-entries in a tensor flow process "neurons" is at best a very loose analogy while comparing LLM "neuron numbers" to animals and humans is flat-out nonsense.
the same foundation that makes the binary model of computation so reliable is what also makes it unsuitable to solving complex problems with any level of autonomy
in order to reach autonomy and handle complexity, the computational model foundation must accept errors
because the real world is not binary
Uhh, yes we do.
I mean sure, we don't know everything, but we know one thing which is very important and which isn't under debate by anyone who knows how current AI works: current AI response quality cannot surpass the quality of its inputs (which include both training data and code assumptions).
> The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.
And it's become abundantly clear that this isn't the important difference between current AI and the human brain for two reasons: 1) there are large scale structural differences which contain implicit, inherited input data which goes beyond neuron quantity, and 2) as I said before, we cannot surpass the quality of input data, and current training data sets clearly do not contain all the input data one would need to train a human brain anyway.
It's true we don't know exactly what would happen if we scaled up a current-model AI to human brain size, but we do know that it would not produce a human brain level of intelligence. The input datasets we have simply do not contain a human level of intelligence.
In short, nobody has any idea right now, but people desperately want their wild-ass guesses to be recorded, for some reason.
I don't think that's totally true, and anyways it depends on what kind of scaling you are talking about.
1) As far as training set (& corresponding model + compute) scaling goes - it seems we do know the answer since there are leaks from multiple sources that training set scaling performance gains are plateauing. No doubt you can keep generating more data for specialized verticals, or keep feeding video data for domain-specific gains, but for general text-based intelligence existing training sets ("the internet", probably plus many books) must have pretty decent coverage. Compare to a human: would a college graduate reading one more set of encyclopedias make them significantly smarter or more capable ?
2) The new type of scaling is not training set scaling, but instead run-time compute scaling, as done by models such as OpenAI's GPT-o1 and o3. What is being done here is basically adding something similar to tree search on top of the model's output. Roughly: for each of top 10 predicted tokens, predict top 10 continuation tokens, then for each of those predict top 10, etc - so for a depth 3 tree we've already generated - scaled compute/cost by - 1000 tokens (for depth 4 search it'd be 10,000 x compute/cost, etc). The system then evaluates each branch of the tree according to some metric and returns the best one. OpenAI have indicated linear performance gains for exponential compute/cost increase, which you could interpret as linear performance gains for each additional step of tree depth (3 tokens vs 4 tokens, etc).
Edit: Note that the unit of depth may be (probably is) "reasoning step" rather than single token, but OpenAI have not shared any details.
Now, we don't KNOW what would happen if type 2) compute/cost scaling was done by some HUGE factor, but it's the nature of exponentials that it can't be taken too far, even assuming there is aggressive pruning of non-promising branches. Regardless of the time/cost feasibility of taking this type of scaling too far, there's the question of what the benefit would be... Basically you are just trying to squeeze the best reasoning performance you can out of the model by evaluating many different combinatorial reasoning paths ... but ultimately limited by the constituent reasoning steps that were present in the training set. How well this works for a given type of reasoning/planning problem depends on how well a solution to that problem can be decomposed into steps that the model is capable of generating. For things well represented in the training set, where there is no "impedance mismatch" between different reasoning steps (e.g. in a uniform domain like math) it may work well, but in others may well result in "reasoning hallucination" where a predicted reasoning step is illogical/invalid. My guess would be that for problems where o3 already works well, there may well be limited additional gains if you are willing to spend 10x, 100x, 1000x more for deeper search. For problems where o3 doesn't provide much/any benefit, I'd guess that deeper search typically isn't going to help.
Scaling experiments are routinely performed (the results are not encouraging). To say we know nothing about this is wrong.
It’s interesting how a relatively small # of synapses can do all abstract reasoning when free from those concerns.
Take the pre-frontal cortex, leave the rest.
For your implication to be plausible, you either need to deny that consciousnes plays a role in reasonability of thinking (making you a physicalist reductionist) or you need to posit that a neural network can have consciousness (some sort of mystical functionalism).
As both of these alternatives imply some heavy metaphysical assumptions and are completely unbased, I'd advise to avoid thinking of neural networks as an analogue of brains with regards to thinking and reasonability. Don't expect they will make more sense with more size. It is and will continue to be mere statistics.
All I'm saying above is that the number of neuron-neuron connections in current AI systems is still tiny, so as of right now, we have no way of knowing in advance what the future capabilities of these AI systems will be if we are able to scale them up by 10x, 100x, 1000x, and more.
Please don't attack a straw-man.
We desperately need to rapidly regulately down fossils usage and production for both electricity generation and transport. The rest of the world needs to follow the example of the EU CO2 emissions policy which guarantees it's progressing at a downwards slope independent of what the CO2 emissions are spent on.
Would I use the code directly in production? No. I always use it as an example and write my own code.
We seem to dancing around a problem in the middle of the room like an elephant no one is acknowledging, and that is the interface to Artificial Intelligence and Generative AI is a place that requires several degrees of innovations.
I would argue that the first winning feat of innovation on interfacing with AI was the "CHAT BOX". And it works well enough for the 40% of use cases. And there is another 20% of uses that WE THE PEOPLE can use our imagination (prompt engineering) to manipulate the chat box to solve. On this topic, there was an article/opinion that said complex LLMs are unnecessary because 90% of people don't need it. Yeah. Because the chat box cannot do much more that would require heavier LLMs.
Complex AI and large data sets need nicer presentation and graphics, more actionable interfaces, and more refined activity concepts, as well as metadata that gives information on the reliability or usability of generated information.
Things like edit sections of an article, enhance articles, simplify articles, add relevant images, compress text to fit in a limited space, generate sql data from these reports, refine patterns found in a page with supplied examples, remove objects, add objects, etc.
Some innovation has to happen in MS Office interfaces. Some innovations have to happen in photoshop-like interfaces.
The author is complaining about utopian systems being incompatible with AI. I would argue AI is a utopian system being used in a dystopian world where we are lacking rich usable interfaces.