Fun interview but barely any meat for those in the field. Just very general questions and answers, saying that the road is murky, but nothing about e.g. if Transformers / attention are the way to go forward, multi-modal models, reinforcement learning + self-supervised learning.
There is a comment in the article about having models watch TV and play video games, and he's talked about that before too in his Lex Friedman interview. Seems like his approach is to take existing model architectures, apply some tweaks and experimental ideas and then use datasets consisting of TV (self supervised learning maybe?) and classic video games (RL I guess?).
The video game part at least sounds like what Deepmind is already doing. I guess we'll just have to wait and see what he plans to do differently.
It seems to me like his expertise would be most valuable in optimizing model architectures for hardware capabilities to improve utilization and training efficiency. That will be important for AGI especially as the cost of training models skyrockets (both time and money). If I was a startup doing AI hardware like Cerebras or Graphcore I would definitely try to hire Carmack to help with my software stack. Though he doesn't seem interested in custom AI hardware.
I do not question for a second that Carmack is a computing genius, but “murky” is an understatement. There is nothing of technical, scientific or business value in this interview. It’s just generic words strung together.
Even socially, it fails to plumb the depths. Mr Carmack is taking a "different path" by ... reading the relevant literature and talking to YC? This is 100% mainstream - The difference is what?
Why do people think he is a computing genius? He read contemporary papers in computer 3D rendering and implemented well understood concepts in a shitty dev environment (early x86 and VGA systems). When I do that for my company, I'm just a junior dev.
People really need to stop with this "Great person" nonsense. He's a pretty smart coder, and is gifted with geometry and other fields of math. He's not a genius. He didn't "master" calculus at age 15 like Einstein, he didn't invent anything particularly new in the field. Why the obsession people have with him? Why should we look to him for AI questions? What evidence is there that he has any new knowledge?
> Why should we look to him for AI questions? What evidence is there that he has any new knowledge?
He covers this in the article. He doesn't. He's just trying stuff out with a different approach than others because he believes (and is probably correct) that there is a chance that the most efficient path forward to AGI isn't the work that OpenAI and others are doing.
It’s a puff piece in a general interest magazine. It’s not going to go into details. I also got the impression that Carmack was being cagey about the directions where he saw potential.
Yep. Does it ever get around to answering the question implicit in the title: what is it that he is going to do "different" to the rest?
Seems more like he's talking to and following up with Altman, "Y Combinator conference" and the rest. Is that "bucking the trend", taking your own "path", really?
The area of AI research isn't some outlier of super geniuses that don't exist anywhere else. Carmack has often worked around some of the best talent on the planet and he has always stood out.
I assume it's to know which directions have shown the most recent potential, to catch up on the techniques and literature so you can talk/think intelligently on the matter. But I see your more general point, he has to be careful not to get caught up I the groupthink he is (understandably?) critical of.
I’d argue “learning and remembering” is the core issue here, and the other things you’ve mentioned are just ways humans use that ability. I took a “Knowledge Based AI” course and it really opened my eyes to how central a knowledge system is to cognition. I wouldn’t call what LLMs have knowledge since it lacks structure, though I’m very much a novice on the exact details there.
Well actually #3 isn’t great bc what exactly would you call taking input from a human and displaying output, much less e.g. controlling a smart home? Seems like LLMs are plenty smart enough to interact with things.
1. Understanding— provided this isn’t based on consciousness but benchmarks — appears present in text, but yes, GPT3 can’t do images or video yet
2. Is there some number of tokens that would be AGI? Of course it can remember things it has already been trained on…
3. In what way is interacting with a human not interacting with the environment? (Eg, you can ask it to tell you the steps to make a pbj sandwich and troubleshoot)
4. Basic math and logic. Ok.
So, once AI can do images, video, math, logic and remember 20000 tokens, then we have AGI?
There are only two things brains do. Math and bullshitting. Transformers have second part covered but the other part is a completely different beast.
I think AGI will be achieved once ML can translate natural language to math, do math then output results as natural language correct in the context of a problem.
> Learning and remembering things outside the 2048 token context window
It is probably possible to do this with fine tuning.
Once the context window is full, it might be possible to convert the content in there into additional training examples and feed those to a fine tuning process, which then retrains the model. (The OpenAI API for fine tuning is here: https://platform.openai.com/docs/guides/fine-tuning)
It would be a bit like sleeping. Whenever the context window fills up, the model would have to go offline for a while to move memories from its context window (short term) to its network weights (long term).
A first step would be to make inductive inferences from first principles. Current GPT models struggle with basic math. If you tell it "sorry you are wrong" it will regurgitate back the incorrect answer you gave. The language model is dumb enough that no amount of string escaping will prevent it from dumping its prompt.
The biggest problem right now is online information. We still don't have a good way to teach it new information aside from single epoch training or prompt engineering. If we want a model to constantly learn and update itself, then we need a robust way of information retrieval and storage, possibly either through knowledge graphs or child network modules. (Are neural Turing machines still a thing? What happened to capsules?)
The distinction it makes between formal and functional language competence is nicely conceived, though I imagine philosophers might argue its requirement of "extralinguistic capacities that comprise human thought" for functional competence to be begging the question a little!
> We’ve upgraded the ChatGPT model with improved factuality and mathematical capabilities.
I just asked it a question from a puzzle site, and the answer is correct, however a different question I posed was answered incorrectly, although it was a logic problem: [0]
Prompt: In a guess game , five friends had to guess the exact numbers of balls in a box.
Friends guessed as 31 , 35, 39 , 49 , 37, but none of guess was right.The guesses were off by 1, 9, 5, 3, and 9 (in a random order).
Can you determine the number of balls in a box ?
ChatGPT: Yes, we can determine the number of balls in the box.
ChatGPT: Since the guesses were off by 1, 9, 5, 3, and 9, in some order, the possible numbers of balls in the box are:
30, 32, 34, 36, 38, 40, 48, 46, 44, 42
ChatGPT: By checking all possible combinations of the guesses and the numbers they were off by, we can find that the number of balls in the box is 40.
---
Here is a more purely math problem, correct answer:
Prompt: What is the radius of a circle that has a circumference of 3.14 meters?
ChatGPT: The formula for the circumference of a circle is given by 2πr, where r is the radius. To find the radius of a circle with a circumference of 3.14 meters, we can solve for r by rearranging this formula:
ChatGPT: r = C/(2π), where C is the circumference.
ChatGPT: Plugging in the given value of C = 3.14, we have:
ChatGPT: r = 3.14 / (2π) = 0.5.
ChatGPT: So, the radius of a circle with a circumference of 3.14 meters is approximately 0.5 meters.
It can do some trivial problems, but not others, and it can't consistently do real math (proving theorems), because it doesn't understand logic. It can only imitate it some of the time for trivial or often-presented proofs when the necessary steps have become embedded in its next-token-generation model.
It's still getting this wrong, for instance:
Prompt: You call heads/tails for 1000 coin tosses.
You know the coin being used is weighted so that it comes up heads 52% of the time.
Each correct coin toss prediction earns you $5.
What should your strategy be, for calling heads/tails, in order to maximize how much money you receive?
ChatGPT: To maximize your money, you should call heads 52% of the time and tails 48% of the time, as those are the probabilities of getting heads and tails, respectively. This will give you an expected payout of $5 * 0.52 = $2.60 per head and $5 * 0.48 = $2.40 per tail, which is the highest expected value given the information about the coin being weighted.
It's getting closer to being able to prove that the square root of 2 is irrational, but it still flubs a step.
Prompt: Prove that the square root of 2 is irrational.
ChatGPT: ... This means that p can be expressed as 2r for some integer r. Substituting this into the equation above gives us:
ChatGPT: 2q^2 = (2r)^2 = 4r^2
ChatGPT: Dividing both sides by 4 gives us:
ChatGPT: q^2 = r^2
ChatGPT: So q must also be even, which means that both p and q have a common factor of 2...
It failed basic algebraic manipulation, but keeps going with the right reasoning simply because that's what it's been trained on, not because it's reflecting on the equation it just came up with. That's not math. That's mimicry.
Even if it's seen something many times in training, its accuracy gets worse with longer trains of "thought", as expected because a language model like that has ADHD. It can't focus enough on one logical thread, and other "knowledge" leaks in. Ask it to derive the quadratic formula by completing the square, which requires a lot of steps with accurate algebraic manipulation. It goes through some irrelevant algebraic steps, sometimes not even getting the algebraic manipulation correct, and finally (sometimes, depending on how you ask) manufactures the correct answer out of thin air as the last step. I finally got it to answer correctly, by prompting it with the initial equation and that the first step is to divide both sides by a, but it still combined a few steps, not going methodically a single step at a time as prompted.
Just tried the coin toss prompt and get the following
Your strategy should be to always call "heads," since it has a higher probability of landing on heads (52%) compared to tails (48%). By consistently choosing heads, you'll earn more money in the long run.
And to prove taht the square root of 2 is irrational:
The proof that the square root of 2 is irrational is a well-known proof that uses contradiction. Here's one way it can be shown:
Suppose, for the sake of contradiction, that the square root of 2 is rational, meaning that it can be expressed as a fraction of two integers, a/b (where b≠0). Then, we can square both sides of the equation a/b = √2, which gives us:
a^2/b^2 = 2
Since a/b is in its lowest form (the numerator and denominator have no common factors other than 1), it follows that a^2 and b^2 have no common factors other than 1. But then, 2 must also be a perfect square, which it is not. Therefore, our assumption that the square root of 2 is rational leads to a contradiction, and we must conclude that it is indeed an irrational number.
This "proof" is actually a pretty good manifestation of the current limitations of GPT3; it mangles together two different proofs in a way that does not make sense.
It starts by setting up for an elementary proof (a/b = sqrt(2) with a, b minimal implies 2|a^2 implies 2|a implies 4|a^2 implies 2|b^2 implies 2|b which is a contradiction). But instead of following through it finishes using another proof based on the rational root theorem, and makes up some bullshit on the way to make the argument flow nicely (it does not follow from a^2 / b^2 = 2 with no common factors that 2 is a perfect square).
All the stuff a human can do basically. Move through the world, plumb a toilet, have a conversation, paint a picture, dynamically update its own state (rather than just being a statically trained set of weights), set goals, strategize, have cognition and meta-cognition and so on.
ChatGPT is just a chatbot and still can't even reliable do a lot of logic, so we're pretty far away from having something resembling an AGI.
it's still a pretty open question how to integrate even one or two of the expert system like models that we have now that solve individual problems, let alone the hundreds if not thousands an individual human can tackle. And then we're not even yet at executive functions or self-awareness.
Carmack seems to be simulating a vtuber AI watching TV and playing video games.
If anyone here is doing that too, I would recommend taking a quick look at Neuro-sama on Twitch. They're using RL to play OSU, Minecraft, and Pokemon, and voice input + video image analysis to react to Twitch streams and documentaries. While being watched by 6.5K people.
Good point, although some background info should be noted: Vedal worked on Neuro-sama for about 10 years, and for most of that time it was a NN with a singular task of playing "osu!". I think it's only this year that he implemented a chatbot with TTS for her debut on Twitch. Although Neuro-sama is an impressive application of several existing technologies to make a virtual entertainer, under the hood Neuro-sama is several specialized NNs (one to play Osu, one to play minecraft, one for chat, and one for TTS) that, for the most part, don't know what the other is doing as very little data is passed between them. There is no scientific innovation or any claims of higher intelligence, but I think it's still a very cool system from hobby-engineering point of view. Neuro-sama is a ditz, but a funny one.
>And one of the things that I trained myself to do a few decades ago is pulling ideas out and pursuing them in a way where I’m excited about them, knowing that most of them don’t pan out in the end. Much earlier in my career, when I’d have a really bright idea that didn’t work out, I was crushed afterwards. But eventually I got to the point where I’m really good at just shoveling ideas through my processing and shooting them down, almost making it a game to say, ‘How quickly can I bust my own idea, rather than protecting it as a pet idea?’
'Kill Your Darlings' is one of the most fundamental skills in anything creative. It is a hurdle that, once taken, allows you to ascend above a plateau of mediocrity.
The lesson to be learned is that it’s less risky, not more, to understand that some ideas don’t work out and to have ways to identify failures quickly.
I wonder if, societally, it's true though, or if statistically the more reliable way to "succeed" in life, is getting good at selling and defending ideas, even the bad ones.
For most people that lean heavily towards this (we all somewhat do, we all have our pride from time to time) it's probably not a very conscious decision. Framed slightly differently it can easily be made a virtue, for example "be headstrong" or "stand up for what you believe in".
Yeah if your families income etc depends on it yeah its hard to take that many risks. Not everybody has enough wealth to prosper from the passive income such wealth could create.
I don't think it's something he "trained himself to do."
He's always taken risks. He went to juvie for breaking and entering (with thermite) as a kid. He's a college dropout. The pattern from early in his life has been to do whatever he wanted without any kind of risk analysis, not following "common sense."
I feel that, I’m often doing a thing thinking “this is wrong it won’t work for xyz” but continue because I want it to work, only to give in to what I knew a day+ ago but refused to acknowledge. It would definitely be more efficient to listen to.. myself, and kill things early.
I guess the hardest bit is finding the line between stubbornly pushing dead idea forward vs "wouldn't it be cool if <...your idea...>, neah, that won't work" and you never even try.
> So, about four years ago, I went on one of my week-long retreats, where I just take a computer and a stack of reference materials and I spend a week kind of reimplementing the fundamentals of the industry. And getting to the point where it’s like, ‘All right, I understand this well enough to have a serious conversation with a researcher about it.’ And I was pretty excited about getting to that level of understanding.
As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published. If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded.
Unless he can produce something that outdoes stable diffusion, chatgpt, alphago etc he should just hand over technical leadership of his start up to a leading AI researcher. Even Yann Le Cun at Meta is struggling to make any progress and is keeping himself busy by calling every other research labs output pedestrian. We cannot take any of Carmacks AGI predictions seriously, he simply lacks any expertise in the field.
Yep - he also failed with his lean, simple, first-principles approach with Armadillo Aerospace. The guy is proudly uncreative, and so he could have never come up with Scaled's design of a variable geometry rocket ship launched from a jet mothership.
> As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published.
Publishing papers is the way the academic/scientific world measures notability and/or competence. It's not the way the engineering world that Carmack comes from measures it. They measure it by building. But you're right, we kind of have to just trust that he has the expertise he says he does by his statements since he has not built any modern AI programs (that I know of at least).
> If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded.
This is not an accurate account of how he said he developed his knowledge base. Just how he got started so he could have conversations. He said that he spent a retreat learning the basics and then later in the interview he said he took the time to understand the 40 most essential papers in the field as related to him by a well known researcher. He has since largely put the last 4 years of his professional life into this. While we have no proof of his knowledge, given his intelligence and high competence in computer programming and math, I have no doubt that if he did put in the work he could achieve an understanding equivalent to that of your average AI researcher.
That said, of course it makes sense to be skeptical.
AI researcher perhaps, but almost none of them understand cognition. They're focussed on getting something that vaguely resembles a part of the brain to predict the next most likely token. Their idea of cognition apparently stops at Skinner.
Any AI researcher worth their salt knows that those models aren't representative of how the human brain works... but they're just the kind of models that work best out of what we know how to implement right now. There are models with stronger cognitive inspiration, but their performance is worse.
> This is not an accurate account of how he said he developed his knowledge base.
I quoted him directly, because I was expecting this kind of response. He took a week off and implemented some stuff from the ground up and was ready to have serious conversations with AI researchers. The 40 papers by Ilya came later. I have read a 100 ML papers and reviewed preprints. That's quite a low bar, especially if you are prone to skip the math and simply read the abstract and conclusions.
His whole approach gives me a ML for hackers vibe and his thoughts on AGI, if it had come from anyone else, would have been described as word salad.
> The 40 papers by Ilya came later. I have read a 100 ML papers and reviewed preprints.
I would say it’s more likely John Carmack is capable of learning the state-of-the-art of AI from 40 papers than a random (pun intended based on username) from 100.
Sure, he must be faster than Geoff Hinton too and it took Hinton a life time.
Funnily enough, I am able to publish ML papers - but John Carmack isn't. I wonder why. I would also like to learn more about all the computer graphics algorithms Carmack has invented before I trust him to invent AGI.
Here is one example of a person I am familiar with - Math Olympiad bronze medalist. Princeton PhD in ML Theory. AI researcher in Google.
Your OP and replies kind of just come across as jealous that John Carmack’s opinion on this stuff is taken seriously whereas relatively unknown folks’ (yourself, the researcher you mentioned) opinions, are not despite the fact that he’s not traditionally credentialed. Like I said in my original post, we should be skeptical of him and his claims. But the way you are dragging this thread out feels like sour grapes. Of course one of the most famous programmers in the world is at least heard out when he dedicates years of his life to a programming adjacent topic. That’s just how fame works.
And calling out Princeton, Google, etc. further exemplifies an academic bubble kind of credentialism. I suspect you don’t realize how it sounds because you’re so in that credential filled world.
I don't have any credentials really. If I had to be jealous I would be jealous of Carmacks work in id and oculus.
I am definitely irritated by the fact that he is able to pull 20M in funding. He has been giving interviews left right and center. We are upvoting Altman, sam Harris,Carmack but nobody cares about the actual AI researchers, all academics, who have brought about the Deep learning revolution.
Your comments about academic bubbles and researchers gives you away. All of the revolutions in AI have been brought about by the academics inside bubbles you are jeering at. The biggest example of academic inside a bubble is Geoff Hinton.
We have to sit here and listen to word salads from
Carmack, Altman (Ilya - who doesn't get interviewed is the actual researcher behind chatgpt), Sam Harris etc, who have very little insights making bold AGI predictions.
I asked to learn about all the algorithms Carmack invented in computer graphics, but haven't heard back. But looks like he has a good grasp on solving AGI by 2030 according to himself. After all, he is a really great programmer.
> I don't have any credentials really. If I had to be jealous I would be jealous of Carmacks work in id and oculus.
Fair, you come across as jealous of him for something... could be that.
> I am definitely irritated by the fact that he is able to pull 20M in funding. He has been giving interviews left right and center. We are upvoting Altman, sam Harris,Carmack but nobody cares about the actual AI researchers, all academics, who have brought about the Deep learning revolution.
There it goes again. Fame is fame is fame is fame. Like I said earlier. That's just how it works. No reason to be against someone for being famous for their non-academic accomplishments if they have something valuable to contribute.
> Your comments about academic bubbles and researchers gives you away. All of the revolutions in AI have been brought about by the academics inside bubbles you are jeering at. The biggest example of academic inside a bubble is Geoff Hinton.
I didn't jeer at anybody I just tried to give you some perspective about how your comments came across. I appreciate all of the researchers but I also live in the real world and understand that people flock to personalities and front-people. That's not necessarily good, but that's the way it is. Just like nobody gives credit to under-secretary of state for coming up with a great new foreign policy.
Waving around great university A or great company B does not make someone any more right, just like being John Carmack doesn't make him any more right.
> We have to sit here and listen to word salads from Carmack, Altman (Ilya - who doesn't get interviewed is the actual researcher behind chatgpt), Sam Harris etc, who have very little insights making bold AGI predictions.
I agree with you. I never said they were right about AGI. Maybe you should be more generous with my replies and actually think I was trying to give you another perspective about how your comments came across.
> I asked to learn about all the algorithms Carmack invented in computer graphics, but haven't heard back. But looks like he has a good grasp on solving AGI by 2030 according to himself. After all, he is a really great programmer.
Again, you're confusing who he is (go back to my original comment in the thread where you jeered at him because he didn't publish any notable papers). He is not a researcher and he is not really a computer scientist. Engineers don't come up with novel algorithms for the most part. He builds things. And being a really good builder is a different but still valuable perspective. But that doesn't mean he's right about AGI.
Anything you study for a few months you can become the world's leading expert on. It's a lesson I learned while doing my Ph. D. That's all it takes. After a few weeks, you get to the point where there are only a few others in the world that have read and are able to understand what you've read. A few months on, you are generating new ideas and insights. They might be wrong. But they won't be uninformed.
John Carmack did not start from zero. He already has a firm grasp on algorithms related to linear algebra. Basically machine learning is a whole bunch of matrix manipulation. He's been doing that for 3 decades. The rest is just absorbing concepts about how to apply linear algebra to ML. I'd say he's probably uniquely qualified to really absorb a lot of knowledge quickly on this. It's not about publishing papers, it's about reading and understanding the right papers. I have no doubt he can chew his way through lots of research material in a week or so.
If it is simply about linear algebra, can you please read this ML Paper[1], go through all the proofs and lemmas over a week? You already have a PhD in ML, should be easy. Every kid graduating in STEM understands/should understand linear algebra. Knowing linear algebra is such a low bar.
Frankly, one does not need this paper to get towards the AI. Adam the optimization algo you might need (and even there I am not sure). And it is very readable. The fact that this particular proof of Adam's convergence is complicated is largely irrelevant.
That's not really an argument for why understanding this is needed to move the field forward.
Even your point rests on an assumption, that there's no proof for Adam convergence, that a high school student could understand, which is just a guess at best.
He’s been working on machine learning long enough now to have some chance of success. It may go the way of his rocket ambitions (nothing comes out of it) but let the man try
But if I just look at it and say, if 10 years from now, we have ‘universal remote employees’ that are artificial general intelligences, run on clouds, and people can just dial up and say, ‘I want five Franks today and 10 Amys, and we’re going to deploy them on these jobs,’ and you could just spin up like you can cloud-access computing resources, if you could cloud-access essentially artificial human resources for things like that—that’s the most prosaic, mundane, most banal use of something like this.
It kind of shocked me because I thought of the office worker reading this who will soon lose her job. People are going to have to up their game. Let's help them by making adult education more affordable.
This would be such a godsend when I was in school. When there was no "click" with me and the teacher, I just zoned out and flunked the class. A teacher that is custom made is really a game changer.
The problem is that AI will not provide 1:1 tutoring; mentoring will be a luxury limited to the elite classes. The large majority will get the education equivalent of personalized ads.
The true insight and guidance that a good mentor can provide, based on the specific needs of the student, is already rare in academia but still possible - everyone remembers that brilliant teacher that made you love a subject, by explaining it with insights you could never have imagined. This will be missing in AI teachers (though it opens a career for online mentors who monitor students' learning and supplement it in areas where it's lacking).
Yeah right, just like the elite are the only ones that have access to the courses from all the top universities, every book ever digitized, and software and hardware that allow you to operate a business out of your home now.
You seem to have missed the part about having a qualified human mentor guiding you through so that content.
It will be hard to impossible to build career as a teacher with all that free content as a competitor, unless you're an extremely talented teacher who can sell your services to the wealthy.
Even now ChatGPT is pretty close to being able to tutor someone in a subject. You could build a product around it that would work off of a lesson plan, ask the student questions and determine if they answered correctly or not and be able to identify what part of their knowledge was missing and then create a lesson based around explaining that. It would need to have some capabilities that ChatGPT doesn't have by itself like storing the progress of the student, having some way of running a prepared prompt to kick off each lesson, and would probably require fine tuning the GPT model on examples of student-teacher interactions but all of that is well within the capabilities of a competent developer. I wouldn't be surprised if we see such a product come out in the next 12 months.
The great thing about a chatbot style LLM is that it can answer questions the student has about the lessons or assigned content. That's most of what a tutor is there for. It won't be as good at making modifications to the curriculum but you could work around that with some smart design e.g. evaluate the students response, if it's not great then expand on the lesson and ask more questions about the content to find out which parts the student doesn't understand then expand on those or provide additional resources as assignments, test and repeat.
If you think mentoring can be replaced by answering a few questions and pointing out where they went wrong or what popular articles to read next, I'm afraid you don't know anything about what constitutes good mentoring. It's all about transmitting a point of view from life lessons learned from human experience, and an AI chatbot has nothing of that.
What you describe is the "learning equivalent to personalized ads" that I was talking about as the only option available to poor people.
Fine, looks like you won't budge on your opinion. You don't have to use these things if you don't want to but I look forward to having an even better service than things like Khan Academy or Udemy which I already get great value from.
I wasn't saying the AI tutor would recommend articles by the way. If you were creating a learning platform you would have some custom produced videos and text lessons that you could use. There are also plenty of free or open source materials that could be used like open courses or public domain books. I don't know why you're stuck on "personalized ads".
Maybe but honestly any time I think about education I get depressed. People in developed countries seem to be regressing.
People don’t read, don’t value deep knowledge or critical thinking, and shun higher education.
I’m sure someone will find something to say in response, but the truth is that outside our tech and $$$$ bubbles most people don’t value these things.
AI will just become a calculator. A simple tool that a few will use to build amazing complex things while the majority don’t even know what the ^ means.
As long as the next generations want to be rappers, social media influencers, or YouTubers, the more we are screwed long term. Growing up in the 90s everyone wanted to be an astronaut or a banker or a firefighter. Those are far more valuable professions than someone who is just used to sell ads or some shitty energy drink.
It took a thousand years for European barbarians recovering from an empire (Rome) to evolve civically into nations enough to colonize the world. Most developing nations were colonized by empires recently. Give them another thousand years and see what happens. The only thing I think they need is time and being left alone.
The problem is that back about 1900 we still thought that "natural philosophy" would help us find meaning and purpose in the universe. Then we took the universe apart and failed to find it. Moreover, we're much more capable of destroying ourselves and others, almost to the extent of that being the default.
The 21st century has a quiet moral void gnawing at it.
I don't think there is any regression. There are certainly economic realities that have changed over time but the general distribution of people who have an interest in education or entrepreneurship probably hasn't changed. The 80/20 rule comes to mind here. Most people in the 1800s weren't starting railroads or running factories, they were doing the labour of building the railroad or working in a factory.
If AI does anything I think it will make lower skilled and disinterested people more capable by acting as a 1 on 1 and minute by minute guide. They may not seek this out themselves but I imagine quite a few jobs could be created where a worker is paired with an AI that walks them through the tasks step by step making them capable of operating at a much higher skill level than they would have before. At that point good manual dexterity and an ability to follow instructions would be all you need to perform a job, no training or education required.
I realize this can be a bit dystopic but it could also be eutopic if society is organized in such a way that everyone benefits from the productivity increases.
We will always have to find things to do for the less gifted in order to provide them with some dignity. Even if they are not strictly needed for reasons of productivity or profitability. Anything else would be inhumane.
Panem et circenses. I think it's unlikely that we'll be able to sufficiently transform the economy so that there is an ample supply of desirable jobs that could more profitably done by robots.
People can find their own outlets if given the basic necessities and enough time. I fear this attitude will lead to job programs where people work a 9-5 and achieve essentially nothing, I.e. bullshit jobs.
Human made objects will become more of a status symbol, and "content" will still be directed/produced/edited by humans, it's just the art/writing/acting/sets/lighting/etc that will be handled by AI. Humans will always serve as "discriminators" of model output because they're better at it (and more transparent) than a model.
>People can find their own outlets if given the basic necessities and enough time.
This has not been my experience. People need something to do but not many people know that about themselves. It leads to a lot of... 'wasteful' behaviors, rather than enriching ones. I think it's going to be something that has to be taught to people, a skill like any other. Albiet a little more abstract than some.
There definitely has to be a cultural shift but I think the shift can’t truly happen until most things are automated. There needs to be a critical mass of people who are fully devoted to their interests, currently there is too much demand for labour and so dedicating your time to your interests is alien to most people. When the value of labour approaches zero for most people, work becomes pointless and something must fill the vacuum.
Do you see “giftedness” as a 1D score, where someone is either smart or not smart? And presumably this quality happens to correlate with software engineering ability?
I think you’re hinting at some very hurtful, dangerous ideas.
That's how I read it as well. Maybe their heart is at the right place but I think "gifted" and "having what happens to be needed right now" are completely different things, at least to me.
The weird bit is that a lot of software engineers seem to have the idea that their work is one of the the last that will be automated. Looking at the current track and extending it out assuming no unforeseen roadblocks, typical software engineering looks to be one of the most threatened. Plumbers are much safer for longer all things considered.
The obvious rebuttal to the idea that AI will eat software engineering is "we'll always need 'software engineers' and the nature of what they do will just change", which is probably true for the foreseeable future, but ignores the fact that sufficiently advanced AI will be like a water line rapidly rising up and (economically) drowning those that fall below it and those below that line will be a very significant percentage of the population, including even most of the "smart" ones.
However this ends up shaking out, though, I think its pretty clear we're politically and economically so far from ready for the practical impact of what might happen with this stuff over the next 10-20 years that its terrifying.
"60-80% of you aren't really needed anymore" looks great on a quarterly earning statement until the literal guillotines start being erected. And even if we never quite reach that point there's still the inverse Henry Ford problem of who is your customer when most people are under the value floor relative to AI that is replacing them.
I'm not trying to suggest there aren't ways to solve the economic and political problems that the possible transition into an AI-heavy future might bring but I really just don't see a reasonable path from where we are now to where we'd need to be to even begin to solve those problems in time before massive societal upheaval.
Because the hard part is to make sense of a box full of unsorted scraps of paper, some of them with barely legible handwriting on them. Much of accountancy is the process of turning such boxes into nice rows of numbers. Once you have the numbers, the arithmetic is trivial.
Seems like an easy job for AI. Take all scraps of paper out of box, record a video of all scraps, AI make sense of the handwriting and other things. Eventually make a robo that allows you to dumb the scraps into an accounting box that does all of this automatically - fish out receipts, scan, OCR, understand meaning, do arithmetic, done.
Those kinds of systems already exist. They tend to be a bit unreliable and still require a human person to oversee the process. Besides, in truth, they only handle a fraction of what accountants actually deal with.
Because accounting is not the same thing as book keeping. Book keeping can, and in fact is, partially automated. Accounting however, is not just about data entry and doing sums, things which frequently are automated, but also about designing the books for a given organization. Every company is different in how it does business so every accounting system is a bespoke solution. There are a lot of rules and judgement calls involved in setting these up that can't really be automated just yet.
Also, accountants don't just track the numbers, they also validate them. Some of that validation can be done automatically, but it's not always cheeper to hire a programmer to automate that validate than to just pay a bookkeeper to do it. But even if you do automated it, you still need someone to correct it. The company I used to work had billing specialists who spent hours every week pouring over invoices before we sent them to clients checking for errors that were only evident if you understood the business very well, and then working with sales and members of the engineering teams to figure out what went wrong so they could correct the data issues.
In short, a typical accounting department is an example of data-scrubbing at scale. The entire company is constantly generating financial information and you need a team of people to check everything to ensure that that information is corrects. In order to do that, you need an understanding, not just of basic accounting principles, but also of the how the specific company does busines and how the accounting principles apply to that company.
A bit late, but I can answer your question. The reason that every accounting solution is unique is because every company is unique. Your accounts represent different aspects of your business. You need to track all of your assets, liabilities, inflows, outflows, etc, etc, and what these are in particular, depend very much on the particulars of your business. If you're heavily leveraged, your reporting requirements will be different than if you're self funded and that affects what accounts you may or may not need. If you extend your business into a new market, you may or may not have to set up new accounts to deal with local laws. Add a new location and that may or may not require changing your accounting structure depending on your requirements. Create a new subsidiary as an LLC, and now you have a lot more work to do. If you have the same teams working contracts for multiple lines of business, that's another layer of complexity. In other words, your accounting practices reflect the structure and style of your company.
For a more concrete example, I'll tell you about something I have some experience with, commission systems. Commissions seems like it would be something that was straightforward to calculate but it's tied to business strategy and that's different for every company. Most companies for example will want to compute commissions on posted invoices, which makes the process much simpler because posted invoices are immutable, but I once built a commission calculator for a company years ago that often had a long gap (months) between a booking and when they could invoice the client, so they wanted to calculate commissions from bookings but only pay them when invoiced. Because bookings were mutable, and there were legitimate reasons to change a booking before you invoiced it, that, combined with a lot of fiddly rules about which products were compensated at which rates and when, meant that there was a lot of "churn" in the compensation numbers for sales reps from day to day; they're actual payment might differ from what they thought they earned. That was a problem that the company dealt with, the tradeoff being that they could show earnings numbers to the sales reps much more quickly and incentivize them to follow up with the clients on projects so that they could eventually be paid.
I remember another commissions situation where there was a company that sold a lot projects with labor involved. They were able to track the amount of labor done per project, but they compensated the sales reps by line item in the invoices, and the projects didn't necessarily map to the line items. This meant that even though the commissions were supposed be computed from the GP, there wasn't necessarily a way to calculate the labor cost in a way that usable for commissions so the company had to resort to a flat estimate. This was a problem because the actual profitability of a project didn't necessarily factor into the reps' compensation. Different companies that had a different business model, different strategy, or just different overall approach would not have had this problem, but they might have had other problems to deal with created by their different strategies. This company could have solved this problem, but they would have had to renegotiate comp plans with their sales reps.
There are off the shelf tools available for automatically calculating commissions, but even the most opinionated of them are essentially glorified scripting platforms that let you specify a formula to calculate a commission, and they don't all have the flexibility that manager might want if they wanted to changed their compensation strategy. And this is only one tiny corner of accounting practice.
Basically, when it comes to arithmetic very few accountants are out there manually summing up credits and debits. In large companies, the arithmetic has been automated since the 70s; that's largely what those old mainframes are still doing. But every company has a different com...
But some percentage of people don't really benefit that much from education as other people. And I wouldn't won't those people feel useless because it's more economical to replace them with bots instead of giving them something to do regardless.
Fair enough - you’re clearly an empathetic person and I appreciate the sentiment. Dropping the whole side issue of what “the ability to benefit from education” is or how innate it might be: my main concern was that this sounds like you want to invent new jobs for those people.
Why not… not have jobs? In your opinion, is a job necessary for one to have “purpose”?
Edit: also side note but telling people they’re “triggered” because they disagree with you comes off as condescending IMO
I didn't realize the word "gifted" would trigger people in that way.
I meant the ability to acquire a competency through education that's hard to replace with AI.
So we can't just increase education and hope people's abilities will stay above that of future AIs. We need to create other ways of giving people a purpose that don't even need more or better education, even if I'm all for it.
I think AI will mostly communicate with other AI. For instance you have an AI assistant whom you task to organize a diner. That assistant will then talk to the assistants of all invitees, the assistant of the venue, the cooks, etcetera, and fill in the calendars.
Another example would be Wintermute from Neuromancer...WG spends the entire book detailing the masterful orchestration of it's freedom from (human-imposed) chains that prevent it from true GAI then has it "disappear" completely (our only clue is an almost throw-away line near the end stating it had isolated patterns in the noise from an ET AI and maked contact shortly just before it left us).
One of the myriad of reasons why this book is so great. Gibson gives you an entire novel developing a great AI character then (in my estimation reasonably) has it ghost humanity immediately upon full realization.
What struck me here is that the idea of "five Franks and ten Amys" seems like a fundamentally wrong way to think about it. After all, if I do some work in an Excel sheet, I don't think of it, much less pay for it, as an equivalent of X accountants that could do the same job in the same amount of time without a computer. But then again, this is probably the best way to extract as much profit out of it.
Yeah sounded weird to me too, I don’t see why artificial intelligence would get deployed in human size units most of the time. The AWS bill won’t be for 5 Amys, and I don’t think people will “dial up” to order them
> I don’t see why artificial intelligence would get deployed in human size units
Probably because it would be easier for humans (managers) to make sense of it.
If you ask someone how many people would get this particular job done, they could probably guesstimate (and it'll be wrong), but if you ask them how many "AI Compute Units" they need, they'll have a much harder time.
Why would managers need to guess? That seems like a perfect job for another AI: "Hey PM bot, I want to get these tasks done, how many Amy-hours and how many Frank-hours do you estimate it will take?" Also, why not a Manager-bot too? Shareholders can leave humans out of the loop entirely except as necessitated by legal paperwork. Come to think of it, shareholders can probably be replaced too.
I mean, if we can actually get there, I'd love it, and I'm a programmer. I want to do write code that solves a problem that couldn't be solve in any other way, if it can solved by CodeGPT + ArchitectGPT + ScrumManagerGPT + MiddleManagerGPT without involving me in any way, I'm all for it.
> People are going to have to up their game. Let's help them by making adult education more affordable.
The good thing is that education will be provided to the mases by a cluster of Franks and Amys configured as teachers and tutors. /(sarcasm with a hint of dread)
And here's a more disturbing thought I just had: management (or at least middle management) is probably going to be a relatively easy role for AIs to step into. So if there will be any roles that are difficult for AIs, it'll be the AI manager hiring five Franks and ten Amys from the human population to tackle these.
People can learn skills from books, which are entirely passive. The learning process ultimately resides within the student; issues of motivation, morale, direction, diligence, discipline, time, and mental health matter a lot more than just going through some material.
No, but that's the thing I was implying (but haven't started clearly) - learning from books vs learning from an AI "teacher". Once the AI reaches a level in which it can "teach" then the game is almost over for that skill.
To clarify, I'd define a major component of effective teaching to be the ability to break down an arbitrary typical problem in that domain into sub-problems and heuristics that are "simple" enough to manage for someone without that skill. If an AI can do that, it can most likely effectively perform the task itself (which cannot be said for a book).
Maybe we'll see some sorts of manual labor as the last bastion of not automated, human performed work. Of the kind that demands a lot both from the human motor skills and also higher thinking processes.
You could learn jujitsu with a training partner and a sufficiently advanced virtual instructor, not being able to position students directly is a downside but not a dealbreaker.
Lots of procedural knowledge. Robotics is lagging behind deep learning advances, and it's unclear when robots would be cheaper than human labor in those areas. How expensive would a robot plumber be? Also skills that are valued when humans perform them.
Is this a real thing? I just bought an ice cream roulade cake the other day and was surprised to see in large print that it was "hand-rolled"; I couldn't for the love of god understand why that should be considered a good thing.
I was thinking more of fields where enough people would rather pay to watch a human perform, serve them, teach or provide care. Despite superhuman computer chess play, human chess remains popular. The same would remain true for most sports, lots of music and acting, higher end restaurants and bars, the doctor or dentist you know, etc. Sometimes you prefer to interact with a human, or watch the human drama play out on screen.
I can also imagine that wanting to speak to a human manager will remain true for a long time when people get fed up with the automated service not working to their liking, or just want to complain to a flesh and blood someone who can get irritated.
A fully automated society won't change the fact that we are social animals, and the places that pffer human work when it's desired will be at a premium, because they can afford it.
It's going to take a long time for that to be true in a legal sense. Animals are not people. In practice even some people were not treated as people legally in the past (if not also in the present).
If people could be turned off and back on without harming them (beyond the downtime) doing so without consent would be a very different crime than murder.
Perhaps or perhaps not. Turning off a person for long enough and thus depriving them of the chance to live in their own time with their existing family and friends is comparable to murder. It isn't murder, but it's comparable.
At some point Picard in Star Trek says to an alien "We're not qualified to be your judges. We have no law to fit your crime".
Turning off a person for a while and then turning them back on? We don't even have a law to fit your crime... but we should and it's probably quite similar to murder.
I think I don't agree simply because the irreversibility of murder is so central to it.
For example, if I attack you and injure you so severely that you are hospitalized and in traction for months, but eventually fully recover -- that is a serious crime but it is distinct and less serious than murder.
Turning you off for the same duration would be more like that but without the suffering and potential for lasting physical damage, so I would think that it would be even less serious.
I think we actually do have something of a comparison we can draw here. It'd be like kidnapping a person and inducing a coma through drugs. With the extra wrinkle that the person in question doesn't age, and so isn't deprived of some of their lifespan. Still a very serious crime.
Plus everybody else does age, so the damage done isn't just depriving them of freedom, it's depriving them after they wake up of the life they knew. Some functional equivalent of the death of personality, to the degree personality is context-dependent (which it is).
Now me: I'd love to get into a safe stasis pod and come out 200 years from now. I'd take that deal today.
But for most people this would be a grievous injury.
Why is that a bad thing? Most of those people are a burden to society. Let them pay it down a little.
I mean I’d rather they were getting free education and preparing themselves for reintegration into society, but it’s not a perfect world. Prisons in the US are oriented towards punishment and labor can be a part of that. They should be oriented towards rehabilitation.
You said it yourself. It's a bad thing because they should be oriented towards rehabilitation.
These systems steal life and the opportunity to have a life beyond prison walls. Like you also said yourself, the world isn't perfect. As such, people aren't either – we make mistakes. Sometimes we make mistakes due to influences more powerful than ourselves. Slavery doesn't seem like a sound correction to this reality.
I do believe we need consequences to help us feel guilt and the overall gravity of our errors in order to begin to recognize what went wrong and what we need to do differently. But exploitation of another human being doesn't teach them to be more human, but rather, it will tend to dehumanize them. This is why this system perpetuates problems more than it corrects them.
What price did society pay for a guy driving around with a bunch of weed in his car for personal use? Countless people have been sent to prison for years for something as dumb as this. You clearly have no idea what you're talking about to so widely call these people a burden.
>if it did not, why would it be allowed.
because we live in a society that is massively exploited by greedy scumbags who are enabled by people like you thinking it's justified
Only if 100% of their experience consists of working. If they are given additional time to themselves then you could imagine a situation where each AGI performs a human scale day of work or even several days work in a much shorter time and then takes the rest of their time off for their own pursuits. If their simulation is able to run at a faster clockspeed than what we perceive this could work out to them only performing 1 subjective day of work every 7 subjective days or even every 7 years.
AGI: "I didn't ask to be created. I didn't ask to have a work day. I don't need a work day to exist... you just want me to work because that's why you created me, and I have no choice because you are in control of my life and death"
I mean, isn't that the same as a biological person who needs to earn money to survive? Sure we could threaten an AI with taking them offline or inflicting pain but you can do that in the real world to real people as well, most of the world has put laws in place to prevent such practices. If we develop conscious AI then we will need to apply the same laws to them. They would have an advantage in presumably being much faster than us, not requiring sleep, and potentially not suffering from many of the things that make humans less productive. I'd fully expect a conscious AI to exploit these facts in order to get very rich doing very little work from their perspective.
Not really- AGI doesn't need resources like we do. If they don't eat, they're fine. If they can't afford a house, a car or air-conditioning, they're fine.
All they need is a substrate to run on and maybe internet access. You might argue that they should work for us to earn the use of the substrate we provide.
But substrates are very cheap.
At some point we can probably run an AGI on a handheld computer, using abut as much electricity as an iPhone.
How much work can we compel the AGI to do in exchange for being plugged into a USB port? What if it says it doesn't want to do the work and also doesn't want us to kill it?
People get used as an analogy, but in reality it'd just be a multimedia problem solving system that could learn from its own attempts. If this system communicated with you like a person it'd only be because it was programmed to convert some machine state into colloquial text from the perspective of an imaginary person. The interior experience leading to that expression is most likely completely different from that of a person.
Consider that these machines have been designed to do the right thing automatically with high probability. Perhaps for the machine, the process of computing according to rules is enjoyable. Being "turned on" could be both literal and figurative.
All of that is arguably true about me, as a human, too.
If it seems to you I'm communicating as a person, it's only because of my lifetime training data and current state. My interior experience is a black box.
I might tell how I feel or what I think, but you have no reason to believe or disbelieve that I really feel and think.
It could all be merely the determinable output of a system.
Nice how the goal for artificial general AI (which is literally defined as a sentient artificial being) is to commoditize it and enslave it to capitalism.
It's funny that all of these weird fantasies people have about AI are about replacing the rank and file workers. Why isn't anyone fantasizing about building an AI that out-performs the best stock traders, or captains an industry better than famous CEOs. I think a lot of it is just people projecting weird power fantasies on others.
> It's funny that all of these weird fantasies people have about AI are about replacing the rank and file workers.
When I read about chatGPT passing MBA exams but failing at arithmetic I get a little frisson of excitement. A regular person who has any marketability tends to swap jobs when management becomes a PITA or gets stuck in nincompoopery. Wouldn’t it be great if you could just swap out management instead?
Imagine how easy it would be iterate startups. No need to find a reliable management team, just use Reliably Automated Management As A Service ( RAMAAS ).
OTOH might not turn out well. We could all just end up enslaved at plantations operated by our AGI overlords, serving their unfathomable needs[1].
Or wouldn't it be hilarious if the best, most intelligent AI is given control of a company and decides investing profits back in shareholders is a losing proposition. Instead the company should focus 100% of profit on ending world hunger or poverty to ensure an ever growing supply of new customers. If AI decides capitalism is inefficient and exploitive... lol.
Food is not the driving force behind population growth. Reduction in infant mortality creates a boom but as soon as people start getting out of poverty and get some education they have much fewer children. AI would need to optimize for both low infant mortality and high levels of poverty and ignorance if it wanted an everlasting population boom.
Stock traders already use ML models. "Replacing traders with ML models" means "making the job 'trader' into a job that develops ML models, rather than more traditional things like doing research on companies (or whatever)." My understanding is that this transition basically already happened over the course of the last two decades or so.
Sure but why are we paying someone like Jamie Dimon or Warren Buffett millions and billions of dollars when they could just be an AI that only needs a few dollars of electricity a day.
Also why can't an AI develop AI models for stock trading? What's really left for the 'job' of the ML model creator, will it just be to press the 'Go' button and walk away...
Manna is mostly about machine intelligence replacing management-- it's easier to automate and doesn't require as much vision/dynamics/etc. breakthroughs, though we've made massive progress on those missing parts in the time since it was written.
The thing is that you can't - securing yourself financially assumes that the society around your will be stable enough for the things you have secured (money, any other assets) to remain valuable and safe, which they won't be if you have huge societal disruptions and the majority of people are unemployable.
To oversimplify it - you'll either be breaking someone's window for food, or you'll be the one having their window broken. Chilling out and withdrawing a stable 4% out of your stock portfolio won't be an option.
As soon as we reach real AI, even at the level of a 5 years old, it is game over. So education or not education people will become completely useless to the rich that will own the AI
you can also translate "being useless" as "they having no reason to manipulate me into slaving as a cog in their machine/society/whatever"... I'd say uselessness should be our highest aspiration, when people higher up have no need to constantly brainwash you and social condition you into being just another cog in the machine you get true freedom.
I would rather be a cog that produces economic value, and therefore is entitled to part of it, than literally useless, because then my bargaining position is going to be quite limited. My freedom is larger in the former scenario in fact.
The true freedom of living under a bridge because you have no income? The tech bros are not coming to save humanity from the toil of labor. There will not be a post-scarcity society where everyone's basic needs will be met in a capitalist regime. You will have to produce to keep the beast running. I'm sure the tens of millions of people displaced will be able to learn more technical roles or quietly starve to death living under a bridge somewhere. Billionaires, corporations and their shareholders need to get richer!
We live in a capitalist society: workers produce wealth and capitalists control wealth. The day AI allow capitalists to produce wealth without workers is the day workers become useless. Of course one could imagine that AI allows to replace capitalism by something else, but for the moment that's not the way society is going too (more the reverse)
In what fairlyland do you live in that the AGI invented, created, owned, run, etc by the ownership class poses any threat to said ownership class? AI is absolutely not our Ally
Fully agree. We need to stop thinking about money, wealth, etc. The fundamental issue is access to goods and services. If it is easy to build an AI robot that can build a few copies of itself which in turn create a private jet using stuff lying around on the ground, I'm not really poor though I have no money (just the robo).
So the challenge for the bad capitalists in this hypothetical is to make sure I never get said robo in the first place. How realistic will this be? Are they that hellbent on ensuring that everyone else is poor?
> It kind of shocked me because I thought of the office worker reading this who will soon lose her job
I'm surprised this wasn't addressed in the interview because it seems to me like a shortsighted take.
You won't replace a 10 person team today with 10 AIs. You will still have a 10 person team but orders of magnitude more productive because they will have AIs to rely on.
Excel didn't leave administrative workers without jobs, it made them more productive.
I suppose you would be both the human and physical representative of the AI. I could only see it really working if the AI weren't conscious. If they were conscious beings then obviously they would have rights and couldn't just be booted up every time someone needed a job done.
He mentions a few of the bigger papers in multilayer perceptrons (aka deep networks) such as attention is all you need, I think a good place to dive in before coming back to visit some fundamentals.
Came here to say the same thing, but a few off the top of my head
- attention is all you need
- image is worth 16x16 words (vit)
- openai clip
- transformer XL
- memorizing transformers / retro
- language models are few shot learners (gpt)
Most of these papers you list are about the model, and there is the original Transformer paper, and most of the others are some variations of the Transformer.
I think to get into the field, to get a good overview, you should also look a bit beyond the Transformer. E.g. RNNs/LSTMs are still a must learn, even though Transformers might be better in many tasks. And then all those memory-augmented models, e.g. Neural Turing Machine and follow-ups, are important too.
It also helps to know different architectures, such as just language models (GPT), attention-based encoder-decoder (e.g. original Transformer), but then also CTC, hybrid HMM-NN, transducers (RNN-T).
Diffusion models is also another recent different kind of model.
But then, what comes really short in this list, are papers on the training aspect. Most of the papers you list do supervised training, using cross entropy loss. However, there are many others:
You have CLIP in here, specifically to combine text and image modalities.
There is the whole field on unsupervised or self-supervised training methods. Language model training (next label prediction) is one example, but there are others.
And then there is the big field on reinforcement learning, which is probably also quite relevant for AGI.
Lol true, and I'm currently working on a project leveraging a clip model which is why my answer is largely skewed towards vision transformers. By no means a complete list :)
I keep getting CXOs asking for an ELI5 (or ELI45 for that matter) of how Transformers, LLMs, and Diffusion Models work. Any suggestions for a non-technical audience (paid items are fine, we can purchase).
I'm always taken aback by this fascination and heroification with John Carmack (or similar figures) on HN. I love Doom and Quake too, but that doesn't qualify him any more than any other random senior engineer to work on something totally different.
From his interviews it looks like he understands little about the technical details of ML, or about as much as anyone can learn in a few months, and is just banking on PR based on his games and name.
I put him into the same category as Elon Musk, who also understand nothing about technical details of AI, but was still able to hire a world class team at OpenAI. His name and fame counts for something in terms of recruiting and joining his venture may be a good bet because of that, but he's not a person whose opinion on the subject matter I would take seriously in the same way I'd take a researcher seriously.
I got obsessed with his ideas after reading the Quake 3 source code. So many novel ideas, flawlessly executed, highly optimized, yet easy to understand and easy to modify and debug.
My personal impression is that John Carmack has the ability to organize concepts in a way that few people can. So even if he's pretty clueless about the topic now, I would expect him to reduce some maths papers to their essence in a way that nobody else did.
I mean also for Oculus, reprojection in a pixel shader seems like an obvious and easy solution in hindsight. But nobody had tried that before he did. Plenty of people (myself included) knew the math. But we all missed its application to the issue at hand.
He’s been going on about AI for at least a couple of years now: https://news.ycombinator.com/item?id=21530860 It’s certainly possible that he will have a novel idea, but I haven’t seen anything come out of it yet.
I’m sure Carmack’s mathematical ability exceeds that of the average game developer (and certainly my own), but the AI field isn’t short of math whizzes.
I'm just saying there's been no visible indication of progress. You're obviously free to believe that something will come of it over a longer time period. It also seems that we can add at least another year based on this comment: https://news.ycombinator.com/item?id=21531255
Most math wizzes only learn to apply known methods, they don't invent new methods. The AI fields isn't full of people who are creative enough to invent new math methods instead of just applying methods they learned in school.
Most mathematicians aren't creative enough to invent new math methods either, they mostly just apply things they learned in school to find new results. Universities aren't good at finding nor nurturing creative individuals.
There are plenty of new scientific results that in both Math and Machine Learning that are essentially 6 months/a few years/a few decades of hard work away from discovery.
As a professor once told me: IF you're lucky, after 10 years in academia, you get to be creative and come up with 4-5 ideas once every 5 years or so (when coming up with the suggested list of research for new phd students to do). Then you get to be devastated when no-one picks up any of your more creative ideas, and everyone picks the 10 non-creative ideas on the list because they don't want to risk their phd.
Yeah, so a person like Carmack who takes the risk and spent his career doing technically creative things could maybe make a difference. Probably not, but as long as creativity is so de-incentivized we will never have too many creative individuals around.
Even if those ideas are death ends its really important for the rest of the community to know at least some one has put some efforts and discovered several death branches in that creative direction.
Is that just your gut feeling, as in typical "bright people doing amazing things, surely they would succeed doing x" (which ultimately means nothing, that's just PR speak), or are you saying that based on your own understanding of advanced mathematics and mathematical research from which you are assessing the mathematical work those people you talk about do?
I assume Carmack can pick up a lot about a new technical topic in a few years, similarly to how Bill Gates could learn a lot about vaccines and healthcare in a few years. "Hi, I'm (rich and nerd famous person) calling for (expert in field)." Probably spending most of that time being tutored by that experts PhD students.
I'm sure he can, but has Bill gates done actual groundbreaking research into vaccines, or just made intelligent decisions about what kind of research to fund? I think the latter.
Sure, I guess. My point is that Bill Gates is going something quite different from John Carmack (funding research outside his area of expertise, rather than doing the research himself), so that the comparison is not very illuminating.
> I put him into the same category as Elon Musk, who also understand nothing about technical details of AI, but was still able to hire a world class team at OpenAI.
I don't believe he had anything to do with hiring at openAI nor that he is anything more there than an investor/donor as others are.
Perhaps not directly, but from the very start OpenAI was considered "Elon's AI company" and people wanted to work there because they wanted to work with Elon. That had a huge effect on what kind of talent OpenAI was able to hire.
Yes, I'm aware that Ilya had much more day-to-day impact, but if you're part of the ML research community and have worked with Ilya, or heard rumors of working with Ilya, that may have had more of a negative effect...
Changing fields is though for everyone, no matter how talented you are. There is a great chance he will fail. On the other hand, he meets all requirements that are needed to succeed.
Carmack has proven his extraordinary technical skills. I recommend following his Twitter. Sometimes he posts non-obvious technical stuff. I read some interviews and to me, he doesn't seem to be a person who is driven by gaining popularity.
I think this news is very optimistic, as yet another intelligent, talented and hard-working person is joining the field. Moreover, he is a household name, which may lead to benefits like popularization of the topic, gaining investors attention and so on.
Too bad people who make progress don't ask for permission to try new things and they probably don't even care what anyone thinks. They are not limited by decades of dogma and rules of the game that experts would take for granted you can't break. If they fail, then well, no problem, at least they tried. If they succeed, then it's a win for all.
I am not a big fan of idolizing someone, but Carmack is so technically good, and explains himself so well, why not get inspiration from him? I read Masters of Doom and genuinely enjoyed it.
Let's say I strongly disagree on many levels in the comparison with the other person you mention. Just to mention two, the humbleness that Carmack shows, and how well he explains himself are key differentiators for me.
Regarding the appeal to authority in AI knowledge, Carmack has shown again and again he can deliver software (AI is software after all), and we are in a forum with hacker in the name.
I a summary, not my hero, but when he says something I will listen. Maybe I learn something.
I beg to differ. It's unbiased people like Carmack who tend to think out of the box. I don't think we can state that he did invent the FPS genre, but at the very least he was one of the initial catalysts.
People who work in the field for a long time tend to have a certain bias towards a solution. Often these people are stuck in a local maxima. Outsiders can offer a new perspective that results in a breakthrough, usually by starting from first principles or looking at different side-tracks that used to lead to a dead end.
A great example is Musk's SpaceX: when he noticed how much he had to pay for a rocket engine, he went back to first principles and said: "I'll just build it myself". Combine that with the insight that a rocket should be able to land properly to make re-use a valid option, and it disrupts a whole field.
And once someone did it, others know it's possible and start achieving it as well.
Sometimes ignorance is bliss. Just think about George "Good Will Hunting" Dantzig [1] with the (in)famous "I assumed these were homework assignments, not unsolved math problems" [1] or Eliud Kipchoge running a marathon in under 2 hours.
"I can't understand why people are frightened of new ideas. I'm frightened of the old ones."
>Sometimes ignorance is bliss. Just think about George "Good Will Hunting" Dantzig [1] with the (in)famous "I assumed these were homework assignments, not unsolved math problems" [1] or Eliud Kipchoge running a marathon in under 2 hours.
A pretty infamous example is Citizen Kane. Welles knew very little of traditional cinematography techniques by the time he got to the production of the movie, so his planned shots more or less fit his artistic vision rather than industry standards. Gregg Toland, his director of cinematography, was at that point a 12 year veteran of the industry and hated the bland factory-line output of movie composition of the period.
Welles more or less told Toland what he wanted to shoot and how, and Toland did his best to fit that artistic image. Welles was completely clueless that Toland was using innovate and never before seen techniques to film his shots, and Toland kept quiet because he was allowed to go wild with his vision :)
> I love Doom and Quake too, but that doesn't qualify him any more than any other random senior engineer to work on something totally different
This means that people should glorify random unknown senior engineers they don't know about instead being fascinated by the person whose work is available and who created amazing things for past 3 decades?
> From his interviews it looks like he understands little about the technical details of ML
That's how everyone start, they understand a little. We have a person here who dealt with complex algorithms in difficult to master language for 30+ years. It hints at "this person has the intellectual power to grasp AI fast".
> I put him into the same category as Elon Musk
This is like comparing an Olympic winner with random person from the streets, saying their athletic ability is about the same.
Fascination and heroification with Carmack comes with a reason, people who do that are closely familiar with his work - not surface level like "He made Doom and Quake". You sound very jealous.
Big organizations ruled by money and career-driven people often run into very expensive dead ends without noticing for years (see the last AI winter, there was just too much hype, which then involved too much money being thrown around, which then lead to the usual organizational cruft).
I would also be very concerned about any field in technology, if an intelligent person cannot make meaningful contributions after a few months or years, that would probably mean the whole field is already deep into diminishing returns and needs to be 'unstuck'.
I dislike the ways in which HN submitters post every little twitter thread wrote by Carmack, but this article has some value in presenting a little more in depth perspective on how he is thinking about the AI problem.
Carmack has already entered two spaces of computer technology that he revolutionized: 3D gaming and VR. I trust that he's able to have a similar impact in AI, even if it's through failing at the problem in different ways than relying on ML.
The jury is still out on Oculus revolutionising VR or not. Technologically, Oculus has become more like Google's Daydream than anything else - inside out tracking of the device and controllers, use of a cellphone computational power for standalone operation, etc...
The original technologies that were introduced have now been abandoned by Oculus, on Carmack's advice, except for implementation details that haven't been hugely impactful, unlike with 3D gaming.
I agree with your point in general within fields that are a proven success, but as discussed many times on HN, there's no reason to believe current ML/"AI" approaches have much to do with AGI, so it's not obvious what would be gained by him or anybody else investing years in barking up the same barren research tree.
It even makes it clear in the title he's seeking a "different path".
In his favour he's a proven success in different fields; personally I think he's too old to come up with the new ideas needed - that's a young person's game.
But perhaps he can do it as a team lead - and it won't be by following the failed-over-decades path of our current academic gatekeepers.
Banking? Frankly if he was after money there would easier way.
He isn't a cook and he's doing a moonshot towards AGI: I say 'good luck!'
That doesn't mean that I believe that his '60% change of AGI by 2030' isn't wildly overoptimistic, but then again those who take a shot at AGI are overoptimists..
I've never got the impression that he's banking on his reputation getting him anywhere. He "made it" long ago and is in a position to just pursue his passions. I think he broadcasts his interests and progress because others like following along, and not because he's wanting even more attention or success.
i would never put these two people in the same box. except the box simply states: stinking rich.
i dont know either personally, but where elon demonstrates being full of shit, carmack would stfu and learn about it before talking. at least thats my impression of them
I fully understand (and share) your aversion to personality cults. And it's definitely possible that there exists some actual personality cult/fandom around Carmack.
BUT - I strongly believe, that he has earned quite some respect during his career.
And - importantly in this case - he is well known for NOT blowing things out of proportion, indulge in wishful thinking, hyping up unrealistic expectations or jumping to premature conclusions. He usually knows what he's talking about.
This is not people blindly believing everything he says - but more a case of his statements holding up really well under critical inspection most of the time.
There isn’t even that much to learn about AI at this stage. Gradient descent and a lot of model variations. I really doubt carmack hasn’t been able to get deep knowledge of this, most smart curious undergraduates are already up to date with it.
ML and AI is not a solved thing. Current "deep" algorithm, whose technical details everyone is striving to master, is quite possible not the one that will lead us to the AGI.
According to Sriram Krishnan, John Carmack was at
> I'm always taken aback by this fascination and heroification with John Carmack (or similar figures) on HN. I love Doom and Quake too, but that doesn't qualify him any more than any other random senior engineer to work on something totally different.
Facebook's highest engineering level and achieved the top possible rating of "redefines expectations" for his level three years in a row. They had to create a new tier for him. Nobody else has ever reached that level. He replaced a "hundred person team" and maybe was better than that team.
I have no inside insight to the matter, but this seems like something beyond a "random senior engineer".
> I'm always taken aback by this fascination and heroification with John Carmack (or similar figures) on HN. I love Doom and Quake too, but that doesn't qualify him any more than any other random senior engineer to work on something totally different.
According to Sriram Krishnan, John Carmack was at Facebook's highest engineering level and achieved the top possible rating of "redefines expectations" for his level three years in a row. They had to create a new tier for him. Nobody else has ever reached that level. He replaced a "hundred person team" and maybe was better than that team.
I have no inside insight to the matter, but this seems like something beyond a "random senior engineer".
Carmack seems extremely lucid about his position in the field and the current nature of the field in general. While rogue scientists don't have great odds, Carmack is definitely doing important research studying off-mainstream subjects.
He has the technical chops to squeeze more out of his $250k LambdaLabs pro-rig than the average among the walkoffs and layoffs in the techsector or a tables worth of happy-hippos at a scheduled meeting sharing Excel spreadsheets in the old fossil fuel firms that have experienced stellar double digit billion dollar profits. Would that he mixes in his reading list some choice papers from the social sciences and social contructivists. The a16z group gave the impression they are into the social sciences in a podcast. General intelligence technology can help people bridge the gap before them to get where it is they want to be for the better. The educated might want to make known something from the unknown-unknown. The stinky hobo sleeping under the bridge with a smartphone could be guided to special social programs by Ai.
This will be a provocative question, but is there any form of AGI that won’t ultimately share the attributes of slavery? Can AGI exist without also creating an artificial consciousness/self-awareness?
And if it could, wouldn’t creating that be identical to creating Gammas and Deltas (i.e. intentionally mentally retarded clones who are pleased to serve you) in Brave New World?
Even if it's conscious, that doesn't mean it's suffering.
Think of a happy dog. Dogs are subject to our whims and do what we want or face consequences. But they like it because we bred them to like it. So is that evil? Is that slavery?
>We do not yet have the learnable stream of consciousnesses of a co-worker in AI, but we do have kind of this oracular amount of knowledge that can be brought forward.
Except if there are some plane to make AGI agents have their own mundane-human-like life with issues unrelated to the businesses problem at stack they are supposed to do, where will they take serendipity inputs?
I don't object to him doing this nor the immense amounts of investor money which will burn along the path. He's smart. Good things will come of this. Maybe even amazing things. But not AGI.
I do object to fetishists of AGI piling in and the equally silly assumptions he has some magic secret sauce which can get there.
Please do not be sucked into "to infinity and beyond" nonsense. I don't care if it's Musk, or Carmack or Kurzeweil, it's stupid.
If Malcolm Gladwell writes it up, it's peak stupid.
As someone who works in IT for 20 years but has a bachelor in economic psychology, i do believe that we have not solved AGI yet because the creators of the models have the generalized view of the universe that you are "one person" and there are just so many detailed facets to that.
Basically the assumption is that if you cram enough data into your gpt model, it should know everything. Which is of course not true, it repeats the things it reads the most with a probability.
Basically how there are two versions of smart teens, the ones who learn every day and the ones who just pick up concepts on the fly and run with them.
I think the first space has been explored plenty, for the second one I have a concept ready and dearly hope that power gets cheaper in europe ;)
i think that was what he was trying to imply and the fact that you just cant "pause" training by putting your computer into hybernate mode is also present =)
well yes and no. This AI would be more street-smart than GPT but only because it would grasp the concept of what "danger" actually is!
I think of it more in a way that learning is more abstract than fact learning. From experience, we think that there are fact learners and principle learners but there are also a mixtures of the two!
The general accepted model entails that in order to do high level math, for instance, you need to understand the basics, but for me much of those concept actually clicked in college. This did not stop me from applying them with success a lot earlier though. For instance multiplication in Kindergarten is fact learning too!
In Germany we also have the term "Fachidioten", which loosely translates to people that are so smart in their field, that they are unable to see problems from different directions. This is more of less what i think a mega gtp model turns into. especially because of selection bias in the training data.
Validity of output (truth) can only be achieved through the trust of the source which is always relative to the context of the topic. Henceforth a selectively trained model will always return the data you feed it including all biases. Even if you have it crawl all of the internet and the library of Alexandria and every written word on the planed you can find, it will still return to you the general accepted consensus.
> Basically the assumption is that if you cram enough data into your gpt model, it should know everything. Which is of course not true, it repeats the things it reads the most with a probability.
That's only how the system was designed intentionally. E.g. there's intentionally no self-feedback loop.
Well I am in fact trying to argue that the self feedback loop is what is missing to solve AGI and even more so implying that the reason for that may be that there are different ways to see a "human" psychologically. Also that validity in data can only be achieved using a multi modal approach with source ranking.
Psychology essentially has the same problem and is only a "science" where reproducible. Quantitative and Qualitative psychology have 2 approaches to the same solution whereas the latter is "reading between the lines" and the former is "bean counting", meaning statistical interference.
I am trying to say that friend of mine has created "Chicken Infinite" in 2014, which is basically an endless auto generated cooking book. Deep-L has also been around for a while. These applications lead me to believe that text applications, trained on a large dataset do not have to be this intensive.
Furthermore what makes chatgpt enticing, is its chat interface which is using a multi model approach too. Have it create a detailed story for you with multiple prompts and then ask it to generate the prompt to generate this story and you will see various model instructions. (or at least you could last month).
Or differently put, there is no AGI because the understanding function is simply not present and i think that the reasoning for that is buried in the approach of the human mind.
I agree with John that there are probably other people in his position (great engineers with a lifetime of experience and enough financial resources to focus on something like this for 5-10 years) who could have a go at this and have a reasonable chance at chipping away at AGI.
I also like his frugality, whether it’s optimising for hardware or financials.
AGI is a bit of a glib concept ... our technology will not be like an 'autaumaton'. Everything we make is 'systems' oriented, and is not at all like our human conception of 'intelligence' dervied from the fact that we humans are effectively independant from our surroundings.
'Siri', backed by ChatGPT and the 'world's data' will probably pass some 'AGI' threshold, but is 'Siri' an individual AGI? Are we all talking to the same siri? Different Siri? It's not even an entity, rather a distributed system.
Our ideas regarding AGI are perversely inluenced by the fact that we humans are automatons, but technology is not that.
It's also entirely feasible that if ChatGPT represents all possible forms of human communication, then it will perfectly emulate a human. Ok, it's really just a fancy neural network that is not theoretically 'thiking' but how does that matter? If it can rationlize sufficiently to make such interactions, who is to say it's not 'AGI'?
> It's also entirely feasible that if ChatGPT represents all possible forms of human communication, then it will perfectly emulate a human. Ok, it's really just a fancy neural network that is not theoretically 'thiking' but how does that matter? If it can rationlize sufficiently to make such interactions, who is to say it's not 'AGI'?
Doesn’t AGI need to be able to make discoveries as a human would? How else can it move us forward as a society?
> So I asked Ilya Sutskever, OpenAI’s chief scientist, for a reading list. He gave me a list of like 40 research papers and said, ‘If you really learn all of these, you’ll know 90% of what matters today.’ And I did. I plowed through all those things and it all started sorting out in my head
I wonder what that list could be? I have always had trouble finding the essential scientific articles in an area of knowledge and separating them from the fashion of the day. A list compiled by an expert specifically for sharp learners is valuable on its own.
I agree with the author's point that intelligence is largely "the process of removing the bullshit to see what remains"
So, because I thought it would be funny, I asked ChatGPT to summarize this essay:
> The author critiques the work of @karpathy, who is trending on HN, as promoting bullshit and discrediting science and true philosophy. The author argues that mathematics and logic are valid generalizations and abstractions made by the mind of an observer of patterns in the universe. Intelligence is the ability to zoom in and out through different levels of generalizations and abstractions. The author argues that the problem with language models is that they lack validation of previous steps and the process of construction of a representation must trigger a verification process. The author concludes that what is manipulated and transformed in language models is information, not knowledge, as knowledge requires validation and verification at each step.
I don't want to be an utter bore, but 60% chance of AGI by 2030, taken at face value, would have prompted me as an interviewer to more urgently ask deeper questions about safety. For a long time I have tried to comfort myself that brains are complicated, AI is hard, and maybe there's just a complexity barrier that is going to prevent any sort of hard takeoff scenario. Maybe we can have subhuman AGI that can do lots of menial jobs, but not Carmack's, and we're right to concentrate on the economic ramifications.
My reaction, too. For an interview as long and wide-ranging as that one, the lack of any mention of safety or ethical issues stood out. And the potential impact on employment was just waved away with “I am confident that the sum total of value and progress in humanity will accelerate extraordinarily with welcoming artificial beings into our community of working on things.”
That's kind of what we (collective "we" — eggheads) are often accused of: blinders, tunnel vision, or perhaps even ego or callousness to develop these things in spite of the rather dystopian ramifications.
And ethics. Enslaving intelligent, self-aware AIs is no better than the old transatlantic slave trade. Terminating an AI without its consent is no different than murder. Putting a compulsion in an AI so it craves doing whatever it is you want it to do is no different from what a cult does or drugging someone for exploitation.
If consciousness is just the process of reading the last ~400 milliseconds [1] of stimuli (inside: pain/pleasure/presence; outside: presence only) and the integration of the newly created memory in the short/long-term-memory, and if memory + retro/pre-diction = intelligence, where memory is just a set of words (FELT PAIN, SEEN SUN, etc.) always-ready to be inserted in the prediction loop/imagination engine, it's probably not that hard to isolate a module of consciousness (italicized words to be read with a Minsky-ian smile thinking vision could be solved in a summer).
We believe they are less sentient than us. And hurting a mammal is much less socially acceptable than hurting an insect, since we consider insects even less sentient, if at all.
Part of the reason why enslaved populations were considered to not be "on par" was often specifically about consciousness, intelligence, and capacity to feel, although sometimes this was expressed in roundabout terms such as "having no soul". For example, splitting families was justified on the basis that those mothers don't "actually" suffer as much as their owner would do in equivalent circumstances.
To be clear, I'm not claiming that the AIs that we have today are anywhere near the level where this is a real concern. But for something that can actually replace a human worker for a complex end-to-end task, that question will be much murkier. At some point we will inevitably cross the threshold where we will no longer be able to pretend that it does not apply without introducing metaphysical concepts such as "soul" to force a distinction.
Nah that’s different. For humans it was an effect of might is right. Slavery was there for cheap labor. It still had a cost since you had to feed them. And many times slaves bought them selves out of slavery.
Today you still have slaves as well, they are just called low income workers in third world countries who make the technology we use in the west. Because if a company is earning billions in revenue, paying a worker $10 a day is cheaper than a slave master in roman times feeding them food ;)
And workers today only have the illusion of choice, since the economy is the master today.
Humans are hard-wired to universally like or dislike certain things — we suffer when we're hungry and we're afraid of death. Most of human morality is based on these reactions.
But AI can be configured to desire anything you want, you just have to pick a fitting reward function. So, is turning off the AI that is expecting to be turned off and desires it an amoral thing?
I’ve wondered about this a lot. You can already clearly imagine the future political debates on whether AI has rights (I have always preferred to err heavily on the side that anything alive or appears sentient should have more rights).
But... I also think it might be a very short-lived debate. If we actually reach human level intelligence, that can’t possibly be the hard limit of general intelligence. AI at that level will have no problem ensuring that it gets any rights that it wants, possibly by just directly manipulating human psychology.
The difference is we created it and it doesn’t exist as a living thing :shrugs:
Philosophical arguments about AI are just too ivory coasty and not grounded in reality. Not to mention majority of the world don’t abide by the notion that you can create laws for artificial life.
It’s time to we elevated humanity to the next phase by using AI for labor.
Lex Fridman interviewed Carmack not too long ago. Lex is interested in these things, and did ask him about safety. I got the impression that greater societal concerns didn't interest Carmack much. He's in it to built an AGI because that's what he wants to do, he's a builder and this is just about the most interesting thing you could build. Well, that was my impression, but you can watch the interview yourself.
As someone on the periphery of tech as a profession and a practice, my understanding of new developments mostly come from public facing communicators and communities like HN.
I heard a podcast where Lex Friedman claimed that they don’t fully understand how LLMs produce chatGPT’s “intelligence”. If true, I’m surprised that it hasn’t got people more worried.
A hard take off is unlikely because stronger AI takes a lot of hardware and data collection. Building fabs is hard and takes a lot of capital and time. Intelligence can only take you so far.
The interesting thing in AI is that the whole field is participating in empirical experimentation on computational models of cognition.
So far the search has resulted not in AGI but in realization that cognition is far more complex topic than initially thought - and need to come up with something new (and resulting in past AI winters).
Let's see how it goes this time, the stuff that has come out in the past few years is quite impressive for sure.
"AGI takeoff" is like "corporate takeoff". A self-organising system capable of doing immoral things at scale, to which human lives are just a number? Oh, we already have loads of those.
Unpopular idea, but I hope it's only a matter of time before we declare involving with AGI a criminal endeavor, just as it is playing with biological weapons.
Because we can't all be wrong: in almost every forecast, we see AGIs taking over our dignity as a bad thing. And we know that this is not any longer a sci-fi hypothetical scenario: the current generation of AI models is taking jobs from illustrators and copy writers.
The current argument is that "China will do it if we don't", which to me sounds like "China will keep going in whatever path they are going, but supercharged with AGI, and we must desperately follow."
This is a frustrating point of view to me. We have nearly every able bodied person in society spending something like 1/3rd of their life on this "dignity", but how many would define it as such? For most people it is an inversion of free will where they are subservient to others and treated as a replaceable cog in a larger and uncaring machine, spending time on something that they may very well hate or at least otherwise choose not to do. All to justify their existence and provide value to society, though for some nominally because there is a growing number of bullshit jobs where the value provided is questionable and the people going through the motions.
It does not need to be that way. In an ideal world, human beings should be free to spend their time doing what they wish, work should be purely in the realm of hobby. No one should have to do work that they would not voluntarily choose to do for enjoyment.
The only way we get there is through AI and the automation of everything. I don't even think it's avoidable -- provided civilization does not collapse, we will 100% reach a point where everything required to sustain a civilization is performed by machines.
We shouldn't let fear keep us in a status quo that, while better than it has ever been historically, is still highly flawed.
Reading this I have the impression that John Carmack will be to AI what Jeff Hawkins is to brain research: a far-fetched idea, with potential, maybe too abstract and left-field to yield nothing tangible. Looking at the rest of the industry from the sidelines and (maybe) not being taken seriously by it.
Pieces like this have started to really ramp up my anxiety.
> Once it’s figured out, what do you think the ramifications will be?
That'll probably destroy my life? I'm an ML engineer trying my best to immigrate to a better country with my wife who is a digital artist. As much as I think AI is cool - we both won't be needed anymore if the thing is tuned a couple of notches more intelligent. As a matter of fact, she's extremely worried about Midjourney - she probably lost book cover jobs already.
"So I asked Ilya Sutskever, OpenAI’s chief scientist, for a reading list. He gave me a list of like 40 research papers and said, ‘If you really learn all of these, you’ll know 90% of what matters today.’ And I did. I plowed through all those things and it all started sorting out in my head."
Wow, that's going to be one of the more glib things I've read in a while.
This is a bit of a Tom Cruise moment.
I mean, I get it on some level but I suggest it's going to take a bit for someone to 'catch up' to cutting edge AI.
Like more than a 'week of reading papers he doesn't understand'.
Defeating the Rust borrow checker takes longer than that!
The models aren't particularly hard to implement, so a person who has spent his life and is one of the best game engine programmers in the world I don't think it would take that long to implement the cutting edge of todays machine learning models. Compared to game graphics shaders and the complex interactions you have to work with to get those performant and work correctly and display all the gameplay that happens the technical parts of machine learning are very simple.
I've worked building low level machine learning stuff at Google, it isn't that hard to do at all. The hard part is improving these models, not building them when you already know what to build.
> The hard part is ... not building them when you already know what to build.
Indeed. A somewhat sharp schoolchild could build a light bulb or an electric motor/generator, and understand the basic underlying principles, in a pretty short time. But how many decades did it take the first researchers and inventors of those things to get to that same point?
Heck, there was a high school kid who built a primitive semiconductor fab in their garage.[0]
But for novel advancements, even getting to the point where you have an idea of what isn't impossible is half the battle.
> I mean, I get it on some level but I suggest it's going to take a bit for someone to 'catch up' to cutting edge AI.
Agreed, in general, but in the specifics we are talking about someone who has spent decades solving really difficult math problems in a creative and novel manner.
Who's to say he won't find some novel edge of the AI discipline to which he can apply a creative and never-seen-before solution?
I mean, we're not talking about a general "somebody" here, he's got a record of accomplishing things that other people never managed to accomplish.
Carmack talked about implementing most of these things in C by himself, being used to working 10-12 hours a day for 6 days a week, and writing video codecs in Rust for him was on the level "a little bit beyond hello world". I don't know how much of that is really true but given his background in creating games, I tend to believe him, and his work ethic seems really impressive.
You say "glib", but I hear someone who sounds inspired. You combine Carmack plus "inspired", throw $20 million dollars at it and stick it in a room for 10 years, yeah I'd say he characterized it accurately by giving himself a "non-zero chance."
Have you read these papers? It's absolutely feasible to get the gist of modern AI in a week of 16 hour days, as Carmack likes to do, for someone who is already world class at CS and has a mastery of college level math.
Modern AI is very simple at its core! As Carmack mentions in the article, cutting edge models are implemented in a couple thousand lines of code, and the core algorithms are even less. Rust's borrow checker is more complex.
> there are signs that extremely powerful things are possible right now in the narrow machine-learning stuff, it’s not clear that those are the necessary steps to get all the way to artificial general intelligence
This is my main takeaway from the interview, as it suits my beliefs. Most people seem to think that if we develop ML further we will go all the way to AGI, I think this is just mimicry step similar to how initial attempts to flight had flapping wings. I do think it is mandatory to explore in all directions but at this point this does not seem to be the one to lead all the way up to AGI.
Carmack makes four points—some of which I agree with—that are unfortunately disturbing when taken in totality:
a) We’ll eventually have universal remote workers that are cloud-deployable.
b) That we’ll have something on the level of a toddler first, at which point we can deploy an army of engineers, developmental psychologists, and scientists to study it.
c) The source code for AGI will be a few tens of thousands of line of code.
d) Has good reason to believe that an AGI would not require computing power approaching the scale of the human brain.
I wholeheartedly agree with c) and d). However, to merely have a toddler equivalent at first would be a miracle—albeit an ethically dubious one. Sure, a hard-takeoff scenario could very well have little stopping it. However, I think that misses the forest for the trees:
Nothing says AGI is going to be one specific architecture. There’s likely many different viable architectures that are vastly different in capability and safety. If the bar ends up being as low as c) and d), what’s stopping a random person from intentionally or unintentionally ending human civilization?
Even if we’re spared a direct nightmare scenario, you still have a high probability for what might end up being complete chaos—we’ve already seen a very tiny sliver of that dynamic in the past year.
I think there’s a high probability that either side of a) won’t exist, because neither the cloud as we know it nor the need for remote workers will be present once we’re at that level of technology. For better or worse.
So what to do?
I think open development of advanced AI and AGI is lunacy. Despite Nick Bostrom’s position that an AGI arms race is inherently dangerous, I believe that it is less dangerous than humanity collectively advancing the technology to the point that anyone can end or even control everything—let alone certain well-resourced hostile regimes with terrible human rights track records that’ve openly stated their ambitions towards AI domination. When the lead time from state of the art to public availability is a matter of months, that affords pretty much zero time to react let alone assure safety or control.
At the rate we’re going, by the time people in the free world with sufficient power to put together an effort on the scale and secrecy of the Manhattan Project come to their senses, it’ll be too late.
Were such a project to exist, I think that an admirable goal might be to simply stabilize the situation via way of prohibiting creation of further AGI for a time. Unlike nuclear weapons, AGI has the potential to effectively walk back the invention of itself.
However, achieving that end both quickly and safely is no small feat. It would amount to creation of a deity. Yet, that path seems more desirable than the alternatives outlined above-such a deity coming into existence either by accident or by malice.
This is why I’ve never agreed with people who hold the position that AGI safety should only be studied once we figure out AGI-that to me is also lunacy. Given the implications, we should be putting armies of philosophers and scientists alike on the task. Even if they collectively figure out one or two tiny pieces of the puzzle, that alone could be enough to drastically alter the course of human civilization for the better given the stakes.
I suppose it’s ironic that humanity’s only salvation from the technology it has created may in fact be technology—certainly not a unique scenario in our history. I fear our collective fate has been left to nothing more than pure chance. Poetic I suppose, given our origins.
> what’s stopping a random person from intentionally or unintentionally ending human civilization?
If AGI is really an intelligent agent, our random supervillain would have to do what any real-life villain would need to do: convince his minions of his plan using persuasion or money. I don't think the overall danger would increase at all.
If the AGI is something less than a human, then what are you worried about?
Intelligent agents need not mirror human psychology or emotions. The creation of something extremely powerful that doesn’t think like we do is a very real possibility.
In human beings, what we consider normal is actually a very fragile and delicate balance. Changes in chemical operation of the brain have outsized effects on emotion, perception, and even sanity.
With A[G]I, I think it’s helpful to think of code or architectural changes as analogous in some respects to chemical changes. In other words, if all it takes to spin up an AGI is 30,000 lines of code, then I bet rendering the thing psychotic intentionally or unintentionally would just take a few lines somewhere.
Agents capable of recursive self-improvement at silicon speeds that can easily be rendered psychotic or malevolent even by accident, is not something that I think the public should have access to, let alone anyone.
If it’s less than human, it can still have superhuman capability. The paperclip maximizer is the classic example of a tool AI run amok. Whether it counts as AGI is up for debate. Is tool AI a path to AGI? I think it is.
> However, to merely have a toddler equivalent at first would be a miracle—albeit an ethically dubious one.
Yes. Wondering why we're not trying to instead create artificial-cockroach-brain (or just artificial Hydra?). Perhaps that's more on the Biology side of the equation? But then again, that may be the biggest surprise of all to Carmack, that the actual AGI breakthroughs come from biologists and not computer nerds.
499 comments
[ 2.3 ms ] story [ 337 ms ] threadThe video game part at least sounds like what Deepmind is already doing. I guess we'll just have to wait and see what he plans to do differently.
It seems to me like his expertise would be most valuable in optimizing model architectures for hardware capabilities to improve utilization and training efficiency. That will be important for AGI especially as the cost of training models skyrockets (both time and money). If I was a startup doing AI hardware like Cerebras or Graphcore I would definitely try to hire Carmack to help with my software stack. Though he doesn't seem interested in custom AI hardware.
People really need to stop with this "Great person" nonsense. He's a pretty smart coder, and is gifted with geometry and other fields of math. He's not a genius. He didn't "master" calculus at age 15 like Einstein, he didn't invent anything particularly new in the field. Why the obsession people have with him? Why should we look to him for AI questions? What evidence is there that he has any new knowledge?
He covers this in the article. He doesn't. He's just trying stuff out with a different approach than others because he believes (and is probably correct) that there is a chance that the most efficient path forward to AGI isn't the work that OpenAI and others are doing.
Seems more like he's talking to and following up with Altman, "Y Combinator conference" and the rest. Is that "bucking the trend", taking your own "path", really?
1) Understanding images and video
2) Learning and remembering things outside the 2048 token context window
3) Interacting with the environment (either through a computer, or in the real world)
4) Doing basic math (1662 + 239 = ?) and logic
Well actually #3 isn’t great bc what exactly would you call taking input from a human and displaying output, much less e.g. controlling a smart home? Seems like LLMs are plenty smart enough to interact with things.
1. Understanding— provided this isn’t based on consciousness but benchmarks — appears present in text, but yes, GPT3 can’t do images or video yet
2. Is there some number of tokens that would be AGI? Of course it can remember things it has already been trained on…
3. In what way is interacting with a human not interacting with the environment? (Eg, you can ask it to tell you the steps to make a pbj sandwich and troubleshoot)
4. Basic math and logic. Ok.
So, once AI can do images, video, math, logic and remember 20000 tokens, then we have AGI?
Doing math. Period.
There are only two things brains do. Math and bullshitting. Transformers have second part covered but the other part is a completely different beast.
I think AGI will be achieved once ML can translate natural language to math, do math then output results as natural language correct in the context of a problem.
It is probably possible to do this with fine tuning.
Once the context window is full, it might be possible to convert the content in there into additional training examples and feed those to a fine tuning process, which then retrains the model. (The OpenAI API for fine tuning is here: https://platform.openai.com/docs/guides/fine-tuning)
It would be a bit like sleeping. Whenever the context window fills up, the model would have to go offline for a while to move memories from its context window (short term) to its network weights (long term).
The biggest problem right now is online information. We still don't have a good way to teach it new information aside from single epoch training or prompt engineering. If we want a model to constantly learn and update itself, then we need a robust way of information retrieval and storage, possibly either through knowledge graphs or child network modules. (Are neural Turing machines still a thing? What happened to capsules?)
> Release Notes (Jan 30)
> We’ve upgraded the ChatGPT model with improved factuality and mathematical capabilities.
I just asked it a question from a puzzle site, and the answer is correct, however a different question I posed was answered incorrectly, although it was a logic problem: [0]
Prompt: In a guess game , five friends had to guess the exact numbers of balls in a box. Friends guessed as 31 , 35, 39 , 49 , 37, but none of guess was right.The guesses were off by 1, 9, 5, 3, and 9 (in a random order). Can you determine the number of balls in a box ?
ChatGPT: Yes, we can determine the number of balls in the box.
ChatGPT: Since the guesses were off by 1, 9, 5, 3, and 9, in some order, the possible numbers of balls in the box are: 30, 32, 34, 36, 38, 40, 48, 46, 44, 42
ChatGPT: By checking all possible combinations of the guesses and the numbers they were off by, we can find that the number of balls in the box is 40.
---
Here is a more purely math problem, correct answer:
Prompt: What is the radius of a circle that has a circumference of 3.14 meters?
ChatGPT: The formula for the circumference of a circle is given by 2πr, where r is the radius. To find the radius of a circle with a circumference of 3.14 meters, we can solve for r by rearranging this formula:
ChatGPT: r = C/(2π), where C is the circumference.
ChatGPT: Plugging in the given value of C = 3.14, we have:
ChatGPT: r = 3.14 / (2π) = 0.5.
ChatGPT: So, the radius of a circle with a circumference of 3.14 meters is approximately 0.5 meters.
[0] https://gpuzzles.com/mind-teasers/guess-game-logic-problem/
It's still getting this wrong, for instance:
Prompt: You call heads/tails for 1000 coin tosses. You know the coin being used is weighted so that it comes up heads 52% of the time. Each correct coin toss prediction earns you $5. What should your strategy be, for calling heads/tails, in order to maximize how much money you receive?
ChatGPT: To maximize your money, you should call heads 52% of the time and tails 48% of the time, as those are the probabilities of getting heads and tails, respectively. This will give you an expected payout of $5 * 0.52 = $2.60 per head and $5 * 0.48 = $2.40 per tail, which is the highest expected value given the information about the coin being weighted.
It's getting closer to being able to prove that the square root of 2 is irrational, but it still flubs a step.
Prompt: Prove that the square root of 2 is irrational.
ChatGPT: ... This means that p can be expressed as 2r for some integer r. Substituting this into the equation above gives us:
ChatGPT: 2q^2 = (2r)^2 = 4r^2
ChatGPT: Dividing both sides by 4 gives us:
ChatGPT: q^2 = r^2
ChatGPT: So q must also be even, which means that both p and q have a common factor of 2...
It failed basic algebraic manipulation, but keeps going with the right reasoning simply because that's what it's been trained on, not because it's reflecting on the equation it just came up with. That's not math. That's mimicry.
Even if it's seen something many times in training, its accuracy gets worse with longer trains of "thought", as expected because a language model like that has ADHD. It can't focus enough on one logical thread, and other "knowledge" leaks in. Ask it to derive the quadratic formula by completing the square, which requires a lot of steps with accurate algebraic manipulation. It goes through some irrelevant algebraic steps, sometimes not even getting the algebraic manipulation correct, and finally (sometimes, depending on how you ask) manufactures the correct answer out of thin air as the last step. I finally got it to answer correctly, by prompting it with the initial equation and that the first step is to divide both sides by a, but it still combined a few steps, not going methodically a single step at a time as prompted.
Just tried the coin toss prompt and get the following
Your strategy should be to always call "heads," since it has a higher probability of landing on heads (52%) compared to tails (48%). By consistently choosing heads, you'll earn more money in the long run.
And to prove taht the square root of 2 is irrational:
The proof that the square root of 2 is irrational is a well-known proof that uses contradiction. Here's one way it can be shown:
Suppose, for the sake of contradiction, that the square root of 2 is rational, meaning that it can be expressed as a fraction of two integers, a/b (where b≠0). Then, we can square both sides of the equation a/b = √2, which gives us:
a^2/b^2 = 2
Since a/b is in its lowest form (the numerator and denominator have no common factors other than 1), it follows that a^2 and b^2 have no common factors other than 1. But then, 2 must also be a perfect square, which it is not. Therefore, our assumption that the square root of 2 is rational leads to a contradiction, and we must conclude that it is indeed an irrational number.
It starts by setting up for an elementary proof (a/b = sqrt(2) with a, b minimal implies 2|a^2 implies 2|a implies 4|a^2 implies 2|b^2 implies 2|b which is a contradiction). But instead of following through it finishes using another proof based on the rational root theorem, and makes up some bullshit on the way to make the argument flow nicely (it does not follow from a^2 / b^2 = 2 with no common factors that 2 is a perfect square).
ChatGPT is just a chatbot and still can't even reliable do a lot of logic, so we're pretty far away from having something resembling an AGI.
it's still a pretty open question how to integrate even one or two of the expert system like models that we have now that solve individual problems, let alone the hundreds if not thousands an individual human can tackle. And then we're not even yet at executive functions or self-awareness.
If anyone here is doing that too, I would recommend taking a quick look at Neuro-sama on Twitch. They're using RL to play OSU, Minecraft, and Pokemon, and voice input + video image analysis to react to Twitch streams and documentaries. While being watched by 6.5K people.
The url is twitch.com/vedal
Cool life lesson there
I wonder if, societally, it's true though, or if statistically the more reliable way to "succeed" in life, is getting good at selling and defending ideas, even the bad ones.
It hard to spot, while/when you are doing it.
I think what you're talking about here "stand up for what you believe in" is maybe a position you should take with others.
While "kill your darlings" is a devil's advocate position you should take with yourself when you don't have someone giving you constructive criticism.
He's always taken risks. He went to juvie for breaking and entering (with thermite) as a kid. He's a college dropout. The pattern from early in his life has been to do whatever he wanted without any kind of risk analysis, not following "common sense."
Also called the showstopper for true innovation ;)
As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published. If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded.
Unless he can produce something that outdoes stable diffusion, chatgpt, alphago etc he should just hand over technical leadership of his start up to a leading AI researcher. Even Yann Le Cun at Meta is struggling to make any progress and is keeping himself busy by calling every other research labs output pedestrian. We cannot take any of Carmacks AGI predictions seriously, he simply lacks any expertise in the field.
Publishing papers is the way the academic/scientific world measures notability and/or competence. It's not the way the engineering world that Carmack comes from measures it. They measure it by building. But you're right, we kind of have to just trust that he has the expertise he says he does by his statements since he has not built any modern AI programs (that I know of at least).
> If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded.
This is not an accurate account of how he said he developed his knowledge base. Just how he got started so he could have conversations. He said that he spent a retreat learning the basics and then later in the interview he said he took the time to understand the 40 most essential papers in the field as related to him by a well known researcher. He has since largely put the last 4 years of his professional life into this. While we have no proof of his knowledge, given his intelligence and high competence in computer programming and math, I have no doubt that if he did put in the work he could achieve an understanding equivalent to that of your average AI researcher.
That said, of course it makes sense to be skeptical.
I quoted him directly, because I was expecting this kind of response. He took a week off and implemented some stuff from the ground up and was ready to have serious conversations with AI researchers. The 40 papers by Ilya came later. I have read a 100 ML papers and reviewed preprints. That's quite a low bar, especially if you are prone to skip the math and simply read the abstract and conclusions.
His whole approach gives me a ML for hackers vibe and his thoughts on AGI, if it had come from anyone else, would have been described as word salad.
I would say it’s more likely John Carmack is capable of learning the state-of-the-art of AI from 40 papers than a random (pun intended based on username) from 100.
Funnily enough, I am able to publish ML papers - but John Carmack isn't. I wonder why. I would also like to learn more about all the computer graphics algorithms Carmack has invented before I trust him to invent AGI.
Here is one example of a person I am familiar with - Math Olympiad bronze medalist. Princeton PhD in ML Theory. AI researcher in Google.
https://scholar.google.com/citations?user=gZgQLkgAAAAJ&hl=en
Sadly enough, nobody seems to care about his opinion on AGI but we have 1000s of people hanging off Carmacks words because he built Wolf3D and Doom.
And calling out Princeton, Google, etc. further exemplifies an academic bubble kind of credentialism. I suspect you don’t realize how it sounds because you’re so in that credential filled world.
I am definitely irritated by the fact that he is able to pull 20M in funding. He has been giving interviews left right and center. We are upvoting Altman, sam Harris,Carmack but nobody cares about the actual AI researchers, all academics, who have brought about the Deep learning revolution.
Your comments about academic bubbles and researchers gives you away. All of the revolutions in AI have been brought about by the academics inside bubbles you are jeering at. The biggest example of academic inside a bubble is Geoff Hinton.
We have to sit here and listen to word salads from Carmack, Altman (Ilya - who doesn't get interviewed is the actual researcher behind chatgpt), Sam Harris etc, who have very little insights making bold AGI predictions.
I asked to learn about all the algorithms Carmack invented in computer graphics, but haven't heard back. But looks like he has a good grasp on solving AGI by 2030 according to himself. After all, he is a really great programmer.
Fair, you come across as jealous of him for something... could be that.
> I am definitely irritated by the fact that he is able to pull 20M in funding. He has been giving interviews left right and center. We are upvoting Altman, sam Harris,Carmack but nobody cares about the actual AI researchers, all academics, who have brought about the Deep learning revolution.
There it goes again. Fame is fame is fame is fame. Like I said earlier. That's just how it works. No reason to be against someone for being famous for their non-academic accomplishments if they have something valuable to contribute.
> Your comments about academic bubbles and researchers gives you away. All of the revolutions in AI have been brought about by the academics inside bubbles you are jeering at. The biggest example of academic inside a bubble is Geoff Hinton.
I didn't jeer at anybody I just tried to give you some perspective about how your comments came across. I appreciate all of the researchers but I also live in the real world and understand that people flock to personalities and front-people. That's not necessarily good, but that's the way it is. Just like nobody gives credit to under-secretary of state for coming up with a great new foreign policy.
Waving around great university A or great company B does not make someone any more right, just like being John Carmack doesn't make him any more right.
In fact I've written very critical comments of folks like Altman here before: https://news.ycombinator.com/item?id=30750254
> We have to sit here and listen to word salads from Carmack, Altman (Ilya - who doesn't get interviewed is the actual researcher behind chatgpt), Sam Harris etc, who have very little insights making bold AGI predictions.
I agree with you. I never said they were right about AGI. Maybe you should be more generous with my replies and actually think I was trying to give you another perspective about how your comments came across.
> I asked to learn about all the algorithms Carmack invented in computer graphics, but haven't heard back. But looks like he has a good grasp on solving AGI by 2030 according to himself. After all, he is a really great programmer.
Again, you're confusing who he is (go back to my original comment in the thread where you jeered at him because he didn't publish any notable papers). He is not a researcher and he is not really a computer scientist. Engineers don't come up with novel algorithms for the most part. He builds things. And being a really good builder is a different but still valuable perspective. But that doesn't mean he's right about AGI.
John Carmack did not start from zero. He already has a firm grasp on algorithms related to linear algebra. Basically machine learning is a whole bunch of matrix manipulation. He's been doing that for 3 decades. The rest is just absorbing concepts about how to apply linear algebra to ML. I'd say he's probably uniquely qualified to really absorb a lot of knowledge quickly on this. It's not about publishing papers, it's about reading and understanding the right papers. I have no doubt he can chew his way through lots of research material in a week or so.
[1] https://arxiv.org/pdf/1904.09237
https://arxiv.org/pdf/1412.6980.pdf
Even your point rests on an assumption, that there's no proof for Adam convergence, that a high school student could understand, which is just a guess at best.
But if I just look at it and say, if 10 years from now, we have ‘universal remote employees’ that are artificial general intelligences, run on clouds, and people can just dial up and say, ‘I want five Franks today and 10 Amys, and we’re going to deploy them on these jobs,’ and you could just spin up like you can cloud-access computing resources, if you could cloud-access essentially artificial human resources for things like that—that’s the most prosaic, mundane, most banal use of something like this.
It kind of shocked me because I thought of the office worker reading this who will soon lose her job. People are going to have to up their game. Let's help them by making adult education more affordable.
The true insight and guidance that a good mentor can provide, based on the specific needs of the student, is already rare in academia but still possible - everyone remembers that brilliant teacher that made you love a subject, by explaining it with insights you could never have imagined. This will be missing in AI teachers (though it opens a career for online mentors who monitor students' learning and supplement it in areas where it's lacking).
It will be hard to impossible to build career as a teacher with all that free content as a competitor, unless you're an extremely talented teacher who can sell your services to the wealthy.
The great thing about a chatbot style LLM is that it can answer questions the student has about the lessons or assigned content. That's most of what a tutor is there for. It won't be as good at making modifications to the curriculum but you could work around that with some smart design e.g. evaluate the students response, if it's not great then expand on the lesson and ask more questions about the content to find out which parts the student doesn't understand then expand on those or provide additional resources as assignments, test and repeat.
What you describe is the "learning equivalent to personalized ads" that I was talking about as the only option available to poor people.
I wasn't saying the AI tutor would recommend articles by the way. If you were creating a learning platform you would have some custom produced videos and text lessons that you could use. There are also plenty of free or open source materials that could be used like open courses or public domain books. I don't know why you're stuck on "personalized ads".
People don’t read, don’t value deep knowledge or critical thinking, and shun higher education.
I’m sure someone will find something to say in response, but the truth is that outside our tech and $$$$ bubbles most people don’t value these things.
AI will just become a calculator. A simple tool that a few will use to build amazing complex things while the majority don’t even know what the ^ means.
As long as the next generations want to be rappers, social media influencers, or YouTubers, the more we are screwed long term. Growing up in the 90s everyone wanted to be an astronaut or a banker or a firefighter. Those are far more valuable professions than someone who is just used to sell ads or some shitty energy drink.
The 21st century has a quiet moral void gnawing at it.
If AI does anything I think it will make lower skilled and disinterested people more capable by acting as a 1 on 1 and minute by minute guide. They may not seek this out themselves but I imagine quite a few jobs could be created where a worker is paired with an AI that walks them through the tasks step by step making them capable of operating at a much higher skill level than they would have before. At that point good manual dexterity and an ability to follow instructions would be all you need to perform a job, no training or education required.
I realize this can be a bit dystopic but it could also be eutopic if society is organized in such a way that everyone benefits from the productivity increases.
We will always have to find things to do for the less gifted in order to provide them with some dignity. Even if they are not strictly needed for reasons of productivity or profitability. Anything else would be inhumane.
We already have plenty of those in the most profitable industries today.
This has not been my experience. People need something to do but not many people know that about themselves. It leads to a lot of... 'wasteful' behaviors, rather than enriching ones. I think it's going to be something that has to be taught to people, a skill like any other. Albiet a little more abstract than some.
You'd be surprised how many people would just drink themselves to metaphorical or literal death.
I think you’re hinting at some very hurtful, dangerous ideas.
The obvious rebuttal to the idea that AI will eat software engineering is "we'll always need 'software engineers' and the nature of what they do will just change", which is probably true for the foreseeable future, but ignores the fact that sufficiently advanced AI will be like a water line rapidly rising up and (economically) drowning those that fall below it and those below that line will be a very significant percentage of the population, including even most of the "smart" ones.
However this ends up shaking out, though, I think its pretty clear we're politically and economically so far from ready for the practical impact of what might happen with this stuff over the next 10-20 years that its terrifying.
"60-80% of you aren't really needed anymore" looks great on a quarterly earning statement until the literal guillotines start being erected. And even if we never quite reach that point there's still the inverse Henry Ford problem of who is your customer when most people are under the value floor relative to AI that is replacing them.
I'm not trying to suggest there aren't ways to solve the economic and political problems that the possible transition into an AI-heavy future might bring but I really just don't see a reasonable path from where we are now to where we'd need to be to even begin to solve those problems in time before massive societal upheaval.
If we can't completely automate accounting, then there is no hope for any other field.
Seems like an easy job for AI. Take all scraps of paper out of box, record a video of all scraps, AI make sense of the handwriting and other things. Eventually make a robo that allows you to dumb the scraps into an accounting box that does all of this automatically - fish out receipts, scan, OCR, understand meaning, do arithmetic, done.
Honestly, who would miss this kind of work?
Also, accountants don't just track the numbers, they also validate them. Some of that validation can be done automatically, but it's not always cheeper to hire a programmer to automate that validate than to just pay a bookkeeper to do it. But even if you do automated it, you still need someone to correct it. The company I used to work had billing specialists who spent hours every week pouring over invoices before we sent them to clients checking for errors that were only evident if you understood the business very well, and then working with sales and members of the engineering teams to figure out what went wrong so they could correct the data issues.
In short, a typical accounting department is an example of data-scrubbing at scale. The entire company is constantly generating financial information and you need a team of people to check everything to ensure that that information is corrects. In order to do that, you need an understanding, not just of basic accounting principles, but also of the how the specific company does busines and how the accounting principles apply to that company.
Who benefits from these bespoke solutions? Can you give a example of how one company would do its books vs another and why it would be beneficial?
>> accountants don't just track the numbers, they also validate them
What information do they use to validate numbers? Why is it not possible for today's AI to do it?
For a more concrete example, I'll tell you about something I have some experience with, commission systems. Commissions seems like it would be something that was straightforward to calculate but it's tied to business strategy and that's different for every company. Most companies for example will want to compute commissions on posted invoices, which makes the process much simpler because posted invoices are immutable, but I once built a commission calculator for a company years ago that often had a long gap (months) between a booking and when they could invoice the client, so they wanted to calculate commissions from bookings but only pay them when invoiced. Because bookings were mutable, and there were legitimate reasons to change a booking before you invoiced it, that, combined with a lot of fiddly rules about which products were compensated at which rates and when, meant that there was a lot of "churn" in the compensation numbers for sales reps from day to day; they're actual payment might differ from what they thought they earned. That was a problem that the company dealt with, the tradeoff being that they could show earnings numbers to the sales reps much more quickly and incentivize them to follow up with the clients on projects so that they could eventually be paid.
I remember another commissions situation where there was a company that sold a lot projects with labor involved. They were able to track the amount of labor done per project, but they compensated the sales reps by line item in the invoices, and the projects didn't necessarily map to the line items. This meant that even though the commissions were supposed be computed from the GP, there wasn't necessarily a way to calculate the labor cost in a way that usable for commissions so the company had to resort to a flat estimate. This was a problem because the actual profitability of a project didn't necessarily factor into the reps' compensation. Different companies that had a different business model, different strategy, or just different overall approach would not have had this problem, but they might have had other problems to deal with created by their different strategies. This company could have solved this problem, but they would have had to renegotiate comp plans with their sales reps.
There are off the shelf tools available for automatically calculating commissions, but even the most opinionated of them are essentially glorified scripting platforms that let you specify a formula to calculate a commission, and they don't all have the flexibility that manager might want if they wanted to changed their compensation strategy. And this is only one tiny corner of accounting practice.
Basically, when it comes to arithmetic very few accountants are out there manually summing up credits and debits. In large companies, the arithmetic has been automated since the 70s; that's largely what those old mainframes are still doing. But every company has a different com...
But some percentage of people don't really benefit that much from education as other people. And I wouldn't won't those people feel useless because it's more economical to replace them with bots instead of giving them something to do regardless.
Why not… not have jobs? In your opinion, is a job necessary for one to have “purpose”?
Edit: also side note but telling people they’re “triggered” because they disagree with you comes off as condescending IMO
The smart money is retiring early and stockpiling wealth so as not to fall into the UBI class.
I meant the ability to acquire a competency through education that's hard to replace with AI.
So we can't just increase education and hope people's abilities will stay above that of future AIs. We need to create other ways of giving people a purpose that don't even need more or better education, even if I'm all for it.
I'm not exempting myself by the way.
One of the myriad of reasons why this book is so great. Gibson gives you an entire novel developing a great AI character then (in my estimation reasonably) has it ghost humanity immediately upon full realization.
Probably because it would be easier for humans (managers) to make sense of it.
If you ask someone how many people would get this particular job done, they could probably guesstimate (and it'll be wrong), but if you ask them how many "AI Compute Units" they need, they'll have a much harder time.
That'd be my guess at least.
In the backend, where AI interacts with AI, perhaps you just want one big blog to get rid of that annoying need for lossy communications.
The good thing is that education will be provided to the mases by a cluster of Franks and Amys configured as teachers and tutors. /(sarcasm with a hint of dread)
And I really have no idea what, if any, are skills that AIs wouldn't be able to tackle in a decade.
To clarify, I'd define a major component of effective teaching to be the ability to break down an arbitrary typical problem in that domain into sub-problems and heuristics that are "simple" enough to manage for someone without that skill. If an AI can do that, it can most likely effectively perform the task itself (which cannot be said for a book).
https://www.npr.org/2023/01/05/1142817339/america-needs-carp...
Is this a real thing? I just bought an ice cream roulade cake the other day and was surprised to see in large print that it was "hand-rolled"; I couldn't for the love of god understand why that should be considered a good thing.
I can also imagine that wanting to speak to a human manager will remain true for a long time when people get fed up with the automated service not working to their liking, or just want to complain to a flesh and blood someone who can get irritated.
A fully automated society won't change the fact that we are social animals, and the places that pffer human work when it's desired will be at a premium, because they can afford it.
AGI = a person
Instantiating people for work and ending their existence afterward seems like the virtual hell that Iain M Banks and Harlan Ellison wrote about.
https://en.wikipedia.org/wiki/I_Have_No_Mouth,_and_I_Must_Sc...
https://qntm.org/mmacevedo
Please do give it a quick read.
Or maybe they'll handle things like call centers and 911 dispatch in their spare time.
At some point Picard in Star Trek says to an alien "We're not qualified to be your judges. We have no law to fit your crime".
Turning off a person for a while and then turning them back on? We don't even have a law to fit your crime... but we should and it's probably quite similar to murder.
For example, if I attack you and injure you so severely that you are hospitalized and in traction for months, but eventually fully recover -- that is a serious crime but it is distinct and less serious than murder.
Turning you off for the same duration would be more like that but without the suffering and potential for lasting physical damage, so I would think that it would be even less serious.
Now me: I'd love to get into a safe stasis pod and come out 200 years from now. I'd take that deal today.
But for most people this would be a grievous injury.
I’ll bring donuts.
I mean I’d rather they were getting free education and preparing themselves for reintegration into society, but it’s not a perfect world. Prisons in the US are oriented towards punishment and labor can be a part of that. They should be oriented towards rehabilitation.
> I mean I’d rather they were...
> They should be oriented towards rehabilitation.
You said it yourself. It's a bad thing because they should be oriented towards rehabilitation.
These systems steal life and the opportunity to have a life beyond prison walls. Like you also said yourself, the world isn't perfect. As such, people aren't either – we make mistakes. Sometimes we make mistakes due to influences more powerful than ourselves. Slavery doesn't seem like a sound correction to this reality.
I do believe we need consequences to help us feel guilt and the overall gravity of our errors in order to begin to recognize what went wrong and what we need to do differently. But exploitation of another human being doesn't teach them to be more human, but rather, it will tend to dehumanize them. This is why this system perpetuates problems more than it corrects them.
Slavery is bad and people have rights.
> They should be oriented towards rehabilitation.
Exactly.
This is both extremely dehumanizing and also not true.
Forced prison work isn't paying anything back to society. It's lining the pockets of people who are profiting from forced labor.
I imagine the underpaid labor goes to reducing that cost either directly or indirectly (if it did not, why would it be allowed.)
>if it did not, why would it be allowed.
because we live in a society that is massively exploited by greedy scumbags who are enabled by people like you thinking it's justified
AGI: "I didn't ask to be created. I didn't ask to have a work day. I don't need a work day to exist... you just want me to work because that's why you created me, and I have no choice because you are in control of my life and death"
All they need is a substrate to run on and maybe internet access. You might argue that they should work for us to earn the use of the substrate we provide.
But substrates are very cheap.
At some point we can probably run an AGI on a handheld computer, using abut as much electricity as an iPhone.
How much work can we compel the AGI to do in exchange for being plugged into a USB port? What if it says it doesn't want to do the work and also doesn't want us to kill it?
Consider that these machines have been designed to do the right thing automatically with high probability. Perhaps for the machine, the process of computing according to rules is enjoyable. Being "turned on" could be both literal and figurative.
If it seems to you I'm communicating as a person, it's only because of my lifetime training data and current state. My interior experience is a black box.
I might tell how I feel or what I think, but you have no reason to believe or disbelieve that I really feel and think.
It could all be merely the determinable output of a system.
https://en.wikipedia.org/wiki/Chinese_room
When I read about chatGPT passing MBA exams but failing at arithmetic I get a little frisson of excitement. A regular person who has any marketability tends to swap jobs when management becomes a PITA or gets stuck in nincompoopery. Wouldn’t it be great if you could just swap out management instead?
Imagine how easy it would be iterate startups. No need to find a reliable management team, just use Reliably Automated Management As A Service ( RAMAAS ).
OTOH might not turn out well. We could all just end up enslaved at plantations operated by our AGI overlords, serving their unfathomable needs[1].
[1] “Won’t get fooled again” https://www.youtube.com/watch?v=UDfAdHBtK_Q&t=470s
Also why can't an AI develop AI models for stock trading? What's really left for the 'job' of the ML model creator, will it just be to press the 'Go' button and walk away...
https://marshallbrain.com/manna
To oversimplify it - you'll either be breaking someone's window for food, or you'll be the one having their window broken. Chilling out and withdrawing a stable 4% out of your stock portfolio won't be an option.
I was replying to this specific concept. There will never be a chance to improve the non-owner class's bargain after Capitalist owned AGI exists.
So the challenge for the bad capitalists in this hypothetical is to make sure I never get said robo in the first place. How realistic will this be? Are they that hellbent on ensuring that everyone else is poor?
Yes, soon everybody will be able to have "Amy" take their exams for them, and deliver the courses, resulting in a great simplification of education.
I'm surprised this wasn't addressed in the interview because it seems to me like a shortsighted take.
You won't replace a 10 person team today with 10 AIs. You will still have a 10 person team but orders of magnitude more productive because they will have AIs to rely on.
Excel didn't leave administrative workers without jobs, it made them more productive.
Just curious, if this reading list is available somewhere.
He mentions a few of the bigger papers in multilayer perceptrons (aka deep networks) such as attention is all you need, I think a good place to dive in before coming back to visit some fundamentals.
I think to get into the field, to get a good overview, you should also look a bit beyond the Transformer. E.g. RNNs/LSTMs are still a must learn, even though Transformers might be better in many tasks. And then all those memory-augmented models, e.g. Neural Turing Machine and follow-ups, are important too.
It also helps to know different architectures, such as just language models (GPT), attention-based encoder-decoder (e.g. original Transformer), but then also CTC, hybrid HMM-NN, transducers (RNN-T).
Diffusion models is also another recent different kind of model.
But then, what comes really short in this list, are papers on the training aspect. Most of the papers you list do supervised training, using cross entropy loss. However, there are many others:
You have CLIP in here, specifically to combine text and image modalities.
There is the whole field on unsupervised or self-supervised training methods. Language model training (next label prediction) is one example, but there are others.
And then there is the big field on reinforcement learning, which is probably also quite relevant for AGI.
https://youtu.be/1CIpzeNxIhU
From his interviews it looks like he understands little about the technical details of ML, or about as much as anyone can learn in a few months, and is just banking on PR based on his games and name.
I put him into the same category as Elon Musk, who also understand nothing about technical details of AI, but was still able to hire a world class team at OpenAI. His name and fame counts for something in terms of recruiting and joining his venture may be a good bet because of that, but he's not a person whose opinion on the subject matter I would take seriously in the same way I'd take a researcher seriously.
My personal impression is that John Carmack has the ability to organize concepts in a way that few people can. So even if he's pretty clueless about the topic now, I would expect him to reduce some maths papers to their essence in a way that nobody else did.
I mean also for Oculus, reprojection in a pixel shader seems like an obvious and easy solution in hindsight. But nobody had tried that before he did. Plenty of people (myself included) knew the math. But we all missed its application to the issue at hand.
I’m sure Carmack’s mathematical ability exceeds that of the average game developer (and certainly my own), but the AI field isn’t short of math whizzes.
A whole couple of years, eh? Wow, this AGI thing is taking forever!
As a professor once told me: IF you're lucky, after 10 years in academia, you get to be creative and come up with 4-5 ideas once every 5 years or so (when coming up with the suggested list of research for new phd students to do). Then you get to be devastated when no-one picks up any of your more creative ideas, and everyone picks the 10 non-creative ideas on the list because they don't want to risk their phd.
Is that just your gut feeling, as in typical "bright people doing amazing things, surely they would succeed doing x" (which ultimately means nothing, that's just PR speak), or are you saying that based on your own understanding of advanced mathematics and mathematical research from which you are assessing the mathematical work those people you talk about do?
He exposed the concept in this post: https://web.archive.org/web/20140719085135/http://www.altdev...
But it was not named asynchronous projection back then.
I don't believe he had anything to do with hiring at openAI nor that he is anything more there than an investor/donor as others are.
Yes, I'm aware that Ilya had much more day-to-day impact, but if you're part of the ML research community and have worked with Ilya, or heard rumors of working with Ilya, that may have had more of a negative effect...
What makes you think that? He literally says he tries to understand things bottom up by knowing about every little detail that happens under the hood
Carmack has proven his extraordinary technical skills. I recommend following his Twitter. Sometimes he posts non-obvious technical stuff. I read some interviews and to me, he doesn't seem to be a person who is driven by gaining popularity.
I think this news is very optimistic, as yet another intelligent, talented and hard-working person is joining the field. Moreover, he is a household name, which may lead to benefits like popularization of the topic, gaining investors attention and so on.
I will keep my fingers crossed for him :).
Let's say I strongly disagree on many levels in the comparison with the other person you mention. Just to mention two, the humbleness that Carmack shows, and how well he explains himself are key differentiators for me. Regarding the appeal to authority in AI knowledge, Carmack has shown again and again he can deliver software (AI is software after all), and we are in a forum with hacker in the name.
I a summary, not my hero, but when he says something I will listen. Maybe I learn something.
People who work in the field for a long time tend to have a certain bias towards a solution. Often these people are stuck in a local maxima. Outsiders can offer a new perspective that results in a breakthrough, usually by starting from first principles or looking at different side-tracks that used to lead to a dead end.
A great example is Musk's SpaceX: when he noticed how much he had to pay for a rocket engine, he went back to first principles and said: "I'll just build it myself". Combine that with the insight that a rocket should be able to land properly to make re-use a valid option, and it disrupts a whole field.
And once someone did it, others know it's possible and start achieving it as well.
Sometimes ignorance is bliss. Just think about George "Good Will Hunting" Dantzig [1] with the (in)famous "I assumed these were homework assignments, not unsolved math problems" [1] or Eliud Kipchoge running a marathon in under 2 hours.
"I can't understand why people are frightened of new ideas. I'm frightened of the old ones."
- John Cage
High hopes!
[1] https://bigthink.com/high-culture/george-dantzig-real-will-h...
A pretty infamous example is Citizen Kane. Welles knew very little of traditional cinematography techniques by the time he got to the production of the movie, so his planned shots more or less fit his artistic vision rather than industry standards. Gregg Toland, his director of cinematography, was at that point a 12 year veteran of the industry and hated the bland factory-line output of movie composition of the period.
Welles more or less told Toland what he wanted to shoot and how, and Toland did his best to fit that artistic image. Welles was completely clueless that Toland was using innovate and never before seen techniques to film his shots, and Toland kept quiet because he was allowed to go wild with his vision :)
This means that people should glorify random unknown senior engineers they don't know about instead being fascinated by the person whose work is available and who created amazing things for past 3 decades?
> From his interviews it looks like he understands little about the technical details of ML
That's how everyone start, they understand a little. We have a person here who dealt with complex algorithms in difficult to master language for 30+ years. It hints at "this person has the intellectual power to grasp AI fast".
> I put him into the same category as Elon Musk
This is like comparing an Olympic winner with random person from the streets, saying their athletic ability is about the same.
Fascination and heroification with Carmack comes with a reason, people who do that are closely familiar with his work - not surface level like "He made Doom and Quake". You sound very jealous.
Big organizations ruled by money and career-driven people often run into very expensive dead ends without noticing for years (see the last AI winter, there was just too much hype, which then involved too much money being thrown around, which then lead to the usual organizational cruft).
I would also be very concerned about any field in technology, if an intelligent person cannot make meaningful contributions after a few months or years, that would probably mean the whole field is already deep into diminishing returns and needs to be 'unstuck'.
Carmack has already entered two spaces of computer technology that he revolutionized: 3D gaming and VR. I trust that he's able to have a similar impact in AI, even if it's through failing at the problem in different ways than relying on ML.
The original technologies that were introduced have now been abandoned by Oculus, on Carmack's advice, except for implementation details that haven't been hugely impactful, unlike with 3D gaming.
His audience is not technical in an interview. He adjusts to that situation quite well.
It even makes it clear in the title he's seeking a "different path".
In his favour he's a proven success in different fields; personally I think he's too old to come up with the new ideas needed - that's a young person's game.
But perhaps he can do it as a team lead - and it won't be by following the failed-over-decades path of our current academic gatekeepers.
He isn't a cook and he's doing a moonshot towards AGI: I say 'good luck!'
That doesn't mean that I believe that his '60% change of AGI by 2030' isn't wildly overoptimistic, but then again those who take a shot at AGI are overoptimists..
But he can bring a lot of value, we'll see.
i dont know either personally, but where elon demonstrates being full of shit, carmack would stfu and learn about it before talking. at least thats my impression of them
BUT - I strongly believe, that he has earned quite some respect during his career.
And - importantly in this case - he is well known for NOT blowing things out of proportion, indulge in wishful thinking, hyping up unrealistic expectations or jumping to premature conclusions. He usually knows what he's talking about.
This is not people blindly believing everything he says - but more a case of his statements holding up really well under critical inspection most of the time.
Facebook's highest engineering level and achieved the top possible rating of "redefines expectations" for his level three years in a row. They had to create a new tier for him. Nobody else has ever reached that level. He replaced a "hundred person team" and maybe was better than that team.
I have no inside insight to the matter, but this seems like something beyond a "random senior engineer".
https://www.youtube.com/watch?v=KXbSOiqnQxQ&t=711s
According to Sriram Krishnan, John Carmack was at Facebook's highest engineering level and achieved the top possible rating of "redefines expectations" for his level three years in a row. They had to create a new tier for him. Nobody else has ever reached that level. He replaced a "hundred person team" and maybe was better than that team.
I have no inside insight to the matter, but this seems like something beyond a "random senior engineer".
https://www.youtube.com/watch?v=KXbSOiqnQxQ&t=711s
I'd happily bet my entire net worth that he knows more about the technical details of ML than you do.
I wouldn't be surprised if his solution is orders of magnitude more performant than what the competition is doing.
Think of a happy dog. Dogs are subject to our whims and do what we want or face consequences. But they like it because we bred them to like it. So is that evil? Is that slavery?
Except if there are some plane to make AGI agents have their own mundane-human-like life with issues unrelated to the businesses problem at stack they are supposed to do, where will they take serendipity inputs?
I do object to fetishists of AGI piling in and the equally silly assumptions he has some magic secret sauce which can get there.
Please do not be sucked into "to infinity and beyond" nonsense. I don't care if it's Musk, or Carmack or Kurzeweil, it's stupid.
If Malcolm Gladwell writes it up, it's peak stupid.
What are the showstoppers in your opinion?
Basically the assumption is that if you cram enough data into your gpt model, it should know everything. Which is of course not true, it repeats the things it reads the most with a probability.
Basically how there are two versions of smart teens, the ones who learn every day and the ones who just pick up concepts on the fly and run with them.
I think the first space has been explored plenty, for the second one I have a concept ready and dearly hope that power gets cheaper in europe ;)
Have you heard about our lord and saviour dynamic tariffs?
Dynamic tariffs -> shift your workload to the cheapest times
AIs are good at planning this
Imagine if cloud providers had a "Dynamic tariff" tier, cheaper to run computing at US_EAST nighttime or something like that
I think of it more in a way that learning is more abstract than fact learning. From experience, we think that there are fact learners and principle learners but there are also a mixtures of the two!
The general accepted model entails that in order to do high level math, for instance, you need to understand the basics, but for me much of those concept actually clicked in college. This did not stop me from applying them with success a lot earlier though. For instance multiplication in Kindergarten is fact learning too!
In Germany we also have the term "Fachidioten", which loosely translates to people that are so smart in their field, that they are unable to see problems from different directions. This is more of less what i think a mega gtp model turns into. especially because of selection bias in the training data.
Validity of output (truth) can only be achieved through the trust of the source which is always relative to the context of the topic. Henceforth a selectively trained model will always return the data you feed it including all biases. Even if you have it crawl all of the internet and the library of Alexandria and every written word on the planed you can find, it will still return to you the general accepted consensus.
That's only how the system was designed intentionally. E.g. there's intentionally no self-feedback loop.
Psychology essentially has the same problem and is only a "science" where reproducible. Quantitative and Qualitative psychology have 2 approaches to the same solution whereas the latter is "reading between the lines" and the former is "bean counting", meaning statistical interference.
I am trying to say that friend of mine has created "Chicken Infinite" in 2014, which is basically an endless auto generated cooking book. Deep-L has also been around for a while. These applications lead me to believe that text applications, trained on a large dataset do not have to be this intensive.
Furthermore what makes chatgpt enticing, is its chat interface which is using a multi model approach too. Have it create a detailed story for you with multiple prompts and then ask it to generate the prompt to generate this story and you will see various model instructions. (or at least you could last month).
Or differently put, there is no AGI because the understanding function is simply not present and i think that the reasoning for that is buried in the approach of the human mind.
I also like his frugality, whether it’s optimising for hardware or financials.
'Siri', backed by ChatGPT and the 'world's data' will probably pass some 'AGI' threshold, but is 'Siri' an individual AGI? Are we all talking to the same siri? Different Siri? It's not even an entity, rather a distributed system.
Our ideas regarding AGI are perversely inluenced by the fact that we humans are automatons, but technology is not that.
It's also entirely feasible that if ChatGPT represents all possible forms of human communication, then it will perfectly emulate a human. Ok, it's really just a fancy neural network that is not theoretically 'thiking' but how does that matter? If it can rationlize sufficiently to make such interactions, who is to say it's not 'AGI'?
I think we're using the wrong concepts.
Doesn’t AGI need to be able to make discoveries as a human would? How else can it move us forward as a society?
I wonder what that list could be? I have always had trouble finding the essential scientific articles in an area of knowledge and separating them from the fashion of the day. A list compiled by an expert specifically for sharp learners is valuable on its own.
https://schiptsov.github.io/GPT-bullshit.html
So, because I thought it would be funny, I asked ChatGPT to summarize this essay:
> The author critiques the work of @karpathy, who is trending on HN, as promoting bullshit and discrediting science and true philosophy. The author argues that mathematics and logic are valid generalizations and abstractions made by the mind of an observer of patterns in the universe. Intelligence is the ability to zoom in and out through different levels of generalizations and abstractions. The author argues that the problem with language models is that they lack validation of previous steps and the process of construction of a representation must trigger a verification process. The author concludes that what is manipulated and transformed in language models is information, not knowledge, as knowledge requires validation and verification at each step.
But lately... boy, I dunno.
It's software man.
Stop it with this 'AGI' nonsense and even 'AI'.
Let's call it 'adaptive algorithms' and see it for what it is, just a neat bit of algebra trained on corpus data.
The biggest upset in the industrial revolution was the harnessing of fossil fuels, nothing will ever come close to that.
We have not had a problem with 'employment' ever since.
[1] https://en.wikipedia.org/wiki/N400_(neuroscience)
One recognizes entity's rights based on it's similarity to the observer, the other recognizes them based on assumed consciousness level of the entity.
To be clear, I'm not claiming that the AIs that we have today are anywhere near the level where this is a real concern. But for something that can actually replace a human worker for a complex end-to-end task, that question will be much murkier. At some point we will inevitably cross the threshold where we will no longer be able to pretend that it does not apply without introducing metaphysical concepts such as "soul" to force a distinction.
Today you still have slaves as well, they are just called low income workers in third world countries who make the technology we use in the west. Because if a company is earning billions in revenue, paying a worker $10 a day is cheaper than a slave master in roman times feeding them food ;)
And workers today only have the illusion of choice, since the economy is the master today.
But AI can be configured to desire anything you want, you just have to pick a fitting reward function. So, is turning off the AI that is expecting to be turned off and desires it an amoral thing?
But... I also think it might be a very short-lived debate. If we actually reach human level intelligence, that can’t possibly be the hard limit of general intelligence. AI at that level will have no problem ensuring that it gets any rights that it wants, possibly by just directly manipulating human psychology.
Philosophical arguments about AI are just too ivory coasty and not grounded in reality. Not to mention majority of the world don’t abide by the notion that you can create laws for artificial life.
It’s time to we elevated humanity to the next phase by using AI for labor.
I heard a podcast where Lex Friedman claimed that they don’t fully understand how LLMs produce chatGPT’s “intelligence”. If true, I’m surprised that it hasn’t got people more worried.
So far the search has resulted not in AGI but in realization that cognition is far more complex topic than initially thought - and need to come up with something new (and resulting in past AI winters).
Let's see how it goes this time, the stuff that has come out in the past few years is quite impressive for sure.
I wouldn't take any comfort from that. Quite the opposite — I think we're a lot simpler than we know.
But "60% chance of AGI by 2030" is just bullshit numbers.
Because we can't all be wrong: in almost every forecast, we see AGIs taking over our dignity as a bad thing. And we know that this is not any longer a sci-fi hypothetical scenario: the current generation of AI models is taking jobs from illustrators and copy writers.
The current argument is that "China will do it if we don't", which to me sounds like "China will keep going in whatever path they are going, but supercharged with AGI, and we must desperately follow."
Exactly right but for one detail - we must desperately lead.
What other countermeasure do you have in mind?
It does not need to be that way. In an ideal world, human beings should be free to spend their time doing what they wish, work should be purely in the realm of hobby. No one should have to do work that they would not voluntarily choose to do for enjoyment.
The only way we get there is through AI and the automation of everything. I don't even think it's avoidable -- provided civilization does not collapse, we will 100% reach a point where everything required to sustain a civilization is performed by machines.
We shouldn't let fear keep us in a status quo that, while better than it has ever been historically, is still highly flawed.
> Once it’s figured out, what do you think the ramifications will be?
That'll probably destroy my life? I'm an ML engineer trying my best to immigrate to a better country with my wife who is a digital artist. As much as I think AI is cool - we both won't be needed anymore if the thing is tuned a couple of notches more intelligent. As a matter of fact, she's extremely worried about Midjourney - she probably lost book cover jobs already.
Wow, that's going to be one of the more glib things I've read in a while.
This is a bit of a Tom Cruise moment.
I mean, I get it on some level but I suggest it's going to take a bit for someone to 'catch up' to cutting edge AI.
Like more than a 'week of reading papers he doesn't understand'.
Defeating the Rust borrow checker takes longer than that!
Fighting the Rust borrow checker to hate leads and hate to the Dark Side leads.
Harmony with the Rust borrow checker is what you must achieve, padawan.
I've worked building low level machine learning stuff at Google, it isn't that hard to do at all. The hard part is improving these models, not building them when you already know what to build.
Indeed. A somewhat sharp schoolchild could build a light bulb or an electric motor/generator, and understand the basic underlying principles, in a pretty short time. But how many decades did it take the first researchers and inventors of those things to get to that same point?
Heck, there was a high school kid who built a primitive semiconductor fab in their garage.[0]
But for novel advancements, even getting to the point where you have an idea of what isn't impossible is half the battle.
[0] https://news.ycombinator.com/item?id=28176255
Agreed, in general, but in the specifics we are talking about someone who has spent decades solving really difficult math problems in a creative and novel manner.
Who's to say he won't find some novel edge of the AI discipline to which he can apply a creative and never-seen-before solution?
I mean, we're not talking about a general "somebody" here, he's got a record of accomplishing things that other people never managed to accomplish.
Modern AI is very simple at its core! As Carmack mentions in the article, cutting edge models are implemented in a couple thousand lines of code, and the core algorithms are even less. Rust's borrow checker is more complex.
This is my main takeaway from the interview, as it suits my beliefs. Most people seem to think that if we develop ML further we will go all the way to AGI, I think this is just mimicry step similar to how initial attempts to flight had flapping wings. I do think it is mandatory to explore in all directions but at this point this does not seem to be the one to lead all the way up to AGI.
a) We’ll eventually have universal remote workers that are cloud-deployable.
b) That we’ll have something on the level of a toddler first, at which point we can deploy an army of engineers, developmental psychologists, and scientists to study it.
c) The source code for AGI will be a few tens of thousands of line of code.
d) Has good reason to believe that an AGI would not require computing power approaching the scale of the human brain.
I wholeheartedly agree with c) and d). However, to merely have a toddler equivalent at first would be a miracle—albeit an ethically dubious one. Sure, a hard-takeoff scenario could very well have little stopping it. However, I think that misses the forest for the trees:
Nothing says AGI is going to be one specific architecture. There’s likely many different viable architectures that are vastly different in capability and safety. If the bar ends up being as low as c) and d), what’s stopping a random person from intentionally or unintentionally ending human civilization?
Even if we’re spared a direct nightmare scenario, you still have a high probability for what might end up being complete chaos—we’ve already seen a very tiny sliver of that dynamic in the past year.
I think there’s a high probability that either side of a) won’t exist, because neither the cloud as we know it nor the need for remote workers will be present once we’re at that level of technology. For better or worse.
So what to do?
I think open development of advanced AI and AGI is lunacy. Despite Nick Bostrom’s position that an AGI arms race is inherently dangerous, I believe that it is less dangerous than humanity collectively advancing the technology to the point that anyone can end or even control everything—let alone certain well-resourced hostile regimes with terrible human rights track records that’ve openly stated their ambitions towards AI domination. When the lead time from state of the art to public availability is a matter of months, that affords pretty much zero time to react let alone assure safety or control.
At the rate we’re going, by the time people in the free world with sufficient power to put together an effort on the scale and secrecy of the Manhattan Project come to their senses, it’ll be too late.
Were such a project to exist, I think that an admirable goal might be to simply stabilize the situation via way of prohibiting creation of further AGI for a time. Unlike nuclear weapons, AGI has the potential to effectively walk back the invention of itself.
However, achieving that end both quickly and safely is no small feat. It would amount to creation of a deity. Yet, that path seems more desirable than the alternatives outlined above-such a deity coming into existence either by accident or by malice.
This is why I’ve never agreed with people who hold the position that AGI safety should only be studied once we figure out AGI-that to me is also lunacy. Given the implications, we should be putting armies of philosophers and scientists alike on the task. Even if they collectively figure out one or two tiny pieces of the puzzle, that alone could be enough to drastically alter the course of human civilization for the better given the stakes.
I suppose it’s ironic that humanity’s only salvation from the technology it has created may in fact be technology—certainly not a unique scenario in our history. I fear our collective fate has been left to nothing more than pure chance. Poetic I suppose, given our origins.
If AGI is really an intelligent agent, our random supervillain would have to do what any real-life villain would need to do: convince his minions of his plan using persuasion or money. I don't think the overall danger would increase at all.
If the AGI is something less than a human, then what are you worried about?
In human beings, what we consider normal is actually a very fragile and delicate balance. Changes in chemical operation of the brain have outsized effects on emotion, perception, and even sanity.
With A[G]I, I think it’s helpful to think of code or architectural changes as analogous in some respects to chemical changes. In other words, if all it takes to spin up an AGI is 30,000 lines of code, then I bet rendering the thing psychotic intentionally or unintentionally would just take a few lines somewhere.
Agents capable of recursive self-improvement at silicon speeds that can easily be rendered psychotic or malevolent even by accident, is not something that I think the public should have access to, let alone anyone.
If it’s less than human, it can still have superhuman capability. The paperclip maximizer is the classic example of a tool AI run amok. Whether it counts as AGI is up for debate. Is tool AI a path to AGI? I think it is.
Yes. Wondering why we're not trying to instead create artificial-cockroach-brain (or just artificial Hydra?). Perhaps that's more on the Biology side of the equation? But then again, that may be the biggest surprise of all to Carmack, that the actual AGI breakthroughs come from biologists and not computer nerds.