The problem is not AI. It's an excellent technology. The problem is there's an underlying power grab (e.g. layoffs). When humans do that to each other they inherently dehumanize/invalidate/insult each other. Implement some kind of labor protections or basic income and a lot of this dog-eat-dog toxicity goes away.
Yes when the sole purpose is to remove the livelihood of people in an attempt to make short term gains for selfish short term reasons....AI is a weapon when you have corporate stooge lackies who have nothing but animous towards fellow humans who dare to work...and try to live. If AI was beneficial, no one would profitize it....but lbr...you know that...
If AI is not beneficial, then those humans could be re-hired soon.
Big companies are always doing bad things, but not because they use AI, because they have legal protections which prevents small companies (which could be anyone like you and me) to compete. The same small companies who could also benefit from using AI.
Because most automations never capture the complete scope of the job/task ( not even close ). Just like neurons, if you don't use it you lose it and when the inevitable problems come, nobody knows the why, the how and the what. At that point someone smart would incorporate all those real costs and opportunity loses on the "automating everything" equation. But they usually don't.
Of course automating tasks is a must, but it's very far from being a black and white situation. These dynamics have been happening for centuries by now, nothing new.
I agree with that, and that’s why it’s better to ask an AI and improve your prompt, instead of hiring a human that will disappear and you will loose all institutional knowledge
You only have as much institutional knowledge as you're willing to cultivate and be liable for. Many companies already don't give a shit about institutional knowledge, indicated by how little they're willing to invest in keeping a strong team together, caused in-part by long-standing toxic incentive structures.
Ask an AI to solve a problem and it may do that, but if you don't understand why or how it works, or what to do with the information in order to keep it useful for even the medium term, then all you've done is taken away the opportunity for someone else to be responsible for something you shouldn't be.
It's not necessarily a mystery how to make good food. You can ask an AI how to make good food, follow the instructions, and you're off to the races. The question then is whether you want to be in that race.
Would you have gone to chef school? Would you work in a kitchen? Are you willing to deal with customers, or risk RSI from so many repeated kitchen movements? Are you willing to practice and be tested?
If the answer to any of those is no, then get the hell out of that kitchen and let the people who have more grit than you do their job. Do what you can to make it easier for you to pay them consistently and well on the back-end.
The risk of hiring a hunan is easy enough to manage. Ensure that the process is documented and new hires are trained. Any disturbance by someone leavingg is then smoothed out in the long term.
Can you show us how it’s better to use an AI to ensure steadiness over a long period of time?
<< Because most automations never capture the complete scope of the job/task ( not even close ).
Management likes to think otherwise for a variety of reasons and peons such as myself know it is rarely that simple. Case in point, our team uses Jira, but Jira is not great at.. capturing efforts that are umm.. less code oriented. In fact, there are times when it is genuinely better to leave some details out for considerations that may simply not be part of coding considerations.
In other words, automation assumes proper capture of the entirety of the task, but, I have seldom seen it capturing anything but simple workflows well.
Depends on the scale and the technology. If the answer is "all of white collar workers" then I would argue it's very toxic. (I don't think it can do that, but it's hard not to get the impression that it's absolutely the goal)
I don't think anyone ever really believed it would replace any ordinary jobs. That angle was meant to appeal to emotion and distract the public from the shady deals and big defense spending.
Think about all the security clearances no longer required to aggregate big data into intelligence reports. The conditions and incentives for LLMs seem almost laser focused on replacing those particular jobs.
Automating a stamp mill that fed 50 families does not obviously correlate with human progress. I'm sure it's nice for an auto manufacturer's bottom line though.
Clearly not what is being said. If you read dehumanizing your workers and equate it with automating a job, then you're already well into the feeling that humans are fungible pawns to be disposed of, no?
Instead, you can and probably should see technology as augmenting you and your coworkers.
I don’t think humans workers should be cogs in the machine, but from my experience unfortunately that’s how people want to be treated. One simple explanation is that there is no freedom / creativity without responsibility, and the latter is extremely expensive in brain resources
AI is completely irrelevant to the power that software wields.
Implementing labor protections and basic income is not seeing the forest for the trees. What we need is a more educated public that can contribute their ideas. All AI can do is lower the barrier to entry, but it does not replace the need for education.
> Perhaps we can get to a system that produces not just the binary but also a machine-verifiable proof that the binary implements some higher-level language description of the program.
We don't even have a solution to the halting problem, and it probably can't be solved. "Proof it implements a spec" is pure science fiction.
Hard agree that we'd all be better off muting Elon Musk though.
> AI can write binary and probably better in a few years.
Assuming you mean an LLM, you'd have to train this LLM entirely on a parameter space of binary tokens. Or are you saying the LLM generating natural language tokens is going to be printing machine language in 3-5 years, because that claim kinda betrays a misunderstanding of LLM functions.
You don't use an LLM to create images, that requires (today) transformer model.
I assume that "AI can write binary" means "AI can use a toolset that results in a binary" because we've already seen GPTs use a combination of LLM and specialized math tools to do the things the original GPTs did.
When I think about the typical team composition needed to build a typical app in a lot of companies, well, actually what do I think about? I think, okay, that’s how many people we need to build a typical app. That’s what we should all think … like, what is that “number” of people we need?
So, today, if I think really really really, and I mean, really fucking hard about that “number”, I really really really come up with a number close to 0.
That means, for everything that we build today, before we even build it, we’ve already mentally replaced people. People be replaced yo, in your mind, in your company, in your reality. That’s it.
I think if you use AI to automate talking to customers, you're going to stop having customers. If you use AI to do your engineering and deployments, it's up to you to fix it when it goes wrong. If you use it for accounting, you BETTER look at the results if you don't want to be audited. If the AI handles legal, same thing. Basically, if it replaces N people, then you, the one person, need to be accountable for being able to handle N people's roles, because the AI can't take accountability.
Right, the one person (Neo?), has to be able to verify the output. Most of us can’t replace the accountant because we can’t really … replace the accountant. We can replace the accountant’s work, but we can’t replace his/her other work which is knowing if the accounting is right.
One programmer can know if the freaking program is right. So N people in that case, can truly be replaced.
I can know the SQL query came back right, I can know the drop down menu looks right, you know what I mean?
I think that's really unlikely on a large software project, or even a medium size one. Mature things usually get big enough that developers specialize in an area. Just as a random example I know a lot about, Unreal Engine has so many incredibly complicated and huge features that expecting even a project manager to know half the features is unlikely, and expecting that a single programmer could maintain it is wild.
You're also making the mistake of thinking developers are interchangeable, or that any given developer can know all of the tech stack. Again with the Unreal Engine example, how many developers understand animation, procedural generation, localization, high end 3d graphics, the math behind physics, gameplay systems, storage systems, consoles.. (I'm probably covering like 5% here)
I was earlier of the belief, Oh my god, now nothing is impossible. Ai can market, ai can create video, ai can sell, i can build and make a great product. we will win.
I realized all this was quite wrong, AI just helps people who already know become more efficient. It just gives confidence to people who dont know anything by the sycophantic nature of AI. AI is just a tool for smart people to become smarter.
Love AI and what it allows us to do. But, it does give some superpowers, but the problems that existed before still do and they need to be solved for. with or without AI.
Thus far, this has been my general experience as well. There are things that it can do under my supervision, but badly if I don't know enough to correct well enough and catch it when it makes mistakes. And it still often does. The question is typically whether I pay enough attention to notice.
I completely agree. The more experienced you are, the more you get out of AI.
A friend of mine is super smart, but he's not a software developer. He tried to use AI to build software. And it worked well for small projects, but he always reached a point where the AI could no longer fix bugs or add new features.
The problems were often simple. An experienced developer could fix them with a few minutes of manual editing. Sometimes it was just an incorrect > in an XML file, but the AI would get completely stuck.
So IMHO AI works best when you know how to guide it and when to step in yourself.
Normies just don’t understand what all is involved in running and evolving a production system. They think all programmers do is write code. Of course they believe that AI could replace programmers.
My in laws smugly asked me what I would be doing for work since AI would take away all the programming jobs and I said its not gonna happen and they laughed in derision.
I would be glad they don't understand what's next. For now, it helps devs keep a low profile as their work transitions into more bureaucracy. You're lucky they do not yet consider you one of the "bad guys" like lawyers, doctors, teachers, etc. (who didn't get replaced by the photocopier, google, or video tape)
Some people who "used to love programming" have jumped ship for precisely this reason. Ordinary software (not R&D) is getting serious. A broken or disagreeable app is no longer a mere inconvenience. There is growing alignment between business and politics and the older generation is retiring out.
It's crazy how defensive (offensive) people can get when you're into programming/math/whatever. Almost like "I don't understand your world, I don't want to, and I hate it and hope you fail".
It's almost as if the tech industry is full of weird racist incel sociopaths who have been smugly condescending to "normies" for decades and openly celebrating AI's inevitable consumption of their livelihoods while dancing on the pile of money they've earned turning the web and society into a cesspool of technofascist hypercapitalism and grift, only to then complain about how the neighborhood they paved over and burned to the ground is no longer quirky like it was in the 90s.
Sam Altman, Elon Musk, Peter Thiel, this whole scene has gotten fucking insufferable and everyone who isn't in the cult hates AI and hates the world techies want to build, and justifiably so.
We really need to renormalize beating up nerds again.
When I was in school I had crippling social anxiety and the nerds beat me up, so no.
But if you read my comment and only saw it as a personal threat then you are expressing exactly the lack of contextual insight and empathy that makes people find techies insufferable.
Let me reassure you that "beating up the nerds" is alive and well.
The nerds are the Wozniaks and Craig Newmarks who are curious and want to build cool things, not the sociopathic billionaires that always find the shortest path to their own selfish goals while burning down the earth.
Be curious and find the source of your pain, not the sacrifical lamb the media throws your way.
I just don't understand why people are incapable of extrapolating. AI can't do this blah blah, my brother in spaghetti monster, have you seen the curve? ChatGPT launched in November, 2022. Opus in December last year. Do you see where this is going?
AI may be a tool and add 50 IQ points to a 100 IQ person today but what happens when its adding 10000 IQ points. The 100 base doesn't quite matter as much does it? What does that world look like?
No, it's not just next word prediction. No you can't just wave your hand that it's probabilistic, so insert bad analogy to compiler or calculator and go about your day. And no, you can't just look at its limitations today, make a conclusion, and go back under your rock.
but you’re just on the other end doing the same thing.
one end is luddites nothing to see here, and the other end is omg humanity is cooked because programming is solved! and consciousness is coding so -> gg
yes there’s a wide surface of emergent behavior we haven’t fully explored, but why do you get the pass to draw a straight line to a conclusion? the naysayers are doing the same thing.
Extrapolation leaves you in a world with no agency. Okay, so the models become superintelligent, you're kind of fucked at that point and there's no value you as a human can add so that reality isn't really productive to think about.
The only thing we can do is look at the current state of things, and we still have the same problems we've always had with these models, they need proper direction and input.
Is there demand for 10000 iq intellegence to fund its construction if we already have +100 iq AI? You are assuming there will be demand that scales as fast as technology can to fund its continued fast pace development.
Yes. If it looks like 10000 iq is possible, then money will be thrown at whoever is most likely to achieve it first, and whoever has the most intelligent models right now strongly predicts who will be first.
This is driven by the belief that whoever gets to 10000 iq first will likely dominate the majority of all economic activity, since it will be more efficient to trade with them than with some less efficient (dumber).
This does not depend on traditional economic demand, at some point this becomes a self-fulfilling prophecy.
People with money to invest will throw money at it to build it, but who is the customer base after that? So one guy eventually controls the stock market, why would I as a human want anything to do with that market? Why would I invest my earning into sustaining a machine that ideally is extracting every cent possible without leeway and funneling it up the pyramid? Unless I start at the top, ill never really go meaningfully higher.
Instead I can spend far less to use my 200 iq machine that already exceeds everything I could ever want. Im not ever going to personally fund a war of intellegence against other intellegences so anything more is of marginal value. The same reason I don't own a 2,000 horsepower truck or tractor, there are almost no uses for me to want it.
That's the fun part, there is none (in the traditional way of people buying things at least).
> Why would I invest my earning into sustaining a machine that ideally is extracting every cent possible without leeway and funneling it up the pyramid.
You shouldn't! Unfortunately everyone acting in their own self interest still results in this getting funded, since if such a machine were to exist, would it not be better to have a share of it?
I agree with your perspective! I just don't think that, as a species, we have a good track record of saying no to the existence of 2000hp trucks with ALMOST no use.
The demand for frontier intelligence is basically infinite. The frontier doesn't just complete a task more effectively, it creates new tasks and new problems and new solutions. It grows the pie exponentially. It's like looking at the internet the first time and trying to imagine the economy that will be built using it: social media, YouTube, Online job boards, Blogs etc.
> ChatGPT launched in November, 2022. Opus in December last year. Do you see where this is going?
Maybe this is tongue-in-cheek, but in case it's not.
No, nobody does. Consider that a person in their lifetime could have seen the Wright Brothers first flight in 1903, then the first jet engine in 1939, then commercial jet travel in the 1950s. That is an amazing 50 years for air travel. If you were living in 1950 and were to extrapolate where air travel would be by 2026, you would think we would be taking routine trips to Mars. Instead what we got was TSA, cramped seats, and better safety. But "progress" leveled off dramatically. Commercial air flight has looked pretty much the same for my entire life.
We have no idea how far transformers and LLMs will take us. It certainly isn't obvious where this is going.
I think it's credible to believe that that a system which is able to meaningfully improve it's own intelligence will lead to a far weirder world than the one imagined by the article.
The parent comment could be interpreted as frustration from a lack of imagination about how weird the future could be.
Perhaps we're not at the recursive self improvement yet, but it seems increasingly naive to believe that it isn't possible.
Of course you might be right. But if we look at our past ability to predict the future by extrapolating from the present and recent past, you might also say it is naive to think that _this time_ is somehow different, that this time the future is clear.
Also, I think you've set up a straw man. I haven't seen anyone arguing that this future isn't possible. What I see is resistance to the idea that we can have any certainty about the future by drawing a straight line from the past.
I agree with you that there is no certainty, and in fact the future will almost certainly not look like anything we imagine. I think that's what makes it exciting.
We think about starships and humans mining Mars, and it'll probably be nothing like that. It might just be humanoids or AI drones that actually end up colonizing the solar system, with the humans never really leaving earth except as mind uploads...
But at the end of the day, things are changing. Things are accelerating. And our mental models of what works or doesn't work will almost certainly break. I don't fear the future, nor do I project any confidence on what that's going to look like, but I know I want to be part of it.
> What I see is resistance to the idea that we can have any certainty about the future by drawing a straight line from the past.
I'm arguing that if recursive self improvement happens, that the trend line will be reasonably predictable. (and that we are close enough to RSI that we should take this possibility seriously).
woa your nephew lives inside a height-optimization machine that gets redesigned every few months (weeks), learns from every growth chart on Earth, and keeps proving it can grow faster?
> I just don't understand why people are incapable of extrapolating.
I guess it's a matter of education. People with a mathematics or computer science background know that unless the dynamics of a process are known to be linear, extrapolation is usually wrong. Since we don't know that the dynamics here are linear and have every reason to believe they aren't, extrpolation is unlikely to teach us anything valuable.
Because most systems in nature and society, including technological progress, aren't linear.
It's certainly plausible that improvements will continue, but the pace is completely unpredictable. It's also plausible that the training material is polluted and progress will not continue. I'm just saying that predicting the rate of technological progress is not easy, and historically, it's rarely been smooth in the long run.
As for the recursive part, currently it's either hypothetical or based on too few data points. Not saying it won't happen, but it's far from being the only plausible trajectory.
no because we have no way to understand the full consequence chain of any stated goal. We see this time and time again in all complex system goals; unintended consequences.
So RSI toward what prompted goal? Then, how can you be so sure that reaching that goal looks like the straight line you envision?
Is a coherent, human-interpretable, goal necessary for recursive improvement? (profit/money making is also a contender).
> how can you be so sure that reaching that goal looks like the straight line you envision?
Because the thing pursuing the goal would be able to improve it's ability to pursue the goal. Doesn't that follow from the premise of RSI?
> we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes.
We are terrible at it! I don't think that will stop us letting something rip in some unknowable direction. I hope that aligned incentives are a prerequisite, but it seems increasingly likely that they are not.
> Because the thing pursuing the goal would be able to improve it's ability to pursue the goal
But not necessarily without needing a huge amount of resources. It is a mathematical certainty that no intelligence can solve computationally intractable problems (including forecasting the weather, or the economy) without access to resources we simply don't have.
> I suspect we just have different beliefs about how close we are to RSI.
I don't have any belief on the matter, but my scepticism isn't necessarily about RSI itself, but about how much it would matter even if it does happen soon. Too many things are limited by lack of resources that no brain in a jar can obtain. And if such a brain in a jar itself is very expensive to operate, it may not be easy for it to justify its existence. My point is that the technical aspect is uncertain, but it is also only a part of a larger system that's has many sources of uncertainty.
This makes sense. Interacting with the world is hard and poses real bottlenecks.
I think that resource acquisition is a solvable problem for such a system, robots seem mainly limited by software.
I concede that I am too certain about the linearity of it. But like, cmon man, in the limit entropy ensures that everything ends. The world could still get very weird very fast.
First, they're not. Second, some resources would take even robots decades or centuries to collect.
> I think that's what original comment was trying to argue, and I think it's possible within the limits you've laid out.
Possible isn't the same as likely, and the reason we don't extrapolate is that different extrapolations lead to very different results.
But regardless of what happens and when, I think that we, people educated in computer science, should remember that many questions are simply not answerable in a short amount of time, and we know with absolute certainty that answering them is not a question of intelligence but of computational power and time. We no that no human or machine, however intelligent, can predict or control the nonlinear systems that are all around us because they are computationally intractable.
We comprehend, but these "scaling laws" have been in effect for less than a decade (and they're more historical observations over a very short period than actual laws), and while some technologies progress exponentially for some amount of time, the complexity of some computational problems grows exponentially forever. For example, if computational resources double every year, it may still be five centuries before some computational problems can become practically computable. It is mathematically proven that no amount of intelligence can compensate for resources, and even if exponential growth could be sustained for a long time, and that's a big if, an exponential curve still grows slowly at the beginning.
For example, suppose AI helps us figure out a way to exploit much more of the sun's energy. Accomplishing that necessary preliminary can, on its own, take many decades, and if we split our efforts among multiple approaches, it may take longer. Intelligence can't break the actual laws of mathematics or of physics. And that's before we consider things like how resources will be allocated when AI tells people that man-made climate change and transgenderism are real.
Of course, improvements in problems that may not grow exponentially also matter a great deal, but there are too many unknowns (look at how many unknowns there are around quantum computing).
It literally is just next word prediction. If you don't understand that, everything else you're thinking and saying is based on an incorrect foundation.
Take a step back and really take a look at what's going on. You are not thinking clearly.
Extrapolate the flight time from NYC to London in 1976 ... have you seen the J-curve?
Or extrapolate the curve for self driving cars once they were only "two years away".
Catch is, tech progress is not one curve. It's overlapping S curves, stair steps of progress and stagnation, with the next iteration starting off worse than the current one, until crossing it as the current one tops out.
It did replace draftsmen and designers, and changed the work that the engineers do. It's now more efficient to let engineers be their own draftsmen and designers. And design work has changed -- a lot less "hard" quantitative engineering, and a lot more "bureaucracy in three dimensions" as things have gotten more complex.
100 comments
[ 2.7 ms ] story [ 74.6 ms ] threadBig companies are always doing bad things, but not because they use AI, because they have legal protections which prevents small companies (which could be anyone like you and me) to compete. The same small companies who could also benefit from using AI.
What? How does that follow?
Because most automations never capture the complete scope of the job/task ( not even close ). Just like neurons, if you don't use it you lose it and when the inevitable problems come, nobody knows the why, the how and the what. At that point someone smart would incorporate all those real costs and opportunity loses on the "automating everything" equation. But they usually don't.
Of course automating tasks is a must, but it's very far from being a black and white situation. These dynamics have been happening for centuries by now, nothing new.
Ask an AI to solve a problem and it may do that, but if you don't understand why or how it works, or what to do with the information in order to keep it useful for even the medium term, then all you've done is taken away the opportunity for someone else to be responsible for something you shouldn't be.
It's not necessarily a mystery how to make good food. You can ask an AI how to make good food, follow the instructions, and you're off to the races. The question then is whether you want to be in that race.
Would you have gone to chef school? Would you work in a kitchen? Are you willing to deal with customers, or risk RSI from so many repeated kitchen movements? Are you willing to practice and be tested?
If the answer to any of those is no, then get the hell out of that kitchen and let the people who have more grit than you do their job. Do what you can to make it easier for you to pay them consistently and well on the back-end.
Can you show us how it’s better to use an AI to ensure steadiness over a long period of time?
I chuckled. Institutional prompt will disappear so much faster than you imagine.
Management likes to think otherwise for a variety of reasons and peons such as myself know it is rarely that simple. Case in point, our team uses Jira, but Jira is not great at.. capturing efforts that are umm.. less code oriented. In fact, there are times when it is genuinely better to leave some details out for considerations that may simply not be part of coding considerations.
In other words, automation assumes proper capture of the entirety of the task, but, I have seldom seen it capturing anything but simple workflows well.
Think about all the security clearances no longer required to aggregate big data into intelligence reports. The conditions and incentives for LLMs seem almost laser focused on replacing those particular jobs.
You're not one of the people who gets to benefit from the gains.
Instead, you can and probably should see technology as augmenting you and your coworkers.
Implementing labor protections and basic income is not seeing the forest for the trees. What we need is a more educated public that can contribute their ideas. All AI can do is lower the barrier to entry, but it does not replace the need for education.
False, Anthropic didn’t claim what was linked in the article.
> OpenAI CEO Sam Altman speculates that enough AI compute could help figure out how to cure cancer
Idk likely true? What’s so wrong here.
> Elon Musk says there will be no need for compilers as AI will write the binary directly
What seems so off about this? AI can write binary and probably better in a few years. Higher level languages would still be efficient probably.
> Anthropic claims that Mythos is super dangerous, the people can’t possibly handle such a powerful cyberweapo
Well it was?
Wouldn’t that make the AI the compiler?
We don't even have a solution to the halting problem, and it probably can't be solved. "Proof it implements a spec" is pure science fiction.
Hard agree that we'd all be better off muting Elon Musk though.
Assuming you mean an LLM, you'd have to train this LLM entirely on a parameter space of binary tokens. Or are you saying the LLM generating natural language tokens is going to be printing machine language in 3-5 years, because that claim kinda betrays a misunderstanding of LLM functions.
I assume that "AI can write binary" means "AI can use a toolset that results in a binary" because we've already seen GPTs use a combination of LLM and specialized math tools to do the things the original GPTs did.
So, today, if I think really really really, and I mean, really fucking hard about that “number”, I really really really come up with a number close to 0.
That means, for everything that we build today, before we even build it, we’ve already mentally replaced people. People be replaced yo, in your mind, in your company, in your reality. That’s it.
It’s a replacement, not a tool.
One programmer can know if the freaking program is right. So N people in that case, can truly be replaced.
I can know the SQL query came back right, I can know the drop down menu looks right, you know what I mean?
I think that's really unlikely on a large software project, or even a medium size one. Mature things usually get big enough that developers specialize in an area. Just as a random example I know a lot about, Unreal Engine has so many incredibly complicated and huge features that expecting even a project manager to know half the features is unlikely, and expecting that a single programmer could maintain it is wild.
You're also making the mistake of thinking developers are interchangeable, or that any given developer can know all of the tech stack. Again with the Unreal Engine example, how many developers understand animation, procedural generation, localization, high end 3d graphics, the math behind physics, gameplay systems, storage systems, consoles.. (I'm probably covering like 5% here)
I realized all this was quite wrong, AI just helps people who already know become more efficient. It just gives confidence to people who dont know anything by the sycophantic nature of AI. AI is just a tool for smart people to become smarter.
Love AI and what it allows us to do. But, it does give some superpowers, but the problems that existed before still do and they need to be solved for. with or without AI.
A friend of mine is super smart, but he's not a software developer. He tried to use AI to build software. And it worked well for small projects, but he always reached a point where the AI could no longer fix bugs or add new features.
The problems were often simple. An experienced developer could fix them with a few minutes of manual editing. Sometimes it was just an incorrect > in an XML file, but the AI would get completely stuck.
So IMHO AI works best when you know how to guide it and when to step in yourself.
My in laws smugly asked me what I would be doing for work since AI would take away all the programming jobs and I said its not gonna happen and they laughed in derision.
Some people who "used to love programming" have jumped ship for precisely this reason. Ordinary software (not R&D) is getting serious. A broken or disagreeable app is no longer a mere inconvenience. There is growing alignment between business and politics and the older generation is retiring out.
It's crazy how defensive (offensive) people can get when you're into programming/math/whatever. Almost like "I don't understand your world, I don't want to, and I hate it and hope you fail".
https://www.smbc-comics.com/comic/what-its-like
Sam Altman, Elon Musk, Peter Thiel, this whole scene has gotten fucking insufferable and everyone who isn't in the cult hates AI and hates the world techies want to build, and justifiably so.
We really need to renormalize beating up nerds again.
If you had the chance to do me harm now, would you?
But if you read my comment and only saw it as a personal threat then you are expressing exactly the lack of contextual insight and empathy that makes people find techies insufferable.
The nerds are the Wozniaks and Craig Newmarks who are curious and want to build cool things, not the sociopathic billionaires that always find the shortest path to their own selfish goals while burning down the earth.
Be curious and find the source of your pain, not the sacrifical lamb the media throws your way.
You should make $ bets with them on whether you will still have a job 2 years from now. An easy way to make money and teach them a lesson :)
AI may be a tool and add 50 IQ points to a 100 IQ person today but what happens when its adding 10000 IQ points. The 100 base doesn't quite matter as much does it? What does that world look like?
No, it's not just next word prediction. No you can't just wave your hand that it's probabilistic, so insert bad analogy to compiler or calculator and go about your day. And no, you can't just look at its limitations today, make a conclusion, and go back under your rock.
one end is luddites nothing to see here, and the other end is omg humanity is cooked because programming is solved! and consciousness is coding so -> gg
yes there’s a wide surface of emergent behavior we haven’t fully explored, but why do you get the pass to draw a straight line to a conclusion? the naysayers are doing the same thing.
Presumably there are a lot of use cases that are essentially “solved” where more powerful models won’t matter.
The only thing we can do is look at the current state of things, and we still have the same problems we've always had with these models, they need proper direction and input.
You seem to be implying that there was nothing before. Obviously not true.
“AI may be a tool and add 50 IQ points to a 100 IQ person today but what happens when its adding 10000 IQ points.”
What is even IQ? What if it adds 50 now, 25 next month, 12.5 the month after, etc. You should check out Zeno.
Yes. If it looks like 10000 iq is possible, then money will be thrown at whoever is most likely to achieve it first, and whoever has the most intelligent models right now strongly predicts who will be first.
This is driven by the belief that whoever gets to 10000 iq first will likely dominate the majority of all economic activity, since it will be more efficient to trade with them than with some less efficient (dumber).
This does not depend on traditional economic demand, at some point this becomes a self-fulfilling prophecy.
Instead I can spend far less to use my 200 iq machine that already exceeds everything I could ever want. Im not ever going to personally fund a war of intellegence against other intellegences so anything more is of marginal value. The same reason I don't own a 2,000 horsepower truck or tractor, there are almost no uses for me to want it.
That's the fun part, there is none (in the traditional way of people buying things at least).
> Why would I invest my earning into sustaining a machine that ideally is extracting every cent possible without leeway and funneling it up the pyramid.
You shouldn't! Unfortunately everyone acting in their own self interest still results in this getting funded, since if such a machine were to exist, would it not be better to have a share of it?
I agree with your perspective! I just don't think that, as a species, we have a good track record of saying no to the existence of 2000hp trucks with ALMOST no use.
Maybe this is tongue-in-cheek, but in case it's not.
No, nobody does. Consider that a person in their lifetime could have seen the Wright Brothers first flight in 1903, then the first jet engine in 1939, then commercial jet travel in the 1950s. That is an amazing 50 years for air travel. If you were living in 1950 and were to extrapolate where air travel would be by 2026, you would think we would be taking routine trips to Mars. Instead what we got was TSA, cramped seats, and better safety. But "progress" leveled off dramatically. Commercial air flight has looked pretty much the same for my entire life.
We have no idea how far transformers and LLMs will take us. It certainly isn't obvious where this is going.
The parent comment could be interpreted as frustration from a lack of imagination about how weird the future could be.
Perhaps we're not at the recursive self improvement yet, but it seems increasingly naive to believe that it isn't possible.
Also, I think you've set up a straw man. I haven't seen anyone arguing that this future isn't possible. What I see is resistance to the idea that we can have any certainty about the future by drawing a straight line from the past.
We think about starships and humans mining Mars, and it'll probably be nothing like that. It might just be humanoids or AI drones that actually end up colonizing the solar system, with the humans never really leaving earth except as mind uploads...
But at the end of the day, things are changing. Things are accelerating. And our mental models of what works or doesn't work will almost certainly break. I don't fear the future, nor do I project any confidence on what that's going to look like, but I know I want to be part of it.
I think you're right, my bad.
> What I see is resistance to the idea that we can have any certainty about the future by drawing a straight line from the past.
I'm arguing that if recursive self improvement happens, that the trend line will be reasonably predictable. (and that we are close enough to RSI that we should take this possibility seriously).
I guess it's a matter of education. People with a mathematics or computer science background know that unless the dynamics of a process are known to be linear, extrapolation is usually wrong. Since we don't know that the dynamics here are linear and have every reason to believe they aren't, extrpolation is unlikely to teach us anything valuable.
Why? Recursive self improvement seems credible, it's just a question of when.
It's certainly plausible that improvements will continue, but the pace is completely unpredictable. It's also plausible that the training material is polluted and progress will not continue. I'm just saying that predicting the rate of technological progress is not easy, and historically, it's rarely been smooth in the long run.
As for the recursive part, currently it's either hypothetical or based on too few data points. Not saying it won't happen, but it's far from being the only plausible trajectory.
> Not saying it won't happen, but it's far from being the only plausible trajectory.
But if it does happen, then wouldn't expected outcome be at least linear?
So RSI toward what prompted goal? Then, how can you be so sure that reaching that goal looks like the straight line you envision?
> how can you be so sure that reaching that goal looks like the straight line you envision?
Because the thing pursuing the goal would be able to improve it's ability to pursue the goal. Doesn't that follow from the premise of RSI?
> we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes.
We are terrible at it! I don't think that will stop us letting something rip in some unknowable direction. I hope that aligned incentives are a prerequisite, but it seems increasingly likely that they are not.
But not necessarily without needing a huge amount of resources. It is a mathematical certainty that no intelligence can solve computationally intractable problems (including forecasting the weather, or the economy) without access to resources we simply don't have.
I don't have any belief on the matter, but my scepticism isn't necessarily about RSI itself, but about how much it would matter even if it does happen soon. Too many things are limited by lack of resources that no brain in a jar can obtain. And if such a brain in a jar itself is very expensive to operate, it may not be easy for it to justify its existence. My point is that the technical aspect is uncertain, but it is also only a part of a larger system that's has many sources of uncertainty.
I think that resource acquisition is a solvable problem for such a system, robots seem mainly limited by software.
I concede that I am too certain about the linearity of it. But like, cmon man, in the limit entropy ensures that everything ends. The world could still get very weird very fast.
First, they're not. Second, some resources would take even robots decades or centuries to collect.
> I think that's what original comment was trying to argue, and I think it's possible within the limits you've laid out.
Possible isn't the same as likely, and the reason we don't extrapolate is that different extrapolations lead to very different results.
But regardless of what happens and when, I think that we, people educated in computer science, should remember that many questions are simply not answerable in a short amount of time, and we know with absolute certainty that answering them is not a question of intelligence but of computational power and time. We no that no human or machine, however intelligent, can predict or control the nonlinear systems that are all around us because they are computationally intractable.
For example, suppose AI helps us figure out a way to exploit much more of the sun's energy. Accomplishing that necessary preliminary can, on its own, take many decades, and if we split our efforts among multiple approaches, it may take longer. Intelligence can't break the actual laws of mathematics or of physics. And that's before we consider things like how resources will be allocated when AI tells people that man-made climate change and transgenderism are real.
Of course, improvements in problems that may not grow exponentially also matter a great deal, but there are too many unknowns (look at how many unknowns there are around quantum computing).
Take a step back and really take a look at what's going on. You are not thinking clearly.
Or extrapolate the curve for self driving cars once they were only "two years away".
Catch is, tech progress is not one curve. It's overlapping S curves, stair steps of progress and stagnation, with the next iteration starting off worse than the current one, until crossing it as the current one tops out.
Being embodied will always be a huge advantage over a “country full of geniuses in a datacenter”.
It did replace draftsmen and designers, and changed the work that the engineers do. It's now more efficient to let engineers be their own draftsmen and designers. And design work has changed -- a lot less "hard" quantitative engineering, and a lot more "bureaucracy in three dimensions" as things have gotten more complex.
In the long run, this is as misleading as saying: Humans did not replace animals.