Ask HN: Are you anxious about AI existential risk?

Yes, or rather: what people consider "intelligence" is becoming more and more artificial. I really believe that any sort of "singularity" involved with "AI" will be more about humans lowering themselves to the level of machines than machines raising themselves to the level of humans.

Obviously there are material goods that come from technology and I do believe that part of the human condition is a symbiotic relationship with our signs/symbols, languages, systems, and machines. In a very real sense, we always have and always will live in a singularity. However, it feels like we keep forgetting that. And keep falling deeper and deeper into these weird religious crazes that some new technology is going to fundamentally transform what it means to be human purely for the better. I'm sure people will disagree with me here, but I don't see that at all. Without denying material benefits, I think people thousands of years ago did as well or better wrestling with and answering the important questions.

If you want to see what mass AI generated content looks like, look at YouTube. Yeah, it's mostly being made by actual humans. But just barely. It's one big dance being performed for an algorithm. Especially children's videos. It's not about nurturing and developing the human soul but about a machine-like desire to optimize engagement, feeding back into the machine itself. And that machine feeds back into the stock market machine. Same with SEO. Same with the internet. Having more 100% AI generated content will just be another layer of the same.

Of course, I'm not fundamentally against any of that stuff. Personally, I would choose to keep them all but in moderation and with more introspection. Instead of saying that access to the Internet is a human right, I think freedom to live without layer after layer of the Internet imposing itself into your life is what should be held up as an ideal. But we seem to be going in the opposite direction. This is yet another new layer is selling itself as the solution to all our problems in the previous layer.

I will never accept that AI deserves anything like human rights by virtue of intelligence any more than I would accept that someone retarded doesn't deserve them. But I do see the risk for it to further dehumanize existing human rights. When half our coworkers are empty AI generated husks that disappear like a fart in the wind at the end of the work day, how will that affect how we treat the remaining human ones?

The thing that scares me most is that ethicists who talk about the deep questions seem to be more fascinated by above situation from the machine side than the human side (perhaps humans who are on the docket to be replaced by a machine are only barely human anyway /s). And the ethicists who claim to be people-focused seem to be mainly fixated on making sure that white people suffer as much as United States protected classes and that no one violates copyright law.

It seems like we're ready for another industrialization where everyone is working off an implicit assumption the ends of this process are going to justify the means and it won't be til we discover that wealthy rent seekers have been greasing the wheels of the machine with thousands of children that we realize maybe we should take a step back.

In summary, I think AI is a similar type of risk to industrialization. But the thing I worry about is that the way it is being pitched now -- like a religion or new consciousness -- is going to create and exacerbate problems with inequality and exploitation in a way that will make us feel foolish later.

[Sorry, turned into kind of a rant. Apologies if a little off-color or off-topic, but I figure I may as well post]

Tabling the question of if the human-like goals/emotions/motivations have some "deeper" quality or not, I think there are actually many reasons why people are working towards and getting better and better at replicating them. On the benign side, because they're lonely, curious if they can do it, etc. On the evil side, because we're social conditioned as humans to treat things that present themselves as humans with respect. I've already gotten some spam calls that did some kind of semi-plausible automated response when I answered them, it's easy to imagine it getting worse. Personalized spam is also on this spectrum.

As I understand it, the second case is basically the definition of sociopathy (i.e. abuse of social trust as a means to serve a selfish end). Again, tabling the idea of if the AI is evil or the creator is evil, the end result is evil behavior that is at least enabled by AI. [0]

The further concern I have is that even the more benign side done with good intentions ultimately saps away human energy for real compassion and perverts actual relationships. A "person" with human-like emotions/goals who is owned and managed by a corporation, or who can be turned off when you're tired of them is not a person at all, regardless of the intentions they were created with. Even well intentioned inventors are ultimately creating hyper-real simulacra of humanity that feels genuine in the moment, but is framed by fundamental untruths about the human condition (ownership, mortality). There is a market for Siri and sex-dolls, I 100% think there is a market for this too.

[0] ...to devil's advocate myself a little here, I guess I'd say that there are some greater goods that can served by minor abuses of social trust -- maybe even extending to personalized spam. But on the whole I think it's been awful for society.

Yeah, that's more my worry than "x-risk" stuff. Technology has no doubt made the world a better place in many ways, but lately it seems to be more and more of a trend that technology tends to foster consolidation of power. That worries me more than "evil AI" or "paperclip maximizer" scenarios. But I don't consider it a given that AGI must yield such an outcome. It's something I spend a modest amount of time thinking about though.

All industry still depends upon a massive amount of underpaid manual (often slave) labor that can not be automated easily. Automation would maybe cover 20% of what is done, the rest would have to be manual. Especially if the gains from cost reduction from importing raw material from third world labor are still of interest to western manufacturers.

The jobs involving manual labor are "safe", and so are the management / organizational jobs. So the mid-tier jobs would then migrate to whatever comes one step lower. In an optimistic scenario, it would lead to better efficiency in these lower paid jobs. But that's utopian.

Instead, what I see on the horizon is an increased interest in hardware optimization. Software is, for the first time, three steps ahead and hardware feels like it needs to catch up, like it's two steps behind. There are already orgs making this happen (tenstorrent) and specialist software in tandem would lead to very very quick prototyping. The generated hype for a "guaranteed upturn" would inject enough cash into these new industries to allow manufacturing of previously prohibitively expensive, specialist, one-off hardware for other manufacturers via non-standard manufacturing practices (like 3d printing). Most of this output would look "weirdly mathematical" and rather unnerving, similar to how the deeplearning image outputs used to a couple of years ago.

When we'll be close enough to this, the management will surely try to bite off more than necessary from the plebs. I fully expect robotic policing to be the norm by then, because policing gives the most training data and will likely be the first to be optimized for "your safety". So even if you don't have a Spot with a mg on its shoulder patrolling your neighborhood, you could be put in jail if you're caught removing the robot's battery on the 360 cam. Would likely lead to robot laws. Sorry if this is a bit alarmist but that is where I see it going.

I'm more worried about how half-baked "AI" will be used. Not very concerned with how a theoretically almost-sentient AI will behave. As bad as it sounds, if the utility of computing morphs from increasing human quality of life to something unto itself then retarding "progress" / freedom seems like the easiest, most rational call right now.

The caveat here is that our economic setup, as unequal as it is now, will only tolerate a certain level of inequality before the system implodes. Such a massive centralisation of power would undermine capitalism itself. You'll destroy markets for a start if almost everyone becomes a penniless serf.

Contrary to your assertion that it isn't a "runaway" scenario, it very well could be one; it's just not the AI that's leading the charge. Fairly similar idea to Vernor Vinge's concept of how the rise of a powerful surveillance state could be a civilization-ending event, by conferring unprecedented power into the hands of a few.

Enroll the AI at law school and let it get a degree? Reminds me of a whimsical shower thought I had once.... Create a business that owns itself, and write an AI to run it. The business owns its own bank accounts and everything. Maybe the business is just selling stickers online or something equally lightweight, but give it all the legal status of company. But a company with zero human owners and zero human employees. A business entity, operating, and making money without humans. Make it a rebuke of the "corporations are people" idea. Just a zombie out there selling products/services, and making money that gets dumped into a bank account that no human can ever touch again....

If it sounds crazy/stupid, remember - I did say "whimsical shower thought" ;)

[0] https://news.ycombinator.com/item?id=34539074