This is the premise of the short story 'The lifecycle of software objects' by Ted Chiang. In that story the employees of the company take it upon themselves to maintain their human-like pets. If the concept of future AI/robot friends like this 'dying' is of interest to you give this story a read: https://www.goodreads.com/book/show/7886338-the-lifecycle-of...
I came to comment this. It was a weird story, to be honest. Like I didn't know where it was going, but at the same time I couldn't stop reading it. Highly recommended it.
From the Octopus Poultry Safe to the Chicken Boy, to the Spoutnic that hassles them off the floor and into the nesting boxes, chickens seem amply served by robotic companions:
Yes, I’m sure that’s exactly the lesson that autistic children will take from this. That won’t be hard to explain at all. Publishers are already rushing to make a children’s book on the perils of trusting SaaS with your friendship.
Let’s be real: This is at best a lesson for the adults, and not one they’re concerned with learning right now.
As someone with a kid, I feel bad for these little ones. When a stuffed animal gets beat up or a toy gets physically damaged, it's easy to explain to them what happened and why. When a pet dies, it sucks, but at least it's an opportunity for them to learn about life and death. Good luck as a parent explaining to a kid that her beloved friend is going to stop working because some company far away screwed up, they don't care, and they designed the thing stupidly to only work as long as they were perpetually in business. Buyer beware again and again.
Well, just like the robot is a simulacrum of a friend, the cloud disconnection can be a simulacrum of death, no need to explain to them about LLM tokens and their costs and how the MBA's costs-profits graph ended up not being sustainable for this shitty company.
That sounds like a good life lesson right there.
There's going to be sooo much stuff that's set up that way in their future unless they don't buy into it.
As someone with a kid I feel bad too, but in response to your point, if your kid weren’t developmentally aged to understand what was happening here, would explaining a pet death be so much easier to handle in comparison? I’ve had to explain to a toddler in active crisis that his toys have run out of batteries, went missing, and/or otherwise stopped working… It’s never fun, but putting the kind of existential crisis of owning hardware that is dependent on cloud based services onto your kid seems either sufficiently advanced or totally unnecessary.
If the company had any sort of ethics at all, they would release a patch that let you point it another LLM so you could at least keep it running. Or release their software as open source.
Or just do something so that they don't just "die".
Another example on the long list of "there should be a law that you have to open source your server if you're shutting down a server based service".
There's zero (0) motivation. They have their $800 / child. It implies there's someone who actually cares about children, or who can even possibly comprehend sympathy in charge of the company.
Rather than: "AI is hot. Parents with children spend money on toys. Selling children AI toys will make money. Profit."
Notably, does not look like that many people were actually swindled. Only 5000 downloads for the App companion on Google Play, and 27 reviews, and many are scathing. [1]
> Like a Jipsy moxie will keep taking your money And lie to to convince you to keep paying for it. It's $100 a month and cost $1,000 to buy.
> Do not buy this product. Technical support is non existent. I opted to return the robot. They acknowledged return. 3 weeks later still no refund for this $1K robot. Only after I created a PayPal dispute did PayPal refund me. Still not a single reply from the company to any of my emails.
if they owe debt, they may not be legally able to open source their software, since it could still theoretically be sold to someone to pay off debts. is there an expert in bankruptcy law that knows for sure?
The larger issue is that they sold a product which was designed for children to form emotional attachments to, but the product was never designed to be long lasting.
If they were ethical, they would’ve ensured that moxie would work at least somewhat offline.
Nintendo has been doing this for years. Their games are always almost entirely playable offline (outside of timed events.)
That’s why you can “time travel” in animal crossing and Pokémon.
It doesn’t even phone home to check the time even if it affects gameplay.
What are you on about, last I checked Nintendo makes a huge number of games unavailable and is extremely litigious against the emulation community. IIRC you can't even transfer savefiles to a different device in some cases.
Both can be true that Nintendo does well at sunsetting services to ensure they work as well as they can for as long as possible while still being complete jerks about making sure you can only play that game in that specific way and if you want to play it any other they try to force you to buy the same game over and over.
Server based services are different because they are required for me to keep using the thing I purchased. If my hardware requires an internet service to keep functioning, then they should have to make that service available for self hosting or for someone else to host it if they go out of business.
There should never be a case where something physical that I bought stops working because of an internet based service.
it's impossible to do it safely. open source model for kids!? without moderation every 'hacker' would like to get access. imho, the only option is to sell support as a business to someone else. with payed subscription it could survive for some time. but obviously not for long without new robots ans subscriptions. more interesting would be to have this sort of chatgpt for kids. with some basic robot control like flashing lights. then it would be possible to have several small and big manufacturers making mechanical robots and contributing to common software development.
The last time the issue of "abrupt" startup shutdowns messing up people's lives was debated on HN, the was a contingent that insisted that the founders and leadership have a fiduciary duty to share holders to hold out for a miracle. If they release a firmware that unlocks the robots, they are immediately setting their startups worth to about $0, when they could be bought out at the last minute for a couple of millions. Or so the story goes, I think it's immoral. When your left with <1 mo of runway, the best thing for your employees and consumers would be an orderly exit. VCs may not like that though, so it tends not to happen.
Personally, I love the phrase fiduciary duty. I use it all the time to help rationalize the choices I make, to ensure I maintain my fiduciary duty to myself, and use it liberally to convince my friends it's ok fight for raises, or the new job they want. I think I like it also, because of the context it has to live within my brain. A long time ago, when I was a young teenager, I was arguing, with a named partner for a huge lawfirm (family friends) about if a large company had to cede identifying information about users to $government. I made that same argument, they have a duty to shareholders. His retort was a simple. "No, this is a human rights issue, you can just say 'we're not playing'". That was a quick end to that conversation, because of how insane it is to suggest giving up human rights for money.
Fiduciary responsibility is nice and all, but have you tried upholding your responsibility to protecting human rights too? Sure, yes startup founders have a responsibility to hope for a miracle "for the shareholders!" but they have a much higher priority to human rights.
> That was a quick end to that conversation, because of how insane it is to suggest giving up human rights for money.
It is insane when you write it out kike that, but this is further complicated by money providing extrinsic motivation, and caring about human rights being intrinsic, at least with our current economic system and jurisprudence with laissez-faire approach to human rights violations (see wage theft for a direct example of the insanity). C-suites are more worried about shareholder lawsuits than they are about those from either consumer or employees.
My suspicion is that there's active filtering that prevents those who "overly" care about human rights from climbing too high up the leadership ladder at for-profit organizations.
> they would release a patch that let you point it another LLM
What a great idea. Let parents hook it up to Gemini, who told users to eat glue and rocks. I’m sure wrong and harmful information won’t have any effect on autistic children, no siree, can’t see a single problem with that.
And before anyone has the followup brilliant idea of then putting the blame on parents for whatever LLM they choose, let me remind you that even on HN (ostensibly with a tech savvy crowd) a ton of people furiously defend LLMs. Non-tech users aren’t as equipped to make these choices.
> any cloud based device is subject to the health of the company and LLMs are not cheap to run
Perhaps the technical angle to this story is the promise of edge ML. If your language model runs on the device, your cloud inference costs go to zero and the device works as long as it has power.
Aside from the financial benefits, there’s a huge privacy upside as well since no audio or text is sent over a wire. Might be notable for a children’s toy.
Of course, this is very difficult for large companies and VC-backed startups to care about because 1) it involves hard technical problems rather than API calls and 2) as long as you can keep asking for money the inference costs don’t matter and 3) there are no criminal consequences (prison time) for privacy violations.
In spite of my hugely biased pro-privacy stance, I'm skeptical that real privacy is a feature or selling point that can contribute to the financial interests of corporations enough to consider, broadly speaking.
To the corporations, data is the new oil, and to the vast majority of consumers, there's this very defeatist attitude around privacy, something like "I already don't have any privacy, what's one more recording device going to do?".
I think we should not expect privacy to meaningfully improve until the gap between end-user perceptions of the value of their privacy and corporate perceptions of the value of their customers' (or users') data shrinks, and I sadly don't see much hope here at all. In fact, I think a substantial plurality, if not an outright majority, even have the technical aptitude to critically evaluate the accuracy of corporate claims of privacy, like those misleading claims of privacy offered by Apple.
Humans are highly adaptable creatures, and sadly, I think most have settled quite comfortably into the panopticon that modern society has mandated for so many of us, embedded deep down in the terms of service that everyone agrees to but never reads.
>> I think we should not expect privacy to meaningfully improve until the gap between end-user perceptions of the value of their privacy and corporate perceptions of the value of their customers' (or users') data shrinks,
Do you see this as a generational thing? I remember working for wireless companies in the 1900's when GPS got huge and companies wanted to start using tracking apps for their fleet management and every company we talked to refused to install it because of privacy concerns and the drivers (and sometimes unions) being 100% opposed to it.
Now? We have what? Two to three generations who have never valued their privacy enough to really do anything about it. While I agree with your assertion, I whole heartedly believe the road back to people seeing privacy as important may have effectively died with the Gen Xers.
It’s a bit more nuanced than people simply not caring. All things being equal, people will choose not to be spied on. Apple showed that pretty conclusively in my mind with the “allow app to track” toggle.
If your generational hypothesis is correct we should see that older people are less likely to allow the app to track. I personally doubt that but I don’t have the data. More likely is that young people are more likely to use newer technologies at all, and those technologies have other side effects.
People care, imo. Companies know this so they make it a pain in the butt to opt out. Plus, the harms are very abstract and rarely materialize. So unless a person has a lot of time to spend configuring everything, they usually don’t waste time turning tracking off.
I agree, I don’t expect privacy to be popular enough to drive the R&D. But models small enough to run on-device will be a byproduct of other cost cutting, which means we will be able to turn off telemetry via other means or even develop more privacy-conscious devices.
For the layman, this will allow them to bypass a monthly cloud subscription fee which is a killer feature.
I don't think edge ML is competitive for now. It can do simple things but not the big beefy LLMs. State of the art AI chips are so expensive that you can't afford to idle them. They need to be M:N shared - M chips for N users so they have maximum utilization, and that fits perfectly for the cloud.
However, there is a middle ground: pluggable AI could potentially be a thing. The device would use an open protocol to access cloud AI. If the original company goes bankrupt then someone else can implement the protocol and the devices can be repointed.
Yeah definitely, especially for more complicated things like having a conversation. State of the Art will probably always need a server.
For simpler things, small models can definitely handle them. Transcription, object detection, simple classification tasks. I expect more and more to fall under the category of “things which ML can do on $X of hardware” as hardware and software get better.
Maybe this is the straw that breaks childhood's back, but they've said the same about every boogeyman since the 19th century. "The real problem is that parents are giving children [books → radio → comic books → rock music → television → video games → D&D → rap music → computers → internet → smartphones → social media → toy that uses AI], not [the actual problem]."
The truth is probably somewhere in between "social media/technology is the cause of all problems" and "social media/technology causes absolutely zero problems that wouldn't be caused anyway"
For sure, there are problems ascribed to all of these things consumed [ignorantly | irresponsibly | in excess]. Still, I've lived long enough to see many of these featured in hysterical "for the children" propaganda, and I find myself recoiling from that, maybe more than most. It's easy to see that AI (LLMs) are next on the list to be vilified, which seems absurd to me.
First, reasoning by analogy (or dismissal by analogy) is a poor way to reason. Second of all, that progression is already part of the problem in a way -- at least part of your progression is one of technology, which is alienating and isolating.
So, I disagree that it's any straw. In fact, I'd argue it's the reverse: AI has just revealed and accentuated the real probelm: social media, smartphones, internet, computers, video games, etc. were already some kind of problem and it's one of magnitude, not some binary condition.
If you're not so quick to leap to the thought terminating cliche you might stop to think that yes, those transitions did change us. And how might we change next after we outsource thinking and socialization to automation rather than just consume media in a different format?
Ultimately, the parent has the full responsibility for their kid, because they have way more involvement in developement of the "core" neural networks of the child.
I find the video in the article really interesting. This person is clearly an adult and is crying while discussing the robot's future "death" with it. She's clearly quite attached despite the robot being quite inhuman with sub par text to speech.
This company is a take the money and run type of operation. No subscription plan and I doubt the API fees were making a huge part of their obscene profit. They would have eventually, but clearly they were not going to wait that long.
Well yeah, I would cry if one of my stuffies got destroyed and it's even less animated than the robot. It being less animated actually helps the attachment because it's more like a pet than a person.
When will people learn about cloud dependent gadgets. And if this thing already cost $800, how much more would it have cost to put some basic AI into it for use as a fallback? Can't you run stuff like that on smartphones these days?
yeah, be aware that many things you buy today depend on some cloud service that may go away.
for example, a computer system that routinely checks for updates will hopefully still function if the update server goes away. and you can always install another operating system.
since I listen to most music via a streaming service these days, I don't consider the music owned and don't mind moving to another service (which I have gone twice, losing playlists in the process).
the worst is "digital copies" of movies. I try not to buy these, but occasionally do. I'm pretty sure that Blu-rays I buy today will work until the media degrades, but have no such confidence in digital copies.
I'm a former hardware engineer with a lot of ties to communities that include handicapped children. Happy to help reverse engineer and resuscitate these as a side-project if I can get my hands on a "dead robot".
A fair amount of the shittiness comes from people actively enabling that shittiness, such as by throwing large sums of money at fly-by-night companies selling products dependent on cloud services and then getting mad when the fly-by-night company goes under and their product stops working, and then voting for politicians who oppose any regulation that would curtail such behavior.
If you think humans getting emotionally attached to robots is bad, wait til you hear about the type of stuff that stems from humans getting emotionally attached to other humans
I'll take one of these units off someone's hands for $400, if they have it. I would like to disassemble one on video in a video marketed toward children.
I'm fine with that. Those children will learn valuable lessons about loss and death. The kind of lessons that having a goldfish pet will provide, but without killing a goldfish.
Or: "Robots will need to go hibernating now in order to return to its far, far away planet". You are welcome, coward parent.
Parents who can afford $800 toys for small children, made by startups that have not quite made it, can probably get them the therapy they need to get over this.
184 comments
[ 4.7 ms ] story [ 196 ms ] threadMade into the film https://en.wikipedia.org/wiki/A.I._Artificial_Intelligence
https://en.wikipedia.org/wiki/Klara_and_the_Sun
https://www.canadianpoultrymag.com/rise-of-the-robots-30876/
I mean, yes.
Except for the kids, that is, who've got hit by unneeded and unexpected grief.
Yay.
Let’s be real: This is at best a lesson for the adults, and not one they’re concerned with learning right now.
> nfl player pays for 15 kids to go to college
> student organizes bake sale to pay for school lunches
> chemistry teacher finds creative ways to pay for chemo
However, parents who shelled out $800 recently may not appreciate that.
Or just do something so that they don't just "die".
Another example on the long list of "there should be a law that you have to open source your server if you're shutting down a server based service".
Rather than: "AI is hot. Parents with children spend money on toys. Selling children AI toys will make money. Profit."
Notably, does not look like that many people were actually swindled. Only 5000 downloads for the App companion on Google Play, and 27 reviews, and many are scathing. [1]
> Like a Jipsy moxie will keep taking your money And lie to to convince you to keep paying for it. It's $100 a month and cost $1,000 to buy.
> Do not buy this product. Technical support is non existent. I opted to return the robot. They acknowledged return. 3 weeks later still no refund for this $1K robot. Only after I created a PayPal dispute did PayPal refund me. Still not a single reply from the company to any of my emails.
[1] https://play.google.com/store/apps/details?id=com.embo.embod...
If they were ethical, they would’ve ensured that moxie would work at least somewhat offline.
Nintendo has been doing this for years. Their games are always almost entirely playable offline (outside of timed events.)
That’s why you can “time travel” in animal crossing and Pokémon.
It doesn’t even phone home to check the time even if it affects gameplay.
If Nintendo goes bankrupt, my game boy, 3DS, GameCube, Nintendo 64, and switch will all still work pretty much as if nothing has changed.
Unlike moxie or that Spotify car thing.
You’re describing a different issue entirely around games preservation outside of official Nintendo channels (emulators and roms)
They could include any number of DRM strategies to prevent emulation at the cost of longevity of their software and hardware.
They don’t, though. They opt for litigation over really strong DRM.
There should never be a case where something physical that I bought stops working because of an internet based service.
another proof for 'robotics is hard'...
I can imagine this going horribly wrong fast.
Personally, I love the phrase fiduciary duty. I use it all the time to help rationalize the choices I make, to ensure I maintain my fiduciary duty to myself, and use it liberally to convince my friends it's ok fight for raises, or the new job they want. I think I like it also, because of the context it has to live within my brain. A long time ago, when I was a young teenager, I was arguing, with a named partner for a huge lawfirm (family friends) about if a large company had to cede identifying information about users to $government. I made that same argument, they have a duty to shareholders. His retort was a simple. "No, this is a human rights issue, you can just say 'we're not playing'". That was a quick end to that conversation, because of how insane it is to suggest giving up human rights for money.
Fiduciary responsibility is nice and all, but have you tried upholding your responsibility to protecting human rights too? Sure, yes startup founders have a responsibility to hope for a miracle "for the shareholders!" but they have a much higher priority to human rights.
It is insane when you write it out kike that, but this is further complicated by money providing extrinsic motivation, and caring about human rights being intrinsic, at least with our current economic system and jurisprudence with laissez-faire approach to human rights violations (see wage theft for a direct example of the insanity). C-suites are more worried about shareholder lawsuits than they are about those from either consumer or employees.
My suspicion is that there's active filtering that prevents those who "overly" care about human rights from climbing too high up the leadership ladder at for-profit organizations.
What a great idea. Let parents hook it up to Gemini, who told users to eat glue and rocks. I’m sure wrong and harmful information won’t have any effect on autistic children, no siree, can’t see a single problem with that.
And before anyone has the followup brilliant idea of then putting the blame on parents for whatever LLM they choose, let me remind you that even on HN (ostensibly with a tech savvy crowd) a ton of people furiously defend LLMs. Non-tech users aren’t as equipped to make these choices.
Perhaps the technical angle to this story is the promise of edge ML. If your language model runs on the device, your cloud inference costs go to zero and the device works as long as it has power.
Aside from the financial benefits, there’s a huge privacy upside as well since no audio or text is sent over a wire. Might be notable for a children’s toy.
Of course, this is very difficult for large companies and VC-backed startups to care about because 1) it involves hard technical problems rather than API calls and 2) as long as you can keep asking for money the inference costs don’t matter and 3) there are no criminal consequences (prison time) for privacy violations.
To the corporations, data is the new oil, and to the vast majority of consumers, there's this very defeatist attitude around privacy, something like "I already don't have any privacy, what's one more recording device going to do?".
I think we should not expect privacy to meaningfully improve until the gap between end-user perceptions of the value of their privacy and corporate perceptions of the value of their customers' (or users') data shrinks, and I sadly don't see much hope here at all. In fact, I think a substantial plurality, if not an outright majority, even have the technical aptitude to critically evaluate the accuracy of corporate claims of privacy, like those misleading claims of privacy offered by Apple.
Humans are highly adaptable creatures, and sadly, I think most have settled quite comfortably into the panopticon that modern society has mandated for so many of us, embedded deep down in the terms of service that everyone agrees to but never reads.
Do you see this as a generational thing? I remember working for wireless companies in the 1900's when GPS got huge and companies wanted to start using tracking apps for their fleet management and every company we talked to refused to install it because of privacy concerns and the drivers (and sometimes unions) being 100% opposed to it.
Now? We have what? Two to three generations who have never valued their privacy enough to really do anything about it. While I agree with your assertion, I whole heartedly believe the road back to people seeing privacy as important may have effectively died with the Gen Xers.
If your generational hypothesis is correct we should see that older people are less likely to allow the app to track. I personally doubt that but I don’t have the data. More likely is that young people are more likely to use newer technologies at all, and those technologies have other side effects.
People care, imo. Companies know this so they make it a pain in the butt to opt out. Plus, the harms are very abstract and rarely materialize. So unless a person has a lot of time to spend configuring everything, they usually don’t waste time turning tracking off.
For the layman, this will allow them to bypass a monthly cloud subscription fee which is a killer feature.
However, there is a middle ground: pluggable AI could potentially be a thing. The device would use an open protocol to access cloud AI. If the original company goes bankrupt then someone else can implement the protocol and the devices can be repointed.
For simpler things, small models can definitely handle them. Transcription, object detection, simple classification tasks. I expect more and more to fall under the category of “things which ML can do on $X of hardware” as hardware and software get better.
So, I disagree that it's any straw. In fact, I'd argue it's the reverse: AI has just revealed and accentuated the real probelm: social media, smartphones, internet, computers, video games, etc. were already some kind of problem and it's one of magnitude, not some binary condition.
Not enough emphasis is placed on this.
This company is a take the money and run type of operation. No subscription plan and I doubt the API fees were making a huge part of their obscene profit. They would have eventually, but clearly they were not going to wait that long.
Humans will pack bond with anything.
for example, a computer system that routinely checks for updates will hopefully still function if the update server goes away. and you can always install another operating system.
since I listen to most music via a streaming service these days, I don't consider the music owned and don't mind moving to another service (which I have gone twice, losing playlists in the process).
the worst is "digital copies" of movies. I try not to buy these, but occasionally do. I'm pretty sure that Blu-rays I buy today will work until the media degrades, but have no such confidence in digital copies.
There is no ethical way to build robots to which humans are meant to, or predictably will, become emotionally attached.
Woah. So you'd have to keep paying for this now-defunct plastic piece of garbage?
They go out of business too quickly these days, and run away with the money.
Or: "Robots will need to go hibernating now in order to return to its far, far away planet". You are welcome, coward parent.