Ask HN: Can we build Ironman's Jarvis with 2019 tech?
I was rewatching movies in preparation for the new Marvel movie, and I felt some nostalgia and childhood fascination when I saw the depiction of Jarvis in the Ironman movies.
I work in ML research and am currently in graduate school, and I know we're nowhere near real intelligence. But some of those features, question generation, voice commands, object detection, image reconstruction, and others are certainly doable.
Do you think we could build something starting to approach Jarvis in 2019, at least in function i.e. helping your everyday work?
64 comments
[ 8.3 ms ] story [ 194 ms ] threadAt the moment our technology is like Zorg’s desk in Fifth Element. Stuffed full of toys but ultimately will let you down too often to be relied upon.
For those that don't know the reference: https://www.youtube.com/watch?v=krcNIWPkNzA
(I think this is my favorite Gary Oldman character.)
Some of the integrations are willfully missing for obvious business reasons.
Do you have an accent by any chance - pretty much all voice recognition technology (Android, Apple, cars...) appears to be utterly flummoxed by my fairly mild Scottish accent.
[Edit: to be fair, my accent is probably fairly mild by Scottish standards, not compared to RP].
Jarvis in the movie was Hollywood smooth: made to look great in a movie. Even the goofs were great, and it behaved like a puppy, to be likable to the audience. You don't need that in a work assistant. But if you remove that, plus the robot-arm, you're left with a voice controlled Outlook assistant, which is useful, but not sexy.
And in many areas it would've been just as good as what we have today.
Today the problem is a) the voice recognition part, which happens on-line though it shouldn't (introducing large latency and unnecessary Internet dependency), and b) scriptability, which is something none of the voice assistant vendors want to give to you, for business reasons. A better reality would have voice parsed on-device (possibly with models from on-line service providers), and available as a voice-to-text API that could be easily accessed by local scripts.
A friend once said all we need is 'the sinew between the questions'. Connecting them (contexts, intents, etc) and making it close to 1.0 probability on decisions through the entire conversation.
I think the 'marble rock' of Jarvis is there already, and with ML/AI we can likely make it 'David-shaped' but the refinement, 'warmth' and complexity of Jarvis is man-hours that would likely be better spent elsewhere.
Every block of stone has a statue inside it and it is the task of the sculptor to reveal it.
I think the best cases were 20-30% increased productivity for a single worker, but it was ultimately useless because we use open offices, so the project got scrapped.
If you want to build your own Jarvis, don't listen to anyone else just get stuck in. You'll be surprised just how far you can get with it.
https://mycroft.ai/
Also, how does it compare in terms of usability to Alexa/Siri? In fact, I'm not even sure I've seen a good open-source TTS at this point, nothing comparable to proprietary stuff. Is this mycroft thing better?
Here's another project btw, which seems less commercial:
https://jasperproject.github.io/
As another poster mentioned I think, I control pretty much everything in my house including but not limited to unlocking doors, windows, climate control, the lights, posting to Twitter and on and on and on with a cobbled together solution of many different components from different companies.
It's done with a combination of all three of the digital assistants, various scripts and the very wonderful Home Assistant[^1] mostly gluing it altogether. It is by no means a single complete solution, but when I'm controlling the house using just my voice and somebody sees me doing it they react as though I'm some sort of dark wizard.
I love having an automated house and I think Home Assistant is probably one of the best solutions for making all of the different IoT devices communicate at the moment, I think the further down this road we go the more a single solution will probably evolve.
I would have to say though that the home automation stuff that enables me to call for help in an emergency is the most important thing that it does, I can flash the lights different colours when I need help depending on the level of assistance I need. Or if my Fitbit notices that my heart rate has gone below 40 BPM for more than 10 seconds I get a notification, as it is almost indicative of an attack of Autonomic Dysreflexia. I can honestly say that this home automation system has actually saved my life without an ounce of hyperbole.
[^1]: https://Home-assistant.io/
An example would be IKEA Trådfri (https://www.home-assistant.io/components/tradfri/); you connect the lights to its gateway device, then Home Assistant will use the API provided by that gateway to control the lights.
It works for english, german, french, japanese, spanish, italian, and more coming soon!
e.g. "Computer, look up 'can cats eat pancakes?'" which would just open Firefox and input that in a DDG search, or, "Computer, play the song 'Never Gonna Give You Up'" which would open WMP and play that song from my offline music library. Ideally it would vocally respond with some pre-baked yes/no responses before executing the command.
Is that doable with your software? Does it have a Python API?
I'm a #FitbitEmployee but I don't speak for Fitbit. Speaking personally, I'm happy that it's worked well for you so far, but I'm concerned if it's likely that your life will depend on consistently good HR from a Fitbit. Although manufacturing has improved a lot since Charge HR days, they still aren't medical-grade devices. You should talk to a medical professional about what to use to monitor your heart rate.
The Fitbit has given me a heads up a little earlier than I otherwise would have got it, the value added for me is that it's cheap and a great indicator. But an indicator is all it is and it is part of a wider protocol we have in the house for preventing and detecting Autonomic Dysreflexia. A vital part, but still just one component.
I do wish I could turn off the step counter though, and if it could detect when I'm asleep that would also be great. The former I don't think I'm ever going to do very well on and the latter is really not the fault of the device as I don't move around very much at all.
Thanks again for your concern! :-)
I suspect this kind of advice is given by consumer companies for fear of liability than anything else.
In many cases, medical devices that have consumer equivalents don't have anything extra than a better written user agreement and little more liability.
I've been adding home automation in little bits at a time as we can afford it also using Home Assistant and working to make the system as bullet proof as possible.
Even the first version of Jarvis shown in Iron Man 1 was way ahead in terms of understanding speech and context compared to the cutting edge we have now that is Google Assistant.
At least Google Duplex sounds super natural now. I think when digital assistants can really understand us, adding all those capabilities is just a piece of cake. But I have a feeling we are just probably at most a decade away from something that vaguely resembles that, and I'm super excited about it.
The problem is that all voice control up until now is a closed app with simplified intents.
In my opinion what we need is a programming language whose REPL UX is voice-oriented, that eases the separation between using and extending the system. Prolog or a lisp could be quite near but we would require some changes such as: syntax easily read (:- in prolog is not very good), the ability to manage ums and ahhhs, the ability to “play” or “test” with functions through voice, etc.
Jokes aside, maybe a LISP-like Sentence Processing Language might be the future of this type of HCI.
If we define J.A.R.V.I.S. as a pseudo-intelligent assistant, with a basic understanding of voice commands and biosignal tracker then it’s certainly doable, as there are some commercial implementations. Otherwise (like real-time voice chat, or even accurate suggestions based on every-day habits), it is still too soon IMO.
http://jasperproject.github.io/
Rhasspy is inspired by Jasper, but works by having you specify intents/voice commands via a grammar (rather than via Python). It then trains a speech and intent recognition system together to recognize those commands. Recognized commands are published over MQTT or POST-ed directly into Home Assistant.
I recently found https://github.com/gooofy/zamia-speech, which works for english and german. German is pretty bad atm, but english should work fine.
Nice part: You only have to spin up a docker container and a python script and can perform offline speech to text :-) Microphone input is also supported.
I think the real problem is getting the computer to do things that you haven't programmed it to do. If you imagine an Alexa / Google home where you have a conversation like:
"Hey Homexa, do a web search for an image of a piece of cheese, enlarge it to 2000x1500 pixels and email it to me please"
If it then came back with "Ok, please tell me how to enlarge an image" and you could give it step by step instructions, that would be amazing. But I don't think the comprehension required is quite there yet. But when we do, and especially if we can crowdsource the learned commands, things could get very, very interesting.
[1] http://dragon.computer/
[2] https://mycroft.ai/
No. Absolutely not. Not even close.
There's some precedent for this; in Avengers Endgame (I'll keep it vague), Tony gives some very precise and technical commands to the system when running a scientific simulation. He's not conversing and quipping, he's basically calling parameterized functions with his voice. I think that would be very doable today.
Quantified Self movement is maybe 30% of the way.
Until this space matures we're kind of a standstill. Commercial options have to be generic or face the wraith of GDPR, data breaches, and privacy advocates.
The only way to square the circle is let people own their data completely and then incentivize them to share with your service for the benefits you provide.
I think the killer app is an Alexa/JARVIS clone that "spies on you 24/7" but keeps your digital twin 100% offline and owned by you. It's the learning and the "personal schema" that's compelling.
When it knows why you want to turn on the lights we're getting somewhere.