Ask HN: What are the challenges of building a personal assistant like Siri?
I'm thinking to build a language-specific personal assistant, like Siri or Cortana. I'm excited about the technical challenge but wondering about its feasibility.
Some of the challenges are:
1. NLP: I am not aware of any non-English library and don't have much background on the subject.
2. Voice recognition: same as the above.
3. Web crawling: there are tons of libraries for doing crawling and I have a decent understanding of the subject.
So few questions:
1. are there any other challenge that I have not considered?
2. is the project feasible?
3. will it succeed?
57 comments
[ 2.7 ms ] story [ 106 ms ] thread2. Have you read any of the latest research in this area? (think Google scholar).
3. Often, apps like siri and cortana do not process the sound on the device, but rather send it to a server for processing. There is, of course, exception in cases of small, well-defined applications with limited pattern matching required.
4. Whether or not it is feasible and/or succeed is based on your ability, performance, and approach. We cannot predict any of that here.
Yes, no and no.
The young composer is surprised. 'You didn't do all that'. 'Ah,' says mozart, 'but I didn't ask anybody how'.
https://groups.google.com/forum/#!msg/comp.lang.java/aSPAJO0...
https://www.quora.com/Will-I-become-a-billionaire-if-I-am-de...
Writing software - even hard, complex software - isn't even remotely comparable to a space program.
http://space-access.org
The vast majority of the pragmatic functionality I'd use on a day to day basis could probably be built by someone with the willingness to learn about the systems involved, no rocket science or all of humanities ML required :) (I certainly didn't know about anything from scraping to relay control to information retrieval at the start of my pet project)
I guess why I'm saying this is to address yours and some other child comments that seemed more dismissive of the idea due to the complexity (Responded to yours largely because you had gone through the steps of "try it yourself" which I see as the crux of all of this). To the op, Try it! What does it matter if it's out of your grasp. Learn new shit. Realize the extents of your knowledge, and in the process, push them a bit further. If you fail, I would be hesitant to ever call it failure if you even learned a little; I can't tell you the number of pipe dream impossible projects (microkernels, distributed OSes, rewriting AML on my own platform, etc) I've taken on, and despite failing at many of them, the learnings were well worth the time, not to mention the once in a while surprising success at someone you thought was out of your reach.
You could also use some of the Google APIs, but those are more-or-less unsupported, and subject to change as Google needs to change them.
From there, it's a matter of transforming intent into action.
It's entirely possible to do, but you'll have a lot of learning to do to implement it. One place you might start looking is at the Microsoft Python blog [1].
[0] https://www.microsoft.com/cognitive-services/ [1] https://blogs.msdn.microsoft.com/pythonengineering/2016/02/1...
If you, singular, are building Siri v2, you need to do as little as possible yourself. Find an OSS library / tool that does the thing you want, use Caffe/Theano/etc. instead of rolling your own code, use a commercial package if you absolutely have to. Concentrate on the "hard" parts, understanding and translating language to action. Build that part, and glue on all the other stuff. After you get a working thing you can peel off the layers and replace them with something of your own, maybe, or more likely something better will come out by that time and you can plug it in with minimal effort.
I set out six months ago to do exactly what you describe (build my own digital assistant) but decided it would be more helpful to convert it into a framework and open-source it.
I'm happy to help you get started via our IRC or the mailing list, or discuss the challenges we've faced and specific solutions we chose and why.
It's a fantastic idea. I wonder if there's a way to share the data generated as people use it. I would imagine that everyone would benefit from this, though it would have to be anonimized effectively.
Yes, we will absolutely make it easy to share training data via opt-in! If I train a plugin, everyone should benefit without needing a GPU rig or 2 TB of data.
You should go for it. Even if you never really get it to work you'll most likely learn a ton along the way. And if you even halfway pull it off it would make an epic "Show HN". Also, a lot of the required technologies are advancing at an incredible rate (both the state of the art and public accessibility). It may well be an order of magnitude easier now than it would have been a few years ago.
I don't have any specific advice other than the usual for any large project: start small and focused, then take it one step at a time. Oh, and read lots of academic research papers.
Good luck!
I'd even go as far you say, use an existing platform like Amazon's Alexa with their hardware to start.
Reading a books like "Calm Technology" and books on lean startups would be good too.
Books: http://www.amazon.com/Calm-Technology-Principles-Patterns-No...
http://www.amazon.com/Running-Lean-Iterate-Works-OReilly/dp/...
Start with any part of the problem and break it up into its component challenges. Start knocking off things one by one. Also start reading all the current papers on topic (a membership in a university library will help with this).
Since we take high school students and turn them into engineers with 4 years of training, figure that two and a half to four years (depending on how much time you will spend on core skills like upgrading your maths understanding) of reading, implementing, and improving.
Always. NLP is a valid way to accept requests. Web crawling is a valid way to get data into the system. How do you tie these two together? How is the data stored? How do create relationships between the data in your system?
2. is the project feasible?
You have not yet defined your project beyond "like Siri or Cortana". What does that mean to you? How will you know when that has been accomplished? Can you first define a more limited scope for testing the feasibility of the different parts of your system?
3. will it succeed?
First you must define success.
First you need a way to get input. Is that a text input or do you need voice to text? Google and others can provide a voice to text API to help along here at least at first.
Second you need to make that text meaningful. But what the heck is "meaningful" in this context? Generically running an NLP over this is going to get you sentence structure, etc which may or may not be all you need.
Essentially you need to take this text and turn it into a graph of decisions which can then be executed.
Basically distill the sentence "send a text to Josh saying hi"
To a graph that might have a branch like: Action -> Send -> SMS -> ("joshbsmith@yahoo.com", "hi")
Categorization is hard. If you can get it reasonable (85% solution) then you'll be pretty good. This is how Siri essentially works.
Things to consider:
- "How long is the Titanic" is this referring to a movie or the boat?
- "Where is Aaron's?" Is this referring to the furniture store chain, a friend in your contact list or even the street name?
Source: been looking into this a ton. Wrote this during a hackathon: http://devpost.com/software/sim. So this is all very possible :)
If you, however, want to learn a lot, then it's a definite "Go for it".
For Deep NLP, you will need to be solid on linear algebra and machine learning. For introduction to machine learning, check out Andrew Ng at Coursera: https://www.coursera.org/learn/machine-learning and my favorite talks on Linear Algebra are the ones done by Gilbert Strang: http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-...
For web crawling, there are plenty of open source libraries. If you are not familiar with it, check out the Common Crawl: http://commoncrawl.org/ This is a great source of data to crawl.
If you focus solely on NLP and data from the Common Crawl (or even Wikipedia), then you will see where you stand as the smoke clears and you feel comfortable with the state of the art techniques.
Ignore all the naysayers. The good news is that it has never been easier to get started in deep NLP. Once you have the experience of training a model and seeing how well it works, you can decide on the next steps. Perhaps, you will find a niche that is not well-served that you can go after as a first step of a personal assistant.
Good luck.
But of course, there are some engineering challenges as well, although you might want to use existing solutions (open source or APIs) for that. For speech-to-text you can find many APIs. Here's a nice demo for doing NLP: http://nlp.stanford.edu:8080/corenlp/
If you simply want to make an app for your own personal use, and you imagine a restricted form of dialog (by which I mean e.g. "query/reply" or "command"-type of dialogs as opposed to open discussions) to trigger a limited set of actions (say, verbatim web searches, control the built-in functionality of a smart phone, etc.) it is feasible.
But that doesn't mean it's easy. But for a project to hack on, why not?
The good news is that there are a lot of tools that can do some of the heavy lifting for you, especially if you restrict yourself to English. You are right that the situation for other languages is not quite as luxurious, but there are tools (of varying quality) for other languages as well, as specially Western European languages.
However, because it's a complex subject matter, expect that you might need to first dig into some linguistic and/or NLP theory in order to get the most out of these tools.
For instance, the Kaldi speech recognition toolkit is a state-of-the-art research software for automatic speech recognition (ASR), and it's open source. The thing is, to get really good recognition results, you might need to train your own acoustic and language models. Hence, you'd need to learn about these things.
For NLU (natural language understanding) there are also a bunch of free software packages available; however, they often follow completely different philosophies and goals. Thus, in order to make an informed decision which one would be the best for you, you'd again have to be prepared to do some reading.
One quite user-friendly service for NLU you might want to check out is wit.ai which was acquired by Facebook last year. They focus on setting the entrance barrier really low for the task of turning spoken input into a domain representation. For example, you can quite easily define rules that turn the utterance "turn down the radio please" into a symbolic representation that you can use in your downstream processing. The big plus here is that they do the ASR for you, so you don't have to worry about that.
If you prefer to have more control over your tool chain, there are a wide variety of scripting languages that you can use to get your feet wet. AIML is sort of popular for writing bots, but it's quite limited and you have to write rules in XML. VoiceXML is a standard that is great for form-filling applications, ie., situations where your system needs to elicit a specific set of information that's required to run a task. A classic example would be traveling: for your system to find a flight for you, it needs to know (a) point of departure, (b) destination, (c) preferred date and time, (perhaps others). So you need to tell the system, or it has to ask about this information.
There are also domain-specific languages like Platon (https://github.com/uds-lsv/platon) that, again, give you more control but also try to make it quite easy to write a simple application.
A next aspect more complex dialog systems typically care about is what the intent of a specific user utterance is. Say, you ask your personal assistant: "do you know when the next bus comes?", you don't want it to answer "yes". That's because your (what is called) "dialog act" was not a yes-no-question, but a request for information. So, you might want to care about how to detect the correct dialog act. Well, first you might want to care about what kinds of dialog acts there are and which of those your system should be prepared to handle.
There are many different dialog act sets developed for different domains and situations. There's also an ISO standard (ISO 24617-2) that defines such a set, but then you'd go into more advanced areas again.
Next, say your system has done all of the above processing, recognized speech, analyzed the meaning, et...
Voice can be a layer that you put on top later, with the extra complexity that the recognition may not be perfect. I guess it really depends which is the most interesting aspect for you.
As the website states, this SDK is only "one of three main components of the Microsoft Bot Framework", and I'm not sure if the other two are open-source, but it shouldn't be all that significant given that it's just integration with Microsoft products, like Office 365, plus the bot directory, which I imagine consists of some ready-to-use bots one could possibly piggyback on.
This is not voice recognition, but once you're past that stage what you have on your hands is a piece of text anyway - which has to be parsed and processed.
If you're motivated by learning, you'll certainly do so. You'll probably learn about topics you hadn't considered learning about. Start small and build incrementally. Be patient, you're in for a lot of reading.
If you're motivated by profit, i.e. starting a company, then you need to be strategic. The Big Guys - Apple, Google, MS, etc. - are mostly looking at "general purpose" assistants. Keep in mind that they're putting massive effort into these assistants because the technology will become a central part of products like automobiles.
Perhaps you can find a niche market they're missing so that you can target your research and coding efforts.
As others said, pick a small, focused goal in building an assistant and add goals as you succeed.
Getting started is not that hard, getting good is. It's a hard problem to parse speech correctly, take numbers for example: Nineteen-Eightyfour can be parsed as 19 and 84, or as 1948, or as 9 10 80 4. There are challenges, certainly, but creating a simple program with things like wit.ai is do-able. Prepare to write a lot of speech parsing logic, and implement every piece of functionality by hand. Magic as Siri, Google Now, and Cortana may seem, most of it is just hardcoded responses and actions. That need not be a problem, but I can promise you, smart assistants will lose a lot of their magic once your realize it's just a bunch of responses and actions hard coded.
Anyway, I don't want to discourage you from trying, because it's really interesting to try and see what challenges you're going to encounter. The getting started pack for speech recognition is mostly wit.ai, or the Google STT engine. Keep in mind: none of the big companies are doing everything from scratch. Sure, Google has their own speech recognition, but recognizing the trigger phrase (jargon for 'OK Google', or 'Hey Siri') is outsourced. Every piece of software that has such a trigger uses the same library. Remember: using libraries is not cheating, it's just focusing on your core task, which is writing the parsing and actions.