This is very cool. One thing I wonder about though is whether small companies will be able to compete with large ones like Google in ML in the future. One reason Google's translator is better is because they have way more data. In the past they digitized tons of books so they have an excellent dataset that has been translated by professional, human translators. This data collection is effectively cross-subsidized by Google's primary business: advertising.
Since most competitors to Google offerings aren't going to have a hugely profitable core business with which to fund all the data collection and normalization that goes into building a high quality ML system, the future for poorly capitalized competitors to compete seems bleak to me. This seems to support some of the growing rumblings about enforcing antitrust laws against the large tech companies.
Access to parallel corpora is a limiting factor in general. A good way to train a language translator is to use an open source dataset (several here http://opus.nlpl.eu/) to train a base model, and then fine-tune it with a smaller dataset specific to your domain.
In this case, the author claims pretty good accuracy, almost on par with Google Brain's!
On my test set of 3,000 sentences, the translator obtained a BLEU score of 0.39. This score is the benchmark scoring system used in machine translation, and the current best I could find in English to French is around 0.42 (set by some smart folks as Google Brain). So, not bad.
Wow, missed that part when I read it. Pretty incredible that using open source data you can outperform the state-of-the-art machine translators of a few years ago.
Yes, it shouldn't have been far off Google as the model in the article is Google's itself. In fact you see in some of the examples how Google Translate's output and this model are almost exactly the same.
Pretraining is a very cool idea, I have also seen some good results with pretrained embeddings after doing language modelling. Fast ai discusses this and I think even has some pretrained embeddings available in their library!
The basic idea is to use word vector embeddings to build a source<->target dictionary, then combine this with a language recognition model to iteratively bootstrap a set of source<->target training examples for use with a conventional ML approach.
Another perspective in similar veins would be the rise of AutoML. Given its absurdly high computational cost, I'd think only enterprises with massive computational power at their disposal would be able to use it.
Despite 'The unreasonable effectiveness of RNNs', it's seeming CNNs and the solely attention-based models are managing to perform the same tasks, but with better results and faster speed!
The transformer paper was quite influential in machine translation space. This resource [0] posted here a while back is a good place to learn and get a better idea how it works.
Machine translation has made some pretty impressive progress over the last decade. Unfortunately no methods will ever cover the very last mile as languages don't have perfect 1 to 1 mappings. Though it is amusing watching the machines try.
12 comments
[ 2.6 ms ] story [ 35.9 ms ] threadSince most competitors to Google offerings aren't going to have a hugely profitable core business with which to fund all the data collection and normalization that goes into building a high quality ML system, the future for poorly capitalized competitors to compete seems bleak to me. This seems to support some of the growing rumblings about enforcing antitrust laws against the large tech companies.
Edit: better, not bigger.
In this case, the author claims pretty good accuracy, almost on par with Google Brain's!
Deep learning only started beating tradition methods in 2016!
Pretraining is a very cool idea, I have also seen some good results with pretrained embeddings after doing language modelling. Fast ai discusses this and I think even has some pretrained embeddings available in their library!
https://www.deepl.com/en/translator
The basic idea is to use word vector embeddings to build a source<->target dictionary, then combine this with a language recognition model to iteratively bootstrap a set of source<->target training examples for use with a conventional ML approach.
[0]: http://nlp.seas.harvard.edu/2018/04/03/attention.html