HN Explorer

Ask HN: Kind of data needed to build a language model?

3 points by goy ↗ HN
On DeepL, Google Translate,etc the quality of the translation for not so popular languages is really bad, especially for oral-only dialects. I think this is due sometimes due to the lack written document or dictionary in these languages. What kind of data do you think I should gather from my local community if I want to build a translation software for our dialect ?

3 comments

[ 2.5 ms ] story [ 15.8 ms ] thread
PaulHoule
See here for some machine translation training data sets

https://metatext.io/datasets-list/translation-task

they might not have what you're looking for but you'll probably need something similar to one of the data sets they have.

Most systems now are trained on parallel corpuses, for instance there is a collection of 30,000 sentences in English and Japanese listed on that site. If you've got enough training examples you don't need a dictionary, a specification of the grammar or anything else. You need a lot of text though.

mikewarot
I shudder at the complexity of the task, but I almost think you need to manually tweak the vector space of a model that already works, and I have no idea how to do that in practice.

The amount of text required for a machine to grind through it millions of times to tease out the shape of a language doesn't sound like something you have. If you have the time of native speakers, it might be possible to build tools for them to correct the most "off" parts of the model interactively.