Oh my, I've wondered about this a lot since I come from a Sanskrit background and I love etymology. I'm also cataloguing the superb similarities between Sanskrit and English despite several mutations spanning millennia.
The optimist in me thinks this is the written script for an early version of Proto-Indo-European. Would love to see what like minded people think :)
> The one who wants to see a script for PIE discovered :)
I didn't quite understand. Is this related to what the article says:
"
The discovery of a civilization of people who lived before the Vedic people upended the story of India. Given that it undermines their claims of indigeneity, proponents of Hindutva — the most mainstream strain of Hindu nationalism — balk at the theory of a pre-Vedic civilization, even as evidence for it accumulates across disciplines, including archaeology, genetics, and linguistics.
"
I don't know how Hindutva is related to this discussion.
I'm just interested in this from a coherence point of view. I believe that learning a super language like PIE would help me understand a lot about human relation to language and vice versa.
The article mentioned Hindutva as I pasted. If I understand correctly, Hindutva people have a vested interest to want this script to be Sanskrit related as opposed to an indigenous Indian language. Is that correct?
Hindutva is a political ideology; Sanskrit and PIE (Proto-Indo-European) were languages, with Sanskrit attested in written form (including an ancient grammar!), but PIE not attested--only reconstructed from the daughter Indo-European languages, Sanskrit being one of them. Sanskrit being rather old, it's closer to PIE than any of today's IE languages; it's very roughly contemporaneous with ancient Greek and Hittite, which are also IE languages.
Anything IE is extremely unlikely as a language for the Indus seals; it only really fits with an out of India model of Indo-European, which is outright rejected by all serious scholars AFAIK. If we're going to look for contemporary relatives of the Indus language, the serious candidate is Dravidian. The Dravidian family was in India around the right time, and was definitely spoken in a larger area prior to the Aryan invasions.
I don’t think the Indus Valley seals are a script at all. They might be more like corporate logos.
Try deciphering the english language from a bunch of billboards and pop/cans. Even if you figure out the alphabet there aren’t enough words to construct a single sentence.
All the AI in the universe won’t help when the problem is lack of real data.
Well, names and titles are still language, but I agree with your actual point: the Indus inscriptions seem to be too short and probably not general enough to allow an in-depth understanding without finding new, longer texts.
One way to think of this: have the computer learn how to translate between any two languages, i.e., learn how to "build translators", and then attempt to translate between the unknown language and every other language.
In general, you could try many encodings roughly in order of algorithmic complexity.
Not sure if that's any use here.
The really big problem is lack of data. With sufficient amounts of data and the assumption that we are dealing with human writing, you can probably figure out the encoding.
The assumption is both that this hasn't been tried and that an AI would perform significantly better at this despite lacking any of the necessary training data.
> It is trivial to decode English (or any other alphabetical language) gone through a substitution cypher.
As far as I know, it's only trivial if the substitution is symbol to symbol. If you mess with the letter frequencies, it becomes a lot harder. If you, say, use an orthography that's more in line with the phonology of language in its current form - or devise one that is as quirky, but in a different way. We are talking about a script with 700+ symbols, mathing it together with one that is written in 26 is nonsensical.
This is emphatically not a cryptographic problem. It would really suit ML'ers to learn to take of their shoes and listen a lot more than they speak when they walk into a new scientific discipline.
(BTW, I don't think many modern linguists would accept the term "alphabetic language"; a language is not defined by its writing system, and many written languages are routinely used with multiple writing systems)
You will maybe be able to "figure it out" if you know English. The use of maths to compare languages hinges on the frequencies being comparable, which is as much a trait of the script used as it is of the language itself.
Along these lines: a script with over 700 symbols is definitely not an alphabetic script. Possibly syllabic, although most syllabic writing systems have fewer symbols than that.
We have no idea what's on those seals. If it's even a script, it might be personal names. The inscriptions are all very short. We have no idea how the script is structured, if it's syllabic or what. We have absolutely no idea what language it's written in.
There is nothing to latch on to, no way to anchor the findings in the real world. The problem with applying AI to this is that it will give an answer, but there's absolutely no way to verify it's a reasonable one.
If you can build a network, that can crack Enigma after being trained on German and preceding encryption methods, then you might have made a step towards this. But note that cracking enigma "only" took a few people a few months, that these scripts have eluded many people for a much longer time, and that the source language of Enigma was known, as was its script.
I get your general point, but the Enigma (or any encryption) is a pretty bad example.
Encrypted text 'wants' to be obscure, even to those who know the system but lack the key. Most writing systems 'want' to be decipherable to people in the know.
In addition, Enigma has a fairly short key length by modern standards, and lots of accidental flaws that make it even easier to crack.
(Eg an automated system that knows a lot of German _might_ be able to figure out that the enigma never encrypts a letter to itself, even without seeing any schematics.)
"the enigma never encrypts a letter to itself": I recall that the clue to one day's cypher was a longish document that contained every letter except 'L'. The analyst guessed that it was produced by somebody running a test who kept typing the letter 'L'--she was right.
"Insights"? When you break an encryption scheme, you get a readable text in a language you understand.
The thing is, the written word as an artifact does not itself contain its meaning. To go from writing to meaning is what's called knowing language. The context needed to do that is in the mind of the reader, it's not in the text. Writing something down is quite simply not the same as encrypting it.
What we have here is a few thousand small inscriptions of less than ten characters each. We assume - but don't know for sure - that the characters correspond to sounds, and, if they do, what kind of sounds - syllables, phonemes, maybe a mixture. And even if you could go to sound, then, in order to understand anything, you have to go from sound to meaning, if indeed there is meaning. Which requires knowing the language. And not only do we not know the language, we don't even know what language it is! It might be related to a language of the region, it might not.
I did not have to flesh out my reply, what I meant was in an information theoretic sense. We have methods such as differential cryptanalysis that can operate with very few "hints" to start.
This is not a cryptographic problem. The relationship between ciphertext and plaintext is nothing at all like the relationship between a text and its meaning or translation.
Welcome to linguistics! That happens all the time and human linguists are trained to do exactly the same thing.
At one point in college, I was frustrated with an assignment and said “it could either be this or that but I need another datapoint to disambiguate it”, and someone suggested I look at the back of the page.
I've been trying to run away from bad computational linguistics for ten years after making a wrong turn in college. Wish I'd stuck to CxG!
Sometimes I actually think people post these kinds of articles here to ruin my mornings. Did you see the one on the Voynich manuscript where they'd decided it was in Hebrew and done a bunch of maths to it and then, when the person they asked who actually knew Hebrew said "this doesn't make sense", they'd put it into Google Translate and accepted the spell checker's suggestion and then put that as the conclusion of their paper? Fun times.
Cool, it can decipher Linear B, like humans, because there's a bunch of data points for Linear B (IIRC around 6.5K inscriptions in the corpus) and because it's known that it's written in Mycenaean Greek, and Ancient Greek gives us a place to start working backwards from.
But then there's Linear A. Only has 1500 inscriptions or so. And no translation.
So, Harappan/Indus script has about 4K known inscriptions. And descendant languages to work backwards from are still unknown.
Well, I wish them good luck. But I don't know if ML can do any better than humans. It might be able to do what humans can faster, but I doubt it can do what we can't.
That said, I would love to be proven wrong, because I love the idea of the Indus valley civilisation being where "all the wise men in boats" (to paraphrase Pratchett) came from, given the obvious proximity of the Indus Valley to the Persian Gulf and thus Mesopotamia.
Bonus points if someone translates a Harappan inscription and it says "By the way, if some guy called Noah turns up in a boat, claiming to have been spared from a global flood, he's a bit nutty, it was just a heavy monsoon."
> Well, I wish them good luck. But I don't know if ML can do any better than humans. It might be able to do what humans can faster, but I doubt it can do what we can't.
Being faster might be enough? Who knows, perhaps some human spending a few centuries in deep contemplation of Linear A might crack it?
Humans have already spent combined person-centuries trying to crack Linear A, and been able to guess a few words from context. If you use the sound values from Linear B (which can be guessed via Greek), you can even try to pronounce them out loud. Unfortunately, that doesn't help at all unless you can find a related language to help guess more words. If it's a language isolate without living relatives, that would be simply impossible.
However, that doesn't mean that trying to solve it with ML is completely useless. If you can do a century's worth of hypothesis-testing in a fraction of the time to discover that none of them work, at least humans will waste less time trying to solve an unsolvable problem.
It's not about deep contemplation, it's about having the necessary information.
If I write down a sentence in a language you don't speak, and make it impossible for you to ever get any information at all about the language it's written in (say, by burning every book about it and killing everyone who knows it), how many centuries of contemplation do you think it would take for your to understand it?
Language decipherment, whether by human or machine, is not done by staring at the words you don't understand. It's done by comparing it to things you do. And here we have nothing.
I don't understand how you could say that it ever would. If you don't have anything at all (a related language, partial translations, accompanying illustrations, anything) to compare to, what can you do? You're trying to extract something (meaning) from an object that just does not contain it (text).
Theoretically, given huge amounts of text, you can try to match it with possible human communication, based on the experience of being human - but you would likely need huge amounts of guesses, so it's unlikely to be truly doable with plausible computation resources.
GPT-3 does not understand language like you and I do with reference to the world. It has no senses, so it is not possible for it to do so. It is also to my knowledge not able to translate anything, so even if we were able to make it "learn" Harrappan writing from these seals, it would not be able to generate English text from them, because the information required to do so does not exist.
To reply to your other comment, rabbit and rocks would occur in different contexts, but you have no way of knowing which is which. You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating.
> You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating.
But you have a grounding! You know it's a language used by humans.
> It is also to my knowledge not able to translate anything
It translates very well actually. I've tried with English/French and English/German with very good results.
I think it is quite well understood that deciphering a language without external information is possible? But GPT-3 was a recent example that I think proves it. There are probably tons of literature about this question. I think aliens would be able to understand lots of things about Earth just by listening in.
> To reply to your other comment, rabbit and rocks would occur in different contexts, but you have no way of knowing which is which. You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating.
Well, GPT-3 translates very well, so this argument can not be correct. I don't see why it would be like that? It seems quite intuitive that it would be possible to extract meaning.
GPT-3 trained on only English can produce French text? Citation please! I've looked for one, but I can't find it. I can, however, find more than a little information suggesting that GPT-3 is trained on multilingual data.
> I think it is quite well understood that deciphering a language without external information is possible?
I don't think this is understood at all, and I would have to ask you for just one example of it. I would, in fact, say it is categorically impossible, if by "no external information" you mean that we have only text or sound.
The reason that it is not possible to extract meaning is that meaning is a substance that is not present in the text. It is more than the relations between words, in that it is connected to the world.
I am looking more into the exact nature of the multilingual data used by GPT-3 and it's a little hard. The main paper mostly focuses on the algorithms which is less interesting to me.
The key is, if you have aligned or parallel texts, then you can use structural similarities in the lexicon to bootstrap a mapping from one language to another without referring to substantive meaning (which I will maintain is how humans do translation), even if you don't have a dictionary as such. This is not surprising, but I would expect it to have severe limits.
Not sure how? As a thought experiment, say I created two languages completely from scratch : symbols, words, sentence order, etc.
One of these languages, I've given meaning to the words, the other is random. However I create massive corpuses for each of hundreds of thousands of documents. Now go ahead and machine learn the hell out of both languages.
First question : which one is the 'random' language? Second question : what is the meaning of the non-random language texts? Assume I have not based my 'real' language on any existing example ...
If the machine or human is sufficiently intelligent and there is enough data and time, it is possible to do so as long as the non-random language is useful for human communication and the language sample describes some experiences or things we understand.
There are principles underlying all human languages. For example, they are used to describe the world or communicate intention and they must be understandable by a human.
Given that the copious data of the non-random language should describe something we understand about the world or social dynamics, we can try to match them with world/domain models and find patterns of plausible interpretations that fit.
The patterns should also satisfy the principle of communication economy and the limits of human cognition, such as the size of working memory. These constraints help narrow down the possibilities.
The human/AI system can test and figure out the most likely patterns first and use them to decipher more patterns.
All above is possible in principle but beyond the current state-of-the-art.
This is true, but you're forgetting that the sign (ie. pairing of an expression and its meaning) is arbitrary. There's no particular reason that the sound 'cow' (or the squiggles c-o-w when we write) refers to cows, and you can see that quite clearly by the fact that it only does so in one out of the Earth's 6-7000 current languages.
Note that this doesn't mean that relationships between words are arbitrary, they are of course highly systematic, else language wouldn't work. It also doesn't mean that language doesn't have history, the changes are also systematic. But if you take a random surface form (sound or writing), there is absolutely no way to relate it with any certainty to any content without a context that allows you to infer it.
Yes, it's hard to figure out cow just from a big corpus; though you might learn enough about 'cow' to form meaningful sentences and answer questions. That's what GPT-3 is doing after all, and language models are still improving quickly.
And we do have context! Just knowing that a language is used by humans gives lots and lots of context. Human cultures and languages have more in common than you might realize.
> In linguistics, deixis (/ˈdaɪksɪs/, /ˈdeɪksɪs/)[1] is the use of general words and phrases to refer to a specific time, place, or person in context, e.g., the words tomorrow, there, and they. Words are deictic if their semantic meaning is fixed but their denoted meaning varies depending on time and/or place. Words or phrases that require contextual information to be fully understood—for example, English pronouns—are deictic. Deixis is closely related to anaphora. Although this article deals primarily with deixis in spoken language, the concept is sometimes applied to written language, gestures, and communication media as well. In linguistic anthropology, deixis is treated as a particular subclass of the more general semiotic phenomenon of indexicality, a sign "pointing to" some aspect of its context of occurrence.
> Although this article draws examples primarily from English, deixis is believed to be a feature (to some degree) of all natural languages.
I suspect with a large enough corpus you can most likely figure out the deixis of the given language.
> However, all languages have some sort of a clause-type thing allowing them to express predication, attribution, etc. See Dixon's Basic Linguistic Theory: Basic Linguistic Theory Volume 1: Methodology . All languages also must have means of expressing cohesion and coherence (texture) although this is much less studied in cross linguistic perspective. Punctuated sentences are a kind of cohesive device.
(A comment on this answer points out that the reality is probabilistic, of course:)
> While I agree with you, it seems to me that linguists who study languages with a strong written tradition often talk about 'sentences', even when their examples are not from writing. Those of us who work on previously unwritten languages (I think) tend to talk about 'texts' or 'utterances'. I think this is an important issue as it relates to how some linguists have ignored the true messiness that is often found in examples of human language.
Of course, all of this refers to natural human languages. Not totally arbitrary constructs.
> Yes, it's hard to figure out cow just from a big corpus; though you might learn enough about 'cow' to form meaningful sentences and answer questions. That's what GPT-3 is doing after all, and language models are still improving quickly.
As I've said in a sibling comment, GPT-3 is able to construct new utterances in the same language that it learns from. It does not learn its language while learning a model of the world because it does not exist in the world, so it does not learn semantics like we do, which is requirement for making a translation. I am familiar with all these things you talk about, but you are not starting from a sound understanding of semiotics and the difference between content and expression.
Say you learn a lot of syntax from text, you are able to write out a list of paradigms. How do you know which verb form is past and which form is present? How do you know which word means day and which word means night? The information required to make that judgement is not present in the text artifact.
> How do you know which verb form is past and which form is present?
There's lots and lots of context. One interpretation will fit the data much better than another.
Eg in spoken languages the equivalent of 'I' is used much more often than any other pronoun. Similarly simpler sentences are perhaps more likely to refer to the present than the past; eg past tense is much more likely to come with an additional specification of _when_ stuff happened. That information is less kind of a given when we are talking about _now_.
I agree that all these connections are only probabilistic and you need enormous amounts of data.
Btw, GTP-3 can translate between English and French to certain very limited extent; but that's probably mostly because it has seen example translations.
To give you a testable prediction of my theory: if you trained something like GPT-3 on a corpus that includes examples of natural languages A and B, but never any piece of text that contains both languages, yet alone example translations, I would expect that nevertheless, the resulting network would share the same internal representation for the concept of 'cow' in both languages.
> Eg in spoken languages the equivalent of 'I' is used much more often than any other pronoun. Similarly simpler sentences are perhaps more likely to refer to the present than the past; eg past tense is much more likely to come with an additional specification of _when_ stuff happened. That information is less kind of a given when we are talking about _now_.
This still rests on a lot of assumptions about the language. Pronouns are used much more in some languages than others, some languages have more pronouns than others, etc.. Tense is not at all universal in the world's languages as you probably know. There are linguistic theories that even question the universality of nouns and verbs as categories, and even if they are to an extent universal, they are certainly not fixed in their boundaries across languages, e.g. most things that in English adjectives are expressed as verbs in Classical Arabic. Even if you could do a perfect job of extracting a set of syntactic categories and morphological paradigms from a language, I don't think you would be able to do anything with it. Word 3849 from category C often combines with words 201, 635, and 9913 from category F in conjugations 1-4, but never 5.
And of course, even for nouns that refer to basic, physical things, often some languages will have several words where another will make do with one, or have none. It quickly becomes a question of culture.
I don't think it would be very hard to detect which of the two is random, because meanings are likely to be somewhat correlated. Obviously you could model your 'random' language to be statistically similar to the meaningful one though, if you were feeling particularly evil, but then you'd be imbuing it with some small degree of meaning, arguably.
Whether you can go from this statistical analysis to anything with absolute meaning, I don't know. I suspect _some_ extremely common concepts (pronouns, verbs like to be or to have) might yield to this sort of direction.
But the meaning is not in the text (text-as-artifact, the ink on the page), it is strictly in the mind of the reader, accessed by reference to the speaker's experience with the language in question. If you don't have something to ground at least part of the symbols in, there is nothing to work from. Any word can be assigned any meaning, and there would be infinite possibilities. No amount of statistics will allow you to work out whether a given word or part of a word refers to a rock, a rabbit or Noam Chomsky's granddaughter. How could it? To be able to do this is called knowing a language, and if all the people who had that skill in the soft tissue of their brain have kicked the bucket, tough luck, you can't get that knowledge back.
All (bar none) decipherments of unknown writing systems have been done by reference to related languages, especially the existence of bilingual texts. Computers and statistics are definitely able to help us do that, but if all you have is a page of text with no context, then the information you're looking for is gone.
I agree to a very high degree! But I still think that you have greater than zero chance (but not much greater) to make some correct guesses about some very, very basic grammatical forms based on frequencies within other languages. You'd never be able to confirm this, and it's not a very important point.
However, I do strongly disagree with the other point in the comment I was responding to, that you could not tell a real language with meaningful words from one with random words. That (entirely made up and not very helpful) problem does seem like it would be amenable to basis statistical analysis.
I maintain that it is possible. A rock and a rabbit would occur in different contexts. GPT-3 solves this problem in a limited but impressive way, as I mentioned in a comment below just now.
Actually, you can get a readout on lexical semantics (the meaning of words) if you have a large enough corpus (much larger than we have for any extinct language). The method uses a vector space of word cooccurrence (look up word2vec). It will tell you what words are semantically similar. What it won't tell you is what those words mean in the real world (unless possibly you have pictures with captions). It's like being able to see mountains in one direction and a lake in another and a valley in another direction; you can say you're going towards one or the other of these features, but (ignoring the sun and stars) you don't know whether that's north, south, east or west.
This is... basically what I'm saying? How will you make a translation without knowing what the words mean in the real world? You can get a vector that says 'oiufd' is to 'flarewq' as 'rye8poq' is to 'nmjrewq'. I don't really think that counts as anything like deciphering the mysteries of Harrappa.
The first question is--if I understand what you're proposing--(relatively) easy. The second is much harder, perhaps impossible.
The reason the first question is relatively easy is because of syntax (sentence order, but also the order of words in phrases). If it's a real language, that order is not random; the English word "the" shows up before "man" much more often than it does before "listens", and so for other words. This is true even in so-called free word order languages, which are really free phrase order languages.
The other thing that could make the first question relatively easy is morphology. If you've decided to make a language that has inflectional morphology (in English, suffixes like -ed and -ing), that also helps determine whether the language is "real", because in a real language not all words will take the same affixes--you can't say 'theseing' (these-ing) in English. In other words, the affixes that you find on a given word are not random.
Of course if you decide not to give your language inflectional morphology, this won't help--but if you do that, then your language's syntax will have to be more rigid: in a language with case marking affixes, like Latin, you don't need word order to tell which noun is subject and which (if any) is object; but you need a more or less fixed word order in English, because (apart from pronouns) we don't have any case marking.
deep contemplation combined with the lack of information may lead to completely wrong conclusion. example is medieval interpretation of Egyptian hieroglyphics - completely and laughably wrong, but coherent and making sense, just based on limited amount of inputs.
Re Linear B, it's also worth pointing out that prior to its decipherment the prevailing view was that it wasn't Greek; Ventris spent most of his time trying to align it to Etruscan, and only made the breakthrough when he tried out a few Greek words more or less on a whim. I imagine that had almost any form of computerised automation been readily available in the 50s it would have made the decipherment vastly simpler, but it would require an existing hypothesis of Greek; an AI trying to fit it to Etruscan would have been merely banging its head against an electronic wall.
I believe that Hoshi Sato developed models for real-time translation of spoken language using very limited data samples without translations. It might be an application of the research mentioned here. The timeframes fit.
Huh. The finding that Iberian is not a close language to Basque is surprising, but then again the Romans did not put the two of them as one nation- like they did with Aquitanians (Southwest France today) and Basque territory hmm.
Pilus, pirus, pilis. This is intriguing. But one word doesn't signify the membership of a language in another language family. Is it possible that Tamil itself adopted the IVC term for elephants? Of course, since in all likelihood proto-Tamil speakers absorbed many refugees from IVC, there should be many terms shared between the two languages, even if they're from different language families
Still interesting to speculate.
It reminds me of the first great decipherment, that of hieroglyphic Egyptian. Young's quantitative approach gave a few insights that proved useful later on, but that approach was ultimately limited and wasn't leading anywhere. It wasn't until Champollion brought Coptic to the table that it was unlocked.
I was going to comment on the Coptic connection; Hieroglyphs would have remained a mystery due to their basically vowel-less structure; it was an interpreted language[0]. I wonder to what extent this AI could crack that connection, since we have a more or less complete knowledge of what it ought to produce.
Betteridge's law of headlines: "Any headline that ends in a question mark can be answered by the word no."
As a kid who could read Phoenician script (similar to ancient Hebrew), I tried to decipher the Ashmanezer tablet, which is written in a Semitic language and has many proper nouns. Despite that, there were different ways to interpret the inscription. Taught me just how amazing and difficult language is.
(In the Wikipedia page[1], there is one translation given as though it is empirical. It's not.)
This article is 3566 words. 15 pages long when copied onto libreoffice.
I like reading as much as the next person, but this kind of articles is starting to get ridiculous.
It's like, it starts with an interesting but otherwise straightforward title, you click, and then you're presented with a novel. One structured in such a way so as to make trying to glance to find the relevant sections yields zero information. Nope, you are required to read all about how someone grew without a dog and how their grandmother had lovely spots on her hands and her skin was like paper, before you can even get an inkling where the information lies in the article, if it's there at all.
I've read countless articles like this by now, where you go through the whole 'novel', and the information you were looking for is on paragraph 67 out of 103, and it's basically the line "Well, we don't really know."
(ノಠ益ಠ)ノ彡┻━┻
Here, let me tell you about the story when I once added 1 + 1 together. You'll never believe the result:
"Jonathan was a keen hiker. When his wife got pregnant in the mid nineties, he could never imagine that his great-great-great-great-great grandson would ever be faced with the 1+1 problem [...]"
> I've read countless articles like this by now, where you go through the whole 'novel', and the information you were looking for is on paragraph 67 out of 103, and it's basically the line "Well, we don't really know."
That's basically the outcome here. It talks briefly about a few different approaches and doesn't really touch on the likelihood of success.
I just grepped for " AI" and read the few paragraphs that popped up. Although looking for "algorithm" proved slightly more interesting:
> For now, the mysteries of the Indus script continue to elude decipherment. Last year, in a follow-up paper to their work automating the decoding of Ugaritic and Linear B, Luo and his team made a small but crucial advance: an algorithm aimed at identifying possible related languages of undeciphered writing systems. Potentially, this could help address the problem of deciphering scripts that don’t yet have a known language they can be compared against. When Luo and his team tested their model on the Iberian language, which has historically been linked to Basque, their findings suggested the two languages were not in fact close enough to be related — a conclusion that corroborated recent scholarship on the matter.
I'd want to see this applied to dolphin clicks. Not sure it would get anywhere, but if an existing human language turns out to have similar grammatical and linguistic constructs, it might get the ball rolling on communicating.
In addition to being a programmer, I have a degree in linguistics and I can read a handful of scripts, and honestly, I don't understand how this "algorithm" could possibly work. When you have two languages written in different alphabets, what are you actually comparing? Even if the underlying languages might be the same, the writing systems can be completely different. And in fact, it seems obvious that a writing system with 80 symbols is not comparable to one with 400.
Honestly, the thing I'd want to see this applied to is something like Chinese in Chinese script vs. the same texts in transliteration.
ML over historical languages is no different from ML over currently spoken (and written) languages, with one crucial exception: if the language is still in use, you can get a bigger corpus.
About dictionary making: there are several programs "out there" that are already set up for making dictionaries, probably better than going from scratch with YAML. SIL's FLEx is very good; their older Shoebox program is also useable, but like the C programming language, you can shoot yourself in the foot. There's another one out of East Africa, but I can't remember its name.
I don't know much about Australian languages, but I think most of them have a lot of morphology. If yours does, then you'll want to do something about inflected forms of words (think English walk, walks, walked, walking); you probably don't want to store all of those forms in your dictionary, instead you'll want to choose a base form from which the others can be derived. Again, tools like FLEx allow you to create morphological parsers that help with that.
Finally, I'd suggest that if you haven't already done so, you should take some courses in linguistics. It will make it much easier to reason about your lexicon, the language's grammar (syntax and morphology), and so forth.
Has anybody come up with AI tactics specifically aimed at simply generating new grammatical sentences from a single-language corpus? Meaning that the results aren't judged by meaning or coherence, but by adherence to a grammar, with an attempt to maximize variety?
The goal would be to do that well with languages where we know the grammar and can evaluate the results, and through that find a general tactic. Maybe find a few general theories on how much input that takes, and the character of that input.
If you can extract the grammatical rules with reasonable certainty (not likely with a collection of seals, of course) figuring out the content would just be Mad Libs.
> Has anybody come up with AI tactics specifically aimed at simply generating new grammatical sentences from a single-language corpus? Meaning that the results aren't judged by meaning or coherence, but by adherence to a grammar, with an attempt to maximize variety?
Just training an LSTM deep network on the corpus is enough to do that. GPT-3 is much the same, with a lot more training data and a less computationally intensive network architecture than LSTM, to make that huge amount of training data useful.
A language model approach will learn grammatical/syntactic and semantic patterns at the same time. So "dog walks man" will be less likely than "man walks dog", even though both are valid, grammatically. I think what op was asking for was a mechanism to purely learn grammar, which is more in the realm of GOFAI/old school NLP.
102 comments
[ 5.8 ms ] story [ 903 ms ] threadThe optimist in me thinks this is the written script for an early version of Proto-Indo-European. Would love to see what like minded people think :)
Meaning a scholar of Sanskrit or something else?
> The optimist in me thinks this is the written script for an early version of Proto-Indo-European
Optimist in what sense?
> Optimist in what sense The one who wants to see a script for PIE discovered :)
I didn't quite understand. Is this related to what the article says: " The discovery of a civilization of people who lived before the Vedic people upended the story of India. Given that it undermines their claims of indigeneity, proponents of Hindutva — the most mainstream strain of Hindu nationalism — balk at the theory of a pre-Vedic civilization, even as evidence for it accumulates across disciplines, including archaeology, genetics, and linguistics. "
Is Sanskrit and Hindutva and PIE related?
I'm just interested in this from a coherence point of view. I believe that learning a super language like PIE would help me understand a lot about human relation to language and vice versa.
Don't know, don't care about this 40 year old concept as much as I do for a 5000 year proto language.
Try deciphering the english language from a bunch of billboards and pop/cans. Even if you figure out the alphabet there aren’t enough words to construct a single sentence.
All the AI in the universe won’t help when the problem is lack of real data.
Not sure if that's any use here.
The really big problem is lack of data. With sufficient amounts of data and the assumption that we are dealing with human writing, you can probably figure out the encoding.
It is trivial to decode English (or any other alphabetical language) gone through a substitution cypher.
Now, of course, it's not that simple with older languages, but this is a good starting step.
As far as I know, it's only trivial if the substitution is symbol to symbol. If you mess with the letter frequencies, it becomes a lot harder. If you, say, use an orthography that's more in line with the phonology of language in its current form - or devise one that is as quirky, but in a different way. We are talking about a script with 700+ symbols, mathing it together with one that is written in 26 is nonsensical.
This is emphatically not a cryptographic problem. It would really suit ML'ers to learn to take of their shoes and listen a lot more than they speak when they walk into a new scientific discipline.
(BTW, I don't think many modern linguists would accept the term "alphabetic language"; a language is not defined by its writing system, and many written languages are routinely used with multiple writing systems)
Yes it will be more difficult if you play around and change the syntax, join letters, etc. But you usually can figure it out.
> "alphabetic language"; a language is not defined by its writing system
What I mean is a language not written with an abjad, or a syllabary. As in, the actual thing you're trying to decode.
And of course, it may not be a script at all.
We have no idea what's on those seals. If it's even a script, it might be personal names. The inscriptions are all very short. We have no idea how the script is structured, if it's syllabic or what. We have absolutely no idea what language it's written in.
There is nothing to latch on to, no way to anchor the findings in the real world. The problem with applying AI to this is that it will give an answer, but there's absolutely no way to verify it's a reasonable one.
Encrypted text 'wants' to be obscure, even to those who know the system but lack the key. Most writing systems 'want' to be decipherable to people in the know.
In addition, Enigma has a fairly short key length by modern standards, and lots of accidental flaws that make it even easier to crack.
(Eg an automated system that knows a lot of German _might_ be able to figure out that the enigma never encrypts a letter to itself, even without seeing any schematics.)
The thing is, the written word as an artifact does not itself contain its meaning. To go from writing to meaning is what's called knowing language. The context needed to do that is in the mind of the reader, it's not in the text. Writing something down is quite simply not the same as encrypting it.
What we have here is a few thousand small inscriptions of less than ten characters each. We assume - but don't know for sure - that the characters correspond to sounds, and, if they do, what kind of sounds - syllables, phonemes, maybe a mixture. And even if you could go to sound, then, in order to understand anything, you have to go from sound to meaning, if indeed there is meaning. Which requires knowing the language. And not only do we not know the language, we don't even know what language it is! It might be related to a language of the region, it might not.
Modern cryptography is really good. Without quantum computers, it's unlikely a 16384-bit RSA key will be broken in your lifetime by a motivated actor.
Well, we have a few models of English (large language models like BERT and friends, or the GPT's etc). What insights have they derived about English?
What insights have we derived about English from such models?
Edit: nice paper to read before forming expectations about insights deriveable from language modelling.
Climbing towards NLU: On Meaning, Form and Understanding in the Age of Data.
https://aclanthology.org/2020.acl-main.463.pdf
https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
At one point in college, I was frustrated with an assignment and said “it could either be this or that but I need another datapoint to disambiguate it”, and someone suggested I look at the back of the page.
I've been trying to run away from bad computational linguistics for ten years after making a wrong turn in college. Wish I'd stuck to CxG!
Sometimes I actually think people post these kinds of articles here to ruin my mornings. Did you see the one on the Voynich manuscript where they'd decided it was in Hebrew and done a bunch of maths to it and then, when the person they asked who actually knew Hebrew said "this doesn't make sense", they'd put it into Google Translate and accepted the spell checker's suggestion and then put that as the conclusion of their paper? Fun times.
But then there's Linear A. Only has 1500 inscriptions or so. And no translation.
So, Harappan/Indus script has about 4K known inscriptions. And descendant languages to work backwards from are still unknown.
Well, I wish them good luck. But I don't know if ML can do any better than humans. It might be able to do what humans can faster, but I doubt it can do what we can't.
That said, I would love to be proven wrong, because I love the idea of the Indus valley civilisation being where "all the wise men in boats" (to paraphrase Pratchett) came from, given the obvious proximity of the Indus Valley to the Persian Gulf and thus Mesopotamia.
Bonus points if someone translates a Harappan inscription and it says "By the way, if some guy called Noah turns up in a boat, claiming to have been spared from a global flood, he's a bit nutty, it was just a heavy monsoon."
Being faster might be enough? Who knows, perhaps some human spending a few centuries in deep contemplation of Linear A might crack it?
However, that doesn't mean that trying to solve it with ML is completely useless. If you can do a century's worth of hypothesis-testing in a fraction of the time to discover that none of them work, at least humans will waste less time trying to solve an unsolvable problem.
If I write down a sentence in a language you don't speak, and make it impossible for you to ever get any information at all about the language it's written in (say, by burning every book about it and killing everyone who knows it), how many centuries of contemplation do you think it would take for your to understand it?
Language decipherment, whether by human or machine, is not done by staring at the words you don't understand. It's done by comparing it to things you do. And here we have nothing.
To reply to your other comment, rabbit and rocks would occur in different contexts, but you have no way of knowing which is which. You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating.
But you have a grounding! You know it's a language used by humans.
It translates very well actually. I've tried with English/French and English/German with very good results.
I think it is quite well understood that deciphering a language without external information is possible? But GPT-3 was a recent example that I think proves it. There are probably tons of literature about this question. I think aliens would be able to understand lots of things about Earth just by listening in.
> To reply to your other comment, rabbit and rocks would occur in different contexts, but you have no way of knowing which is which. You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating.
Well, GPT-3 translates very well, so this argument can not be correct. I don't see why it would be like that? It seems quite intuitive that it would be possible to extract meaning.
> I think it is quite well understood that deciphering a language without external information is possible?
I don't think this is understood at all, and I would have to ask you for just one example of it. I would, in fact, say it is categorically impossible, if by "no external information" you mean that we have only text or sound.
The reason that it is not possible to extract meaning is that meaning is a substance that is not present in the text. It is more than the relations between words, in that it is connected to the world.
The key is, if you have aligned or parallel texts, then you can use structural similarities in the lexicon to bootstrap a mapping from one language to another without referring to substantive meaning (which I will maintain is how humans do translation), even if you don't have a dictionary as such. This is not surprising, but I would expect it to have severe limits.
One of these languages, I've given meaning to the words, the other is random. However I create massive corpuses for each of hundreds of thousands of documents. Now go ahead and machine learn the hell out of both languages.
First question : which one is the 'random' language? Second question : what is the meaning of the non-random language texts? Assume I have not based my 'real' language on any existing example ...
There are principles underlying all human languages. For example, they are used to describe the world or communicate intention and they must be understandable by a human.
Given that the copious data of the non-random language should describe something we understand about the world or social dynamics, we can try to match them with world/domain models and find patterns of plausible interpretations that fit.
The patterns should also satisfy the principle of communication economy and the limits of human cognition, such as the size of working memory. These constraints help narrow down the possibilities.
The human/AI system can test and figure out the most likely patterns first and use them to decipher more patterns.
All above is possible in principle but beyond the current state-of-the-art.
Note that this doesn't mean that relationships between words are arbitrary, they are of course highly systematic, else language wouldn't work. It also doesn't mean that language doesn't have history, the changes are also systematic. But if you take a random surface form (sound or writing), there is absolutely no way to relate it with any certainty to any content without a context that allows you to infer it.
Yes, it's hard to figure out cow just from a big corpus; though you might learn enough about 'cow' to form meaningful sentences and answer questions. That's what GPT-3 is doing after all, and language models are still improving quickly.
And we do have context! Just knowing that a language is used by humans gives lots and lots of context. Human cultures and languages have more in common than you might realize.
For example, have a look at Deixis (https://en.wikipedia.org/wiki/Deixis):
> In linguistics, deixis (/ˈdaɪksɪs/, /ˈdeɪksɪs/)[1] is the use of general words and phrases to refer to a specific time, place, or person in context, e.g., the words tomorrow, there, and they. Words are deictic if their semantic meaning is fixed but their denoted meaning varies depending on time and/or place. Words or phrases that require contextual information to be fully understood—for example, English pronouns—are deictic. Deixis is closely related to anaphora. Although this article deals primarily with deixis in spoken language, the concept is sometimes applied to written language, gestures, and communication media as well. In linguistic anthropology, deixis is treated as a particular subclass of the more general semiotic phenomenon of indexicality, a sign "pointing to" some aspect of its context of occurrence.
> Although this article draws examples primarily from English, deixis is believed to be a feature (to some degree) of all natural languages.
I suspect with a large enough corpus you can most likely figure out the deixis of the given language.
See https://linguistics.stackexchange.com/a/4046 for some more structure shared with all human languages:
> However, all languages have some sort of a clause-type thing allowing them to express predication, attribution, etc. See Dixon's Basic Linguistic Theory: Basic Linguistic Theory Volume 1: Methodology . All languages also must have means of expressing cohesion and coherence (texture) although this is much less studied in cross linguistic perspective. Punctuated sentences are a kind of cohesive device.
(A comment on this answer points out that the reality is probabilistic, of course:)
> While I agree with you, it seems to me that linguists who study languages with a strong written tradition often talk about 'sentences', even when their examples are not from writing. Those of us who work on previously unwritten languages (I think) tend to talk about 'texts' or 'utterances'. I think this is an important issue as it relates to how some linguists have ignored the true messiness that is often found in examples of human language.
Of course, all of this refers to natural human languages. Not totally arbitrary constructs.
As I've said in a sibling comment, GPT-3 is able to construct new utterances in the same language that it learns from. It does not learn its language while learning a model of the world because it does not exist in the world, so it does not learn semantics like we do, which is requirement for making a translation. I am familiar with all these things you talk about, but you are not starting from a sound understanding of semiotics and the difference between content and expression.
Say you learn a lot of syntax from text, you are able to write out a list of paradigms. How do you know which verb form is past and which form is present? How do you know which word means day and which word means night? The information required to make that judgement is not present in the text artifact.
There's lots and lots of context. One interpretation will fit the data much better than another.
Eg in spoken languages the equivalent of 'I' is used much more often than any other pronoun. Similarly simpler sentences are perhaps more likely to refer to the present than the past; eg past tense is much more likely to come with an additional specification of _when_ stuff happened. That information is less kind of a given when we are talking about _now_.
I agree that all these connections are only probabilistic and you need enormous amounts of data.
Btw, GTP-3 can translate between English and French to certain very limited extent; but that's probably mostly because it has seen example translations.
To give you a testable prediction of my theory: if you trained something like GPT-3 on a corpus that includes examples of natural languages A and B, but never any piece of text that contains both languages, yet alone example translations, I would expect that nevertheless, the resulting network would share the same internal representation for the concept of 'cow' in both languages.
This still rests on a lot of assumptions about the language. Pronouns are used much more in some languages than others, some languages have more pronouns than others, etc.. Tense is not at all universal in the world's languages as you probably know. There are linguistic theories that even question the universality of nouns and verbs as categories, and even if they are to an extent universal, they are certainly not fixed in their boundaries across languages, e.g. most things that in English adjectives are expressed as verbs in Classical Arabic. Even if you could do a perfect job of extracting a set of syntactic categories and morphological paradigms from a language, I don't think you would be able to do anything with it. Word 3849 from category C often combines with words 201, 635, and 9913 from category F in conjugations 1-4, but never 5.
And of course, even for nouns that refer to basic, physical things, often some languages will have several words where another will make do with one, or have none. It quickly becomes a question of culture.
Whether you can go from this statistical analysis to anything with absolute meaning, I don't know. I suspect _some_ extremely common concepts (pronouns, verbs like to be or to have) might yield to this sort of direction.
All (bar none) decipherments of unknown writing systems have been done by reference to related languages, especially the existence of bilingual texts. Computers and statistics are definitely able to help us do that, but if all you have is a page of text with no context, then the information you're looking for is gone.
However, I do strongly disagree with the other point in the comment I was responding to, that you could not tell a real language with meaningful words from one with random words. That (entirely made up and not very helpful) problem does seem like it would be amenable to basis statistical analysis.
The reason the first question is relatively easy is because of syntax (sentence order, but also the order of words in phrases). If it's a real language, that order is not random; the English word "the" shows up before "man" much more often than it does before "listens", and so for other words. This is true even in so-called free word order languages, which are really free phrase order languages.
The other thing that could make the first question relatively easy is morphology. If you've decided to make a language that has inflectional morphology (in English, suffixes like -ed and -ing), that also helps determine whether the language is "real", because in a real language not all words will take the same affixes--you can't say 'theseing' (these-ing) in English. In other words, the affixes that you find on a given word are not random.
Of course if you decide not to give your language inflectional morphology, this won't help--but if you do that, then your language's syntax will have to be more rigid: in a language with case marking affixes, like Latin, you don't need word order to tell which noun is subject and which (if any) is object; but you need a more or less fixed word order in English, because (apart from pronouns) we don't have any case marking.
I was only elaborating the more general point that in practice merely being faster can let you do new things.
-- John Tukey
Pilus, pirus, pilis. This is intriguing. But one word doesn't signify the membership of a language in another language family. Is it possible that Tamil itself adopted the IVC term for elephants? Of course, since in all likelihood proto-Tamil speakers absorbed many refugees from IVC, there should be many terms shared between the two languages, even if they're from different language families Still interesting to speculate.
[0]https://www.youtube.com/watch?v=J-K5OjAkiEA
As a kid who could read Phoenician script (similar to ancient Hebrew), I tried to decipher the Ashmanezer tablet, which is written in a Semitic language and has many proper nouns. Despite that, there were different ways to interpret the inscription. Taught me just how amazing and difficult language is.
(In the Wikipedia page[1], there is one translation given as though it is empirical. It's not.)
[1]: https://en.wikipedia.org/wiki/Eshmunazar_II
I like reading as much as the next person, but this kind of articles is starting to get ridiculous.
It's like, it starts with an interesting but otherwise straightforward title, you click, and then you're presented with a novel. One structured in such a way so as to make trying to glance to find the relevant sections yields zero information. Nope, you are required to read all about how someone grew without a dog and how their grandmother had lovely spots on her hands and her skin was like paper, before you can even get an inkling where the information lies in the article, if it's there at all.
I've read countless articles like this by now, where you go through the whole 'novel', and the information you were looking for is on paragraph 67 out of 103, and it's basically the line "Well, we don't really know."
Here, let me tell you about the story when I once added 1 + 1 together. You'll never believe the result:"Jonathan was a keen hiker. When his wife got pregnant in the mid nineties, he could never imagine that his great-great-great-great-great grandson would ever be faced with the 1+1 problem [...]"
That's basically the outcome here. It talks briefly about a few different approaches and doesn't really touch on the likelihood of success.
I just grepped for " AI" and read the few paragraphs that popped up. Although looking for "algorithm" proved slightly more interesting:
> For now, the mysteries of the Indus script continue to elude decipherment. Last year, in a follow-up paper to their work automating the decoding of Ugaritic and Linear B, Luo and his team made a small but crucial advance: an algorithm aimed at identifying possible related languages of undeciphered writing systems. Potentially, this could help address the problem of deciphering scripts that don’t yet have a known language they can be compared against. When Luo and his team tested their model on the Iberian language, which has historically been linked to Basque, their findings suggested the two languages were not in fact close enough to be related — a conclusion that corroborated recent scholarship on the matter.
I'd want to see this applied to dolphin clicks. Not sure it would get anywhere, but if an existing human language turns out to have similar grammatical and linguistic constructs, it might get the ball rolling on communicating.
Honestly, the thing I'd want to see this applied to is something like Chinese in Chinese script vs. the same texts in transliteration.
When I'm finished, I wanted to play around with some ML exercises to see if I could derive anything useful in understanding it.
Besides this article does anyone know of any practical/simple attempts at running ML over historical languages?
About dictionary making: there are several programs "out there" that are already set up for making dictionaries, probably better than going from scratch with YAML. SIL's FLEx is very good; their older Shoebox program is also useable, but like the C programming language, you can shoot yourself in the foot. There's another one out of East Africa, but I can't remember its name.
I don't know much about Australian languages, but I think most of them have a lot of morphology. If yours does, then you'll want to do something about inflected forms of words (think English walk, walks, walked, walking); you probably don't want to store all of those forms in your dictionary, instead you'll want to choose a base form from which the others can be derived. Again, tools like FLEx allow you to create morphological parsers that help with that.
Finally, I'd suggest that if you haven't already done so, you should take some courses in linguistics. It will make it much easier to reason about your lexicon, the language's grammar (syntax and morphology), and so forth.
The goal would be to do that well with languages where we know the grammar and can evaluate the results, and through that find a general tactic. Maybe find a few general theories on how much input that takes, and the character of that input.
If you can extract the grammatical rules with reasonable certainty (not likely with a collection of seals, of course) figuring out the content would just be Mad Libs.
Just training an LSTM deep network on the corpus is enough to do that. GPT-3 is much the same, with a lot more training data and a less computationally intensive network architecture than LSTM, to make that huge amount of training data useful.