Definitely; I think it's originally from Friends (possibly in a slightly different form; Xey McXerson). Haven't tracked down the exact reference, but it's well over a decade old.
I wonder what's with that X McXface thing. Yes, I've heard something about Boaty McBoatface (whatever that is), but is the Boatface thing original, or is it some kind of obscure American meme?
It wasn't the original, and yes there is a sort of meme like it. Not always -face though. I've heard Beardy McBeardson, the pattern is usually X-y Mc-X-somthing.
:). It seems to have come from an Internet poll. Assuming everything on the (English-language parts of the) Internet is American is usually a safe heuristic ;).
echo 'Bob brought the pizza to Alice.' | syntaxnet/demo.sh
Input: Bob brought the pizza to Alice .
Parse:
brought VBD ROOT
+-- Bob NNP nsubj
+-- pizza NN dobj
| +-- the DT det
+-- to IN prep
| +-- Alice NNP pobj
+-- . . punct
Really, the mechanism of all these parsers, including SyntaxNet, is the same in that they use statistical training data to set up a neural network. Here's a paper on the Stanford CoreNLP parser, which you can compare with Google's paper: http://cs.stanford.edu/people/danqi/papers/emnlp2014.pdf
So, really all of the above parsers are weak in that they only output a single best parsing, when in reality sentences can have more than one valid structure, the principal example being the second sentence you've provided. I don't think Google's model has a better sense of humor than the others, no. I anticipate that they all have used relatively similar training data.
However, there is probably a trivial way to get the second sentence to parse as
Subject --- Verb --- Object
Noun Verb Article Noun
| \ | | |
Fruit flies like a banana .
and that is to provide training data with more occurrences of ...
The more sentences using simile that the parser finds, the less likely the neural net is to consider 'like' as a verb. It's also impacted by all of the uses of [flies like].
That's the nature of statistical language tools.
The stock parser debuted here gives the same answer as CoreNLP, by the way.
flies VBZ ROOT
+-- Fruit NNP nsubj
+-- like IN prep
| +-- banana NN pobj
| +-- a DT det
+-- . . punct
Evidence that this is the most accurate parser is here;
the previous approach mentioned is a March 2016 paper, "Globally Normalized Transition-Based Neural Networks," http://arxiv.org/abs/1603.06042
"On a standard benchmark consisting of randomly drawn English newswire sentences (the 20 year old Penn Treebank), Parsey McParseface recovers individual dependencies between words with over 94% accuracy, beating our own previous state-of-the-art results, which were already better than any previous approach."
From the original paper, "Our model achieves state-of-the-art accuracy
on all of these tasks, matching or outperforming
LSTMs while being significantly faster.
In particular for dependency parsing on the Wall
Street Journal we achieve the best-ever published
unlabeled attachment score of 94.41%."
This seems like a narrower standard than described, specifically being better at parsing the Penn Treebank than the best natural language parser for English on the Wall Street Journal.
The statistics listed on the project GitHub actually contradict these claims by showing the original March 2016 implementation has higher accuracy than Parsey McParseface.
Those are the part-of-speech tag accuracies. spaCy's accuracy on the PTB evaluation is 92.2% --- so it makes 20% more errors than P. McP. On the other hand, spaCy is about 200x faster.
I've been watching the line of research in SyntaxNet closely, and have been steadily working on replacing spaCy's averaged perceptron model with a neural network model. This is one of the main differences between spaCy and Parser McParseface.
The key advantage of the neural network is that it lets you take advantage of training on lots and lots more text, in a semi-supervised way. In a linear model, you grow extra parameters when you do this. The neural network stays the same size --- it just gets better. So, you can benefit from reading the whole web into the neural network. This only works a little bit in the linear model, and it makes the resulting model enormous.
Another difference is that spaCy is trained on whole documents, while P. McP. is trained in the standard set-up, using gold pre-processing. I speculate this will reduce the gap between the systems in a more realistic evlauation. Of course, P. McP can do the joint training too if they choose to. I've reached out to see whether they're interested in running the experiment: https://github.com/tensorflow/models/issues/65
spaCy doesn't use the GPU. Not sure what their speed is on GPU. I wouldn't be surprised if it's hard to use the GPU well for their parser, because minibatching gets complicated. Not sure.
> The statistics listed on the project GitHub actually contradict these claims by showing the original March 2016 implementation has higher accuracy than Parsey McParseface.
So you're referring to this LSTM?
"Andor et al. (2016)* is simply a SyntaxNet model with a larger beam and network. For futher information on the datasets, see that paper under the section "Treebank Union"."
After spending a few months hand coding a NLP parser, am rather intrigued by LSTM. I like the idea of finding coefficients, as opposed to juggling artificial labels.
Yes, my mistake. Their claim is that SyntaxNet (originally described in the paper and improved over one month) is the best in field, whereas Parsey McParseface is just one trained instance.
The paper you mention is the world's best results and is macparseface with broader beam search and more hidden layers.
This is an opensourcing of the March 2016 method (syntaxnet, note that in the paper there are results from several trained models) as well as a trained model that is comparable in performance but faster (macparseface).
It is very hard to separate those two things from the way they write.
better at parsing the Penn Treebank than the best
natural language parser for English on the Wall
Street Journal
I'm pretty sure "the 20 year old Penn Treebank" and "the Wall Street Journal" are referring to the same dataset here. In the early 1990s the first large treebanking efforts were on a corpus from the WSJ, and they were released as the Penn Treebank: https://catalog.ldc.upenn.edu/LDC95T7 People report results on this dataset because that's what the field has been testing on (and overfitting to) for decades.
Yes, the press release is (actually) pretty difficult to parse and really opaque in how the comparison is measured, which is why I wanted to throw into question the blog's headline, "The World's Most Accurate Parser." It seems more clear now but obviously Google doesn't feel the need to overtly prove that they are the best in the world at tasks, which is a bit questionable considering their number of followers. In all, it seems they have tested against several other dependency parsers, but clearly not all of them, and it's fair to say that it is "highly accurate," but this parser still falls victim to some of the same issues that most statistical parsers do, and while faster than some dependency parsers, it is not faster than all of them.
The point about overfitting is valid, too, which is another reason why this "most accurate such model in the world" claim is obnoxious.
It's also fair to note that their advance is in fractions of percentage points on this specific dataset over models that are 5-10 years older.
Coincidentally, I had a parent/teacher conference with my 1st grader's teacher yesterday afternoon. Regarding reading level & comprehension, she remarked that current research indicates anything below about 98% comprehension isn't sufficient for reading "fluency". Before the past few years, the standard was 95% comprehension = fluency, but that extra few percentage points apparently make an enormous difference (probably because of colloquial & jargon edge case usages that carry specific meanings in specific contexts, but which aren't easy to programmatically detect, but that's just my supposition).
Sorry, but that just doesn't make any sense to me. Practically 70% seems like enough to understand most narrative. I've read some really difficult texts (translated German theology) and for anything of meaningful complexity 98% is unreachable without a huge vocabulary and understanding of both oddities of grammar and the construction of narrative or argument.
Yes, we derive syntactic meaning from grammatical structure. It's one thing getting a machine to understand grammar and another to get a human to understand. If anyone is interested, Doing Grammar by Max Morenberg is an excellent source of knowledge about grammar.[0]He approaches grammar very systematically which is helpful if people want to train machines.
Lojban is unambiguous and interesting. The problem isn't the language, though. It's ego. When you can insert "I am the best" before every internet comment, the language used is not the problem. It's the ego.
This is really cool, and props to Google for making it publicly available.
The blog post says this can be used as a building block for natural language understanding applications. Does anyone have examples of how that might work? Parse trees are cool to look at, but what can I do with them?
For instance, let's say I'm interested in doing text classification. I can imagine that the parse tree would convey more semantic information than just a bag of words. Should I be turning the edges and vertices of the tree into a feature vectors somehow? I can think of a few half-baked ideas off the top of my head, but I'm sure other people have already spent a lot of time thinking about this, and I'm wondering if there are any "best practices".
The typical approach is something like a tree kernel (https://en.wikipedia.org/wiki/Tree_kernel). Looked into them briefly for a work project that never got off the ground, can't say too much about using them in practice.
This is actually really useful for a project I'm working on. I'm trying to detect bias in news sources using sentiment analysis and one of the problems I've run into is identifying who exactly is the subject of a sentence. Using this could be really helpful in parsing out the noun phrases and breaking them down in order to find the subject.
I've been experimenting with Stanford's CoreNLP to identify named entities for analyzing RSS feeds and I was really impressed by how well it worked, having known nothing about the state of NLP research before I started. Especially things like being able to identify coreferences.
Yes, I've used that before. I'm currently using Textacy for python which is also really good. However, extracting the named entities from a sentence is still a ways off from determining what's the subject of the sentence, although it gives a good indication. Using NER + quality POS tagging and tree building should do the trick for me I think.
Are you using BOW for sentiment analysis? Also, have you tried tinkering with Watson's sentiment analyzer?
I'm working on a project that analyzes sentiment from speech, and I've been meaning to start on text sentiment analysis, but I'm not sure where to start.
I was actually pretty disappointed with the NER in CoreNLP - I fed a few articles (including this one) into it, and while it's impressive that a computer can do this at all, it's pretty far away from being able to build a usable product. It seems to over-recognize Persons, for example - Parsey McParseFace was tagged as a person, as were Alice and Bob, as was Tesla (in another article), and while all of these are understandable, they weren't the intended meanings in the articles. I was also pretty disappointed with the date parser: while it gets some tricky ones like "Today" and "7 hours ago", it misses very common abbreviations like 7m or 7min or even "7min ago".
Yeah, I looked at SuTime, but it fell down on many common cases (the CoreNLP online demo is actually integrating SuTime into the annotations it produces).
Another option is Natty [1], but it also seems to fail on the same examples. Natty at least has an ANTLR grammar that's reasonably easy to understand, though.
Thanks 'nl, 'nostrademons and 'rcpt for the links! I've been using Chronicity[0] in my project, and I hand-hacked a Polish-to-English regexp "translator" to make it work with Polish language[1]. I'll be looking at the sources of the libraries you provided as well as papers they reference; maybe I'll manage to steal some code :).
This would be very interesting when applied to Biblical Studies. Any serious academic discussion of biblical texts will involve syntactical breakdown of the text being discussed. Most of the times the ambiguities are clear, but its still quite common for a phrase to have several possible syntactical arrangements that are not immediately clear. These ambiguities are also challenging becuase the languages are dead (at least as used in the biblical texts). So the type of ambiguity of "Alice drove down the street in her car" can lead to some significant scholarly disagreement.
I could see Parsey McParseface helping identifying patterns in literature contemporaneous to the biblical texts. Certain idiomatic uses of syntax, which would have been obvious to the original readers, could be identified much more quickly.
I was going to say... my main interest in this project is precisely for Biblical studies... I could talk about analyzing the Bible for hours, but let's just say there's way more depth than many even realize. The Aleph Tav in relation to the Book of Revelation is one such example, many translations omit it, but the Aleph Tav Study Bible explores it in depth. There could be many discoveries made with these kind of projects that are missed by just about anyone only reading a translation.
There are a ton of Jewish Idioms in the Bible that many don't understand at all, including "No man knows the day or the hour" which is a traditional Jewish Wedding Idiom. Lots and lots of things could be explored with enough data and resources.
I'd think that the advantage of machine translation is on corpora that are not known up front (i.e. user-supplied text) or corpora that are exceptionally large.
If you have a small (ish), well-known text, I don't think you will get much insight from machine translation. Certainly there are plenty of uses for computer text analysis/mining in biblical studies, but I doubt translation is one of them. And for obscure idioms or hapax legomena, machine translation definitely can't help you because by definition there are no other sources to rely on.
Well I wouldn't look for idioms, but it would be interesting to throw in information such as "Strong's Concordance" into the mix, I've yet to really think of an application for this library fully, but it would be fun to play around with it nonetheless. I would be analyzing the Hebrew / Greek / Syriac scripts, seeking verses omitted, or missing, etc. It would make for interesting studying if anything.
You might be interested in Andrew Bannister's research on computer analysis of the Quran. He wrote a book on it [1], and there's also this paper which gives a high-level overview [2].
With a sufficient level of precision, there's room for machine analysis to "reveal" things we are ignoring out of custom. A lot of text analysis done by people is full of biases and deferral to authorities.
E.g. I remember from school getting in into an argument with a teacher over the interpretation of a poem. "His" interpretation, which was really the interpretation of some authority who'd written a book was blatantly contradicted by the text if you assumed that the author hadn't suddenly forgotten all his basic grammar despite all the evidence to the contrary everywhere else that he was always very precise in this respect.
Of course, in some of these kind of instances, it will be incredibly hard to overcome the retort that any "revelation" is just a bug.
In a more general sense, people are typically exceedingly bad at parsing text, judging by how often online debates devolve into bickering caused largely by misunderstanding the other party's argument. Often to the extent of even ending up arguing against people who you agree with. Having tools that help clarify the parsing for people might be interesting in that respect too.
> Any serious academic discussion of biblical texts will involve syntactical breakdown of the text being discussed.
I once interned for a company that's been doing this for years. They have all kinds of features tracing individual words through various different languages, etc.
Actually it's not very appropriate for studying bible text. In Biblical Studies you would prefer not to have any errors at all, and since you work with a limited corpus you can afford to annotate by hand. People have in fact done this and I collaborated with a group that has been working on this for decades.
For actual syntactical breakdown of the Bible, I agree. Biblical Scholars, and even competent pastors, can syntactically analyze the the Bible sufficiently well.
I would think the technology could be helpful in a fairly narrow way: identifying syntactical constructions outside the bible to help explain ambiguous syntactical constructions within it (For example, Ugaritic texts, another ancient Semitic language similar to Hebrew, are often studied to aid in understanding portions of the Old Testament). Scholars have been doing this without computers for some time and have begun to do this type of analysis with software. I would imagine more sophisticated software would yield at least some new insights.
Most of the really good applications are part of larger systems. Parsing is good in machine translation, for instance. You transform the source text so that it's closer to the target language. Parsing is also useful for question answering, information extraction, text-to-speech...
Here's an example of using information from a syntactic parser to decorate words, and create an enhanced bag-of-words model: https://spacy.io/demos/sense2vec
> Parse trees are cool to look at, but what can I do with them?
One really simple and obvious thing is word sense disambiguation. Plenty of homonyms are different parts of speech (e.g. the verb "lead" and the noun "lead"). I'm sure there's lots of more sophisticated stuff you can do as well, but this might be the lowest-hanging fruit.
However, for that you just need PoS tags (which is also provided by this Google thing, yes).
And of course the hard part of WSD is detecting whether "bank" refers to the bank of a river, or the financial institution, or the building where the institution is located, or [you name it].
I use parse trees as a kind of "advanced language model" for when I need to replace a word in a sentence (see for example: http://www.aclweb.org/anthology/P13-1142 ), it's so much better than using just simple n-grams.
Here is an application of parse trees: sentiment analysis with recursive neural networks based on how components of the parse tree combine to create the overall meaning.
They are useful as a preprocessing step for a lot of downstream NLP tasks. It shouldn't be hard to find more papers that take advantage of the tree structure of language.
Idea: point this at political speeches / security breach notifications / outage postmortems / etc, and rate them by how many ambiguities with starkly different dependancy parses there are... (Well of _course_ we mean the roads inside Alice's car when we made that commitment!)
> Humans do a remarkable job of dealing with ambiguity, almost to the point where the problem is unnoticeable; the challenge is for computers to do the same. Multiple ambiguities such as these in longer sentences conspire to give a combinatorial explosion in the number of possible structures for a sentence.
Isn't the core observation about natural language that humans don't parse it at all? Grammar is a secondary, derived construct that we use to give language some stability; I doubt anyone reading "Alice drove down the street in her car" actually parsed the grammatical structure of that sentence, either explicitly or implicitly.
I'm not sure about the claim on implicit lack of parsing structure. I read your example as who did what, where, in what. There must be some level of structural parsing and recognition so we understand it was Alice who drove in a car, that the car is owned by Alice, and that she, Alice, drove down the street, in her car. That we automatically understand all this seems to indicate some level of implicit parsing, right? Admittedly, it's been many years since I did any study of linguistics and language acquisition, so I'm pretty ignorant of the current state of knowledge here. Am I just layering my grammatical parsing atop an existing understanding that doesn't parse at all?
I think observing how children learn their native language is pretty informative. They can speak and understand it very well, whether or not they were taught formal grammar at school. Personally, I know very, very little of Polish grammar (i.e. of my native language), and only little bit more of English grammar - and that is only because foreign language courses are pretty heavily grammar-laden.
I'm not a linguist, but seeing how people a) can understand sentences that are grammatically malformed perfectly well, b) can easily derive meaning out of "sentences" stripped out of verbs ("I her dinner cinema Washington"), it seems to me that most of the work is being done by pattern-matching to known words and phrases. E.g. "drove down the street" is a kind of semantic unit on its own.
Again, I'm not a linguist, but a lot of introspection as well as observing other people strongly suggest to me that humans do anything but parsing grammatical structures.
Ah, sure. Those are excellent points. I wasn't really thinking about how we bridge grammatical incorrectness. For myself, perhaps because I'm a grammar nerd, I feel like I always parse someone's mistaken statements into their grammatically correct forms. But I can recognize doing that after I've already figured out what they were intending to say. Same happened with my kids. That's a helpful vector for thinking about the problem, for sure. Thanks!
I'm not sure what you're implying. The fact that we are not consciously aware of parsing grammatical structures doesn't mean it doesn't happen.
For example we know for sure that the brain applies sophisticated mathematical algorithms to signals coming from the ears to locate sound in 3D space, yet we are certainly not consciously aware of it - we just "know" where the source is located
Regarding grammar, there is a theory called Universal Grammar from Chomsky that we are born with grammar structures in the brain.
It's precisely how strongly we conform to grammar, without having been taught it, which shows that it's key to our internal representations/to how we learn language.
Here is the undeniable proof that syntactic structure exists. Consider the sentence `The magician pointed at the man with the hat.' This is a perfectly natural sentence, of which there are two likely interpretations. One is that the magician used a hat to point at the man. The other is that the man who was pointed at wore a hat.
What distinguishes these sentences? Only the underlying syntactic structure, of whether to parse it as `the magician pointed at (the man) with the hat' or as `the magician pointed at (the man with the hat)'. This `hierarchical structure' of our sentences is syntactic structure at its essence.
You argue that humans can understand sentences with whatever grammar, and parsing is pretty much pattern-matching of words. But what about the sentence pair : `Benny chased Jenny' versus `Jenny chased Benny'? These have the same words, and mean different things. It is only our syntactic understanding of how words are ordered in English that allows us to understand these sentences.
Here is the undeniable proof that syntactic structure exists.
There are multiple hypothesis of what a sequence of words can mean, which is not the same thing as 'we form explicit syntax trees in our heads when reading a sentence'.
I could also give you the bag of words
magician point man hat
You would derive meaning from this bag of words, probably the same interpretations as in your example. However, the sentence is utterly ungrammatical. Note that I am not contending that we don't use some form of syntax at all. E.g., I think that someone whose native language has a freer word order than English will assign more hypotheses to the bag of words above (e.g., my brain also considers the less likely option that the magician is the object).
Another problematic aspect of this hypothesis is that a longer sentence will have so many possible parses that it would take a long time to construct and consider all parses. Moreover, I find it unlikely that we have thousands of exact syntax trees in our head that we compare.
> There are multiple hypothesis of what a sequence of words can mean, which is not the same thing as 'we form explicit syntax trees in our heads when reading a sentence'.
Yeah. I'm playing with a different idea now - maybe that "tree structure" that "undeniably exists" in our brains isn't an explicit syntax tree, but an artifact of recursive, adaptive pattern-matching? I.e. if you look at things like reading speed or "understanding" speed, you'll notice that people tend to process stuff in large blocks until something "does not click", and they have to focus and process the block in detail. That sort of feels like a recursive refinement, and any process that recurses in more than one place generates a tree structure as a side effect.
Various syntactic theories (HPSG, GPSG, minimalism, construction grammars) from linguistics are certainly derived constructs, but most researchers would agree that they all reflect real abstractions that humans make. I think the NLP community has good a job of harvesting the substantive aspects (which tend to be fairly conventionalized upon across theories) without overfitting on specific cases. "Alice drove down the street in her car" is easy for people to process, "The horse raced past the barn fell" is not, because it requires a pretty drastic reinterpretation of the structure when you get to the last word.
That said, there is some interesting work on "good-enough" language processing, which suggests that people maintain some fuzziness and don't fully resolve the structure when they don't need to. [1]
but most researchers would agree that they all reflect real abstractions that humans make
They reflect a particular language in its well-written form. However, humans are extremely robust against syntax errors. I am not a linguist, but I think this speaks in favor of lexicalist approaches: we can be very free in word order, as long as our brain can match up e.g. verbs with their expected arguments.
No, the academic consensus is pretty much the opposite. For example by trying to rigorously state the way we form yes/no sentences in english - the process that converts "the man who has written the book will be followed" to "will the man who has written the book be followed?" instead of the incorrect "has the man who written the book will be followed?" - you will find that the rules must involve imposing some sort of tree structure on the original sentence. The fact that we do it correctly all of the time on sentences we've never seen before means that we must have parsed the original sentence.
(Example sentences taken from https://he.palgrave.com/page/detail/syntactic-theory-geoffre..., although any introductory linguistics/syntax textbooks will spend a few pages making the case that humans understand language by first parsing it into some kind of tree structure).
> the process that converts "the man who has written the book will be followed" to "will the man who has written the book be followed?" instead of the incorrect "has the man who written the book will be followed?"
And yet the following is also correct - in terms of real-world usage, not some prescriptive definitions:
"The man who has written the book will be followed, right mate?"
> you will find that the rules must involve imposing some sort of tree structure on the original sentence.
The rules are, and the brain may be, but I feel those are different tree structures. Moreover, I wonder if the "tree structures" of our brains aren't just artifacts of recursive pattern matching - we also know that when reading, humans process whole groups of words at a time, and only if there's some mismatch they process pieces in more detail. Any recursive process like this will generate a tree structure as its side effect.
Anyway, thanks for the examples. I might pick a linguistic book at some point. Right now the idea of understanding natural language by parsing it into "NOUN PHRASES" and "VERB PHRASES" and stuff seems completely backwards, given how humans have no trouble parsing "invalid" sentences, or using them - especially in spoken language.
(Not to mention our ability to evolve the language, and how the grammatically invalid constructs tend to be introduced, used, understood with no trouble and at some point they become grammatically accepted - see e.g. recent acceptance of "because <noun>").
Yes, of course your "right mate" example is also grammatically correct. The point is that people routinely and naturally do the complicated transformation to "will the man who has written the book be followed?", and that transformation can't be done by simple pattern matching. Hence, humans who are able to do the complicated transformation must be mentally parsing the sentence. The fact that there is an alternative simple transformation to form the yes-no question is irrelevant because the ability to use the complicated transformation still exists.
> given how humans have no trouble parsing "invalid" sentences
I think you misunderstand slightly - the claim linguists make is not "humans are unable to understand invalid sentences because they can't parse them", the claim is that when you see an invalid (cannot be parsed into a proper tree) sentence, you have a gut feeling that it "sounds off", and if you're a native speaker you would never accidentally produce such ill-formed sentences. You can still understand the meaning of a sentence like "I this morning fish eat" but you also immediately notice that it's "off" - and that's the phenomena that syntax tries to explain.
Furthermore, the way you understand sentences like "I this morning fish eat" is different from the way you understand "I ate fish this morning", in the former it feels like you're guessing. It could work for communicating simple thoughts, but I doubt an english non-speaker who has an english dictionary could convey a complicated thought requiring many words by that same guessing process. In fact the reason why language evolved tree syntax is probably because it is needed to convey long, complicated thoughts.
> because <noun>
I'm glad you mentioned that! First, modern linguistics is very far from prescriptive. In fact the first thing they teach you (at around the same time they make the claim that "humans parse sentences into tree structure") is that linguistics is a descriptive field - language changes over time, the study of the rules of language and how these rules change is interesting and important, but it's pointless to "enforce" the rules. Even new constructions like "because X" have rules that govern them, eg see http://allthingslinguistic.com/post/72252671648/why-the-new-... - constructions like "because want" and "because need" exist, but no one says "because adore", and something interesting explains why. (to be fair, I haven't really internalized the "because X" construction so I can't claim that I find "because adore" unnatural, but the article says it's the same reason why "omg want" and omg need" are currently grammatical but "omg adore" is not, and even if you're not familiar with the "omg X" construction, it gives independent evidence in that "omg adore" has no tumblr tags; of course, it may become grammatical in the future, but that would be because the rules have changed over time, not because there are no rules). To that point,
> or using [ill-formed sentences] - especially in spoken language.
actually, if a sentence is used in spoken language routinely and non-accidentally, linguists take it as evidence that it's grammatical and then work backwards to find the rules that explain why it is so. How else could they do it?
Thanks for your answers. You've raised a lot of good points, and I need to think them through.
> the claim is that when you see an invalid (cannot be parsed into a proper tree) sentence, you have a gut feeling that it "sounds off", and if you're a native speaker you would never accidentally produce such ill-formed sentences. You can still understand the meaning of a sentence like "I this morning fish eat" but you also immediately notice that it's "off" - and that's the phenomena that syntax tries to explain.
I see. Yeah, most of the way I think about how mind processes language comes from focusing on that "gut feeling", that on one hand tells you that this perfectly understandable sentence is somehow "off", and on the other hand lets you form perfect sentences without ever explicitly thinking about grammar.
> First, modern linguistics is very far from prescriptive. In fact the first thing they teach you (at around the same time they make the claim that "humans parse sentences into tree structure") is that linguistics is a descriptive field
It seems to me that I've been operating under invalid assumption that linguistics is mostly prescriptive. Thanks for that. Any recommendation for an intro book I could grab to read in my spare time?
> Any recommendation for an intro book I could grab to read in my spare time?
Unfortunately I think the field suffers from a lack of such books.
1. You could try Steven Pinker's "The Language Instinct", although it's a general-audience book that doesn't really try to teach you linguistics proper
> I doubt anyone reading "Alice drove down the street in her car" actually parsed the grammatical structure of that sentence, either explicitly or implicitly.
You do need to analyze a sentence to understand it. Think of a classical attachment ambiguity such as "the boy saw the girl with the telescope". There are two readings of the sentence, and just like a Gestalt, you're typically perceiving it as one or the other. This involves a process of disambiguation, which is evidence that you have parsed the sentence.
I'm pretty sure the borders between the two are fluid at google.
I also wish middle management were a bit less afraid of such things. You can usually get cool ideas approved in the higher echolons (because a CEO doesn't have to fear too much, or possibly because these people actually are more imaginative).
Names that have meaning, tell a story are incredibly useful for marketing, even if they sometimes sound unprofessional. Exp: 'Plan B' (morning after pill), 'CockroachDB', 'Virgin'.
It's beyond me how anyone could have chosen the predictable public outcry instead of naming that boat Boaty McBoatface. That's probably the least offending name that ever resulted from an internet poll.
You know, it's a brilliant name. It's a person's name, like Siri or Cortana. But if they'd named it Jane or John, the mass media would have a fit. So they side-stepped that by making it sound computer-y, while remaining a proper name. Brilliant.
I used to have a woman in my life, pumpkin latte sipping, wanna-be graphic designer and that is how she referred to everything xx-y mc-xx-yface seemingly in an attempt to remain a teenager well into her third decade.
I am so glad to be rid of her vapid nonsense.
It is a silly name and makes me think it is not a serious effort. But names can be deceiving.
(INB4 yes, that's the reference to an old joke about Lufthansa pilot in Frankfurt complaining he's forced to speak in English to the control tower)
Honestly though, English is a quite good starting language due to its simplicity, but I do hope more attention will be expended on making the technology like this trainable on other languages as well.
According to the section titled "Why is Parsing So Hard For Computers to Get Right?" in the blog post, determining the difference between "An English parser" and "An English parser" .. is Parsey's great strength.
According to their paper (http://arxiv.org/pdf/1603.06042v1.pdf), the technique can also be applied to sentence compression. It would be cool if Google publishes that example code/training-data as well.
Say, I wanted to use this for English text with a large amount of jargon. Do have to train my own model from scratch or is it possible to retrain Parsey McParseface?
How expensive is it to train a model like Parsey McParseface?
One of the projects I'd love to develop is an automated peer editor for student essays. My wife is an english teacher and a large percentage of her time is taken up by grading papers. A large percentage of that time is then spent marking up grammar and spelling. What I envision is a website that handles that grammar/spelling bit. More importantly, I'd like it as a tool that the students use freely prior to submitting their essays to the teacher. I want them to have immediate feedback on how to improve the grammar in their essays, so they can iterate and learn. By the time the essays reach the teacher, the teacher should only have to grade for content, composition, style, plagiarism, citations, etc. Hopefully this also helps to reduce the amount of grammar that needs to be taught in-class, freeing time for more meaningful discussions.
The problem is that while I have knowledge and experience in the computer vision side of machine learning, I lack experience in NLP. And to the best of my knowledge NLP as a field has not come as far as vision, to the extent that such an automated editor would have too many mistakes. To be student facing it would need to be really accurate. On top of that it wouldn't be dealing with well formed input. The input by definition is adversarial. So unlike SyntaxNet which is built to deal with comprehensible sentences, this tool would need to deal with incomprehensible sentences. According to the link, SyntaxNet only gets 90% accuracy on random sentences from the web.
That said, I might give SyntaxNet a try. The idea would be to use SyntaxNet to extract meaning from a broken sentence, and then work backwards from the meaning to identify how the sentence can be modified to better match that meaning.
Thank you Google for contributing this tool to the community at large.
SyntaxNet is, by definition, for syntactic analysis - it would likely not help you much with semantics, to extract meaning. It could maybe help you automatically determine is a sentence is grammatically correct, though.
I think this is still risky if used in a context where the student might think that the computer is somehow always right. Great English writers often deliberately use sentence fragments or puns, or use a word with a nonstandard part-of-speech interpretation (especially using a noun as a verb). They may also sometimes use a sentence that's difficult for readers to parse and then explain the ambiguity after the fact.
If a teacher gave students a grammar-checking tool to check their writing, they might assume that the tool knew better than they did, which is only sometimes true.
Knowing the rules is not the same thing as having formal training.
With then-innovated art like Cubism, which in some sense broke earlier rules, the point is that those artists, like Picasso, were able to do representative art in correct proportions -- they were just going beyond that.
This is a nontrivial issue, because there are always students who think they can skip learning boring mundane old fashioned art and go straight to something like cubism, but in 99% of cases that doesn't work well at all compared with learning "the rules" first.
With writing, many great writers have broken "the rules" with punctuation, spelling, grammar, etc. But the important thing is that they do so onpurpose.
Whereas if one doesn't know the rules in the first place, one doesn't have the choice of whether to follow them or break them.
Such a person will always break those rules they don't know (subconscious knowledge counts btw) -- but not for aesthetic reasons, only out of ignorance.
My perhaps favourite example of knowing when and how to break the rules is Franz Schuberts "Erlkönig" [1] because it is so stark.
If you listen to it without paying attention to the text (based on a poem by Goethe by the same name; both the German text and an English translation is found at [1]), parts of it sounds like horrible jammering and poor harmonies and it's easy to write it off as not sounding very nice.
Here's [2] a much clearer rendition (two singers, with much stronger delineation of the three different characters) than the one linked from Britannica.
If you do pay attention to the text, it is very clear that the unpleasant parts are very deliberate:
The singer(s) switches between the role of a father, his sick dying child, and the Erl-king that occurs in the hallucinations of the child while the father is riding to bring the child to a doctor.
The big difference between the unpleasant-sounding parts of this song and a bad composer is the clear intent and delineation - Schubert made things sound bad intentionally explicitly at the points he wanted to illustrate pain and fear, rather than because he didn't know how to make things sound pleasant when he wanted to.
The song clearly proves this by setting the childs jammering and the fathers fearful attempts to soothe him up against much more pleasant segments where the Erl-king speaks and tries to seduce the child to come with him.
You only get that clear separation if you know how to evoke each effect precisely. Arguably a bad particularly composer wouldn't even know how to make things sound bad the "right way" - there's a big difference between random bad sounds and making things evoke a child in pain.
Your logic is flawed. "Artist A did X before doing Y, therefore X is necessary to do Y". It doesn't compute.
I've heard great painters say that the only thing that matters is that you paint. Plenty of wonderful painters did not study their predecessors in depth.
I'm assuming that your 99% number is fabricated? Incidentally 99% of statistics are made up.
Sorry for not being clear. This is intended for use in teaching English. What you describe falls under the purview of Creative Writing. Specifically it'd target English in grade school up until the early college college classes (the ones most people skip out of based on tests). After that, yes you'd be right and this tool wouldn't be appropriate.
I am also afraid of the misuse of such software. It is also possible that the teacher does not know that much and will look at the software as correct.
And come to think of it, isn't there a saying: Did stupidity require smart computers or did smart computers allow for stupidity?
An intelligent software (or one that pretends to be intelligent) might allow for any unqualified bloke to be in a position where they can teach.
Doesn't Grammarly[0] already do this? It analyzes the input for common grammar mistakes and proposes ways to fix them. As a student, I occasionally use Grammarly to proofread a paper for me, and it has worked pretty well so far.
"A large percentage of that time is then spent marking up grammar and spelling."
As an aside, I don't think this is the optimal way to teach people how to write. What were the ideas in those papers? How were they organized? Do the student's arguments make sense? I think that's what most students spend most of their time thinking about when writing an essay, and it can be a bit demoralizing to see the teacher care just as much about whether the grammar was right. Most students can fix grammar mistakes relatively easily once they notice them anyway.
> Most students can fix grammar mistakes relatively easily once they notice them anyway.
You would be surprised! Especially with people whose first languages aren't English. Having something to provide feedback on grammar problems early on would be really useful. Of course, once they get grammar down, the next step is rhythm and flow, as well as reducing redundancy, and the biggest problem, as you say, is always the story, but you have to peel through lots of grammar problems before you get to that point. I edit a lot of research papers for my Chinese peers (most have PhDs, I work in a China-based research lab, so that isn't weird), so I'm pretty clear on the problems.
I don't think I remember actually being taught how to write in primary school. I think they should reteach grammar in high school from the beginning. Most people's brains just can't pick up a systematic treatment of some of the finer points when younger. I went and did all the grammar quizzes over at http://grammar.ccc.commnet.edu/grammar/quiz_list.htm a couple of years ago. It improved my writing a lot, and only took a weekend.
I honestly didn't learn how to write until I got to graduate school. My very patient adviser had to beat it into my skull. Writing is a very practical art...you need to practice at it a lot vs. studying and memorizing rules, and my pre-grad school education didn't really force that.
Can you tell what other resources you've used to learn, what practices were helpful in studying grammar, what was not helpful in your study of grammar?
Not sure how they work exactly, but have you looked at http://noredink.com (and as another commenter mentioned, http://grammarly.com/)? I'd be interested in your thoughts.
I've bounced my idea off my wife before and asked whether such a thing existed. I do recall her mentioning things like noredink and commenting that they weren't a match for one reason or another (the specific reasons escape me at the moment).
Visiting noredink.com now ... I can't actually figure out what it is the site offers from a cursory glance, so I can't even begin to figure out whether it matches my idea and what issues it may have had that excluded it from her classroom.
I'll bounce grammarly off her later though, in case that's new.
This looks fantastic. I've been fascinated with parsers ever since I got into programming in my teens (almost always centered around programming language parsing).
Curious - The parsing work I've done with programming languages was never done via machine learning, just the usual strict classification rules (which are used to parse ... code written to a strict specification). I'm guessing source code could be fed as data to an engine like this as a training model but I'm not sure what the value would be. Does anyone more experienced/smarter than me have any insights on something like that?
As a side-point:
Parsy McParseface - Well done. They managed to lob a gag over at NERC (Boaty McBoatface) and let them know that the world won't end because a product has a goofy name. Every time Google does things like this they send an unconscious remind us that they're a company that's 'still just a bunch of people like our users'. They've always been good at marketing in a way that keeps that "touchy-feely" sense about them and they've taken a free opportunity to get attention for this product beyond just the small circle of programmers.
As NERC found out, a lot of people paid attention when the winning name was Boaty McBoatface (among other, more obnoxous/less tasteful choices). A story about a new ship isn't going to hit the front page of any general news site normally and I always felt that NERC missed a prime opportunity to continue with that publicity and attention. It became a topic talked about by friends of mine who would otherwise have never paid attention to anything science related. It would have been comical, should the Boaty's mission turn up a major discovery, to hear 'serious newscasters' say the name of the ship in reference to the breakthrough. And it would have been refreshing to see that organization stick to the original name with a "Well, we tried, you spoke, it was a mistake to trust the pranksters on the web but we're not going to invoke the 'we get the final say' clause because that wasn't the spirit of the campaign. Our bad."
Curious - The parsing work I've done with programming languages was never done via machine learning,
Artificial languages (such a programming languages) are usually designed to be unambiguous. In other words, there is a 1:1 mapping from a sentence or fragment to its abstract representation.
Natural language is ambiguous, so there is usually 1:N mapping from a sentence to abstract representations. So, at some point you need to decide which of the N readings is the most likely one.
Older rule-based approaches typically constructed all readings of a sentence and used a model to estimate which reading is the most plausible. In newer deterministic, linear-time (transition-based) parsers, such ambiguities (if any) are resolved immediately during each parsing step.
In the end it's a trade-off between having access to global information during disambiguation and having a higher complexity. So, naturally, the rule-based systems have been applying tricks to aggressively prune the search space, while transition-based parsers are gaining more and more tricks to incorporate more global information.
> At Google, we spend a lot of time thinking about how computer systems can read and understand human language in order to process it in intelligent ways.
There is 6 links in this sentence in the original text. I get it can help to get more context around it, but I think it's actually making the text harder to "human" parse. It also feels they have hired a cheap SEO consultant to do some backlink integrations.
How would you feed a sentence to a neural net? As I understand, the inputs are usually just floating point numbers in a small range, so how is the mapping performed? And what if the sentence is longer than the number of input neurons? Can that even happen, and pose a problem?
There's a bunch of blogs, tutorials, etc, around word2vec and other methods of generating vectors from a training set of words.
Also, in the tensorflow models codebase where this syntaxnet code lives, there is an another tensorflow-using-method of generating word embeddings with demonstration code called Swivel
I think Syntaxnet is just using the NN to guide search instead of doing end-to-end parsing. That said, you can feed "sequences of stuff" into recurrent neural nets! See
As someone who has published work in the NLP area, I always take claimed results with a grain of salt. With that said, I still will have to read the paper to know the implementation details, although my problem with generic linguistic approaches such as this one seems to be is that it is usually hard to "port" to other languages.
For instance, the way they parse sequences of words may or may not be too specific to the English language. It is somewhat similar to what we call "overfitting" in the data-mining area, and it may invalidate this technique for other languages.
When I worked on this area (up to 2014), I worked mainly in language-independent statistical approaches. As with everything, it has its cons as you can extract information from more languages, but, in general, with less certainties.
But in general, it is good to see that the NLP area is still alive somewhere, as I can't seem to find any NLP jobs where I live! :)
Edit: I've read it in the diagonal, and it is based on a Neural Network, so in theory, if it was trained in other languages, it could return good enough results as well. It is normal for English/American authors to include only english datasets, but I would like to see an application to another language.. This is a very specialized domain of knowledge, so I'm quite limited on my analysis..
254 comments
[ 2.4 ms ] story [ 265 ms ] thread[0] http://www.theatlantic.com/international/archive/2016/05/boa...
http://www.duetsblog.com/2016/03/articles/trademarks/namey-m...
[0]: https://news.ycombinator.com/item?id=11642618
Vote for boaty! He's keepin hope afloat!
http://shirt.woot.com/offers/vote-for-boaty
Just kidding! As an American, I can say it's a decent assumption :D
For other non-Parsey McParseface dependency parsers and POS taggers that are web accessible, see http://corenlp.run/ and http://nlp.stanford.edu:8080/parser/.
Time flies like an arrow. Fruit flies like a banana.
So, really all of the above parsers are weak in that they only output a single best parsing, when in reality sentences can have more than one valid structure, the principal example being the second sentence you've provided. I don't think Google's model has a better sense of humor than the others, no. I anticipate that they all have used relatively similar training data.
However, there is probably a trivial way to get the second sentence to parse as
and that is to provide training data with more occurrences of ... than occurrences of The more sentences using simile that the parser finds, the less likely the neural net is to consider 'like' as a verb. It's also impacted by all of the uses of [flies like].That's the nature of statistical language tools.
The stock parser debuted here gives the same answer as CoreNLP, by the way.
So much for Parsey McParseface's sense of humor."On a standard benchmark consisting of randomly drawn English newswire sentences (the 20 year old Penn Treebank), Parsey McParseface recovers individual dependencies between words with over 94% accuracy, beating our own previous state-of-the-art results, which were already better than any previous approach."
From the original paper, "Our model achieves state-of-the-art accuracy on all of these tasks, matching or outperforming LSTMs while being significantly faster. In particular for dependency parsing on the Wall Street Journal we achieve the best-ever published unlabeled attachment score of 94.41%."
This seems like a narrower standard than described, specifically being better at parsing the Penn Treebank than the best natural language parser for English on the Wall Street Journal.
The statistics listed on the project GitHub actually contradict these claims by showing the original March 2016 implementation has higher accuracy than Parsey McParseface.
There is a simplified educational 200 lines python version [2] of it. It claims 96.8% for the WSJ corpus.
What am I missing here?
[1] https://news.ycombinator.com/item?id=8942783
[2] https://spacy.io/blog/part-of-speech-pos-tagger-in-python
Actually I found an educational 500 lines python parser from the same author as well [1]. It claims an accuracy of 92.7 for the WSJ corpus.
[1] https://spacy.io/blog/parsing-english-in-python
I've been watching the line of research in SyntaxNet closely, and have been steadily working on replacing spaCy's averaged perceptron model with a neural network model. This is one of the main differences between spaCy and Parser McParseface.
The key advantage of the neural network is that it lets you take advantage of training on lots and lots more text, in a semi-supervised way. In a linear model, you grow extra parameters when you do this. The neural network stays the same size --- it just gets better. So, you can benefit from reading the whole web into the neural network. This only works a little bit in the linear model, and it makes the resulting model enormous.
Another difference is that spaCy is trained on whole documents, while P. McP. is trained in the standard set-up, using gold pre-processing. I speculate this will reduce the gap between the systems in a more realistic evlauation. Of course, P. McP can do the joint training too if they choose to. I've reached out to see whether they're interested in running the experiment: https://github.com/tensorflow/models/issues/65
Just installed syntaxnet - tests passed in the following setup.
https://gist.github.com/Hendler/61831e411069815ee4ed490f553f...
INFO: Elapsed time: 908.048s, Critical Path: 640.26s
//syntaxnet:arc_standard_transitions_test PASSED in 0.0s
//syntaxnet:beam_reader_ops_test PASSED in 20.9s
//syntaxnet:graph_builder_test PASSED in 16.3s
//syntaxnet:lexicon_builder_test PASSED in 1.8s
//syntaxnet:parser_features_test PASSED in 0.0s
//syntaxnet:parser_trainer_test PASSED in 46.1s
//syntaxnet:reader_ops_test PASSED in 5.7s
//syntaxnet:sentence_features_test PASSED in 0.0s
//syntaxnet:shared_store_test PASSED in 0.5s
//syntaxnet:tagger_transitions_test PASSED in 0.0s
//syntaxnet:text_formats_test PASSED in 1.7s
//util/utf8:unicodetext_unittest PASSED in 0.0s
Some other notes:
Also using Keras with Theano. Before spaCy, StanfordNLP, Freeling, and/or NLTK.
So you're referring to this LSTM?
"Andor et al. (2016)* is simply a SyntaxNet model with a larger beam and network. For futher information on the datasets, see that paper under the section "Treebank Union"."
After spending a few months hand coding a NLP parser, am rather intrigued by LSTM. I like the idea of finding coefficients, as opposed to juggling artificial labels.
Cool. I wonder how much how much Human effort (vs machine time) went into tweaking SyntaxNet versus tweaking Parsey M.?
This is an opensourcing of the March 2016 method (syntaxnet, note that in the paper there are results from several trained models) as well as a trained model that is comparable in performance but faster (macparseface).
It is very hard to separate those two things from the way they write.
(I worked on a successor project, OntoNotes, that involved additional treebank annotation on broader corpora: https://catalog.ldc.upenn.edu/LDC2013T19)
The point about overfitting is valid, too, which is another reason why this "most accurate such model in the world" claim is obnoxious.
It's also fair to note that their advance is in fractions of percentage points on this specific dataset over models that are 5-10 years older.
Analysis of the structure of a piece of text is the first step to understanding its meaning. IBM are doing some good work in this area. http://www.alchemyapi.com/products/demo/alchemylanguage
Anything in the pipeline for this project to help with classifying sentiment, emotion etc. from text?
[0] http://www.amazon.com/Doing-Grammar-Max-Morenberg/dp/0199947...
[1] - https://en.wikipedia.org/wiki/Garden_path_sentence
The blog post says this can be used as a building block for natural language understanding applications. Does anyone have examples of how that might work? Parse trees are cool to look at, but what can I do with them?
For instance, let's say I'm interested in doing text classification. I can imagine that the parse tree would convey more semantic information than just a bag of words. Should I be turning the edges and vertices of the tree into a feature vectors somehow? I can think of a few half-baked ideas off the top of my head, but I'm sure other people have already spent a lot of time thinking about this, and I'm wondering if there are any "best practices".
I'm working on a project that analyzes sentiment from speech, and I've been meaning to start on text sentiment analysis, but I'm not sure where to start.
The code is pretty readable but relies heavily on a ruleset which might need to be tweaked for one's need.
For the date parser you want http://nlp.stanford.edu/software/sutime.html
The code and rules aren't fun to customize though.
Another option is Natty [1], but it also seems to fail on the same examples. Natty at least has an ANTLR grammar that's reasonably easy to understand, though.
[1] http://natty.joestelmach.com/
One day someone will build a neural net model to do this rather than hand written rules.
[1] https://github.com/leondz/timen
[2] https://github.com/HeidelTime/heideltime
[0] - https://github.com/chaitanyagupta/chronicity
[1] - it's surprising how easy is to get 80% there with hacks like these: https://github.com/TeMPOraL/alice/blob/master/language.lisp#...
https://github.com/HeidelTime/heideltime
I could see Parsey McParseface helping identifying patterns in literature contemporaneous to the biblical texts. Certain idiomatic uses of syntax, which would have been obvious to the original readers, could be identified much more quickly.
There are a ton of Jewish Idioms in the Bible that many don't understand at all, including "No man knows the day or the hour" which is a traditional Jewish Wedding Idiom. Lots and lots of things could be explored with enough data and resources.
If you have a small (ish), well-known text, I don't think you will get much insight from machine translation. Certainly there are plenty of uses for computer text analysis/mining in biblical studies, but I doubt translation is one of them. And for obscure idioms or hapax legomena, machine translation definitely can't help you because by definition there are no other sources to rely on.
[1] http://www.amazon.com/Oral-Formulaic-Study-Quran-Andrew-Bann...
[2] http://www.academia.edu/9490706/Retelling_the_Tale_A_Compute...
E.g. I remember from school getting in into an argument with a teacher over the interpretation of a poem. "His" interpretation, which was really the interpretation of some authority who'd written a book was blatantly contradicted by the text if you assumed that the author hadn't suddenly forgotten all his basic grammar despite all the evidence to the contrary everywhere else that he was always very precise in this respect.
Of course, in some of these kind of instances, it will be incredibly hard to overcome the retort that any "revelation" is just a bug.
In a more general sense, people are typically exceedingly bad at parsing text, judging by how often online debates devolve into bickering caused largely by misunderstanding the other party's argument. Often to the extent of even ending up arguing against people who you agree with. Having tools that help clarify the parsing for people might be interesting in that respect too.
I once interned for a company that's been doing this for years. They have all kinds of features tracing individual words through various different languages, etc.
https://www.logos.com/
I would think the technology could be helpful in a fairly narrow way: identifying syntactical constructions outside the bible to help explain ambiguous syntactical constructions within it (For example, Ugaritic texts, another ancient Semitic language similar to Hebrew, are often studied to aid in understanding portions of the Old Testament). Scholars have been doing this without computers for some time and have begun to do this type of analysis with software. I would imagine more sophisticated software would yield at least some new insights.
Here's an example of using information from a syntactic parser to decorate words, and create an enhanced bag-of-words model: https://spacy.io/demos/sense2vec
Here's a very terse explanation of using them in a rule-based way: https://spacy.io/docs/tutorials/syntax-search
One really simple and obvious thing is word sense disambiguation. Plenty of homonyms are different parts of speech (e.g. the verb "lead" and the noun "lead"). I'm sure there's lots of more sophisticated stuff you can do as well, but this might be the lowest-hanging fruit.
I use parse trees as a kind of "advanced language model" for when I need to replace a word in a sentence (see for example: http://www.aclweb.org/anthology/P13-1142 ), it's so much better than using just simple n-grams.
http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf
They are useful as a preprocessing step for a lot of downstream NLP tasks. It shouldn't be hard to find more papers that take advantage of the tree structure of language.
Isn't the core observation about natural language that humans don't parse it at all? Grammar is a secondary, derived construct that we use to give language some stability; I doubt anyone reading "Alice drove down the street in her car" actually parsed the grammatical structure of that sentence, either explicitly or implicitly.
Anyway, some impressive results here.
I'm not a linguist, but seeing how people a) can understand sentences that are grammatically malformed perfectly well, b) can easily derive meaning out of "sentences" stripped out of verbs ("I her dinner cinema Washington"), it seems to me that most of the work is being done by pattern-matching to known words and phrases. E.g. "drove down the street" is a kind of semantic unit on its own.
Again, I'm not a linguist, but a lot of introspection as well as observing other people strongly suggest to me that humans do anything but parsing grammatical structures.
For example we know for sure that the brain applies sophisticated mathematical algorithms to signals coming from the ears to locate sound in 3D space, yet we are certainly not consciously aware of it - we just "know" where the source is located
Regarding grammar, there is a theory called Universal Grammar from Chomsky that we are born with grammar structures in the brain.
Some recent news on it - http://www.medicaldaily.com/noam-chomskys-theory-universal-g...
It's a very good study, but does NOT prove `UG' once and for all.
Here is the undeniable proof that syntactic structure exists. Consider the sentence `The magician pointed at the man with the hat.' This is a perfectly natural sentence, of which there are two likely interpretations. One is that the magician used a hat to point at the man. The other is that the man who was pointed at wore a hat.
What distinguishes these sentences? Only the underlying syntactic structure, of whether to parse it as `the magician pointed at (the man) with the hat' or as `the magician pointed at (the man with the hat)'. This `hierarchical structure' of our sentences is syntactic structure at its essence.
You argue that humans can understand sentences with whatever grammar, and parsing is pretty much pattern-matching of words. But what about the sentence pair : `Benny chased Jenny' versus `Jenny chased Benny'? These have the same words, and mean different things. It is only our syntactic understanding of how words are ordered in English that allows us to understand these sentences.
There are multiple hypothesis of what a sequence of words can mean, which is not the same thing as 'we form explicit syntax trees in our heads when reading a sentence'.
I could also give you the bag of words
magician point man hat
You would derive meaning from this bag of words, probably the same interpretations as in your example. However, the sentence is utterly ungrammatical. Note that I am not contending that we don't use some form of syntax at all. E.g., I think that someone whose native language has a freer word order than English will assign more hypotheses to the bag of words above (e.g., my brain also considers the less likely option that the magician is the object).
Another problematic aspect of this hypothesis is that a longer sentence will have so many possible parses that it would take a long time to construct and consider all parses. Moreover, I find it unlikely that we have thousands of exact syntax trees in our head that we compare.
Yeah. I'm playing with a different idea now - maybe that "tree structure" that "undeniably exists" in our brains isn't an explicit syntax tree, but an artifact of recursive, adaptive pattern-matching? I.e. if you look at things like reading speed or "understanding" speed, you'll notice that people tend to process stuff in large blocks until something "does not click", and they have to focus and process the block in detail. That sort of feels like a recursive refinement, and any process that recurses in more than one place generates a tree structure as a side effect.
That said, there is some interesting work on "good-enough" language processing, which suggests that people maintain some fuzziness and don't fully resolve the structure when they don't need to. [1]
[1] http://csjarchive.cogsci.rpi.edu/proceedings/2009/papers/75/...
They reflect a particular language in its well-written form. However, humans are extremely robust against syntax errors. I am not a linguist, but I think this speaks in favor of lexicalist approaches: we can be very free in word order, as long as our brain can match up e.g. verbs with their expected arguments.
(Example sentences taken from https://he.palgrave.com/page/detail/syntactic-theory-geoffre..., although any introductory linguistics/syntax textbooks will spend a few pages making the case that humans understand language by first parsing it into some kind of tree structure).
And yet the following is also correct - in terms of real-world usage, not some prescriptive definitions:
"The man who has written the book will be followed, right mate?"
> you will find that the rules must involve imposing some sort of tree structure on the original sentence.
The rules are, and the brain may be, but I feel those are different tree structures. Moreover, I wonder if the "tree structures" of our brains aren't just artifacts of recursive pattern matching - we also know that when reading, humans process whole groups of words at a time, and only if there's some mismatch they process pieces in more detail. Any recursive process like this will generate a tree structure as its side effect.
Anyway, thanks for the examples. I might pick a linguistic book at some point. Right now the idea of understanding natural language by parsing it into "NOUN PHRASES" and "VERB PHRASES" and stuff seems completely backwards, given how humans have no trouble parsing "invalid" sentences, or using them - especially in spoken language.
(Not to mention our ability to evolve the language, and how the grammatically invalid constructs tend to be introduced, used, understood with no trouble and at some point they become grammatically accepted - see e.g. recent acceptance of "because <noun>").
> given how humans have no trouble parsing "invalid" sentences
I think you misunderstand slightly - the claim linguists make is not "humans are unable to understand invalid sentences because they can't parse them", the claim is that when you see an invalid (cannot be parsed into a proper tree) sentence, you have a gut feeling that it "sounds off", and if you're a native speaker you would never accidentally produce such ill-formed sentences. You can still understand the meaning of a sentence like "I this morning fish eat" but you also immediately notice that it's "off" - and that's the phenomena that syntax tries to explain.
Furthermore, the way you understand sentences like "I this morning fish eat" is different from the way you understand "I ate fish this morning", in the former it feels like you're guessing. It could work for communicating simple thoughts, but I doubt an english non-speaker who has an english dictionary could convey a complicated thought requiring many words by that same guessing process. In fact the reason why language evolved tree syntax is probably because it is needed to convey long, complicated thoughts.
> because <noun>
I'm glad you mentioned that! First, modern linguistics is very far from prescriptive. In fact the first thing they teach you (at around the same time they make the claim that "humans parse sentences into tree structure") is that linguistics is a descriptive field - language changes over time, the study of the rules of language and how these rules change is interesting and important, but it's pointless to "enforce" the rules. Even new constructions like "because X" have rules that govern them, eg see http://allthingslinguistic.com/post/72252671648/why-the-new-... - constructions like "because want" and "because need" exist, but no one says "because adore", and something interesting explains why. (to be fair, I haven't really internalized the "because X" construction so I can't claim that I find "because adore" unnatural, but the article says it's the same reason why "omg want" and omg need" are currently grammatical but "omg adore" is not, and even if you're not familiar with the "omg X" construction, it gives independent evidence in that "omg adore" has no tumblr tags; of course, it may become grammatical in the future, but that would be because the rules have changed over time, not because there are no rules). To that point,
> or using [ill-formed sentences] - especially in spoken language.
actually, if a sentence is used in spoken language routinely and non-accidentally, linguists take it as evidence that it's grammatical and then work backwards to find the rules that explain why it is so. How else could they do it?
> the claim is that when you see an invalid (cannot be parsed into a proper tree) sentence, you have a gut feeling that it "sounds off", and if you're a native speaker you would never accidentally produce such ill-formed sentences. You can still understand the meaning of a sentence like "I this morning fish eat" but you also immediately notice that it's "off" - and that's the phenomena that syntax tries to explain.
I see. Yeah, most of the way I think about how mind processes language comes from focusing on that "gut feeling", that on one hand tells you that this perfectly understandable sentence is somehow "off", and on the other hand lets you form perfect sentences without ever explicitly thinking about grammar.
> First, modern linguistics is very far from prescriptive. In fact the first thing they teach you (at around the same time they make the claim that "humans parse sentences into tree structure") is that linguistics is a descriptive field
It seems to me that I've been operating under invalid assumption that linguistics is mostly prescriptive. Thanks for that. Any recommendation for an intro book I could grab to read in my spare time?
Unfortunately I think the field suffers from a lack of such books.
1. You could try Steven Pinker's "The Language Instinct", although it's a general-audience book that doesn't really try to teach you linguistics proper
2. The first textbook I used was https://linguistics.osu.edu/research/pubs/lang-files and it's pretty good. However, it's quite hard to obtain.
Edit:
3. If you just want to look at syntax http://web.mit.edu/norvin/www/24.902/24902.html is advanced but good
You do need to analyze a sentence to understand it. Think of a classical attachment ambiguity such as "the boy saw the girl with the telescope". There are two readings of the sentence, and just like a Gestalt, you're typically perceiving it as one or the other. This involves a process of disambiguation, which is evidence that you have parsed the sentence.
https://github.com/tensorflow/models/tree/master/syntaxnet/s...
Doubt it had any other meaning other than the team just having fun making a quip at the Boaty Mcboatface poll for a ship's name.
I also wish middle management were a bit less afraid of such things. You can usually get cool ideas approved in the higher echolons (because a CEO doesn't have to fear too much, or possibly because these people actually are more imaginative).
Names that have meaning, tell a story are incredibly useful for marketing, even if they sometimes sound unprofessional. Exp: 'Plan B' (morning after pill), 'CockroachDB', 'Virgin'.
It's beyond me how anyone could have chosen the predictable public outcry instead of naming that boat Boaty McBoatface. That's probably the least offending name that ever resulted from an internet poll.
If Helpy McHelpdesk got into widespread use, it'd probably soon become more referred to as the HMH.
Indeed. e.g. http://knowyourmeme.com/memes/events/dub-the-dew
Remotey Remoteface?
Remotey Remoteface?
What kind of parser is it?
An English parser.
... :)
You know, it's a brilliant name. It's a person's name, like Siri or Cortana. But if they'd named it Jane or John, the mass media would have a fit. So they side-stepped that by making it sound computer-y, while remaining a proper name. Brilliant.
I am so glad to be rid of her vapid nonsense.
It is a silly name and makes me think it is not a serious effort. But names can be deceiving.
Seriously though, this is just a joke on an Internet meme. Quit being a buzzkill.
#otherlanguagesmatters
(INB4 yes, that's the reference to an old joke about Lufthansa pilot in Frankfurt complaining he's forced to speak in English to the control tower)
Honestly though, English is a quite good starting language due to its simplicity, but I do hope more attention will be expended on making the technology like this trainable on other languages as well.
Why would they have a fit if Google named it Jane or John?
According to the section titled "Why is Parsing So Hard For Computers to Get Right?" in the blog post, determining the difference between "An English parser" and "An English parser" .. is Parsey's great strength.
Five stars, would pun again.
How expensive is it to train a model like Parsey McParseface?
The problem is that while I have knowledge and experience in the computer vision side of machine learning, I lack experience in NLP. And to the best of my knowledge NLP as a field has not come as far as vision, to the extent that such an automated editor would have too many mistakes. To be student facing it would need to be really accurate. On top of that it wouldn't be dealing with well formed input. The input by definition is adversarial. So unlike SyntaxNet which is built to deal with comprehensible sentences, this tool would need to deal with incomprehensible sentences. According to the link, SyntaxNet only gets 90% accuracy on random sentences from the web.
That said, I might give SyntaxNet a try. The idea would be to use SyntaxNet to extract meaning from a broken sentence, and then work backwards from the meaning to identify how the sentence can be modified to better match that meaning.
Thank you Google for contributing this tool to the community at large.
EDIT: but sure, this is only the first step and semantic parsing is far from solved
If a teacher gave students a grammar-checking tool to check their writing, they might assume that the tool knew better than they did, which is only sometimes true.
"Learn the rules like a pro, so you can break them like an artist."
-Picaso
"And that quote is almost certainly made up."
- Willem Shakespere
With then-innovated art like Cubism, which in some sense broke earlier rules, the point is that those artists, like Picasso, were able to do representative art in correct proportions -- they were just going beyond that.
This is a nontrivial issue, because there are always students who think they can skip learning boring mundane old fashioned art and go straight to something like cubism, but in 99% of cases that doesn't work well at all compared with learning "the rules" first.
With writing, many great writers have broken "the rules" with punctuation, spelling, grammar, etc. But the important thing is that they do so on purpose.
Whereas if one doesn't know the rules in the first place, one doesn't have the choice of whether to follow them or break them.
Such a person will always break those rules they don't know (subconscious knowledge counts btw) -- but not for aesthetic reasons, only out of ignorance.
If you listen to it without paying attention to the text (based on a poem by Goethe by the same name; both the German text and an English translation is found at [1]), parts of it sounds like horrible jammering and poor harmonies and it's easy to write it off as not sounding very nice.
Here's [2] a much clearer rendition (two singers, with much stronger delineation of the three different characters) than the one linked from Britannica.
If you do pay attention to the text, it is very clear that the unpleasant parts are very deliberate:
The singer(s) switches between the role of a father, his sick dying child, and the Erl-king that occurs in the hallucinations of the child while the father is riding to bring the child to a doctor.
The big difference between the unpleasant-sounding parts of this song and a bad composer is the clear intent and delineation - Schubert made things sound bad intentionally explicitly at the points he wanted to illustrate pain and fear, rather than because he didn't know how to make things sound pleasant when he wanted to.
The song clearly proves this by setting the childs jammering and the fathers fearful attempts to soothe him up against much more pleasant segments where the Erl-king speaks and tries to seduce the child to come with him.
You only get that clear separation if you know how to evoke each effect precisely. Arguably a bad particularly composer wouldn't even know how to make things sound bad the "right way" - there's a big difference between random bad sounds and making things evoke a child in pain.
[1] http://www.britannica.com/topic/Erlkonig
[2] https://soundcloud.com/sean_contretenor_lee/erlkonig
I've heard great painters say that the only thing that matters is that you paint. Plenty of wonderful painters did not study their predecessors in depth.
I'm assuming that your 99% number is fabricated? Incidentally 99% of statistics are made up.
And come to think of it, isn't there a saying: Did stupidity require smart computers or did smart computers allow for stupidity?
An intelligent software (or one that pretends to be intelligent) might allow for any unqualified bloke to be in a position where they can teach.
[0]: http://grammarly.com
As an aside, I don't think this is the optimal way to teach people how to write. What were the ideas in those papers? How were they organized? Do the student's arguments make sense? I think that's what most students spend most of their time thinking about when writing an essay, and it can be a bit demoralizing to see the teacher care just as much about whether the grammar was right. Most students can fix grammar mistakes relatively easily once they notice them anyway.
You would be surprised! Especially with people whose first languages aren't English. Having something to provide feedback on grammar problems early on would be really useful. Of course, once they get grammar down, the next step is rhythm and flow, as well as reducing redundancy, and the biggest problem, as you say, is always the story, but you have to peel through lots of grammar problems before you get to that point. I edit a lot of research papers for my Chinese peers (most have PhDs, I work in a China-based research lab, so that isn't weird), so I'm pretty clear on the problems.
Visiting noredink.com now ... I can't actually figure out what it is the site offers from a cursory glance, so I can't even begin to figure out whether it matches my idea and what issues it may have had that excluded it from her classroom.
I'll bounce grammarly off her later though, in case that's new.
> This suggests that we are approaching human performance—but only on well-formed text.
It may fall down on exactly the bad writing you want to process. GIGO?
Curious - The parsing work I've done with programming languages was never done via machine learning, just the usual strict classification rules (which are used to parse ... code written to a strict specification). I'm guessing source code could be fed as data to an engine like this as a training model but I'm not sure what the value would be. Does anyone more experienced/smarter than me have any insights on something like that?
As a side-point:
Parsy McParseface - Well done. They managed to lob a gag over at NERC (Boaty McBoatface) and let them know that the world won't end because a product has a goofy name. Every time Google does things like this they send an unconscious remind us that they're a company that's 'still just a bunch of people like our users'. They've always been good at marketing in a way that keeps that "touchy-feely" sense about them and they've taken a free opportunity to get attention for this product beyond just the small circle of programmers.
As NERC found out, a lot of people paid attention when the winning name was Boaty McBoatface (among other, more obnoxous/less tasteful choices). A story about a new ship isn't going to hit the front page of any general news site normally and I always felt that NERC missed a prime opportunity to continue with that publicity and attention. It became a topic talked about by friends of mine who would otherwise have never paid attention to anything science related. It would have been comical, should the Boaty's mission turn up a major discovery, to hear 'serious newscasters' say the name of the ship in reference to the breakthrough. And it would have been refreshing to see that organization stick to the original name with a "Well, we tried, you spoke, it was a mistake to trust the pranksters on the web but we're not going to invoke the 'we get the final say' clause because that wasn't the spirit of the campaign. Our bad."
Artificial languages (such a programming languages) are usually designed to be unambiguous. In other words, there is a 1:1 mapping from a sentence or fragment to its abstract representation.
Natural language is ambiguous, so there is usually 1:N mapping from a sentence to abstract representations. So, at some point you need to decide which of the N readings is the most likely one.
Older rule-based approaches typically constructed all readings of a sentence and used a model to estimate which reading is the most plausible. In newer deterministic, linear-time (transition-based) parsers, such ambiguities (if any) are resolved immediately during each parsing step.
In the end it's a trade-off between having access to global information during disambiguation and having a higher complexity. So, naturally, the rule-based systems have been applying tricks to aggressively prune the search space, while transition-based parsers are gaining more and more tricks to incorporate more global information.
There is 6 links in this sentence in the original text. I get it can help to get more context around it, but I think it's actually making the text harder to "human" parse. It also feels they have hired a cheap SEO consultant to do some backlink integrations.
Can't work out if this is a joke.
So for example :
Yes = (0,0,1,0,0,...) No = (0,0,0,1,0,....)
Convolutional and recurrent nets can handle inputs of arbitrary lengths.
https://www.tensorflow.org/versions/r0.7/tutorials/word2vec/...
There's a bunch of blogs, tutorials, etc, around word2vec and other methods of generating vectors from a training set of words.
Also, in the tensorflow models codebase where this syntaxnet code lives, there is an another tensorflow-using-method of generating word embeddings with demonstration code called Swivel
https://github.com/tensorflow/models/tree/master/swivel
https://papers.nips.cc/paper/5346-sequence-to-sequence-learn...
For instance, the way they parse sequences of words may or may not be too specific to the English language. It is somewhat similar to what we call "overfitting" in the data-mining area, and it may invalidate this technique for other languages.
When I worked on this area (up to 2014), I worked mainly in language-independent statistical approaches. As with everything, it has its cons as you can extract information from more languages, but, in general, with less certainties.
But in general, it is good to see that the NLP area is still alive somewhere, as I can't seem to find any NLP jobs where I live! :)
Edit: I've read it in the diagonal, and it is based on a Neural Network, so in theory, if it was trained in other languages, it could return good enough results as well. It is normal for English/American authors to include only english datasets, but I would like to see an application to another language.. This is a very specialized domain of knowledge, so I'm quite limited on my analysis..
Nlp is very much alive and well.
Look at the March 2016 paper they cite.
Surprisingly, this thing is written in C++.