I might be wrong, but based on the parser home page it looks to me like they provide a Shift-reduce parser but it is not the default. I even says on the page Parser: englishPCFG.ser.gz.
I’m currently writing a paper on improving natural language interfaces for databases, and I’ve ran both Parsey McParseface and Stanford’s CoreNLP for that, and both have horrible performance.
Their results for even the most simple english sentences are horrible, often with multiple mistagged words.
As soon as you add relative clauses, they break down entirely.
Well, it's not really a solved problem, is it? Saying the performance is "horrible" does some injustice to it. It's actually the state of the art in solutions to a very difficult problem.
I've used the Stanford parser for university work (at Masters level). It was part of the pipeline for a sentiment extractor. I didn't get the feeling it performed badly, quite the contrary- the parses it generated were quite useful as features in my extractor. It had this rare feeling of a tool that you can actually use to do something interesting and useful.
Then again, I do have my expectations set very lowly, for this sort of thing, especially after completing my Masters thesis (on grammar induction). Language learning is hard.
I mean, it's surprising how well it works, but if you come in, and expect to get a perfect parse tree out of everything, so you can then easily pattern match on top of that, well, you'll be in for a rouch surprise.
Any every slightly more complicated sentence was completely misparsed by CoreNLP and ParseyMcParseface. As I mentioned before, as soon as you introduce relative clauses it breaks down.
It's hard when your research was supposed to discuss how to better handle the common knowledge problem of NLIDBs, but you have to spend a lot of time just to get a kinda useful parse out in the first place.
You usually can't tell from one single sentence if a parser handles garden path sentences, which involve backtracking (hence the name).
Plus, there is a part-of-speech tagger that provides an intermediate layer of nlp analysis, which makes a backtracking involving a change of part-of-speech («houses» being either a verb of a noun) quite harder (yet not impossible).
POS taggers improve parsers performance dramatically but usually are an indication that the parser have limited backtracking capabilities.
You can try it with «the complex houses married soldiers», houses is erroneously marked as a noun.
But, honest, that's a rotten way to try out NL parsers. Nobody claims Stanford's, or anyone's, parser can handle that sort of thing. It throws human language organs because it's specially engineered to do that. And if it can confuse humans, there's no program that it can't confuse, except for one where the right parse is hard-coded.
It is crazy slow and uses absurd amounts of memory, as well as being simultaneously an object lesson in both over-engineering, and under-engineering. Built by academics, so it's not unexpected.
Nice example of the application of recent NLP thesis topics to parsing problems, though. Quite suitable as documentation and reference for, e.g. future work on SpaCy - which is much more usable in productive contexts.
Yes, it does use a reasonable chunk of memory (I wouldn't run in with less than 3GB available)
However, speed wise, its only slow in my experience, if you try to integrate via a shell script - if you run it as a service and make http calls it's been amazingly fast in my usage.
I believe they include a "service" mode as part of the package now, but for a project a little while ago I forked & extended a project that provides a JSON/XML http service: https://github.com/Koalephant/StanfordCoreNLPHTTPServer
In my testing it struck a good balance between accuracy and performance. I was also impressed by the lengths the group goes to help people on their mailing list. That said, their licensing is not friendly so I generally have to rule it out.
I understand that -- as a parser, of course it's not its job to understand sarcasm or wit.
Just referencing the old Chomskian bit about statistical correlates and how relevant it is to understanding the fundamental principles behind the expression of language (rather than merely its syntax).
That's great. But let me ask, technically, how is the sentence "a grammatically correct sentence in American English" without punctuation? For NLP, shouldn't this first be regarded as a 'special case' of unpunctuated sentences? (and BTW, the parser doesn't handle the punctuated version any better).
It is in fact grammatically correct without punctuation, just somewhat awkward. Consider this analogous sentence with the same meaning, with optional parentheticals added to help you parse:
New York oxen (that) New York oxen bully (themselves) bully New York oxen.
the fact that nobody can understand without a preposition and a pronoun even after you replaced the uncommon words with common ones tells me this is a bunch a buffalo excrement that barely entertain linguist phds.
That sentence is virtually the same parse tree as "The man I saw crossed the street," which is a totally reasonable sentence to most English speakers. The difference in the parsing is that the "I" in this case corresponds to "Buffalo buffalo", a noun phrase--and the sentence is confusing because we have a tendency to add the "that" for noun phrases instead of single-word noun/pronouns, which the sentence is missing.
For me it was confusing the first time I read it because I didn't know that "buffalo" can mean "to bully". Without that information it doesn't make sense at all.
So I never thought there's something inherently difficulty in the syntax of the sentence- I thought it's just that unexpected third use of "buffalo" that makes it so strange.
Of course, now that I think of it, what you say makes sense also. For an NL parser, the variant use of "buffalo" shouldn't be an issue. You could even say that lacking the context of what a "buffalo" really is, a parser should even find it easier to parse the sentence than a human, who must pause to try and understand what it means.
Maybe it's not such a bad sentence to test parsers after all.
The Wikipedia article above says that though uncommon in usage, the buffalo verbs in that sentence mean "to bully, harass, or intimidate" or "to baffle".
AIUI, it's only "to bully" in the sense of verbally overwhelming someone into believing what you what them to believe. It's like the word "to snow" (in the sense that means "to con").
I would read "buffaloing" somebody as a less vulgar version of bullshitting, which is also a way of lying about something, mostly to get them to buy something they don't actually need.
Crucially, what is a reasonable sentence depends heavily on words.
"History lamps traffic people house cry Michigan algebra" also has the same parse tree, but if I utter this, people will probably think I need medical attention.
I would not have been able to tell which buffalos were verbs, which were nouns and which were proper nouns without the rephrasing of the sentence on Wikipedia. In fact even with that rephrasing I still find it difficult to tell which buffalo in the original sentence is what.
Not so. I've tried this on a few dozen students (post-secondary, many of them post-baccalaureate, perhaps 50% ESL) over the years; of those who have not heard it previously, all others say "what? that's not a sentence". I don't recall a single counter-example.
That's surprising to hear. I don't think I've ever found someone that understood it without an explanation. I like using it as an example of how confusing English can be to parse.
My main use case is unsupervised feature extraction from blob text. One application may be to extract all noun entities from a news article, so you can automatically figure out who the article is about.
Features you can use for other stuff, like entity detection.
E.g. AlchemyAPI has a cool demo, where they can detect names of diseases, cities, etc. I suspect the parts of speech would improve the accuracy for tagging words that have multiple meanings.
The assumptions we make, the more performance goes down in tasks like machine translation, and that includes the assumption that parts of speech matter to machines.
How difficult is it to use NLP to do keyword extraction for non-English languages? Any pointer to get started? (not NLP in general, but how to tailor it for non-English)
As long as you have a tokenizer, i.e. you can split words in a sentence, language doesn't matter for some basic techniques.
Two examples that easily come to mind: TF-IDF [1] and TextRank [2,3].
These are good to get started, but if you want to know more about the state of the art, searching on Google Scholar for "keyword extraction $language" is your best bet (and maybe "keyword extraction overview", also in Scholar).
Keyword extraction is not easy for other languages, but POS tagging and parsing is easy. There's a project called UDPipe which has pretrained models for most languages. That's the best "general" solution I've found.
My undergraduate research used Stanford Parser for getting word tags in a sentence before calculating its similarity with another sentence. Stanford Parser results are good, but it fails in some sentences. If you need a better alternative than this (for JVM languages), I would recommend NLP4J [1].
69 comments
[ 4.8 ms ] story [ 127 ms ] thread1: https://en.wikipedia.org/wiki/Garden_path_sentence
https://demos.explosion.ai/displacy/?text=The%20complex%20ho...
vs.
https://demos.explosion.ai/displacy/?text=The%20complex%20co...
Their results for even the most simple english sentences are horrible, often with multiple mistagged words.
As soon as you add relative clauses, they break down entirely.
I've used the Stanford parser for university work (at Masters level). It was part of the pipeline for a sentiment extractor. I didn't get the feeling it performed badly, quite the contrary- the parses it generated were quite useful as features in my extractor. It had this rare feeling of a tool that you can actually use to do something interesting and useful.
Then again, I do have my expectations set very lowly, for this sort of thing, especially after completing my Masters thesis (on grammar induction). Language learning is hard.
Any every slightly more complicated sentence was completely misparsed by CoreNLP and ParseyMcParseface. As I mentioned before, as soon as you introduce relative clauses it breaks down.
It's hard when your research was supposed to discuss how to better handle the common knowledge problem of NLIDBs, but you have to spend a lot of time just to get a kinda useful parse out in the first place.
Plus, there is a part-of-speech tagger that provides an intermediate layer of nlp analysis, which makes a backtracking involving a change of part-of-speech («houses» being either a verb of a noun) quite harder (yet not impossible).
POS taggers improve parsers performance dramatically but usually are an indication that the parser have limited backtracking capabilities.
You can try it with «the complex houses married soldiers», houses is erroneously marked as a noun.
It's an unfair test, is what I'm saying.
You could use this Dockerfile to run a local copy if you're interested: https://gist.github.com/charlieegan3/910276eef0f8658b44b42af...
Nice example of the application of recent NLP thesis topics to parsing problems, though. Quite suitable as documentation and reference for, e.g. future work on SpaCy - which is much more usable in productive contexts.
However, speed wise, its only slow in my experience, if you try to integrate via a shell script - if you run it as a service and make http calls it's been amazingly fast in my usage.
I believe they include a "service" mode as part of the package now, but for a project a little while ago I forked & extended a project that provides a JSON/XML http service: https://github.com/Koalephant/StanfordCoreNLPHTTPServer
I haven't seen too many things for NLP on .NET. I have run across a ton things on Python or C++/C. Would love to see more inn .NET.
Just referencing the old Chomskian bit about statistical correlates and how relevant it is to understanding the fundamental principles behind the expression of language (rather than merely its syntax).
I guess it was a bad joke gone over. ah well
Unfortunately with zero comments.
Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo
Stanford fails miserably on this one, even at the tagging step:
... but then, most humans also find that sentence really hard to parse.Check it out:
https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffal...
New York oxen (that) New York oxen bully (themselves) bully New York oxen.
So I never thought there's something inherently difficulty in the syntax of the sentence- I thought it's just that unexpected third use of "buffalo" that makes it so strange.
Of course, now that I think of it, what you say makes sense also. For an NL parser, the variant use of "buffalo" shouldn't be an issue. You could even say that lacking the context of what a "buffalo" really is, a parser should even find it easier to parse the sentence than a human, who must pause to try and understand what it means.
Maybe it's not such a bad sentence to test parsers after all.
"History lamps traffic people house cry Michigan algebra" also has the same parse tree, but if I utter this, people will probably think I need medical attention.
I put the following phrase
> I nut on my girlfriends face.
and the parser correctly tagged 'nut' as a VBP (i.e. Verb, non-3rd Person Singular resent) i.e. that 'nut' in this context is a verb, and not a noun.
Here's what the tagging returned (all words with tags)
I/PRP nut/VBP on/IN my/PRP$ girlfriends/JJ face/NN ./.
Tag Descriptions: http://stackoverflow.com/a/1833718/325521
I wonder how the other NLP tools parse that sentence. These are the results:
* Spacy [1]
* NLP4J [2] Well, that sentence really makes NLP tools confused.[1]: https://spacy.io/
[2]: https://emorynlp.github.io/nlp4j/
E.g. AlchemyAPI has a cool demo, where they can detect names of diseases, cities, etc. I suspect the parts of speech would improve the accuracy for tagging words that have multiple meanings.
I used the disease bit on AlchemyAPI for https://www.findlectures.com, to make facets for a health topics: https://www.findlectures.com/?p=1&class1=Science&category_l2...
http://nlpviz.bpodgursky.com/
How difficult is it to use NLP to do keyword extraction for non-English languages? Any pointer to get started? (not NLP in general, but how to tailor it for non-English)
Two examples that easily come to mind: TF-IDF [1] and TextRank [2,3].
These are good to get started, but if you want to know more about the state of the art, searching on Google Scholar for "keyword extraction $language" is your best bet (and maybe "keyword extraction overview", also in Scholar).
[1] http://www.joyofdata.de/blog/tf-idf-statistic-keyword-extrac...
[2] http://digital.library.unt.edu/ark%3A/67531/metadc30962/m2/1...
[3] https://github.com/davidadamojr/TextRank
[1]: https://emorynlp.github.io/nlp4j/