> We also come up with a new version of BERT, called NumBERT, with improved numerical reasoning by replacing numbers in the pretraining text corpus with their scientific notation, which more readily exposes the magnitude to the model, and demonstrate that NumBERT representations capture scale significantly better than all those previous text representations.
That's really cool. I like the writing in this post. Concise and dense with info.
That's a heck of a question to expect an AI to answer. It's weird (we almost never discuss half animals), oddly phrased ("on the moon" is much more standard), and runs afoul of an imprecision in English (we normally given masses when asked about weight, this requires an answer in force, or else by analogy a mass with equivalent Earth-weight).
Bleh.. .English and it's weird grammar .. Sometimes I find it such a messed up language compared to my native tongue..... I guess it's the separation of modifiers and nouns that's part of the problem.
Thanks! I'm indeed not native :)
In my own defense it "felt" like it was missing something. However I am confident there are many moons and just one Moon, our own.
Also, the answer is 1.62kN to 4.8kN, I know Wolfram doesn't employ a language model or anything of a sorts, but with all the other NLP magic I've seen I sort of expected a valid answer.
It's because the name of the moon is "The Moon", not "Moon". I think SF writers sometimes pretend it's called "Luna" so it'll have a more interesting name.
It shouldn't really be that difficult. If it's capable of structuring "How heavy is _ on the moon?" The answer would be start out the same way looking for weight of _.
It seems to me that these kinds of questions could be handled but require some adjustments or tuning more than we need fundamentally different approaches. We can't do everything at once so there's going to be simple things that don't work for quite a while.
It is a tough question, but we keep hearing from AI-boosters about how the Singularity is right around the corner and have you seen this AI that can make plausible-looking text?
Well, yes. A B+ high-school student can also reliably recognise the number 710. If you think that AI should be able to do something just because a high-school student can, I think it's your expectations which are out of touch.
Needs a supervisor for cheeky monkey curve ball playful question to interpret answer on gradient of seriousness from thousands of mind model with access to knowledge library and language as commonly used. The part before "on" doesn't sound out of place from a template for language exchange in a butcher's shop and "on Moon" is a playful reframing adjusting the physics model. A child lucky to have a parent with training in physics would get an answer easy to this question.
Or it simply needs to be augmented with something other than GPT. The form of my question is very easily solvable in a programmatic way, just not by a GPT based neural net.
Someone mentioned multi model training, that sounds awesome!
Nice answer, but it actually supports the point being made here. NLP is an attempt to see how far we can go in processing natural language, without doing this sort of translation, and without structured knowledge bases. As (I'm pretty sure) Wolfram Alpha contains a structured knowledge base and draws on it for this sort of question (though not without help in this case), its abilities are orthogonal to the issues addressed in the article.
Well even if you take the most symmetric halves by volume, the density of an ideal elephant is not uniform. Some organs don’t come in pairs and all that
I guess it depends on how many hairs you want to split and how precise you want the answer
However most unpaired organs don’t contribute to a mass asymmetry. It’s thought this has evolved to aid with locomotion as any asymmetry will negative impact on efficiency
Evaluating the quality of language models is a challenge in itself. This post just presents another way to see how much your model can understand alongside a new model. This is typical when presenting new tech for which old evaluation methods might not tell you anything.
It's not really about getting the answer to that question as it is about figuring out how much information your model can glean from text.
I feel like it's fairly well-established that these text-only models cannot not have the capability of common sense reasoning because they don't incorporate sensorimotor/spatial/temporal/etc. knowledge. They can't because they are not trained with any of that, just text.
I did read the post. I guess I should have been more explicit. These types of incremental improvements seem to have limited utility and researchers who want to improve the models substantially may find more payoff in pursuing the integration of other types of data.
It's a clever technique, not data, so anyone can incorporate it. As for improvement the post shares its metrics and anyone could experiment on their own tasks to see if it brings improvement or not.
I find it interesting how hard it is to think of a question that truly cannot be answered by just reading lots of text. Stuff like "how can you carry water without a vessel?" (you freeze it!), which might stump a lot of AI models. But then you think... there should be texts that explain that ice is water and that ice can be carried. Similarly, the problem with numbers (lenghth, height, etc) isn't a lack of data, it's messy notation which is still easily parsed by human readers and probably could be solved.
If anything, I've become more convinced that text alone has enough information to solve some eerily specific problems. It's quite a powerful tool.
In an alternate universe, a really “smart” AI would be able to answer the “how do you carry water without a vessel” question, still not in the normal human way (you freeze it), but in the same vein as what you would find in Randall Munroe’s “How To” book.
That question is hard for current language models.
Theoretically you could get there but you'd need to connect a lot of dots in your algorithm. Throwing more text at existing algorithms won't reliably answer this kind of question.
One answer can be found in the issue being addressed here, which is that the models were not, just from reading lots of text, performing well on questions involving scale.
Fairly well-established, but not widely accepted; discussion around language models tends to ascribe all sorts of complex capabilities to them without evidence.
Yes. The combination of the two is quite a difficult task in itself, known as "grounding" - a mapping of an observation to a language concept. Progress is slow but it's being worked on (including by me)
I'm just starting to write my master's in robotics, and this is the topic I'm working on. Basically, grounding is the idea of trying to correlate (draw a connection between) a visual set of features (outputs from a vision system) and a set of identifiers (outputs from a language model). There are several techniques to do this!
A good starting point is a system that has a pretrained list of objects it can recognise. For example, we can train our image model what a book is, so when a user asks for a book, we can correlate the "bookness" feature (trained part of the output vector from the vision system) to it 1-1, and connect the bounding box from the vision system to the named entitity "book". However, say a user instead asked for a "novel". While this is also a book, we need a way for the two to be connected. For this, we use a quite different type of language model to connect the named entity "novel" to a list of known objects, one of which is "book". It does this by scoring how similar words are in the language model, and picking the closest match that is in the list of known objects. Over some threshold of similarity (measured by distance in abstract word-vector space), it is determined that this is a new, unknown object. What happens next varies, but is usually some variation of the system guiding the user to teach it what the object is that they meant (something like: point to the object, or put the object on top of a special evenly lit blank background for the visual system to learn it's features), so it is known for next time, and given an identifier from what the user called it.
This is the most basic form, but (as with everything) it gets more complicated. Two areas I am looking into at the moment are past reference language learning, which is about learning new visual features from users in operation. For example: user refers to this particular book as an "open textbook", later asks for a "closed textbook". Model knowns "open" and "closed" are mutually exclusive, so looks for a second match for book that is dissimilar in some visual features to the first, and learns some weak relation in those features to "open" and "closed", possibly to be reinforced later by repeated user expressions.
The other area I am looking at is learned spacial relation grounding. That is, trying to be able to resolve something like "The third book from the left" or "the book on top of the blue table". This is interesting because it breaks a lot of current grounding approaches, in that it requires some recursion (you can chain spatial descriptors in natural language e.g. "The book on the table in the room in the leftmost house on the hill...") and self-referential techniques. Ways of doing it aren't as well researched, and we're working on a method for doing it atm :)
This sounds so amazing! In a way you can use language relationships to identify visual training data.
Say someone asks your model to find pictures of a leash. If it’s never been trained on leashes it can use the language model to know they’re associated with dogs, attach to the collar, etc and actually find pictures of leashes and start training itself.
The possibilities are endless. Let me know if there is further reading I can do.
I don't know of any well written texts about this for laymen, but if you're up for a slightly tough (not too bad) read, this paper provides a nice introduction to the field from 2018, although a small part of it requires a little understanding of maths and statistics symbols.
Grounded Language Learning: Where Robotics and NLP Meet*
58 comments
[ 3.8 ms ] story [ 126 ms ] threadThat's really cool. I like the writing in this post. Concise and dense with info.
Wolfram alpha will literally answer.
Also while "the moon" could mean anything, there's only one Moon. I gave it a fighting chance.
I would expect any language model to be robust to minor errors like that though so I doubt it makes any difference.
Also, the answer is 1.62kN to 4.8kN, I know Wolfram doesn't employ a language model or anything of a sorts, but with all the other NLP magic I've seen I sort of expected a valid answer.
It seems to me that these kinds of questions could be handled but require some adjustments or tuning more than we need fundamentally different approaches. We can't do everything at once so there's going to be simple things that don't work for quite a while.
If that's beyond the capabilities of AI, then it obviously isn't living up to the hype.
I think you're lowering expectations for an AI to zero.
> 160 to 520 kg
There you go. Wolfram Alpha literally answered, just needed to translate your question to math ^_^
Our query to Wolfram Alpha should thus be:
(mass of elephant)/2 * moon gravity
I guess it depends on how many hairs you want to split and how precise you want the answer
I was looking for an average from the get-go, but a half is a half!
Half of the weight of an element would be well defined since it is just a number.
It's not really about getting the answer to that question as it is about figuring out how much information your model can glean from text.
If anything, I've become more convinced that text alone has enough information to solve some eerily specific problems. It's quite a powerful tool.
What? Even I couldn't answer that. I thought it was a brainteaser. "Water" in common parlance refers to its liquid form, as opposed to ice or steam.
Theoretically you could get there but you'd need to connect a lot of dots in your algorithm. Throwing more text at existing algorithms won't reliably answer this kind of question.
Aren’t people working on combining language and visual models now? It will be interesting to see how those do.
A good starting point is a system that has a pretrained list of objects it can recognise. For example, we can train our image model what a book is, so when a user asks for a book, we can correlate the "bookness" feature (trained part of the output vector from the vision system) to it 1-1, and connect the bounding box from the vision system to the named entitity "book". However, say a user instead asked for a "novel". While this is also a book, we need a way for the two to be connected. For this, we use a quite different type of language model to connect the named entity "novel" to a list of known objects, one of which is "book". It does this by scoring how similar words are in the language model, and picking the closest match that is in the list of known objects. Over some threshold of similarity (measured by distance in abstract word-vector space), it is determined that this is a new, unknown object. What happens next varies, but is usually some variation of the system guiding the user to teach it what the object is that they meant (something like: point to the object, or put the object on top of a special evenly lit blank background for the visual system to learn it's features), so it is known for next time, and given an identifier from what the user called it.
This is the most basic form, but (as with everything) it gets more complicated. Two areas I am looking into at the moment are past reference language learning, which is about learning new visual features from users in operation. For example: user refers to this particular book as an "open textbook", later asks for a "closed textbook". Model knowns "open" and "closed" are mutually exclusive, so looks for a second match for book that is dissimilar in some visual features to the first, and learns some weak relation in those features to "open" and "closed", possibly to be reinforced later by repeated user expressions.
The other area I am looking at is learned spacial relation grounding. That is, trying to be able to resolve something like "The third book from the left" or "the book on top of the blue table". This is interesting because it breaks a lot of current grounding approaches, in that it requires some recursion (you can chain spatial descriptors in natural language e.g. "The book on the table in the room in the leftmost house on the hill...") and self-referential techniques. Ways of doing it aren't as well researched, and we're working on a method for doing it atm :)
Say someone asks your model to find pictures of a leash. If it’s never been trained on leashes it can use the language model to know they’re associated with dogs, attach to the collar, etc and actually find pictures of leashes and start training itself.
The possibilities are endless. Let me know if there is further reading I can do.
Grounded Language Learning: Where Robotics and NLP Meet*
https://www.ijcai.org/Proceedings/2018/0810.pdf