Yes and no. I personally feel like multi-modality and embodiment is the next step and has both incredible challenges as well as opportunities. The limitation for embodiment data is two-fold. Synthetic embodiment is distributionally limited. Physical embodiment is hardware limited. Either way, it's an interesting space to be in and I wouldn't be surprised if embodiment training has positive spillover effects in language comprehension along with improvements in other domains.
By synthetic embodiment, do you mean placing the agent into a simulated physical environment? E.g. something like mindcraft (https://github.com/kolbytn/mindcraft).
There is nearly an unlimited amount of data. For example, feed real world video/sensory data into LLMs and they can learn how real world physical objects behave. Simulate physics with a physics engine and then feed that data into the LLM.
The internet represented easy/low hanging fruit data. It doesn't seem like the transformer algorithm is limited to pure text training data.
Mining visual data—whether from CCTV, smartphone videos, or other sources—is the next big frontier. Given its immense information density, the potential payoffs could far exceed those of previous data revolutions.
6 comments
[ 3.2 ms ] story [ 22.5 ms ] thread1. https://waabi.ai/waabi-world/
The internet represented easy/low hanging fruit data. It doesn't seem like the transformer algorithm is limited to pure text training data.