Reinforcement learning has an issue currently with modeling complicated aspects of an agent’s given simulation (their “world model”) over long time horizons.
This is probably because there is a vast amount of information in any given “frame” of this simulation and the agent has no baked-in priors about its environment (or if they do, it’s because they’ve been hand-programmed, which isn’t ideal in a field premised on the automation of such things).
If you instead pretrain _another_ model using the same tech as GPT3 on the agent’s environment, the agents can use this model during their simulations to model the world they see, essentially giving them something that looks kind of like common sense (although I certainly wouldn’t call it that to another researcher).
You wonder if that means intelligence really works like other bodily systems. Animals work so well not because of a single trick, but due to a big bag of tricks. Not "one single gene" or algorithms learning good performance at everything, but 1000 different algorithms and a good way of selecting between them. Not 1 or 2 or 10, but a large number.
11 comments
[ 3.0 ms ] story [ 43.0 ms ] threadThis is probably because there is a vast amount of information in any given “frame” of this simulation and the agent has no baked-in priors about its environment (or if they do, it’s because they’ve been hand-programmed, which isn’t ideal in a field premised on the automation of such things).
If you instead pretrain _another_ model using the same tech as GPT3 on the agent’s environment, the agents can use this model during their simulations to model the world they see, essentially giving them something that looks kind of like common sense (although I certainly wouldn’t call it that to another researcher).
Does that include Montezuma's Revenge? Because that's the Atari game RL agents completely fail at.
Edit: Yup, Atari 100k doesn't include MR.
https://twitter.com/arankomatsuzaki/status/14553550319019089...
If you allow MCTS, then EfficientZero performs +85% better. And if you look at the median (instead of the mean), then SPR performs +37% better.
But for some games, like ChopperCommand or Krull, IRIS (this paper) performs beyond comparison to other methods.