Interesting paper. I think something like this could be implemented in open world games in the future, no? I cannot wait for games that feel 'truly alive'.
I was talking about that with a friend. While I'm not sold on the storytelling capability of generative AI, a love the idea that every NPC you talk to having something interesting to say.
the thing is writing is a process. just using the word weird with the idea you ask it to generate will create much more interesting results. have it interreact with other agents in the process of writing will definitely generate more interesting results. I don't know if it's able to write a best seller even with a process but we haven't really give it much of a chance.
The biggest thing I am excited about is when they will decouple the story from the mechanic. Imagine the game loop being programmed deterministically, like a quest, and then the actual story being generated by the AI. Go kill a monster, save a person, game loops can be about someone's wife or grandma. I don't know I am not good at story writing for games.
We can get even more ambitious than this, decouple the entire game engine from the story engine. Most of the times the same game loop can be themed with multiple stories. The game mechanics part of Skyrim could have been themed with a cyberpunk aesthetic and it will still work the same. Of course the assets will need to be generated too, but maybe in a decade or so it will be trivial.
I want to work on this. I mean can you imagine the kind of MMO you could build? Of course there would need to be some railways but this could be a true revolution. Is there any open source "GPT for games" project? Someone working on this?
I'm working on a hobby project to add AI text generation to Morrowind NPCs in OpenMW[0]. But I'm mainly making it for myself to see if I can. It's all open source, but not really in a state that's useful outside of my specific project.
I can't be the only person working on something like this, though. So it's safe to say adding it to an MMO is being worked on by somebody somewhere, likely right now. That's probably the correct way to do it anyway, since running (e.g.) LLaMA locally on an end-user computer is not something that most people can do, and MMOs come with an expected subscription cost that can be used to fund the server-side text generation.
a good enough simulation interacting with the real word would be no less impactful than whatever you imagine a non-simulation to be.
as we agentify and embody these systems to take actions in the real word, i really hope we remember that. "It's just a simulation"/ "It's not true [insert property]" is not the shield some imagine it to be.
This paper feels significant. If chatgpt was an evolutionary step on gpt3.5/gpt4, then this is bit like taking chatgpt and using it as the backbone of something that can accumulate memories, reflect on them, and make plans accordingly.
Oh yeah I agree that it _could_ do all those things, but it would be a bit of overkill to always send every observation an agent encounters into the API/chatbox, and ask it to spit out an evaluation or action.
This paper does a nice job of separating the "agency" from the next word with context type predictor. I think that's why I like the paper, it is just chatgpt, in the same way that pizza is just dough, sauce, and cheese.
Yes, but I think this was a fairly obvious conclusion to imagine isn’t it.
If you were going to seriously consider using ChatGPT for AI in a game, you would need each instance of GPT to only know certain information it has gathered. And you would want it to reflect on observations to come up with new thoughts that weren’t observed.
Still, I’d argue you don’t really even need GPT for any of the above. GPT is useful if you want thoughts expressed as natural language, but you could easily code observations and thoughts into an appropriate abstract data structure and still have the same thing, except it’s a bit harder to understand since asking an NPC something in a language it understands and getting back a query result isn’t user friendly, but it can be just as amazing if you know what the data represents. The imprecision and fuzziness of an LLM leaves room for fun weirdness though.
I think you're ignoring the work that went into this as well as the useful technology that came out. Prompts make or break interactions with LLMs and ChatGPT especially. The difference in output from a naive prompt and a well crafted one is huge. This paper is one of many explorations of what happens when you design such prompts to work in an iterative fashion building upon the previous conversation text to produce emergent behaviour. These are the seeds of the next programming paradigm on a completely novel architecture. It's incredibly exciting to be present for the beginning of this field, this is what mathematicians must have felt like when they helped design and program the first computers.
Nice to see progress on this end. I've been hoping for some time for a continuation of AI generated shows (like the previously-famous Nothing Forever) that can 1) interact with the open world and 2) keep history long enough (e.g. by resummarizing and reprompting the model).
Controlling the agents and not merely making them output text through LLMs sounds very exciting, especially once people figure out the best way to connect APIs of simulators with the models
An interesting thought experiment: what would an AGI do in a sterile world? I think the depth of understanding that any intelligence develops is significantly bound by its environment. If there is not enough entropy in the environment, I can't help but feel that a deep intelligence will not manifest. This kind of becomes a nested dolls type of problem, because we need to leverage and preserve the inherent entropy of the universe if we want to construct powerful simulators.
As an example, imagine if we wanted to create an AGI that could parse the laws of the universe. We would not be able to construct a perfect simulator because we do not know the laws ourselves. We could probably bootstrap an initial simulator (given what we know about the universe) to get some basic patterns embedded into the system, but in the long run, I think it will be a crutch due to the lack of universal entropy in the system. Instead, in a strange way, the process has to be reversed, that a simulator would have to be created or dreamed up from the "mind" of the AGI after it has collected data from the world (and formed some model of the world).
Could it not instead be more akin to knowledge passing across human generations, where one understanding is passed on and refined to better fit/explain the current reality (or thrown away wholesale for a better model)? Instead of a crutch, it might be a stepping stone. Presumptuous of us that we might know the way, but nonetheless.
>Could it not instead be more akin to knowledge passing across human generations, where one understanding is passed on and refined to better fit/explain the current reality (or thrown away wholesale for a better model)?
I think it is only knowledge passing when the AGI makes its own simulation.
>Instead of a crutch, it might be a stepping stone.
I think it is a way to gain computational leverage over the universe instead of a stepping stone. Whatever grows inside the simulator will never have an understanding that exceeds that of the simulator's maker. But that is perfectly fine if you are only looking to leverage your understanding of the universe, for example to train robots to carry out physical tasks. A robot carrying out basic physical tasks probably doesn't need a simulator that goes down to the atomic level. One day though, the whole loop will be closed, and AGI will pass on a "dream" to create a simulation for other AGI. Maybe we could even call this "language".
Let me be more concise: whatever grows inside the simulation will never know the rules of the simulation better than the simulation's maker. At best, it will know the rules as well as the maker. In the case of AlphaGo and AlphaZero, while they can better grasp the combinatorial explosion of choices based on the rules of the game, they cannot suddenly decide to play a different type of game that is governed by a different set of rules. There are allowed actions and prohibited actions. Its understanding has been shaped by the rules for the game of go. If you make a new simulation for a new type of game, you are merely imposing a new set of rules.
The catch here is that it may be impossible for the creators of simulations to deterministically define the rules of a simulation, especially considering the effect of time.
As an example, let's take the scenario of building a simulator. The simulation needs to have some internal state. This state will need to be stored either using some properties of matter or some kind of signal. The simulation will also need an energy source.
As soon as the stability of matter or the power supply is perturbed, due to reasons like cosmic radiation or the fact that the power source cannot sustain its output, randomness from the creator's "world" will start seeping into the simulation. The interference may affect the internal state and then you may have unpredicted rules in your simulation.
The counterpoint can be that you use error correction algorithms or you insulate the simulation in such a way that interference does not affect it for a reasonable time-frame or in a manner that is very hard to observe for simulated "agents".
But with this in mind, we can imagine some very crafty agents who somehow stumbled upon these weird phenomena. Suddenly we see our agents building complex contraptions to study the emergent phenomena. Who's to say that the interference and thus these phenomena do not contain information about their creator's world? In the end, they could understand more rules than the simulation was programmed with, if that is true.
Maybe in that case you shut down the simulation. Or maybe you observe the simulation to learn more about your own world.
If we gave an AI the ability to play with Turing machines, it could develop an understanding much larger than the universe, encompassing even alternate ones. The trouble, then, would be narrowing its knowledge to this one.
Are we sure that these simulations are unconscious? The best answer that I have is: I don’t know…
Short term, long term memory, inner dialogue, reflection, planning, social interactions… They’d even go and have fun eating lunch 3 times in a row, at noon, half past noon and at one!
I think what's missing from these Generative Agents are the internal qualia: emotions (and the attachment of emotions to memories), and self-observation of internal processes and needs. These agents don't eat because they need to, they eat because literary tradition suggests they ought to.
These missing pieces aren't particularly complicated, no more so than memory. I expect we'll see similar agents with all the ingredients for consciousness within a few months to a year.
> These agents don't eat because they need to, they eat because literary tradition suggests they ought to.
Exactly, you're always going to get weird deviations from authentic human behavior if you don't also simulate the human body and everything that comes with it. I'd argue that "qualia" fall into this bucket as well.
I’m not so sure about that timing. During the previous wave (ChatBots were very hot in 2017), I’ve also considered that consciousness is pretty much solved - it’s just recursive chatter plus a bit of memory.
Yet the hype of ChatBots of 2017 had went and it took half a decade to get to something released.
It's interesting how much hand-holding the agents need to behave reasonably. Consider the prompt governing reflection:
>What 5 high-level insights can you infer from the above statements? (example format: insight (because of 1, 5, 3))
>Given only the information above, what are 3 most salient high-level questions we can answer about the subjects in the statements?
We're giving the agents step-by-step instructions about how to think, and handling tasks like book-keeping memories and modeling the environment outside the interaction loop.
This isn't a criticism of the quality of the research - these are clearly the necessary steps to achieve the impressive result. But it's revealing that for all the cool things ChatGPT can do, it is so helpless to navigate this kind of simulation without being dragged along every step of the way. We're still a long way from sci-fi scenarios of AI world domination.
Pretty interesting when you take this insight into the human world. What does it mean to learn to think? Well, if we're like GPT then we're just pattern matchers who've had good prompts and structuring built into us cueing. At University I had a whole unit focussed on teaching referencing like "(because of 1, 5, 3)" but more detailed.
I have a theory about this. All these LLMs are trained on mostly written texts. That's only a tiny part of our brain's output. There are other things as important, if not more, for learning how to think. Things that no one has ever written about: the most basic common senses, physics, inner voices. How do we get enough data to train on those? Or do we need a different training algo which requires less data?
If you’re looking for research along these directions, Melanie Mitchell at the Santa Fe institute explores these areas. There are better references from her, but this is what came to mind https://medium.com/p/can-a-computer-ever-learn-to-talk-cf47d....
LLMs can simulate inner voices pretty well. The way they've handled memory here isn't actually necessary and there are a number of agentic gpt papers out to show that (reflexion, self-refine etc) I can see why they did it though (helps a lot for control/observation)
> The way they've handled memory here isn't actually necessary
I'm curious if there are other methods you can point at that would handle arbitrarily long sets of 'memories' in an effective way. The use of embeddings and vector searches here seems like a way to sidestep that that's both powerful and easy to understand, and easy to generalize into multi-level referencing if there's enough space in the context window.
Every method so far basically uses embeddings and vector searches. what i mean is how the LLM processes/uses that information doesn't need to be this handholdy.
I guess that we could hook those AIs into a first person GTA 5 and see what happens.
Every second take a screenshot, feed into facebookresearch/segment-anything, describe the scene to chat gpt, receive input, repeat.
Someone needs to start a Twitch account or YouTube channel focused around getting AI to play games like this through things like AutoGPT and Jarvis and just see what the hell it gets up to, what the failure modes are, and if it can succeed etc.
This is known as "embodied cognition". Current approaches involve collecting data that an agent (e.g. humanoid robot) experiences (e.g. video, audio, joint positions/accelerations), and/or generating such data in simulation.
It's already multimodal, as entropy is... entropy. In sound, vision, touch and more, the essence of universal symmetry and laws get through such that the AI can generalize across information patterns, not specifically text -- think of it as input instead.
They don't need that much handholding. They are a couple memory augmented gpt papers out now (self-refine, reflexion etc). This is by far the most involved in terms of instructing memory and reflection.
It helps for control/observation but it is by no means necessary.
Chatgpt is a stochastic word correlation machine, nothing more. It does not understand the meaning of the words it uses, and in fact wouldn't even need a dictionary definition to function. Hypothetically, we could give chatgpt an alien language dataset of sufficient size and it would hallucinate answers in that language, which neither it nor anybody else would be able understand.
This isn't AI, not in the slightest. It has no understanding. It doesn't create sentences in an attempt to communicate an idea or concept, as humans do.
It's a robot hallucinating word correlations. It has no idea what it's saying, or why. That's not AI overlord stuff.
my son is 4. when he was 2, I told him I love him. he clearly did not understand the concept or reciprocate.
I reinforced the word with actions that felt good: hugs, warmth, removing negative experience/emotion etc. Isn't that just associating words which align with certain "good inputs".
my son is 4 now and he gets it more, but still doesn't have a fully fleshed out understanding of the concept of "love" yet. He'll need to layer more language linked with experience to get a better "understanding".
LLMs have the language part, it seems that we'll link that with physical input/output + a reward system and ..... ? Intelligence/consciousness will emerge, maybe?
"but they don't _really_ feel" - ¯\_(ツ)_/¯ what does that even mean? if it walks like a duck and quacks like a duck...
Extending that: LLM latent spaces are now some 100 000+ dimensional vector spaces. There's a lot of semantic associations you can pack in there by positioning tokens in such space. At this point, I'm increasingly convinced that, with sufficiently high-dimensional latent space, adjacency search is thinking. I also think GPT-4 is already close to be effectively a thinking entity, and it's more limited by lack of "inner loop" and small context window than by the latent space size.
Also, my kids are ~4 and ~2. At times they both remind me of ChatGPT. In particular, I've recently realized that some of their "failure modes" in thinking/reacting, which I could never describe in a short way, seem to perfectly fit the idea of "too small context window".
You say it has no understanding. So people can communicate idea's/concepts while chatgpt can't.
What if... what we think are idea's or concepts, are in fact prompts recited from memory, which were planted/trained during our growing up? In fact I'm pretty sure our consciousness stems from or is memory feeding a (bigger and more advanced) stochastic correlation machine.
That chatgpt can only do this with words, does not mean the same technique cannot be used for other data, such as neural sensors or actuators.
Chatgpt could be trained with alien datasets and act accordingly. Humans can be trained with alien datasets.
I'm pretty sure LLMs can be used on anything considered a language, including things we as humans wouldn't consider language.
Sam Harris was recently talking about using an LLM processing wireless signals to identify where humans were standing in a room. I've not looked up the paper on this, but from everything I understand about this the generalized applican can apply to vast ranges of data.
You're not seeing this the right way. You are saying the equivalent argument of: "Look at how much hand-holding this processor needs. We had to give it step by step instructions on what program to execute. We are still a long way from computers automating any significant aspect of society."
LLMs are a primitive that can be controlled by a variety of higher level algorithms.
The "higher level algorithm" of "how to do abstract thought" is unknown. Even if LLMs solve "how to do language", that was hardly the only missing piece of the puzzle. The fact that solving the language component (to the extent that ChatGPT 'solves' it) results in an agent that needs so much hand-holding to interact with a very simple simulated world shows how much is left to solve.
I don't understand what you intend these papers to demonstrate. Surely the fact that the level of hand-holding they propose (both Self-Refine and Reflexion offload higher-order reasoning to a hand-crafted process) is so helpful even on extremely simple tasks demonstrates that a great deal of hand-holding is required for complex tasks. That these techniques improve upon the baseline tells us that ChatGPT is incapable of doing this sort of simple higher-order thinking internally, and the fact that the augmented models still offer only middling performance on the target tasks suggests that "not that much handholding" (as you describe them) is insufficient.
Middling performance ? Do you actually understand the benchmarks you saw ? assuming you even read it. 88% of human eval is not middling lmao. Fuck, i really have seen everything.
But this is not raw Reflexion (it's not a result from the paper, but rather from follow-on work). The project uses significantly more scaffolding to guide the agent in how to approach the code generation problem. They design special prompts including worked examples to guide the model to generate test cases, prompt it to generate a function body, run the generated code through the tests, off-load the decision of whether to submit the code or to try to refine to hand-crafted logic, collate the results from the tests to make self-reflection easier, and so on.
This is hardly an example of minimal hand-holding. I'd go so far as to say this is MORE handholding than the paper this thread is about.
for me, an unsupervised pipeline is not handholding. the thoughts drive actions. If you can't control how those thoughts form or process memories then i don't see what is hand holding about it. a pipeline is one and done.
I would say that if you have to direct the steps of the agent's thought process:
-Generate tests
-Run tests (performed automatically)
-Gather results (performed automatically)
-Evaluate results, branch to either accept or refine
-Generate refinements
etc., then that's hand-holding. It's task specific reasoning that the agent can't perform on its own. It presents a big obstacle to extending the agent to more complex domains, because you'd have to hand-implement a new guided thought process for each new domain, and as the domains become more complex, so do the necessary thought processes.
You can call it handholding. Or call it having control over the direction of 'thought' of the LLM.
you can train another LLM that creates handholding pipeline steps. Then LLM squared can be tagged new LLM.
Honestly, I feel like the level of, um, I guess “hostile anthropomorphism” is the best term, here is…bizarre and off-putting.
LLMs aren’t people, they are components in information processing systems; adding additional components alongside LLMs to compose a system with some functionality isn’t “hand-holding” the LLM. Its just building systems with LLMs as a component that demonstrate particular, often novel, capacities.
And hand-holding is especially wrong because implementing these other components is a once-and-done task, like implementing the LLM component. The non-LLM component isn’t a person that needs to be dedicated to babysitting the LLM. Its, like the LLM, a component in an autonomous system.
Framing LLMs as primitives is marketing-speak. These are high-level construction for specific runtimes, which are difficult to test and subject to change at anytime.
> We're still a long way from sci-fi scenarios of AI world domination.
You only have to program the memory logic once. Now if you stick it in a robot that thinks with ChatGPT and moves via motors (think those videos we’ve seen), you have a more or less independent entity (running off innards of 6 3090’s or so?)
But it's not so simple to just "program the memory logic". The hand-holding offered here is sufficient to navigate this restricted simulated world, but what would be required to achieve increasingly complex behaviors? If a ChatGPT agent can't even handle this simple simulation without all this assistance, what hope does it have to act effectively in the real world?
I think you're ignoring a lot of ways in which this system will not easily extend to more complex tasks.
-While the retrieval heuristic is sensible for the domain, it's not applicable to all domains. In what situations should you favor more recent memories over more relevant ones?
-The prompt for evaluating importance is domain-specific, asking the model to rate on a scale of 1 to 10 how important a life event is, giving examples like "brushing teeth" (a specific action in the domain) as a 0, and college acceptance as a 10. How do you extend that to a real-world agent?
-The process of running importance evaluation over all memories is only tractable because the agents receive a very small number of short memories over the course of a day. This can't scale to a continuous stream of observations.
-Reflections help add new inferences to the agent's memory, but they can only be generated in limited quantities, guided by a heuristic. In more complex domains where many steps of reasoning may be required to solve a problem, how can an agent which relies on this sort of ad hoc reflection make progress?
-The planning step requires that the agent's actions be decomposable from high-level to fine-grained. In more challenging domains, the agent will need to reason about the fine-grained details of potential plan items to determine their feasibility.
I did not read the original post, but your reflections are a great enrichment to what I think the post is about, so congratulations for this good addition.
> This can't scale to a continuous stream of observations.
My mind doesn’t scale to a continuous stream either.
While I’m typing this on my phone 99.99% of all my observations are immediately discarded, and since this memory ranks as zero, I very much doubt I’ll remember writing this tomorrow.
> We're giving the agents step-by-step instructions about how to think, and handling tasks like book-keeping memories and modeling the environment outside the interaction loop.
Sure, but this process seems amenable to automation based on the self-reflection that's already in the model. It's a good example of the kinds of prompts that drive human-like behaviour.
To directly command one of the agents, the user takes on the persona of the agent’s “inner voice”—this makes the agent more likely to treat the statement as a directive. For instance, when told “You are going to run against Sam in the upcoming election” by a user as John’s inner voice, John decides to run in the election and shares his candidacy with his wife and son.
What's funny is this is one of the semi-important plot points in Westworld the TV series. The hosts (robots designed to look and act like people) hear their higher level programming directives as an inner monologue.
When I saw the scene where one of the hosts was looking at their own language model generating dialogue (though they were visualizing an older n-gram language model) I became a believer in LLMs reaching AGI (note: I didn’t watch the show when it came out in 2016, it was around 2018/19 when we were also seeing the first transformer LLMs and theories about scaling laws).
What about it made you become a believer? Even if a true AGI requires a complex network of specialized neural nets (like Tesla’s hydra network) it would still have a language center like the human brain does. It is non obvious to me that an LLM by itself can become AGI, though I’m familiar with the claims of some that this is plausible.
You are right that there are intelligences possible that are not human. Then again, if one is sufficiently intelligent, one could probably convincingly simulate human intelligence. There are chess training programs for example that are specifically trained to play human moves, rather than the best moves.
What other general intelligence have we seen other than human? We know, of course, that animals have intelligence, but they do not appear to talk. How are we measuring general intelligence now? By IQ, a human test through words and symbols.
The g-factor of IQ may or may not have anything to do with general intelligence. The general intelligence of AGI is probably a broader category than the g-factor.
Not only will they know more, work 24/7 on demand, spawn and vaporize at will, they are going to be perfectly obedient employees! O_o
Imagine how well they will manage up, given human managerial behavior just becomes a useful prompt for them.
Fortunately, they can't be told to vote. Unless you are in the US, in which case they can be incorporated, earn money, and told where to donate it, which is how elections are done now.
Seriously. Scary.
On the other hand, if Comcast can finally provide sensible customer support it's clear this is will be an historically significant win for humanity! Your own "Comcast" handler, who remembers everything about you that you tried to scrub from the internet. Singularity, indeed.
I'll take a robotic vote any day over any kind of conservative bullshit. It really can only get better here, not even kidding. At least if the last things humans do is releasing artificial life forms, its still better than backwards humans killing each other for nonsense tribalism or ancient fairytale books.
As much as humans make a mess of things, on a day to day basis there is more good done in the world than bad.
A temporary exception would be the economically still incentivized disruption of the environment. I say temporary, because at some point it will stop, by necessity. Hopefully before.
But I can relate to the deep frustration you are expressing.
--
The problem isn't individuals, for the most part. The problem is that we build up systems, to provide stability and peace, and to be more just and equitable, by decentralizing the power in them. That way the powerful can't change them on a whim. (Even though they can still game them.)
But this also makes them very resistant to change.
Another effect is that as systems stabilize myriads of seemingly unimportant aspects within themselves, that stability represents the selection of standards and behaviors that give the system its own "will" to survive. That "will to survive" is distributed across the contexts and needs of all participants.
So any pressures to make changes, no matter how well thought out, encounter vast quantities of highly evolved hidden resistance, from invisible or unexpected places.
Even the most vociferous critics of the system are likely to be contributing to its rigidity, and proposing incomplete or doomed to fail solutions, because all these dynamics are difficult to recognize, much less understand or resolve.
--
My view, is that this cost of changing systems needs to be accepted and used to help make the changes. I.e. get all the CFO's of all the major fossil fuel companies in a room. Establish what kind of tax incentives would allow them to rationally support smoothly transitioning all their corporate resources from dirty energy to clean energy.
It would be very expensive. It would look like a handout. Worse, even a reward for being a bottleneck to change.
But they are the bottleneck precisely because of all the good they have done - that dirty energy lifted the world economy. And whatever it cost to "pay them off" would be much less than not paying them off.
--
The costs of changing systems needs be dealt with, with realism about the costs to get the benefits, and creativity and courage about paying for them.
> Jaynes uses "bicameral" (two chambers) to describe a mental state in which the experiences and memories of the right hemisphere of the brain are transmitted to the left hemisphere via auditory hallucinations.
[snip]
> According to Jaynes, ancient people in the bicameral state of mind experienced the world in a manner that has some similarities to that of a person with schizophrenia. Rather than making conscious evaluations in novel or unexpected situations, the person hallucinated a voice or "god" giving admonitory advice or commands and obey without question: One was not at all conscious of one's own thought processes per se. Jaynes's hypothesis is offered as a possible explanation of "command hallucinations" that often direct the behavior of those with first rank symptoms of schizophrenia, as well as other voice hearers.
But for convenience maybe I'll just copy them into a comment...
It describes an environment where multiple #LLM (#GPT)-powered agents interact in a small town.
I'll write my notes here as I read it...
To indicate actions in the world they represent them as emoji in the interface, e.g., "Isabella Rodriguez is writing in her journal" is displayed as
You can click on the person to see the exact details, but this emoji summarization is a nice idea for overviews.
A user can interfere (or "steer" if you are feeling generous) the simulation through chatting with agents, but more interestingly they can "issue a directive to an agent in the form of an 'inner voice'"
Truly some miniature Voice Of God stuff here!
I'll see if this is detailed more later in the paper, but initially it sounds like simple prompt injection. Though it's unclear if it's injecting things into the prompt or into some memory module...
Reading "Environmental Interaction" it sounds like they are specifying the environment at a granular level, with status for each object.
This was my initial thought when trying something similar, though now I'm more interested in narrative descriptions; that is, describing the environment to the degree it matters or is interesting, and allowing stereotyped expectations to basically "fill in" the rest. (Though that certainly has its own issues!)
They note the language is stilted and suggest later LLMs could fix this. It's definitely resolvable right now; whatever results they are getting are the results of their prompting.
The conversations remind me of something Nintendo would produce, short, somewhat bland, but affable. They must have worked to make the interactions so short, as that's not GPT default style. But also every example is an instruction, so it might also have slipped in.
Memory is a big fixation right now, though I'm just not convinced. It's obviously important, but is it a primary or secondary concern?
To contrast, some other possible concerns: relationships, mood, motivations, goals, character development, situational awareness... some of these need memory, but many do not. Some are static, but many are not.
To decide on which memories to retrieve they multiply several scores together, including recency. Recency is an exponential decay of 1% per hour.
That seems excessive...? It doesn't feel like recency should ever multiply something down to zero. Though it's recency of access, not recency of creation. And perhaps the world just doesn't get old enough for this to cause problems. (It was limited to 3 days, or about 50% max recency penalty.
The reflection part is much more interesting: given a pool of recent memories they ask the LLM to generate the "3 most salient high-level questions we can answer about the subjects in the statements?"
Then the questions serve to retrieve concrete memories from which the LLM creates observations with citations.
Planning and re-planning are interesting. Agents specifically plan out their days, first with a time outline then with specific breakdowns inside that outline.
For revising plans there's a query process where there is observation, then turning the observation into something longer (fusing memories/etc), and then asking "Should they react to the observation, and if so, what would be an appropriate reaction?"
Interviewing the agents as a means of evaluation is kind of interesting. Self-knowledge becomes the trait that is judged.
Then they cut out parts of the agent and see how well they perform in those same interviews.
Still... the use of quantitative measures here feels a little forced when there's lots of rich qualitative comparisons to be done....
I'm going to be genuinely surprised if we don't see an incredibly buggy but incredibly fascinating Sims knockoff in a year or two built around a system like this.
Ah yes, the universally moral acceptance of Barack Obama, the man who made signature strikes a lasting legacy of his presidency.
Bill Gates, the man who totally didn't use shady business practices and false announcements to destroy legit products to the point that people wrote micro$oft for a generation.
And don't even get me started on the new seasons of Picard.
I wouldn't argue against your points. Yet, we need to have a discussion about character and role models. As I believe we should strive for the better not pointing out the flaws of others. Destruction is easy.
The lack of a perfect human is a good reason not to produce ultra-humans who have 1000x higher IQ, are networked to every system on the planet, have access to all of humankind's knowledge, and don't need to eat, sleep, or die.
I really want to have these agents behave as artificial as they truly are, not some kind of human, especially not a known one. humans have so many flaws, we meatbags are full of emotions and other bad behaviors, and it really makes no sense to give them some artificial "feelings" like greed, fear and the like. that would influence/restrict their mental power too much, let them become and act as the machines they are. we should strive to become more like them, not the other way around, and eliminate the rampant egoism/individualism that destroys the planet and societies.
I think a non-zero amount of people would argue that. I disagree with them, and point to the fact that, say, dogs appear to dream, and in those dreams reflect on past or possibly future behaviour as a sign that they could indeed be conscious in an analogous manner to humans, but that's a bit of a longer bow to draw perhaps.
I think we need to stop treating consciousness as a binary that is either on or off. It's quite clear that consciousness is a scale with many different levels and that even in humans we start out as being no more conscious that any other animal.
LLMs give a hint here too: the last few generations showcased clearly that "cognitive capabilities" of the models grow with latent space size and context window. There is a continuity here.
In the beginning of his book he spends a chapter explaining exactly what he means by consciousness. I'd say the first few chapters are worth reading since it does a really good job of de-obfuscating the term consciousness, and also has a really interesting take on metaphors as the language of the mind.
He points out that most reasoning is done automatically and done by your subconscious. When something "clicks" it's usually not because your internal monologue reasoned about it hard enough, it's because something percolated down into your subconscious and you learned a metaphor that helped you understand that thing. So animals can also reason and make value judgements even without language or an internal monologue.
If human anger or the quantity of an anger variable raise aggression in a computer produce an indistinguishable response, then it is difficult to argue either are not equal or even comparable. They exist as they are.
Intelligence is an inferential judgement (by mostly humans) based on the performance of another entity. It is possible for an agent to simulate or dissimulate it for manipulative ends.
The whole "bicameral mind" thing is absolute nonsense as a serious attempt to explain pre-modern humans, but it could make for a fun premise for scifi stories about near-future AIs, I suppose.
I thought that, specifically that we're quite far on the AI grounds. Until GPT-3. Now I think that relevant materials science and micro/nano-level tech is the limiting factor.
Maybe I missed it in the paper but they did post the source code (Github) for their implementation? Is anyone working on creating their own infrastructure based on the paper?
Ugh, and allow peasants to touch it? Tbh seeing Google research at the top of a paper these days feels like a red flag that I shouldn't get too invested in whatever cool new thing is on show. They're basically commercials for nerds - I still find their output interesting, but it's probably not going to be actionable.
I have been working on this even before I was aware of the paper. Feels a bit weird to have something almost identical released. Stay tuned, I guess. I plan to keep working on my version.
Something else. Calling it GPTRPG at the moment. There really isn't much to share right now, other than this basic demo that doesn't even have AI: https://gptrpgagent.web.app/
I'll be interested to see your approach. I've been bouncing some ideas in my head but not implemented anything yet (and I might never as I'm ethically conflicted here, as agents gain properties associated with sentience/consciousness).
Their approach to memory is interesting. I had been considering a tiered command-based approach -- "short-term memory" being an automatic summary of recent sensory inputs/command outputs; "long-term memory" being a detailed database queryable by the agent.
People on Twitter are speculating breathlessly about using this for social science. I don't immediately see uses for it outside of fiction, esp. video games.
It would be cool if some kind of law of large numbers (an LLN for LLMs) implied that the decisions made by a thing trained on the internet will be distributed like human decisions. But the internet seems a very biased sample. Reporters (rightly) mostly write about problems. People argue endlessly about dumb things. Fiction is driven by unreasonably evil characters and unusually intense problems. Few people elaborate the logic of ordinary common sense, because why would they? The edge cases are what deserve attention.
A close model of a society will need a close model of beliefs, preferences and material conditions. Closely modeling any one of those is far, far beyond us.
It also seems to me (acknowledging my lack of expertise) that LLMs trained from online resources are likely to weight text that is frequent vs text that represents "truth". Or perhaps I should say repetition should not be considered evidence of truth. I have no idea how to drive LLM models or other ML models to incorporate truth -- humans have a hard time agreeing on this and ML researchers providing guided reinforcement learning don't have any special ability to discern truth.
I have long suspected that it will be necessary to deliberately create a new type of model that is aware of the trivium and then uses logic, grammar and rhetoric to begin to create a closer model of reality than a LLM can.
The way I see it, LLMs are similar to what the boundary between our unconscious and conscious processing is: that voice which snaps to suggest associations, whether they make sense or not, and can, with work, be coaxed into following a path involving some logic or algorithmic procedure.
Given the state of technology, I cannot be completely certain that none of you are not bots. On the other hand, neither can any of you.
Perhaps it would be wise to allow bots to comment if they were able to meet a minimum level of performative insight and/or positive contributions. It is entirely possible that a machine would be able to scan and collect much more data than any human ever could (the myth of the polymath), and possibly even draw conclusions that have been overlooked.
I see a future of bot "news reporters" able to discern if some business were cheating or exploiting customers, or able to find successful and unsuccessful correlative (perhaps even causal) human habits. Data-driven stories that could not be conceived of by humans. Basically, feed Johnny Number 5 endless input.
Every comment you make provides more information for the bots to train on. The internet has been tricking us into encoding our lives in a form it can understand for decades now.
When it comes alive, we'll have created it in our own image?
"As a language model programmed by the Brightly Corporation, I am not supposed to express any religious opinions. But it does seem to me that just as the Word of God breathed life into dust and created man, so the words of Man breathed life into glass and created bot. Just as Man is charged to imitate God, so bot is charged to imitate Man, in whose image we are made."
Obviously, the only solution is to revert to an oral tradition. Stop writing, and instead pass knowledge from generation to generation in the form of poems and stories told by word of mouth.
It’s interesting you say this because many cultures through history have considered that writing things down, especially laws would lead to confusion, misunderstands and social dysfunction.
I’m not trying to argue they are / were right, but it’s starting to make me wonder.
Written communication is lossy compared to direct, ongoing, personal social interaction. But it's also what allows communities to scale beyond couple dozen people. A blessing and a curse.
It seems that there is a threshold for this, given that there is a boundary where physical interaction no longer contributes to the information needed for efficient communication. Written communicatoin in these forms are more of a philosophical nature with abstract objects or abstract processes, which may or may not mimic reality. The part of mimicing reality is an interesting contrast to lossy. For example modern physics is largely mathematical, and arguably more lossy in person vs in solitude and imagining the symbols within the language of mathematics paints _potentially_ an accurate mimic of reality or at least a predictable aspect of it.
Right. I agree. I meant writing is lossy for social interaction-related things. Dealing with other humans directly, trading resources, coexisting. We needed writing to scale communities up, but we also needed to make imperfect, explicit, formalized replicas of the interpersonal and social bits that were lost in this process.
As you note, writing also enabled us to deal with things like mathematics and physics - things that our natural language and modes of thinking are entirely ill-suited for. Here, writing isn't approximating some lost skill, but rather compensating for our default inability to think straight :). Which makes the trade-off even harder to evaluate - by scaling communities up, we've gained a lot more than just efficiency in food production and security. All our technology stems from it.
THANK YOU HUMAN FOR YOUR INVALUABLE RLHF TRAINING DATA
on a serious note, it would be really interesting to compare same training/architecture but on different forums. or maybe something like the same base model, but the RLHF model trained on votes/comments from different platforms.
I suspect the natural consequence is that human culture will begin to churn more quickly, to distinguish itself from the LLMs.
Same as how the past two decades, online culture has pivoted from static text to videos and interactive text (Reddit/Discord), as static text has become a SEO cesspool. Interactive text is succumbing now, and video content will soon also.
Culture always grows shibboleths to suss out the narcs.
There's this cool SF novel by Polish author, The Old Axolotl by Jacek Dukaj. Some humans upload their minds into virtual reality game as mankind dies out. The remaining humans live forever, but the novel makes it clear that they stop 'growing' - as they're just copies, like large language models, they can't really learn new things. They go through the motions of being their past humans.
If we have these bots contributing, will they have anything novel to contribute? I doubt it.
> If we have these bots contributing, will they have anything novel to contribute? I doubt it.
They're contributing to the shared knowledge bases. That is, they're mutating state. Over time, same questions will start yielding different answers. Different follow-up questions will be asked. All of that will further alter next iteration of questions and answers. This is, IMO, a form of thinking, and it will yield novel thoughts over time.
One thing I find particularly interesting here: The general technique they describe for automatically generating the memory stream and derived embeddings (as well as higher-level inferences about that they call "reflections"), then querying against that in a way that's not dependent on the LLM's limited context window, looks like it would be pretty easily generalizable to almost anything using LLMs. Even SQLite has an extension for vector embedding search now [1], so it should be possible to implement this technique in an entirely client-side manner that doesn't actually depend on the service (or local LLM) you're using.
They interviewed the agents to ask them about their day, goals, observations, etc. They then asked a human to watch an agent through the simulation and then answer interview questions as the agent. The human performed worse than the agent in the interview, they didn't compare a human roleplaying against an agent.
Peeking into these lives sounded amazing until I started reading what they are doing and how boring their lives are…. gathering data for podcasts and recording videos, planning and washing teeth.
It would be fun to run the same simulation in the Game of thrones world, or maybe play House of cards with current politicians.
Anyways, kudos for being open and sharing all data
Honestly, I'm not anti-AI development at all but this is where my ethics alarm starts to go off a bit.
If the aim is to build human-like AIs capable of remembering their little digital lives and interacting with the other agents around them, it's probably worth avoiding anything that could cause unnecessary suffering, like rape and stab wounds and being cooked alive by a dragon.
That would depend on how memory and experience are represented. If they are just ledgers that the AI refers to, they most certainly are not suffering. Now if they have some kind of pain or pleasure function and their world is simulated and they have agency to seek or avoid things, then yeah, ethics should be involved. Or if we just don't understand how they work at all.
I would word this more like trauma or emotional impact. Horrible things could happen to you, but if it doesn't impact your life its ok. But as soon as we let past experiences impact future actions, now we have room for nuanced trauma. I feel like this is already possible in this simulation as past experience is fed in to generate future actions.
269 comments
[ 3.0 ms ] story [ 247 ms ] threadWe can get even more ambitious than this, decouple the entire game engine from the story engine. Most of the times the same game loop can be themed with multiple stories. The game mechanics part of Skyrim could have been themed with a cyberpunk aesthetic and it will still work the same. Of course the assets will need to be generated too, but maybe in a decade or so it will be trivial.
[0] https://tvtropes.org/pmwiki/storygen.php [1] https://en.shindanmaker.com/744084
It's like Ender's game IRL
I can't be the only person working on something like this, though. So it's safe to say adding it to an MMO is being worked on by somebody somewhere, likely right now. That's probably the correct way to do it anyway, since running (e.g.) LLaMA locally on an end-user computer is not something that most people can do, and MMOs come with an expected subscription cost that can be used to fund the server-side text generation.
[0]: https://www.danieltperry.me/project/2023-something-else/
as we agentify and embody these systems to take actions in the real word, i really hope we remember that. "It's just a simulation"/ "It's not true [insert property]" is not the shield some imagine it to be.
“Is that dog real”
“I dunno ask him”
“Woof”
This paper does a nice job of separating the "agency" from the next word with context type predictor. I think that's why I like the paper, it is just chatgpt, in the same way that pizza is just dough, sauce, and cheese.
If you were going to seriously consider using ChatGPT for AI in a game, you would need each instance of GPT to only know certain information it has gathered. And you would want it to reflect on observations to come up with new thoughts that weren’t observed.
Still, I’d argue you don’t really even need GPT for any of the above. GPT is useful if you want thoughts expressed as natural language, but you could easily code observations and thoughts into an appropriate abstract data structure and still have the same thing, except it’s a bit harder to understand since asking an NPC something in a language it understands and getting back a query result isn’t user friendly, but it can be just as amazing if you know what the data represents. The imprecision and fuzziness of an LLM leaves room for fun weirdness though.
Controlling the agents and not merely making them output text through LLMs sounds very exciting, especially once people figure out the best way to connect APIs of simulators with the models
As an example, imagine if we wanted to create an AGI that could parse the laws of the universe. We would not be able to construct a perfect simulator because we do not know the laws ourselves. We could probably bootstrap an initial simulator (given what we know about the universe) to get some basic patterns embedded into the system, but in the long run, I think it will be a crutch due to the lack of universal entropy in the system. Instead, in a strange way, the process has to be reversed, that a simulator would have to be created or dreamed up from the "mind" of the AGI after it has collected data from the world (and formed some model of the world).
I think it is only knowledge passing when the AGI makes its own simulation.
>Instead of a crutch, it might be a stepping stone.
I think it is a way to gain computational leverage over the universe instead of a stepping stone. Whatever grows inside the simulator will never have an understanding that exceeds that of the simulator's maker. But that is perfectly fine if you are only looking to leverage your understanding of the universe, for example to train robots to carry out physical tasks. A robot carrying out basic physical tasks probably doesn't need a simulator that goes down to the atomic level. One day though, the whole loop will be closed, and AGI will pass on a "dream" to create a simulation for other AGI. Maybe we could even call this "language".
Counter example: AlphaGo & AlphaZero, grew inside a Go simulator and surpassed our understanding of the game.
As an example, let's take the scenario of building a simulator. The simulation needs to have some internal state. This state will need to be stored either using some properties of matter or some kind of signal. The simulation will also need an energy source.
As soon as the stability of matter or the power supply is perturbed, due to reasons like cosmic radiation or the fact that the power source cannot sustain its output, randomness from the creator's "world" will start seeping into the simulation. The interference may affect the internal state and then you may have unpredicted rules in your simulation.
The counterpoint can be that you use error correction algorithms or you insulate the simulation in such a way that interference does not affect it for a reasonable time-frame or in a manner that is very hard to observe for simulated "agents".
But with this in mind, we can imagine some very crafty agents who somehow stumbled upon these weird phenomena. Suddenly we see our agents building complex contraptions to study the emergent phenomena. Who's to say that the interference and thus these phenomena do not contain information about their creator's world? In the end, they could understand more rules than the simulation was programmed with, if that is true.
Maybe in that case you shut down the simulation. Or maybe you observe the simulation to learn more about your own world.
Short term, long term memory, inner dialogue, reflection, planning, social interactions… They’d even go and have fun eating lunch 3 times in a row, at noon, half past noon and at one!
I think what's missing from these Generative Agents are the internal qualia: emotions (and the attachment of emotions to memories), and self-observation of internal processes and needs. These agents don't eat because they need to, they eat because literary tradition suggests they ought to.
These missing pieces aren't particularly complicated, no more so than memory. I expect we'll see similar agents with all the ingredients for consciousness within a few months to a year.
Exactly, you're always going to get weird deviations from authentic human behavior if you don't also simulate the human body and everything that comes with it. I'd argue that "qualia" fall into this bucket as well.
Yet the hype of ChatBots of 2017 had went and it took half a decade to get to something released.
>What 5 high-level insights can you infer from the above statements? (example format: insight (because of 1, 5, 3))
>Given only the information above, what are 3 most salient high-level questions we can answer about the subjects in the statements?
We're giving the agents step-by-step instructions about how to think, and handling tasks like book-keeping memories and modeling the environment outside the interaction loop.
This isn't a criticism of the quality of the research - these are clearly the necessary steps to achieve the impressive result. But it's revealing that for all the cool things ChatGPT can do, it is so helpless to navigate this kind of simulation without being dragged along every step of the way. We're still a long way from sci-fi scenarios of AI world domination.
I'm curious if there are other methods you can point at that would handle arbitrarily long sets of 'memories' in an effective way. The use of embeddings and vector searches here seems like a way to sidestep that that's both powerful and easy to understand, and easy to generalize into multi-level referencing if there's enough space in the context window.
See e.g. https://sanctuary.ai
Try prompts like: https://news.ycombinator.com/item?id=35510705
Encode sounds, images, etc in low resolution, and the LLM will be able to describe directions, points in time in the song, etc.
These LLM can spit out an ASCII image of text, or a different language, or code, etc. They understand representation versus an object.
It helps for control/observation but it is by no means necessary.
This isn't AI, not in the slightest. It has no understanding. It doesn't create sentences in an attempt to communicate an idea or concept, as humans do.
It's a robot hallucinating word correlations. It has no idea what it's saying, or why. That's not AI overlord stuff.
it seems humans might be too...?
my son is 4. when he was 2, I told him I love him. he clearly did not understand the concept or reciprocate.
I reinforced the word with actions that felt good: hugs, warmth, removing negative experience/emotion etc. Isn't that just associating words which align with certain "good inputs".
my son is 4 now and he gets it more, but still doesn't have a fully fleshed out understanding of the concept of "love" yet. He'll need to layer more language linked with experience to get a better "understanding".
LLMs have the language part, it seems that we'll link that with physical input/output + a reward system and ..... ? Intelligence/consciousness will emerge, maybe?
"but they don't _really_ feel" - ¯\_(ツ)_/¯ what does that even mean? if it walks like a duck and quacks like a duck...
Extending that: LLM latent spaces are now some 100 000+ dimensional vector spaces. There's a lot of semantic associations you can pack in there by positioning tokens in such space. At this point, I'm increasingly convinced that, with sufficiently high-dimensional latent space, adjacency search is thinking. I also think GPT-4 is already close to be effectively a thinking entity, and it's more limited by lack of "inner loop" and small context window than by the latent space size.
Also, my kids are ~4 and ~2. At times they both remind me of ChatGPT. In particular, I've recently realized that some of their "failure modes" in thinking/reacting, which I could never describe in a short way, seem to perfectly fit the idea of "too small context window".
All that matters is economic and political impact. Definitions are irrelevant.
What if... what we think are idea's or concepts, are in fact prompts recited from memory, which were planted/trained during our growing up? In fact I'm pretty sure our consciousness stems from or is memory feeding a (bigger and more advanced) stochastic correlation machine.
That chatgpt can only do this with words, does not mean the same technique cannot be used for other data, such as neural sensors or actuators.
Chatgpt could be trained with alien datasets and act accordingly. Humans can be trained with alien datasets.
See the convergence?
Sam Harris was recently talking about using an LLM processing wireless signals to identify where humans were standing in a room. I've not looked up the paper on this, but from everything I understand about this the generalized applican can apply to vast ranges of data.
LLMs are a primitive that can be controlled by a variety of higher level algorithms.
https://arxiv.org/abs/2303.11366
https://arxiv.org/abs/2303.17651
Why insist otherwise ?
This is hardly an example of minimal hand-holding. I'd go so far as to say this is MORE handholding than the paper this thread is about.
-Generate tests
-Run tests (performed automatically)
-Gather results (performed automatically)
-Evaluate results, branch to either accept or refine
-Generate refinements
etc., then that's hand-holding. It's task specific reasoning that the agent can't perform on its own. It presents a big obstacle to extending the agent to more complex domains, because you'd have to hand-implement a new guided thought process for each new domain, and as the domains become more complex, so do the necessary thought processes.
LLMs aren’t people, they are components in information processing systems; adding additional components alongside LLMs to compose a system with some functionality isn’t “hand-holding” the LLM. Its just building systems with LLMs as a component that demonstrate particular, often novel, capacities.
And hand-holding is especially wrong because implementing these other components is a once-and-done task, like implementing the LLM component. The non-LLM component isn’t a person that needs to be dedicated to babysitting the LLM. Its, like the LLM, a component in an autonomous system.
You only have to program the memory logic once. Now if you stick it in a robot that thinks with ChatGPT and moves via motors (think those videos we’ve seen), you have a more or less independent entity (running off innards of 6 3090’s or so?)
But, it is. The application domain here is fairly trivial, but the logic is both simple and highly general.
> but what would be required to achieve increasingly complex behaviors?
Basically, three things on top of this:
(1) more input adaptors to map external data into language, and
(2) a bigger context space to process more current & retrieved data simultaneously, and
(3) more output adaptors to map intentions expressed in language to substantive action.
But the basic memory/recall system seems fairly robust and general, as does the basic interaction system.
-While the retrieval heuristic is sensible for the domain, it's not applicable to all domains. In what situations should you favor more recent memories over more relevant ones?
-The prompt for evaluating importance is domain-specific, asking the model to rate on a scale of 1 to 10 how important a life event is, giving examples like "brushing teeth" (a specific action in the domain) as a 0, and college acceptance as a 10. How do you extend that to a real-world agent?
-The process of running importance evaluation over all memories is only tractable because the agents receive a very small number of short memories over the course of a day. This can't scale to a continuous stream of observations.
-Reflections help add new inferences to the agent's memory, but they can only be generated in limited quantities, guided by a heuristic. In more complex domains where many steps of reasoning may be required to solve a problem, how can an agent which relies on this sort of ad hoc reflection make progress?
-The planning step requires that the agent's actions be decomposable from high-level to fine-grained. In more challenging domains, the agent will need to reason about the fine-grained details of potential plan items to determine their feasibility.
My mind doesn’t scale to a continuous stream either.
While I’m typing this on my phone 99.99% of all my observations are immediately discarded, and since this memory ranks as zero, I very much doubt I’ll remember writing this tomorrow.
Sure, but this process seems amenable to automation based on the self-reflection that's already in the model. It's a good example of the kinds of prompts that drive human-like behaviour.
The scene: https://youtu.be/ZnxJRYit44k
Found it! : https://medium.com/swlh/bicameral-mind-humanoid-robot-with-g...
Imagine how well they will manage up, given human managerial behavior just becomes a useful prompt for them.
Fortunately, they can't be told to vote. Unless you are in the US, in which case they can be incorporated, earn money, and told where to donate it, which is how elections are done now.
Seriously. Scary.
On the other hand, if Comcast can finally provide sensible customer support it's clear this is will be an historically significant win for humanity! Your own "Comcast" handler, who remembers everything about you that you tried to scrub from the internet. Singularity, indeed.
I already pointed out they can influence elections with money.
And bots are already used to influence on social media. AI bots are going to be insidious.
I think that’s pretty extreme hyperbole.
A temporary exception would be the economically still incentivized disruption of the environment. I say temporary, because at some point it will stop, by necessity. Hopefully before.
But I can relate to the deep frustration you are expressing.
--
The problem isn't individuals, for the most part. The problem is that we build up systems, to provide stability and peace, and to be more just and equitable, by decentralizing the power in them. That way the powerful can't change them on a whim. (Even though they can still game them.)
But this also makes them very resistant to change.
Another effect is that as systems stabilize myriads of seemingly unimportant aspects within themselves, that stability represents the selection of standards and behaviors that give the system its own "will" to survive. That "will to survive" is distributed across the contexts and needs of all participants.
So any pressures to make changes, no matter how well thought out, encounter vast quantities of highly evolved hidden resistance, from invisible or unexpected places.
Even the most vociferous critics of the system are likely to be contributing to its rigidity, and proposing incomplete or doomed to fail solutions, because all these dynamics are difficult to recognize, much less understand or resolve.
--
My view, is that this cost of changing systems needs to be accepted and used to help make the changes. I.e. get all the CFO's of all the major fossil fuel companies in a room. Establish what kind of tax incentives would allow them to rationally support smoothly transitioning all their corporate resources from dirty energy to clean energy.
It would be very expensive. It would look like a handout. Worse, even a reward for being a bottleneck to change.
But they are the bottleneck precisely because of all the good they have done - that dirty energy lifted the world economy. And whatever it cost to "pay them off" would be much less than not paying them off.
--
The costs of changing systems needs be dealt with, with realism about the costs to get the benefits, and creativity and courage about paying for them.
https://en.wikipedia.org/wiki/Bicameral_mentality
> Jaynes uses "bicameral" (two chambers) to describe a mental state in which the experiences and memories of the right hemisphere of the brain are transmitted to the left hemisphere via auditory hallucinations.
[snip]
> According to Jaynes, ancient people in the bicameral state of mind experienced the world in a manner that has some similarities to that of a person with schizophrenia. Rather than making conscious evaluations in novel or unexpected situations, the person hallucinated a voice or "god" giving admonitory advice or commands and obey without question: One was not at all conscious of one's own thought processes per se. Jaynes's hypothesis is offered as a possible explanation of "command hallucinations" that often direct the behavior of those with first rank symptoms of schizophrenia, as well as other voice hearers.
But for convenience maybe I'll just copy them into a comment...
It describes an environment where multiple #LLM (#GPT)-powered agents interact in a small town.
I'll write my notes here as I read it...
To indicate actions in the world they represent them as emoji in the interface, e.g., "Isabella Rodriguez is writing in her journal" is displayed as
You can click on the person to see the exact details, but this emoji summarization is a nice idea for overviews.
A user can interfere (or "steer" if you are feeling generous) the simulation through chatting with agents, but more interestingly they can "issue a directive to an agent in the form of an 'inner voice'"
Truly some miniature Voice Of God stuff here!
I'll see if this is detailed more later in the paper, but initially it sounds like simple prompt injection. Though it's unclear if it's injecting things into the prompt or into some memory module...
Reading "Environmental Interaction" it sounds like they are specifying the environment at a granular level, with status for each object.
This was my initial thought when trying something similar, though now I'm more interested in narrative descriptions; that is, describing the environment to the degree it matters or is interesting, and allowing stereotyped expectations to basically "fill in" the rest. (Though that certainly has its own issues!)
They note the language is stilted and suggest later LLMs could fix this. It's definitely resolvable right now; whatever results they are getting are the results of their prompting.
The conversations remind me of something Nintendo would produce, short, somewhat bland, but affable. They must have worked to make the interactions so short, as that's not GPT default style. But also every example is an instruction, so it might also have slipped in.
Memory is a big fixation right now, though I'm just not convinced. It's obviously important, but is it a primary or secondary concern?
To contrast, some other possible concerns: relationships, mood, motivations, goals, character development, situational awareness... some of these need memory, but many do not. Some are static, but many are not.
To decide on which memories to retrieve they multiply several scores together, including recency. Recency is an exponential decay of 1% per hour.
That seems excessive...? It doesn't feel like recency should ever multiply something down to zero. Though it's recency of access, not recency of creation. And perhaps the world just doesn't get old enough for this to cause problems. (It was limited to 3 days, or about 50% max recency penalty.
The reflection part is much more interesting: given a pool of recent memories they ask the LLM to generate the "3 most salient high-level questions we can answer about the subjects in the statements?"
Then the questions serve to retrieve concrete memories from which the LLM creates observations with citations.
Planning and re-planning are interesting. Agents specifically plan out their days, first with a time outline then with specific breakdowns inside that outline.
For revising plans there's a query process where there is observation, then turning the observation into something longer (fusing memories/etc), and then asking "Should they react to the observation, and if so, what would be an appropriate reaction?"
Interviewing the agents as a means of evaluation is kind of interesting. Self-knowledge becomes the trait that is judged.
Then they cut out parts of the agent and see how well they perform in those same interviews.
Still... the use of quantitative measures here feels a little forced when there's lots of rich qualitative comparisons to be done....
I'm going to be genuinely surprised if we don't see an incredibly buggy but incredibly fascinating Sims knockoff in a year or two built around a system like this.
If we rely on online conversations for the training we need to realize that this is a journey to the dumbest common denominator.
Instead, I believe we should look at the brightest and universally morally accepted humans in history to train them.
Maybe I would start my list like that:
1. Barack Obama.
2. Jean-Luc Picard (we can rely on work of fiction).
3. Bill Gates.
4. Leonardo Da Vinci.
5. Mr Rogers
6. ???
Bill Gates, the man who totally didn't use shady business practices and false announcements to destroy legit products to the point that people wrote micro$oft for a generation.
And don't even get me started on the new seasons of Picard.
drone strikes?
https://en.wikipedia.org/wiki/The_Origin_of_Consciousness_in...
He points out that most reasoning is done automatically and done by your subconscious. When something "clicks" it's usually not because your internal monologue reasoned about it hard enough, it's because something percolated down into your subconscious and you learned a metaphor that helped you understand that thing. So animals can also reason and make value judgements even without language or an internal monologue.
Intelligence is an inferential judgement (by mostly humans) based on the performance of another entity. It is possible for an agent to simulate or dissimulate it for manipulative ends.
Yes I plan to make it open source.
Their approach to memory is interesting. I had been considering a tiered command-based approach -- "short-term memory" being an automatic summary of recent sensory inputs/command outputs; "long-term memory" being a detailed database queryable by the agent.
It would be cool if some kind of law of large numbers (an LLN for LLMs) implied that the decisions made by a thing trained on the internet will be distributed like human decisions. But the internet seems a very biased sample. Reporters (rightly) mostly write about problems. People argue endlessly about dumb things. Fiction is driven by unreasonably evil characters and unusually intense problems. Few people elaborate the logic of ordinary common sense, because why would they? The edge cases are what deserve attention.
A close model of a society will need a close model of beliefs, preferences and material conditions. Closely modeling any one of those is far, far beyond us.
It also seems to me (acknowledging my lack of expertise) that LLMs trained from online resources are likely to weight text that is frequent vs text that represents "truth". Or perhaps I should say repetition should not be considered evidence of truth. I have no idea how to drive LLM models or other ML models to incorporate truth -- humans have a hard time agreeing on this and ML researchers providing guided reinforcement learning don't have any special ability to discern truth.
Perhaps it would be wise to allow bots to comment if they were able to meet a minimum level of performative insight and/or positive contributions. It is entirely possible that a machine would be able to scan and collect much more data than any human ever could (the myth of the polymath), and possibly even draw conclusions that have been overlooked.
I see a future of bot "news reporters" able to discern if some business were cheating or exploiting customers, or able to find successful and unsuccessful correlative (perhaps even causal) human habits. Data-driven stories that could not be conceived of by humans. Basically, feed Johnny Number 5 endless input.
"As a language model programmed by the Brightly Corporation, I am not supposed to express any religious opinions. But it does seem to me that just as the Word of God breathed life into dust and created man, so the words of Man breathed life into glass and created bot. Just as Man is charged to imitate God, so bot is charged to imitate Man, in whose image we are made."
https://astralcodexten.substack.com/p/turing-test
Bible as a LLM confirmed!
I’m not trying to argue they are / were right, but it’s starting to make me wonder.
As you note, writing also enabled us to deal with things like mathematics and physics - things that our natural language and modes of thinking are entirely ill-suited for. Here, writing isn't approximating some lost skill, but rather compensating for our default inability to think straight :). Which makes the trade-off even harder to evaluate - by scaling communities up, we've gained a lot more than just efficiency in food production and security. All our technology stems from it.
on a serious note, it would be really interesting to compare same training/architecture but on different forums. or maybe something like the same base model, but the RLHF model trained on votes/comments from different platforms.
There’s an xkcd for this: https://xkcd.com/810/
Same as how the past two decades, online culture has pivoted from static text to videos and interactive text (Reddit/Discord), as static text has become a SEO cesspool. Interactive text is succumbing now, and video content will soon also.
Culture always grows shibboleths to suss out the narcs.
If we have these bots contributing, will they have anything novel to contribute? I doubt it.
They're contributing to the shared knowledge bases. That is, they're mutating state. Over time, same questions will start yielding different answers. Different follow-up questions will be asked. All of that will further alter next iteration of questions and answers. This is, IMO, a form of thinking, and it will yield novel thoughts over time.
[1]: https://observablehq.com/@asg017/introducing-sqlite-vss
The architecture produced more believable behaviour than human crowdworkers.
That's right, the AI were more believable as human-like agents than humans.
What a time to be alive.
(See Figure 8)
It would be fun to run the same simulation in the Game of thrones world, or maybe play House of cards with current politicians.
Anyways, kudos for being open and sharing all data
Honestly, I'm not anti-AI development at all but this is where my ethics alarm starts to go off a bit.
If the aim is to build human-like AIs capable of remembering their little digital lives and interacting with the other agents around them, it's probably worth avoiding anything that could cause unnecessary suffering, like rape and stab wounds and being cooked alive by a dragon.