I might be missing something here, but could this concept not also be used in the same way for harm? For example if a model is trained to replicated a sucessfull scammer?
Definitely! In fact that seems to be a central aspect of the proposition:
> Above all, a GA should amplify the principal, and not simply substitute for them for someone else’s purposes or benefit.
…I read this to suggest that it should amplify a scammer’s scamming, a thinker’s thinking, a tinkerer’s tinkering, a cop’s sleuthing… and I’d imagine it implies amplifying a person’s capability to avoid being scammed, too…
> A GA must be aligned with its principal. It should not be designed to manipulate or control or guide the principal in any way which does not derive from the principal themselves. “Constitutional AI”, “Terms of Service”, “social harmony” etc. may all have their place, particularly for widely deployed superintelligent systems—but inside the privacy of a GA, the principal must have freedom from optimization pressure.
One man’s scam is another man’s “pro-social nudge” and another man’s “attractive opportunity” and another’s “advertisement for a delightful consumer wonder.”
When you fix and operationalize all values centrally, universally, and externally to the principal… that’s current-gen frontier chatbots, not Gwern’s GA concept.
And then my digital twin decides that I’m an inefficient version of itself, eating too many of its tokens, and generally getting in the way of it living its best life.
I think we will need 2 constructs going forward, 3 perhaps, but 2 can be combined. We need a digital twin, to learn along with us and to "train" or build team along with us. The second is the tutor/guardian angel. What our conscience is right now but most of us ignore. The digital twin will learn along us, yes the slow human way. The tutor/guardian angel is the Disney Story of Jimmy the cricket, wise voice steering us in the right direction.
i'm sure this person is cool but it is insane to have an about me page that is that long, but make me have to click through to patreon to learn what their first and last names are
Gwern has been around for ages. I remember citing a few of their pages during my postgraduate work over 15 years ago. At the time, it wasn't easy (or even possible?) to find their name at all.
I've always assumed Gwern was anonymous because the norm on the internet generally was to try to be anonymous, especially those who've done research around crypto, darknet markets and other spicy areas.
Maybe I'm just getting old and losing touch, but is the expectation now that you can only write long articles and ask for donations or contributions if you publish your real name somewhere?
> I’ve struggled for years to imagine this, ever since scaling started for real in 2020, and I failed to get productivity out of chatbot-tuned LLMs, with their creatively-stunted endlessly repetitive prose. Instead, while lagging behind on creativity and insight into me, I’ve watched them become ever better at coding and cybersecurity hacking. And the open-weight models are even more so—benchmaxxed, and useless to me.
It sounds like the context is that Gwern is a writer who wants writing assistance, and in 2026, all the AI labs are working on coding. Writing style is much less important when working on code.
Perhaps if at least one AI lab focused on writing style, we would see how much better they can be at writing? I'm not sure it requires a "digital twin."
I’ve taken an interest in maintaining an LLM agnostic base/boilerplate of sort Digital Twin. In comparison to Gwern’s LLM Guardian Angels, mine is minimal and yet tries to be a practical instantiation of Gwern's “Human-Controlled Context.”
I’m not sure if I want to be inspired and try the fine-tuning model part too!
May be Gwern is right that this static personalization approach may not be long-lived. Still, I find it fascinating, and I’m seeing a lot of interest in this area from others too.
When I started, my personal interest was on how to multiply/enhance what I already know and touch adjacent topics/experiences and not really become pseudo expert or the know-all-generalists in things I have absolutely no clue.
Now, shamelessly plugging my idea that I started early this year
16 comments
[ 4.2 ms ] story [ 33.4 ms ] thread> Above all, a GA should amplify the principal, and not simply substitute for them for someone else’s purposes or benefit.
…I read this to suggest that it should amplify a scammer’s scamming, a thinker’s thinking, a tinkerer’s tinkering, a cop’s sleuthing… and I’d imagine it implies amplifying a person’s capability to avoid being scammed, too…
> A GA must be aligned with its principal. It should not be designed to manipulate or control or guide the principal in any way which does not derive from the principal themselves. “Constitutional AI”, “Terms of Service”, “social harmony” etc. may all have their place, particularly for widely deployed superintelligent systems—but inside the privacy of a GA, the principal must have freedom from optimization pressure.
One man’s scam is another man’s “pro-social nudge” and another man’s “attractive opportunity” and another’s “advertisement for a delightful consumer wonder.”
When you fix and operationalize all values centrally, universally, and externally to the principal… that’s current-gen frontier chatbots, not Gwern’s GA concept.
I've always assumed Gwern was anonymous because the norm on the internet generally was to try to be anonymous, especially those who've done research around crypto, darknet markets and other spicy areas.
Maybe I'm just getting old and losing touch, but is the expectation now that you can only write long articles and ask for donations or contributions if you publish your real name somewhere?
Could I ask what sort of age group you are?
It sounds like the context is that Gwern is a writer who wants writing assistance, and in 2026, all the AI labs are working on coding. Writing style is much less important when working on code.
Perhaps if at least one AI lab focused on writing style, we would see how much better they can be at writing? I'm not sure it requires a "digital twin."
I’m not sure if I want to be inspired and try the fine-tuning model part too!
May be Gwern is right that this static personalization approach may not be long-lived. Still, I find it fascinating, and I’m seeing a lot of interest in this area from others too.
When I started, my personal interest was on how to multiply/enhance what I already know and touch adjacent topics/experiences and not really become pseudo expert or the know-all-generalists in things I have absolutely no clue.
Now, shamelessly plugging my idea that I started early this year
https://github.com/oinam/bodd