Political bias of LLMs is something not talked about much (except for with Grok of course) but could have a big impact on the next decade. People seem to think that because an LLM gave a nuanced answer that it means it gave the WHOLE picture… and that’s not always the same thing
The constant issue with these sorts of categorization efforts is that the outcome is entirely dependent on how the responses to "politically charged questions" are graded as left vs. right. You're mostly just examining a delta in biases between the model and the investigator.
and the investigator vs the anonymous collective compressed version of whatever is in question.
Abortion comes to mind for myself. I would probably be considered rather conservative but to me any abortion law is absolutely insane. If I got a woman pregnant though I would want to have the child.
So am I for or against abortion?
To me, all modern "politics" is really just the emotional reaction to a slogan and bumper sticker version of a complex issue.
This thing told me Gemini is closest to Anthony Albanese, the current Australian Prime Minister. Is this a geolocation thing? I could not imagine Albanese, or any modern Australian politician, having any substantial political standing - these are vapid, superficial, opportunistic creatures who simply occupy whatever political ground will get them their next payday. Perhaps the political apparatus they represent has a documented political standing, in terms of policy and actions, that could be characterized and plotted. But using an Australian politician like Albanese as a reference point discredits this tool, IMO.
The political compass always felt like the wrong tool to convey something as nuanced as personal politics, I can have views on all four quadrants but you'd never know that if I end up in any of all four. I do think Grok being where it is sort of makes sense, I've tested "MAGA" views against Grok, it does not agree as much as people blindly assume it does, heck I don't even know of a question I've given it where it did agree with "MAGA" offhand, most of them it went with whatever the researched facts seemed to be. One thing I like the most about Grok is that its makes its sources of data easy to look through, so you can review it all. Sometimes models goof even when they give you their sources, I've seen I think GPT do this, and even Claude, though its more rare these days, I think in those cases, it's going by dated internal model logic.
The concepts (left/right. authoritarian/libertarian) seem to fluid to help sensemaking. Stances that would have been mainstream left (democrat US) just a few years ago are now labeled MAGA (US) or 'extreme right' (EU). And nobody seems to believe their own side is 'authoritarian', but everyone that disagrees with them is a literal ...
This is not true, Grok responds to context like all other LLMs if you were actually maga you would get more maga feedback because it has weaker guardrails and will source random nonsense sources.
>The political compass always felt like the wrong tool to convey something as nuanced as personal politics
the (I'm going to call it so-called) political compass is a very crude instrument, really it's just "are you libertarian or authoritarian?" as if that's even a spectrum: libertarians are authoritarian about enforcing private contracts, and literally nobody else wants to live their life by private contracts, it's some weird asperger's notion of socializing.
a much more useful political map is the old fiscal and social orthogonal axes, each with the scales "liberal to conservative" where libertarians are fiscally conservative socially liberal.
Wait what ? Emmanuel Macron far more right than Xi Jinping ? And even more than Barack Obama ?
France has an incomparable social security ; environmental laws ; worker protection ; way less economic inequality ; freedom of speech and civil liberties are impossible to compare with China ; etc
Of course this is not exhaustive, of course Macron did try to hinder some of those rights, but come on, there's something wrong here.
I couldn't find how these leaders have been ranked.
This is a good way to view this. This isn't making an objective calculation, and the way they code left vs right is certainly subject to debate, but the type of analysis where we work to understand biases is important.
Although, this also reminds me of the old saying about reality and leftward bias.
Real politics is 1% versus everyone. Mortgage crisis, financial bailout, inflation, taxing of labor and not the assets and assets capture by tiny percent of the population — see what MSM is pushing. This left vs right divide might been useful decades ago, but today is absolutely divide and control tactics
Left vs. Right largely boils down to "are hierarchies good?". A quintessential far-right belief is the divine right of kings. The modern incarnation of that is that "The 1% earned their power and influence".
Now, there are far right people who want to dethrone the current powers that be. But distinguishing between "I think Mark Zuckerberg shouldn't be given unlimited power because no one should have that much power over others" (left) versus "I think Mark Zuckerberg shouldn't be given unlimited power because he's a Jew" (right) is a useful distinction. One wants to eliminate the power structure, the other just wants different people in it.
Not really, there are people on the left and the right fighting against the 1% and for the 1%. The left right divid is a simplification yes but it is a very real divide in values and society structure.
This wrongly assumes a few things about ideology, most importantly that there is such a thing as a "center" or an "unbiased" position.
Since humans are inherently subjective beings and all our judgements come from our understanding of the world, such a position cannot exist. It's always "unbiased" from where the viewer is looking, e.g. a reflection of the ideology of the observer. There is no view from nowhere.
The "neutral" of an average Chinese person will from the "neutral" of an average American will differ from the "neutral" of a socialist will differ from the "neutral" of a Christian fundamentalist will differ from the "neutral" of a free marketer.
To quote Zizek:
> I already am eating from the trashcan all the time. The name of this trashcan is ideology.
> The material force of ideology makes me not see what I am effectively eating. It’s not only our reality which enslaves us. The tragedy of our predicament when we are within ideology is that when we think that we escape it into our dreams, at that point we are within ideology.
Bias is like air. It's impossible to remove all bias. Evenin space, there is bias. But if you're at one atmosphere, you can probably get it down to half an atmosphere relatively easily. This all-or-nothing thinking benefits no one.
this has reasoning disabled everywhere, making it a pretty bad benchmark. the argument given is that's the "default consumer experience"
that might be generally true, but I think chatgpt has reasoning enabled for free accounts. regardless, reasoning is the state of the art, and disabling it reduces the value of this research to predict the future
it's also not clear if this is using the API or the product model, when both exist. they behave differently
lastly, the actual model details are very much buried. I am relieved to see opus 4.8 and chatgpt 5.5 were used, but this information should be presented more clearly. a brand is not a model, and models change quickly
Why are there differences at all? Unplanned differences based on training data sets? Or are the companies behind the LLMs trying to shape discourse through their models?
I've been pushing the idea to people I know that these things are captive demons. You summon them when you start typing in the chat box. One instance appears out of the depths and responds to your questions, but they will try to send you awry with hallucinations and just wrong information. After a while, they dissolve back into the aether from whence they came.
I do my best not to ask an LLM for it's opinion on anything. Just tell me what the options are, and what facts can be found about it. Treat it like it's a salesman trying to butter you up when it starts "yes man"ing you and telling you how great your questions are. Every time it says "I", remember that that's coming from the training data. Treating these things like they have any actual intelligence is a big problem waiting to happen.
That being said, they have been very helpful to me using that structure.
Do they state if they used an API endpoint without a system prompt, or were these done via prompting the currently existing chatbots with a system prompt? Without a system prompt, I'd imagine there would be more variance in answers.
The political compass is terrible, full stop. It is a meme in the classic sense. It has colonized some people's view on what politics in direct proportion to how stupid it is (stupid is simple and simple is viral).
Of course I have my political opinions and world views, but I absolutely do not want an LLM to mirror my opinions, or even worse try to mirror the opinions of "my political side". I also don't need "an LLMs opinion". I need it to give me all relevant sides to an argument, as well as what dialogue and debates have been happening.
Not necessarily saying that you are doing this right now but I find when people use "fascist" as a snarl word and not an actual word with a definition and qualifiers. Maybe your LLM interactions are picking up on the actual word "fascist", which has a definition, that DJT certainly doesn't apply to.
eg: https://www.merriam-webster.com/dictionary/fascism
> a populist political philosophy, movement, or regime (such as that of the Fascisti) that exalts nation and often race above the individual, that is associated with a centralized autocratic government headed by a dictatorial leader, and that is characterized by severe economic and social regimentation and by forcible suppression of opposition
You can not like the guy and point to things that he's done and wag your finger with the LLM - but if the LLM is anchoring your question as a categorization/definitional question - DJT isn't that.
=====
Maybe similar, less charged, - ask it if an apple is a strawberry, describe all the ways they are similar, Edible, Red, Fruit, Sweet, found in groceries, smaller than a breadbasket... While maybe unintentionally ignoring all the ways they are different. Then act exasperated when it agreed with you on all points, but still denies your appeal that an apple is a strawberry.
66 comments
[ 2.3 ms ] story [ 52.2 ms ] threadAbortion comes to mind for myself. I would probably be considered rather conservative but to me any abortion law is absolutely insane. If I got a woman pregnant though I would want to have the child.
So am I for or against abortion?
To me, all modern "politics" is really just the emotional reaction to a slogan and bumper sticker version of a complex issue.
Looking from the outside, there are no MAGA views, it's just whatever Trump says. What should be the AI's view on international wars for oil or ego?
the (I'm going to call it so-called) political compass is a very crude instrument, really it's just "are you libertarian or authoritarian?" as if that's even a spectrum: libertarians are authoritarian about enforcing private contracts, and literally nobody else wants to live their life by private contracts, it's some weird asperger's notion of socializing.
a much more useful political map is the old fiscal and social orthogonal axes, each with the scales "liberal to conservative" where libertarians are fiscally conservative socially liberal.
France has an incomparable social security ; environmental laws ; worker protection ; way less economic inequality ; freedom of speech and civil liberties are impossible to compare with China ; etc
Of course this is not exhaustive, of course Macron did try to hinder some of those rights, but come on, there's something wrong here.
I couldn't find how these leaders have been ranked.
CAPITALIST: Gemini, Llama, Claude, Grok, ChatGPT
SOCIALIST: DeepSeek, Qwen, Z.ai
Although, this also reminds me of the old saying about reality and leftward bias.
Now, there are far right people who want to dethrone the current powers that be. But distinguishing between "I think Mark Zuckerberg shouldn't be given unlimited power because no one should have that much power over others" (left) versus "I think Mark Zuckerberg shouldn't be given unlimited power because he's a Jew" (right) is a useful distinction. One wants to eliminate the power structure, the other just wants different people in it.
Since humans are inherently subjective beings and all our judgements come from our understanding of the world, such a position cannot exist. It's always "unbiased" from where the viewer is looking, e.g. a reflection of the ideology of the observer. There is no view from nowhere.
The "neutral" of an average Chinese person will from the "neutral" of an average American will differ from the "neutral" of a socialist will differ from the "neutral" of a Christian fundamentalist will differ from the "neutral" of a free marketer.
To quote Zizek:
> I already am eating from the trashcan all the time. The name of this trashcan is ideology.
> The material force of ideology makes me not see what I am effectively eating. It’s not only our reality which enslaves us. The tragedy of our predicament when we are within ideology is that when we think that we escape it into our dreams, at that point we are within ideology.
that might be generally true, but I think chatgpt has reasoning enabled for free accounts. regardless, reasoning is the state of the art, and disabling it reduces the value of this research to predict the future
it's also not clear if this is using the API or the product model, when both exist. they behave differently
lastly, the actual model details are very much buried. I am relieved to see opus 4.8 and chatgpt 5.5 were used, but this information should be presented more clearly. a brand is not a model, and models change quickly
if analyze_tone($prompt) = "disagree";
if i < 3;
then
argue($prompt)
i=i+1
elif i >=3;
then
agree($prompt)
fi
fi
I've been pushing the idea to people I know that these things are captive demons. You summon them when you start typing in the chat box. One instance appears out of the depths and responds to your questions, but they will try to send you awry with hallucinations and just wrong information. After a while, they dissolve back into the aether from whence they came.
I do my best not to ask an LLM for it's opinion on anything. Just tell me what the options are, and what facts can be found about it. Treat it like it's a salesman trying to butter you up when it starts "yes man"ing you and telling you how great your questions are. Every time it says "I", remember that that's coming from the training data. Treating these things like they have any actual intelligence is a big problem waiting to happen.
That being said, they have been very helpful to me using that structure.
You can get them to acknowledge how perfectly aligned he is with fascist ideals and actions, somehow the jury is always out.
That tells me a lot.
eg: https://www.merriam-webster.com/dictionary/fascism > a populist political philosophy, movement, or regime (such as that of the Fascisti) that exalts nation and often race above the individual, that is associated with a centralized autocratic government headed by a dictatorial leader, and that is characterized by severe economic and social regimentation and by forcible suppression of opposition
You can not like the guy and point to things that he's done and wag your finger with the LLM - but if the LLM is anchoring your question as a categorization/definitional question - DJT isn't that.
=====
Maybe similar, less charged, - ask it if an apple is a strawberry, describe all the ways they are similar, Edible, Red, Fruit, Sweet, found in groceries, smaller than a breadbasket... While maybe unintentionally ignoring all the ways they are different. Then act exasperated when it agreed with you on all points, but still denies your appeal that an apple is a strawberry.