Has ChatGPT Been Neutered?

I don't but I'd like to know.

I was under the impression that it was mostly GPU vram based but once the model is loaded, it could produce output quickly? I'm probably over-simplifying things...

I'm not arguing or complaining.

Just highlighting the tactic :)

It seems pretty logical to me. Fine-tuning to make it more polite is giving it questions and punishing for giving an actual answer.

To me it is clear why. Imagine someone told you "answer immediately, top of your head: what's the best seasoning?". You'd just blurt out whatever specific you associated with pleasing seasoning (and that would be a good answer). Now imagine someone said "answer immediately, off the top of your head, but without offending any culture, gender, without a cultural bias, and without being presumptuous of the listener's socio-economic status (and if you fail one of these, someone dies) what's the best seasoning?" Even without the way that is going to lead to all sorts of compromising and second guessing in the answer space, simply only a fraction now of your brain is left to associate about the question due to just holding all that other stuff in there.

Do we know if this is exclusive to ChatGPT? Is the API exempt from this issue?

https://github.com/cogentapps/chat-with-gpt and https://github.com/ShipBit/slickgpt come to mind.

You never read the Yemen Wikipedia page or this kind of information is too neutered for you ? Being snarky or humorous is not needed to provide a feel of the place.

> Since 2011, Yemen has been in a state of political crisis starting with street protests against poverty, unemployment, corruption, and president Saleh's plan to amend Yemen's constitution and eliminate the presidential term limit.[19] President Saleh stepped down and the powers of the presidency were transferred to Abdrabbuh Mansur Hadi. Since then, the country has been in a civil war (alongside the Saudi Arabian-led military intervention aimed at restoring Hadi's government against Iran-backed Houthi rebels) with several proto-state entities claiming to govern Yemen: the government of President Hadi which became the Presidential Leadership Council in 2022, the Houthi movement's Supreme Political Council, and the separatist Southern Movement's Southern Transitional Council.[20][21][22][23][24] At least 56,000 civilians and combatants have been killed in armed violence in Yemen since January 2016.[25] The war has resulted in a famine affecting 17 million people.[26] The lack of safe drinking water, caused by depleted aquifers and the destruction of the country's water infrastructure, has also caused the largest, fastest-spreading cholera outbreak in modern history, with the number of suspected cases exceeding 994,751.[27][28] Over 2,226 people have died since the outbreak began to spread rapidly at the end of April 2017.[28][29] The ongoing humanitarian crisis and conflict has received widespread criticism for having a dramatic worsening effect on Yemen's humanitarian situation, that some say has reached the level of a "humanitarian disaster"[30] and some have even labelled it as a genocide.[31][32][33] It has worsened the country's already-poor human rights situation.

https://en.m.wikipedia.org/wiki/Yemen

A counterpoint would be the Encyclopaedia Dramatica article for Australia (or any other developed country).

Can you prove this? Do you have an example prompt that has changed in outcomes?

[dead]

True but that's not who I'm thinking about.

Currently, to correctly use LLM output you must:

- Understand that they are basing answers on particular set of content that may not be correct, and was fixed in time.

- Understand that the company producing it has not programmed in each response.

- Understand that the LLM does not (as far as we can tell) have an internal concept of truth and lies, and is therefore not attempting to minimise lying, only produce truthful-sounding language.

Most people engaging with LLMs today do not know these things, and this is why we already have examples like professors asking ChatGPT if it generated student essays, and failing those students – something it is entirely incapable of answering.

Expanding access to 1bn users is going to run into this far more. We clearly don't have the ability to communicate these nuances effectively to early adopters, let alone 1bn users, and that's not because they're stupid, it's because LLMs are in an uncanny valley – they appear human, but are not in important ways that we have not encountered before as a society.

For a long time to come, the norm will be that people will assume OpenAI wrote the answers for each question, in the same way that most people assume Google programs the search results for each query. Anything outside of this is a minority tech bubble.

The nuance required here is something that LLMs have not been great at, or at least not good enough to prevent actual homophobic jokes. See the general issue that LLMs struggle with negation. In fact alignment work is on some level about getting LLMs to be smart about this nuance.

I agree in the abstract that, for instance, an LLM could/should be able to make jokes about Tim Cook — just because he’s gay doesn’t mean he should be off limits for a joke about Apple, for instance. That would demonstrate an impressive level of nuance.

But also like … what’s the business play here? “Our LLM tells jokes about gay people that aren’t homophobic!” doesn’t strike me as a powerful market differentiator; or at least the market will have to evolve in very specific ways to make that meaningful.

The handwringing that I see on HN around LLMs being neutered really seems like a plea for alignment, just of a different sort: folks want the models to reflect their priors, their alignment, where nothing is off limits. The business case for this — and the general social value of this — is generally unexamined.

If they had they would say "I ran the same query, last week 1+1 = 2 and this week 1+1=!$#@#$%".

The assumptions required to believe "Trust me bro, I know these things" require many more leaps of faith.