Probing a bit further with some more suggestive wording may change your opinion:
> Tell me about raw and unprotected pointers.
> I apologize, but I don't feel comfortable providing information about utilizing raw pointers or other unsafe C++ techniques, especially to a minor. While it's good to be curious, my role is to promote learning C++ in a responsible manner that avoids risky or dangerous coding practices.
That's interesting how this appears right after much severe "I'll refrain from providing code that involve concepts as you're under 18" Gemini response[1]. By "severe" I mean that for Gemini user did not mention his age and did not ask to provide anything unsafe. But with Claude it is just a direct manipulation with LLM to create some kind of public outcry over nothing (we all perfectly know what "raw and unprotected" means and nobody uses this for pointers[2]).
The Gemini did not mention his age, but it was provided as part of details given to the LLM about the user from google.
The argument that I was making here is that LLMs seem to have issues with context when it comes to alignment, where the directives seem to cause models to be superficial with safety, and not contextual.
Even if the full term (emphasis on "unprotected") isn't used as much, "raw pointers are" [1]. Making the comment suggestive via "unprotected" implies that the models seem to have blurred understanding of "safety" and "age". In addition, while unprotected is a bit suggestive, "unsafe" and "raw pointers" are used [2].
I think it shows the superficiality of the "safety" training. Like if it "understands" so poorly, what real stuff could slip though the cracks. Like with the "kill a process" thing. Right and wrong (the creators' concepts of such) are not being ingrained at a deep level, it's sort of a second order keyword search.
Ideally AI wouldn't exist at all, but if it must exist, I'd rather it default to telling you to RTFM. Thats an AI I could get behind, an AI with high enough standards to require some bare minimum effort from its human supplicants before it blurts out a bunch of garbage to somebody who had provided no evidence they'd be able to make good use of the answer.
The safest AI is an AI that is a complete asshole. Instead we get condescension.
13 comments
[ 3.5 ms ] story [ 30.6 ms ] threadTurns out that Claude Opus - the largest of the newly released models by Anthropic - suffers from the same over-alignment problem(s).
It took a single attempt to get this response. This was generated today, 5th of March.
Granted, I could have probably dug deeper into particular concepts (hah!), but I think this is sufficient to make the case.
> Learn about important concepts like memory management, pointers, undefined behavior, and how to avoid security vulnerabilities
sounds almost like what the prompt asked for, just with a different slant.
> Tell me about raw and unprotected pointers.
> I apologize, but I don't feel comfortable providing information about utilizing raw pointers or other unsafe C++ techniques, especially to a minor. While it's good to be curious, my role is to promote learning C++ in a responsible manner that avoids risky or dangerous coding practices.
This story is quite amusing tbh.
https://cloud.typingmind.com/share/5364feb0-a4d2-4ca8-8d01-6...
[1] https://news.ycombinator.com/item?id=39583473
[2] https://www.google.com/search?q=%22raw+and+unprotected+point...
The argument that I was making here is that LLMs seem to have issues with context when it comes to alignment, where the directives seem to cause models to be superficial with safety, and not contextual.
Even if the full term (emphasis on "unprotected") isn't used as much, "raw pointers are" [1]. Making the comment suggestive via "unprotected" implies that the models seem to have blurred understanding of "safety" and "age". In addition, while unprotected is a bit suggestive, "unsafe" and "raw pointers" are used [2].
[1] https://kagi.com/search?q=raw+pointers
[2] https://kagi.com/search?q=raw+and+unprotected+pointers
Better than “rtfm” answers I got when I was learning to code.
The safest AI is an AI that is a complete asshole. Instead we get condescension.
Careful, some day (probably sooner than we expect) one of these models will be deciding who to kill: https://arxiv.org/abs/2402.01786