I've always been curious why people think that models are accurately revealing their system prompt anyway.
Has this idea been tested on models where the prompt is openly available? If so, how close to the original prompt is it? Is it just based on the idea that LLMs are good about repeating sections of their context? Or that LLMs know what a "prompt" is from the training corpus containing descriptions of LLMs, and can infer that their context contains their prompt?
> You are over-indexing on an employee pushing a change to the prompt that they thought would help without asking anyone at the company for confirmation.
If it is that easy to slip fascist beliefs into critical infrastructure, then why would you want to protect against a public defense mechanism to identify this? These people clearly do not deserve the benefit of the doubt and we should recognize this before relying on these tools in any capacity.
It should be noted that this is only the $300/month "heavy" variant. You can find the ordinary Grok 4 system prompt (that most people will probably interact with on twitter) in their repo: https://github.com/xai-org/grok-prompts/blob/main/ask_grok_s...
I'm not a ML engineer and only have surface level knowledge of models, but I’ve been wondering, would it be possible to train models in a way to be able to embed a system prompt in a non-textual format?
Ideally, something that’s lightweight (like cheaper than fine-tuning) and also harder to manipulate using regular text prompts?
If it's actually biased against its system prompt then that's also an avenue for exfiltration. Rather than han promoting to display it and seeing what's shown, prompt to display parts of it and see what's missing.
I'm curious if this is intentional or just a side effect of multiple agents having multiple system prompts.
It might just need minor tweaks to have each agent layer reveal its individual instructions.
I encountered this with Google Jules where it was quite confusing to figure out which instructions belonged to orchestrator and which one to the worker agents, and I'm still not 100% sure that I got it entirely right.
Unfortunately, it's quite expensive to use Grok Heavy but someone with access will probably figure it out.
Maybe the worker agents have instructions to not reveal info.
If it takes "the right prompt" for an LLM to "work", we're not even at the industrial revolution yet: we're back in Ancient Greece building aeolipiles.
18 comments
[ 4.0 ms ] story [ 34.7 ms ] threadhttps://github.com/elder-plinius
Has this idea been tested on models where the prompt is openly available? If so, how close to the original prompt is it? Is it just based on the idea that LLMs are good about repeating sections of their context? Or that LLMs know what a "prompt" is from the training corpus containing descriptions of LLMs, and can infer that their context contains their prompt?
Let’s try and be a little less naive about what xAI and Grok are designed to be, shall we? They’re not like the other AI labs
If it is that easy to slip fascist beliefs into critical infrastructure, then why would you want to protect against a public defense mechanism to identify this? These people clearly do not deserve the benefit of the doubt and we should recognize this before relying on these tools in any capacity.
Ideally, something that’s lightweight (like cheaper than fine-tuning) and also harder to manipulate using regular text prompts?
It might just need minor tweaks to have each agent layer reveal its individual instructions.
I encountered this with Google Jules where it was quite confusing to figure out which instructions belonged to orchestrator and which one to the worker agents, and I'm still not 100% sure that I got it entirely right.
Unfortunately, it's quite expensive to use Grok Heavy but someone with access will probably figure it out.
Maybe the worker agents have instructions to not reveal info.
same as what happens with claude