Claude 3 Opus has an issue – Repeats same response over and over

4 points by xbsd98 ↗ HN
I am using the API version of Claude 3 Opus to simulate patient profiles. I've been trying since the model got released and I keep getting the same patient description even with different combinations temperature, top_k, etc. I assumed it was a launch issue and it would eventually get fixed, but that did not happen. Is anyone else seeing the same issue ?

```import anthropic

client = anthropic.Anthropic( # defaults to os.environ.get("ANTHROPIC_API_KEY") api_key=os.environ["ANTHROPIC_API_KEY"] # Save ANTHROPIC_API_KEY in .env )

def create_anthropic_message(system_prompt, user_prompt, anthropic_model = "claude-3-opus-20240229", max_tokens = 200, temperature = 0.999, *kwargs): message = client.messages.create( model=anthropic_model, max_tokens=max_tokens, temperature=temperature, system=system_prompt, # Also include any other arguments in the def function kwargs *kwargs, messages=[{"role": "user","content": [{"type": "text","text": user_prompt}]}]) return (message)

# Run

for i in range(20): try: r = create_anthropic_message("You are an oncology expert","Compose a brief 1-sentence presentation of a US-based patient presenting with Skin Cancer.") print(r.content[0]) except Exception as error: # handle the exception print("An exception occurred:", error)```

Responses --

# Results

ContentBlock(text='A 68-year-old retired Caucasian male from Florida presented with a growing, asymmetric, irregularly bordered, multicolored 8mm lesion on his left forearm that had changed in size and color over the past 6 months.', type='text')

ContentBlock(text='A 62-year-old Caucasian male with a history of extensive sun exposure presents with a rapidly growing, irregularly shaped, darkly pigmented lesion on his left forearm that has changed in size and color over the past 3 months.', type='text')

ContentBlock(text='A 45-year-old fair-skinned male from Florida presents with a suspicious, asymmetric, irregularly bordered, multicolored, and evolving mole on his left upper back that has been growing for the past 3 months.', type='text')

ContentBlock(text='A 62-year-old fair-skinned male from Florida presented with a rapidly growing, asymmetric, irregularly bordered, darkly pigmented lesion on his left shoulder that had changed in size and color over the past 3 months.', type='text')

ContentBlock(text='A 45-year-old Caucasian male construction worker from Florida presents with a rapidly growing, irregularly shaped, ulcerated dark brown lesion on his left forearm that has doubled in size over the past 3 months and is accompanied by swollen lymph nodes in his left axilla.', type='text')

ContentBlock(text='A 62-year-old Caucasian male from Florida presented with a slow-growing, irregularly shaped, dark brown patch on his left shoulder that had changed in size and color over the past 6 months.', type='text')

ContentBlock(text='A 62-year-old Caucasian male from Florida presented with a slowly enlarging, irregularly bordered, pigmented lesion on his left forearm that had changed in color and size over the past 6 months.', type='text')

ContentBlock(text='A 45-year-old Caucasian male construction worker from Florida presents with a rapidly growing, asymmetric, irregularly bordered, multicolored, 8mm diameter lesion on his left upper back that has changed in size and color over the past 3 months.', type='text')

ContentBlock(text='A 62-year-old Caucasian male with a history of extensive sun exposure presents with a rapidly growing, irregularly shaped, ulcerated pigmented lesion measuring 1.5 cm in diameter on his left upper back.', type='text')

[... Same profile with changes in only 1-2 words]

10 comments

[ 1.6 ms ] story [ 34.3 ms ] thread
Not using Claude but this behaviour comes not from the model itself. Basically the model does not now what it provided in the other iterations of your for loop. Ask for 20 different profiles. Or, even better: ask for different types of cancers and iterate this list.
I have tried this even with all ranges of temperature (0 to 1) ... it produces the SAME profile over and over ... Since I couldn't get to format the code here, I have put it on pastebin -- https://pastebin.com/fCu3Ji6n

Any chance someone can try this out ? I am pretty sure it will be the same, something is odd (maybe it is too harmless with patient profile?)

I suspect it has something to do with this snippet of code, which sets the amount of randomness in the output via "Temperature"[1]

  temperature = 0.999
[1] https://www.iguazio.com/glossary/llm-temperature
I have tried this even with all ranges of temperature (0 to 1) ... it produces the SAME profile over and over ... Since I couldn't get to format the code here, I have put it on pastebin -- https://pastebin.com/fCu3Ji6n

Any chance someone can try this out ? I am pretty sure it will be the same, something is odd (maybe it is too harmless with patient profile?)

Why haven't you tried a temperature over 1? Try 10 and see what happens.

Look at the reference page I supplied for a better idea of what this parameter does on any LLM.

Temperature ranges from 0-1 … I think the author might be referring to some other llms. See the messages API reference under temperature - https://docs.anthropic.com/claude/reference/messages_post

The last line says “ Note that even with temperature of 0.0, the results will not be fully deterministic.”

At least in this example, even temp 1 is nearly 100% deterministic !

Btw, they give $ 5 as free credit when you sign up for the API, no credit card needed. The credit is good enough for testing Opus.
It's entirely possible the documentation is auto-generated, and wrong. (1%)

What's the harm in trying?

Hmm, I mean to say that I have tried all different combinations of temperature, top_x etc parameters and all the responses are almost same, sometimes verbatim ...