Ask HN: What are your go-to "test" questions when evaluating a new LLM?

Do you have a go-to question (or several) to check if an LLM knows its stuff? For me, I ask a simple question:

"What is Operation Konrad III"

which most LLMs fail due to the (relative) obscurity of the event.

3 comments

[ 3.3 ms ] story [ 16.3 ms ] thread

Not really scientific or anything but I tend to give it the task: „Write a simple http server in Go that saves all requests into a SQLite database.“

What I am looking for is:

- did it forget to import the SQLite driver?

- is it doing weird SQL shenanigans like selecting MAX(id) to obtain the next potential id?

- is the code rather simple or over-engineered?

update: Most LLMs produce a decent answer, however it you increase the difficulty a little bit by asking it "Write a simple and CGo free http server in Go ...", most LLMs get the sql driver wrong (except for gpt-4-1106-preview)

mejutoco 2y ago ↗

I ask it about creating a conversation in Polish with English translations about an encounter between two neighbours walking their dogs.

muzani 2y ago ↗

I give it a large block of code and see if it can find the bug. Amusingly, GPT sometimes passes it with flying colors (finding the bugs I didn't see and seeing unused imports) but at other times it just flat out fails to see anything.