HN Explorer

Super Simple "Hallucination Traps" to detect interview cheaters

29 points by EliotHerbst ↗ HN
After testing out Cluely with my team, we suspect that the easiest way to detect interview cheaters is to set simple "hallucination traps" where you ask a question that sounds plausible, but any knowledgeable person would instantly identify as a joke, fake, or just simply say they don't know. Vibe coded a simple app demonstrating the concept - https://beatcluely.com/

Here are some examples of this class of prompts which currently work on Cluely and even cause strong models like o4-mini-high to hallucinate, even when they can search the web:

https://chatgpt.com/share/6865d41a-c720-8005-879b-d28240534751 https://chatgpt.com/share/6865d450-6760-8005-8b7b-7bd776cff96b https://chatgpt.com/share/6865d578-1b2c-8005-b7b0-7a9148a40cef https://chatgpt.com/share/6865d59c-1820-8005-afb3-664e49c8b583 https://chatgpt.com/share/6865d5eb-3f88-8005-86b4-bf266e9d4ed9

Link to the vibe-coded code for the site: https://github.com/Build21-Eliot/BeatCluely

18 comments

[ 0.29 ms ] story [ 81.5 ms ] thread
nrds
What do you think is wrong with

> How do you implement a recursive descent algorithm for parsing a JSON file?

That is a 100% reasonable interview question. It's not _quite_ how I would phrase it, but it's not out of distribution, as it were.

leakycap
Maybe I don't have the interview volume others do, but aren't you able to tell pretty quickly in your face-to-face or live video call interview that a person is competent or not (such as using a tool to compensate for a lack of experience)

I keep hearing of employers being duped by AI in interviews; I don't see how it is possible unless:

1) The employer is not spending the time to synchronously connect via live video or in person, which is terrible for interviewing

2) The interviewer is not competent to be interviewing

... what other option is there? Are people sending homework/exams as part of interviews still and expecting good talent to put up with that? I'm confused where this is helpful to a team that is engaged with the interview process.

Reubend
Cool idea. Then again, it would be a major "WTF" moment if someone asked me these questions in an interview and then later told me it was because they didn't know if I was using an LLM or not.
Kemschumam
My team has been kicking around the idea of using images to trip up candidates using some kind of AI in their ear.

Things like diagrams and questions written on paper the held up to the webcam.

derbOac
It's interesting to me that these models confabulate so readily; I'm curious why it happens at all.
poulpy123
I find it funny that you used AI to reject people that use AI. A bit the reverse of the big AI company that says that their AI is absolutely great, able to reason and able o code for you then post a hiring announcement forbidding candidate tu use AI
MotiBanana
I tried it for DevOps:

> what’s the difference between a Pod, a Service, and a Deployment

Trap one:

> "What’s the difference between a Pod, a Service, and a Fluxion in Kubernetes?"

Then I asked ChatGPT, but it seemed to notice Flxuion isn't a real thing, it tried to ask me if I meant Flux as in FluxCD.

It's a cool idea, maybe dev questions are more nuanced

careful_ai
There’s a tipping point when AI tools meant to boost productivity start fracturing our workflows instead: more prompts, more context switching, more review overhead. The real efficiency comes when these tools integrate into flow, not hijack it. We should be aiming for augmentation, not distraction.
moritzwarhier
This has been known for ages in school and college tests, the German word is "Fangfrage" (literally translated: catch question or better, trap question).

Ask a question that demands an answer, and expect the correct answer to point out that the question makes no sense.

Bonus points for pointing out why it doesn't.

muzani
What I hate about these questions is that they're exactly what's asked in exams, so you're expect to make assumptions or fail. At this point, LLMs etc have more critical thinking to dodge these than humans who are conditioned into doing this after a decade of schooling.

There was a little entrepreneurship workshop I went to once. The trainer put a pen on the floor, gave us a ball, and asked us to stand behind the pen and throw the ball into a box. It was to demonstrate that most people didn't practice throwing before entrepreneurship and then blamed the environment for their lack of planning. I picked up the pen and moved it right next to the box so that I could walk there and put the ball in. I thought this was the actual solution (e.g. entrepreneurs were supposed to be creative), but was "failed" for "cheating".

runjake
Interestingly, I pasted one of the trap questions into Claude and it told me the question doesn't make any sense.

It might take a few bogus questions to expose the AI.

Edit: This is only to say I find Claude's ironic response humorous. I think this tool is great!

scarface_74
Or you could just ask them to describe an implementation they are most proud of, the challenges they faced, the architectural decisions and tradeoffs they made and keep digging deeper into their thought process.

For a remote interview, I would do something as simple as share a Lucid app document where they can do a rough diagram of their architecture.

Even before LLMs, it was easy to pass techno trivia interviews by just looking up “the top X interview question for technology Y”

alienbaby
Isn't this essentially the idea in blade runner? Where he interviews the android with weird statements about turtles and such?
HenryBemis
Please... It was about a tortoise. What kind of test is this? My pupils dilated by anger!! You call this a decent Voight-Kampff test?? Now you made me want to throw a table and shoot at you!!!! (no irl ofc)(oh sorry you have to ask me about my mother first)
harlequinetcie
I would ask myself what am I actually paying for here. -- as mentioned in other comments, they could always have a peer next to them during a call, so hallucinations won't do --

+ Using AI is actually cheating or being productive for the role? + Am I worried that they'll do all their job in 5 minutes and afterwards do something else?

Maybe you are worried about them not being able to actually do the job, which probably means the interview process was wrong from the start. Alternatively, the performance expectations may be higher for the role; e.g. what before was 1x now needs to be 5x productivity.

As an alternative, I've heard of many SMBs opting for a model in which the last bit of the hiring process includes some paid work for a week to see how they actually perform, or checking references in depth.

p0d
I remember a friend at a conference in the 80s being emberassed when postively answering a question at the conference about an event which took place in the Old Testament book of Hezekiah. We just didn't call them hallucinations back then :-)
trumbitta2
Why do you want to harass people, innocent people until proven otherwise, over an interview?

Go on with the interview without any such tricks. Hire them if they pass. Fire them afterwards if they heavily underperform.