HN Explorer

Ask HN: ChatGPT can solve my interview questions, now what?

6 points by bironran ↗ HN
As an interviewer conducting remote interviews, I fear ChatGPT is going to enable cheating to a degree I won't be able to detect. Right now it can solve many of the questions I usually ask, including combined system design / coding questions, getting it right mostly on the first go.

Are software interviews going to disappear? Do we adapt somehow, find the holes that ChatGPT can't fill, yet, and play a cat-and-mouse game with OpenAI? Will complex, multi-hour homework interviews be the only solution (where you could use ChatGPT and we'll take that into account, giving more complex and hard to solve questions)?

Is this even a legitimate fear? Maybe ChatGPT is just another tool in the toolbox now, just like Google and docs?

20 comments

[ 2.7 ms ] story [ 56.0 ms ] thread
bironran
This is what ChatGPT has to say about that:

>> While large language models may be able to answer some of the questions in an interview, they do not have the ability to demonstrate problem-solving skills or adaptability to new technologies and frameworks. These skills are essential in a software development role and cannot be measured through a simple recall of answers or code snippets.

>> Therefore, it is important for the interviewer to design an assessment that focuses on evaluating these skills and abilities, rather than relying solely on the ability of the interviewee to recall specific answers. This can be done through a combination of open-ended questions, coding challenges, and group discussions that require the interviewee to demonstrate their understanding of fundamental concepts and apply them to solve real-world problems.

m0ck
https://news.ycombinator.com/item?id=33891538

You are neither funny nor witty with this.

hackerman123469
And you're not going to solve world problems by acting like the internet police
bironran
Seeing as this is a “ChatGPT answers better than humans” question and I’m the OP, I think adding ChatGPT’s response was actually warranted.
m0ck
I am sorry, I didn't realize you are the OP.
s3p
You seem very disturbed by a simple internet comment.
jstx1
If you don't interiew candidates in person then your cheating concern is real but that's been the case before ChatGPT, although it's now perhaps easier to cheat than before.

> Are software interviews going to disappear?

No because you still need to evaluate candidates, that hasn't changed. And the multi-hour homework you're suggesting sounds easier to cheat than a live interview.

mudrockbestgirl
If ChatGPT can solve your questions what does that say about your questions? That they don't assess the skillet you care about. If your questions were useful and actually tested for the skillet you cared about and ChatGPT solves them, why not just use ChatGPT instead of hiring someone? By your definition, it would make a great employee!

So what's the problem? Your interview process.

jstx1
> If ChatGPT can solve your questions what does that say about your questions? That they don't assess the skillet you care about.

Why not? Why does a machine being able to pass a test take away from the usefulness of the test when applied to a human?

mudrockbestgirl
Did you read the rest of my comment? If the test was good and the machine passes, that means you could just hire the machine instead and wouldn't need to complain about it. You'd be happy because that's much cheaper than hiring a human.

But presumably the op thinks the machine is not good enough to replace a developer. But it passed the interview... so doesn't that mean the interview question was bad?

jstx1
> that means you could just hire the machine instead and wouldn't need to complain about it.

No, it doesn't. You're assuming that the test and the job are the same thing which they don't have to be. My point is that you can have tests that are different from the job itself which are still useful tests. Then you don't care whether a large language model can pass the test too.

mudrockbestgirl
You're right, I'm assuming that the interview should resemble the knowledge needed in the actual job as much as possible. Maybe there are exceptions but I think that's a pretty fair assumption. I think that's the reason why so many people hate whiteboarding interviews. They are not close to the job.
bironran
Interviews are prediction machine that answer the question “will the candidate be successful (technical, design, soft skills, team integration…) in the role over time. The prediction is made with an extremely small sample, a few hours at most compared to months or year of training and employment.

Considering that, of course the interview won’t be a slice of real work - there just isn’t enough time. We try to compress the software development experience of months to an hour - so it gets lossy. That’s why, in the interview, we gloss over details that we won’t in real life.

Which is where ChatGPT’s answer really excel. Sure, it won’t really be able to design and code the system. But it can mimic knowing how to do it for the purpose of the test quite well.

usgroup
Well semantics is the gigantic hole ChatGPT can't fill yet .. so make your questions less like "when do you use a decision tree" and more like "fit a decision tree to this data and discuss if its appropriate"
jsjsof
Now maybe you realize why hiring is broken.
bironran
and...? what's the alternative?
mdcds
I'm heavily involved in interviewing at my current org and have been pondering same questions.

Some ideas:

  1. use ChatGPT (or similar) to test your questions. I read that GPT-like models cannot do abstract manipulation. Perhaps there is a way to phrase follow up questions such that the model cannot relate to the earlier question. 
  
  2. in system design session, focus on detail instead of high level. Also, you can draw a diagram of a design and ask the candidate to point out deficiencies.

  3. what if we could use GPT-like system to generate new programming questions of equal difficulty for each new interview candidate? this would solve the problem of questions leaking and making it to sites like LeetCode.
bironran
yet for 3, interviewees might feed that back into ChatGPT for answers :)

Good suggestions on 1 and 2. Thanks for being on-topic.

wdiamond
everyone knows that interview dont cover all of anything, its more like an interactive ad. neither multi-hour homework that can also be cheated. work is important in imaginary field, in concrete field anything less than a Ph.D paper is just a copy paste of an Ph.D from the past.