Ask HN: What's the point of automated skill assessment tests in the age of AI?

I guess it tests the ability to successfully apply ChatGPT. Which is a fairly relevant skill now, I guess?

Ideally, companies would evaluate what answers ChatGPT is capable of and/or give tests which contain enough space for creativity that they'd be hard to pass this way. (Ask people to show off their knowledge of some domain without explicit instructions) If you get old style tests, they just haven't caught up to what's happening yet. Hopefully they'll realise after a few second stage interviews and change it.

an interesting question is how to filter out the folks who just feed it through chatgpt, because obviously the people who just lie to get out of any mild annoyance are the ones you least want as coworkers (though obviously people who can effectively use large language models will be incredibly valuable)

i asked gpt-4 some programming questions today, and although it gave clear and convincing explanations in english, it made a lot of really dumb mistakes in writing the code. but this was in a mode where it couldn't call out to an external interpreter; maybe a lot of those will go away once it can test the code

most programmers would have done worse, but is there a pattern to the dumb mistakes chatgpt makes that is clearly different from the dumb mistakes junior programmers make?

like for example i gave it this

    #!/usr/bin/perl
    # See also cifs.py.  Edited GPT-4 output; see <https://bin.gy/clignitter>
    # Try watch -n .1 ./wing.pl.
    for(1..9999){$t=0|rand 3;($y,$x)=$t?($y/2,$x/2+($t-1?40:10)):(.3*$x-.4*$y+12,
    .4*$x+(.2+abs($$%99/99-.5))*$y);$a[$x][$y]=1}map{$r=$_*2;print map({substr
    " .':",2*$_->[$r]+$_->[$r+1],1}@a[0..79]),"\n"}(0..18);

and asked it to translate it to lua, getting the following nonsense in response

    math.randomseed(os.time())
    local a = {}
    for i=1,9999 do
        local t = math.floor(math.random() * 3)
        local x,y
        if t == 0 then
            x,y = .3*a[1][2]-.4*a[2][1]+12, 
    .4*a[1][2]+(.2+math.abs(os.time()%99/99-.5))*a[2][1]
        else
            x,y = a[1][2]/2, a[2][1]/2+(t-1 and 40 or 10)
        end
        if not a[x] then a[x] = {} end
        a[x][y] = true
    end
    local str = " .':"
    for r=0,18 do 
        for c=0,79 do 
            local val = (a[c*2] and (a[c*2][r*2] or 0) or 0) + (a[c*2+1] and 
    (a[c*2+1][r*2+1] or 0) or 0)
            io.write(string.sub(str,val,val))
        end 
        print()
    end 

of course, i've seen worse from interviewees, and understanding the original program is a high (and arguably counterproductive) bar, but perhaps the particular pattern of bonehead errors here could identify this as llm output? maybe the very fact of being so spectacular at understanding the original code is a red flag, at least combined with being so incompetent at writing the lua version?

What's even the point of a lot of these jobs in "the Age of AI"?

I don't know. You don't know. Let's hope it all keeps going a little longer so some of us can keep a roof over our heads.

Some companies now ask to do a coding test with screen sharing. It's not a hard test but it's interesting because you can see the reasoning and ask for more details if necessary. Live sessions means the company is interested in humans but it requires potentially lots of wasted ressources to find the right person.

Hopefully eventually we fallback to the more analogue solutions that have always been better anyway. Things like actually talking with applicants, seeing things they've actually built that have existed for some amount of time, talking to people in their networks, etc.

Companies will interview you on a white board, like many already do. There's nothing new here. This is why we've been taking tests in class and not at home where we could use friends or family to do them for us.

Here is the solution to this:

1. Use AI to do the full on-site interview loop. At least for SWEs it is heavily structured, so something a good wrapper around GPT-4 can handle. It can even have an avatar if need be.

2. The AI-led "on-site" is done at your own time. But you must have a camera enabled.

3. HR just goes over the footage and other features to see if you have been using aids. In any case, the way you respond will be highly indicative of whether you used any aids.

4. If you pass and there is no cheating detected, then go straight to meeting the hiring manager. This interview is mostly behavioral and guessing whether you are a good match.

The problem today is that interviewing is expensive for both parties. Companies in particular get drowned with applications so they put all types of hurdles and auto-rejections.

But with AI, you could flip the process around. You could give _everyone_ the chance to do the full technical interview! Then you leave the human matching components for the best performing candidates.

My opinion: if it was this easy to make interviews completely pointless then they were completely pointless before AI.

Why would a random skill assessment be able to tell if you are going to be a good employee? Your value isn't tied to you knowing a bunch of random facts. We have the vastness of the Internet for that.

All AI has done is point out how useless our current interview process is.