"We are doing what we can to hang on to relevancy as gatekeepers who already held way too much authority over a field". They are going to use AI on the job anyway.
This also applies to universities. The world has changed but they have not and they will make sure to try and stay relevant as much as they can to continue to take money.
Edit: looks like it will take a while for some people to accept that we are not going back from this. The cat is out of the bag and your certificates are increasingly irrelevant. Sorry if you spent a lot of money and time to get it.
Get back to me when AI is actually reliably correct about any technical field.
Accounting exams are gatekeeping, yes. The good kind of gatekeeping where you make sure the people doing the job are actually capable. And you have avenues to punish those who fail their clients.
> This also applies to universities
Eh. I’d say the actual academics are about 1/3 of the university experience. The rest is networking and teaching you how to think and solve problems on a more abstract level. I’d say the people who farm that (and particularly the abstract thinking part) out to AI are going to be the ones left at disadvantage in the future. You’re completely replaceable.
I've had no end of problems with accountants regardless of their certifications, they operate in a domain with an incoherent body of contradictory and highly subjective rules yet make it out to be a science.
My conclusion as a whole is that accountancy as a profession rarely delivers any actual value to their customers, where much of the job is compliance theater at best.
I don’t think it will be too long before the pendulum swings back towards “real people who actually know the subject”. At that point, I might feel bad for everyone who coasted on AI.
Certifications have always been irrelevant for me, but that's only because my goal has always been what I'm capable of doing on my own AND (this one is a biggie) I was unbelievably fortunate to have several people in my career who trusted that I could get the job done.
Certifications are about low trust. With the advent of modern LLM tech, trust levels are probably not going up.
Nobody needs to hire someone who can use an LLM because if that is the skill they're looking for they can just use the LLM themselves.
So if you need to hire someone because the LLM isn't cutting it, then you'll by definition need to be hiring someone who isn't using an LLM. Someone who isn't just using an LLM to make you think that they aren't using an LLM.
How is that going to be done? Sounds like a job for certifications to me. Not today's certifications, but a much more in depth, in person, and gatekeepery certification.
My guess would be that certifications, unfortunately, will be significantly more relevant in the days of LLMs. Not less.
Ask an examiner from 20 years ago the risk of allowing people to take exams in their own home. They would have said 'cheating', even with no concept of AI.
Here is what happened. ACCA, one of several accountancy bodies in the UK, charge their students extraordinary sums of money to take their exams. When I took accountancy exams there were 9of 3 hour written exams, in a real building, with real invigilators. All of the bodies at the same time realised that they could charge the same amount, pay Pearson to administer an electronic test and make more money out of their students. It was a disgrace then and it is a disgrace now
Seriously. Kids are going to cheat. It's already easy enough to just throw the test material into the LLM and get a bunch of flash cards on relevant content and memorize that. I Wish I had AI in college.
> Ask an examiner from 20 years ago the risk of allowing people to take exams in their own home. They would have said 'cheating', even with no concept of AI.
AI has taken it to the next level. Previously, with many exams you would still have to know how to identify the concepts and related keywords in a word problem to even know what words to look for in the index of the books on hand before you could get to the right page to start cheating.
Some of the certification exams I had to take back in the day even came with their own little reference manual that everyone got and was free to use to look up concepts and equations like you would in the real world. The book wasn’t helpful if you didn’t know how to recognize the way to solve the problem and look it up, though.
AI changes that. Now you don’t need to know anything at all. You don’t even need to parse the question or even speak the same language. Copy the problem into ChatGPT with a prompt attached. Copy the answer into the solution box.
Anecdotally, the rise of ChatGPT has also normalized the concept of cheating among students. The common thinking is that everyone is using ChatGPT, therefore you’ll be left behind if you don’t cheat.
> Ask an examiner from 20 years ago the risk of allowing people to take exams in their own home.
Isn't this like an "open-book" exam? We had them 50 years ago when I was doing my A-levels in the UK, and I always thought it was a good system. The trouble now is of course that you can ask the book to look up the answer, unless the question is very well thought out, which is hard. The open-book thing worked best IMHO for things like practical chemistry, where you needed the technique as well as the theory.
Some exams in the UK (GCSEs) moved away from coursework because of the problem of cheating. If it happens with GCSEs, why would it not happen with high stakes professional exams?
There are some IGCSEs that you can take remotely (with a camera on you and a hefty extra fee) and I am wondering what problems those will run into. Pearson are offering them.
It's only a "disgrace" because you still believe the BS at face value. All of these licensing schemes are self (the organization running them) serving rackets to various extents.
Subtract a thousand from all the dates, call it a guild, sprinkle some nobility into the org chart. That'll make it all make sense. Same shit, different day.
Of course they jumped at the chance to charge the same for less. At the time it didn't look like there was a serious downside. "Everyone" was doing it. And to some extent that forced their hand. If you're running a licensing racket and you don't stay up to date with the rest of the licensing rackets and your license becomes relatively a worse value for whatever the upside is then supply will be constrained, prices will go up, perhaps enough to make people not well versed in you trade ask tough questions like "why are these people licensed in this manner" that could be a serious threat to the status quo.
Until quite recently, it was trivial to cheat on remotely proctored exams. All you had to do is spin up a VM, take the exam inside the VM, and use the host system to look up answers. I believe the main proctoring services now have crude VM checks. You can probably still use a KVM switch or a DP splitter and a buddy...
My wife is a teacher of physics and math for an online highschool. Its very common for kids to go into the in person exam with a mark in the 80s and 90s and get a failing grade on the exam.
The web wasn't alwasy that useful for cheating on timed exams as it was essentially like being able to bring in a formula sheet.
LLM's changed this such that you can type in the question and get a fully correct answer in a lot of cases.
The only solution that I see in education is that in person exams start to represent a larger and larger portion of a students grade such that the mid term and final will be more than 50% of a students grade for most classes going forward due to the gratuitous use of llms by students.
It can be hard to prevent cheating in person too: A criminal enterprise was uncovered in 2019 in Sweden. They had targeted the local SAT variant (högskoleprovet).
Their end customer equipment consisted of a modified mobile phone hidden somewhere private, a necklace that acts like a magnetic coil and small magnets that you place against the eardrum. Then the operation would call the phone while the customer was in the auditorium and give them the correct answers via voice.
The answers had been provided by some back office team based on a copy of the test that they had obtained in realtime from some planted source taking the test at the same time, somehow.
I have a dog in this fight as a professor, but I think the AI era may actually (and ironically) help reestablish colleges as a useful tool for employers. We have a significant amount of legacy infrastructure to support in-person testing, and non-digital written exams may be the best way to determine actual competency going forward.
I have historically done my computer science classes entirely online, but I am switching to in-person on-paper tests and increasing their weight in my classes to deal with the cheating.
In person exams are useless too. I was at a UK university as a mature student about 8 years ago. When exams came around a significant number of people went to the bathroom repeatedly during the 2-3 hour exam clearly to check notes on their phones. There isn't really anything that can be done to stop this other than doing some sort of spot check/search for phones on people mid-exam which would obviously be horribly disruptive.
Just ban bathroom breaks and cap exams two 90-120 minutes tops. If longer examination is needed split it into multiple sessions. I genuinely believe that being able to plan around bathroom breaks or manage discomfort otherwise is a reasonable expectation.
In the end of the day academia in general should stop relying on exams based on memorization of random facts and start using real world examples of what kind of work student would be working with as an employee. And if student can deliver correct result even when using AI or any other method, and then explain why those results are the way they are, then student has passed.
In real world outside of academia, nobody cares how did you get to the result, only thing which matters is if result is correct and if you can explain why it is correct.
This has more to do with being remote than AI, but it's worth asking: If the test meaningfully measures the skills of being an accountant, and AI will get you good grades, why not use AI to "cheat" at doing the job itself?
I think this will go full circle (as the article indicates). Ultimately , the parents are the “customer” of education institutions as they send their kids there with the goal of them getting good grades. For now, getting a good grade regardless of whether you got it honestly or not should result in you getting a job and therefore the parents are happy.
But as soon as employers understand that the grades don’t mean anything, they will start prioritising students from places that are more strict e.g. through in person only exams etc.
Once this happens, parents, and therefore schools, will start to prioritise this more.
The sad part is that a generation of kids are going to pass through school and come out dumb and ill prepared for life while the systems corrects
This was inevitable, and it's not about being strict, but about banal technical limitations. Remote proctoring has always been security theater, but Vision LLMs combined with hardware solutions put the final nail in the coffin. The fundamental problem is simple: you cannot software-secure a device to which the attacker has physical access.
ACCA simply admitted defeat in an arms race where the attack became orders of magnitude cheaper and more effective than the defense. The only reliable "air gap" from AI today is a physical room and paper
Do you need quick and reliable hacker service
Do you need to catch or hack your cheating spouse and more
Henryclarkethicalhacker at g mail com is the solution
30 comments
[ 4.2 ms ] story [ 60.9 ms ] threadCalculations must be getting accurate now. Not only questions about vocabulary or domain concepts.
This also applies to universities. The world has changed but they have not and they will make sure to try and stay relevant as much as they can to continue to take money.
Edit: looks like it will take a while for some people to accept that we are not going back from this. The cat is out of the bag and your certificates are increasingly irrelevant. Sorry if you spent a lot of money and time to get it.
Accounting exams are gatekeeping, yes. The good kind of gatekeeping where you make sure the people doing the job are actually capable. And you have avenues to punish those who fail their clients.
> This also applies to universities
Eh. I’d say the actual academics are about 1/3 of the university experience. The rest is networking and teaching you how to think and solve problems on a more abstract level. I’d say the people who farm that (and particularly the abstract thinking part) out to AI are going to be the ones left at disadvantage in the future. You’re completely replaceable.
My conclusion as a whole is that accountancy as a profession rarely delivers any actual value to their customers, where much of the job is compliance theater at best.
Certifications are about low trust. With the advent of modern LLM tech, trust levels are probably not going up.
Nobody needs to hire someone who can use an LLM because if that is the skill they're looking for they can just use the LLM themselves.
So if you need to hire someone because the LLM isn't cutting it, then you'll by definition need to be hiring someone who isn't using an LLM. Someone who isn't just using an LLM to make you think that they aren't using an LLM.
How is that going to be done? Sounds like a job for certifications to me. Not today's certifications, but a much more in depth, in person, and gatekeepery certification.
My guess would be that certifications, unfortunately, will be significantly more relevant in the days of LLMs. Not less.
Here is what happened. ACCA, one of several accountancy bodies in the UK, charge their students extraordinary sums of money to take their exams. When I took accountancy exams there were 9of 3 hour written exams, in a real building, with real invigilators. All of the bodies at the same time realised that they could charge the same amount, pay Pearson to administer an electronic test and make more money out of their students. It was a disgrace then and it is a disgrace now
AI has taken it to the next level. Previously, with many exams you would still have to know how to identify the concepts and related keywords in a word problem to even know what words to look for in the index of the books on hand before you could get to the right page to start cheating.
Some of the certification exams I had to take back in the day even came with their own little reference manual that everyone got and was free to use to look up concepts and equations like you would in the real world. The book wasn’t helpful if you didn’t know how to recognize the way to solve the problem and look it up, though.
AI changes that. Now you don’t need to know anything at all. You don’t even need to parse the question or even speak the same language. Copy the problem into ChatGPT with a prompt attached. Copy the answer into the solution box.
Anecdotally, the rise of ChatGPT has also normalized the concept of cheating among students. The common thinking is that everyone is using ChatGPT, therefore you’ll be left behind if you don’t cheat.
Isn't this like an "open-book" exam? We had them 50 years ago when I was doing my A-levels in the UK, and I always thought it was a good system. The trouble now is of course that you can ask the book to look up the answer, unless the question is very well thought out, which is hard. The open-book thing worked best IMHO for things like practical chemistry, where you needed the technique as well as the theory.
Who sits in front of the PC, who is nearby?
The rest is kind of besides the point then.
There are some IGCSEs that you can take remotely (with a camera on you and a hefty extra fee) and I am wondering what problems those will run into. Pearson are offering them.
Subtract a thousand from all the dates, call it a guild, sprinkle some nobility into the org chart. That'll make it all make sense. Same shit, different day.
Of course they jumped at the chance to charge the same for less. At the time it didn't look like there was a serious downside. "Everyone" was doing it. And to some extent that forced their hand. If you're running a licensing racket and you don't stay up to date with the rest of the licensing rackets and your license becomes relatively a worse value for whatever the upside is then supply will be constrained, prices will go up, perhaps enough to make people not well versed in you trade ask tough questions like "why are these people licensed in this manner" that could be a serious threat to the status quo.
There will be a lot of COVID-era qualifications that are treated with a hint of suspicion in the future.
Take a look at A-level scores: https://schoolsweek.co.uk/a-level-results-2024-future-exams-...
( direct link to graph: https://schoolsweek.co.uk/wp-content/uploads/2023/04/Overall... )
It's unfortunate for those affected either way. It was a difficult time when drastic measures needed to be taken at short notice.
It's right to go back to in-person testing if there is a problem keeping remote exams fair.
The web wasn't alwasy that useful for cheating on timed exams as it was essentially like being able to bring in a formula sheet.
LLM's changed this such that you can type in the question and get a fully correct answer in a lot of cases.
The only solution that I see in education is that in person exams start to represent a larger and larger portion of a students grade such that the mid term and final will be more than 50% of a students grade for most classes going forward due to the gratuitous use of llms by students.
My guess is the number of exceptional circumstances is about to explode...
Their end customer equipment consisted of a modified mobile phone hidden somewhere private, a necklace that acts like a magnetic coil and small magnets that you place against the eardrum. Then the operation would call the phone while the customer was in the auditorium and give them the correct answers via voice.
The answers had been provided by some back office team based on a copy of the test that they had obtained in realtime from some planted source taking the test at the same time, somehow.
I have historically done my computer science classes entirely online, but I am switching to in-person on-paper tests and increasing their weight in my classes to deal with the cheating.
As paul graham said: do things that don't scale.
In real world outside of academia, nobody cares how did you get to the result, only thing which matters is if result is correct and if you can explain why it is correct.
But as soon as employers understand that the grades don’t mean anything, they will start prioritising students from places that are more strict e.g. through in person only exams etc.
Once this happens, parents, and therefore schools, will start to prioritise this more.
The sad part is that a generation of kids are going to pass through school and come out dumb and ill prepared for life while the systems corrects