AI Detection services are the oiliest of snake oils. Seems like education is going to need to change drastically in the next few years. Maybe we'll go all the way back to oral examinations again.
I doubt oral examinations come back in general. The undergrad model relys on massive classes, I don’t see how you can do oral exams in those 300 person classes without massive cheating as the first students tell everyone else what to expect.
ChatGPT could have the exact same set of questions to ask. All students are independently taking the same exam at the same time, responding conversationally. ChatGPT asks all of them the same question, and then probes based on weaknesses in the responses. Grades the answer to that question, then moves on to the next.
Everyone got equal questions. Is there going to be inconsistency in how minor wording trips different followups? Sure. But it isn't clear that this randomness is worse than the subjective differences in how humans grade an essay response to a question.
You've missed the point of curves. Curves exist as a different solution to the problem of "we don't know whether everybody understands the material or not, and we don't know how to evaluate whether they do". It's like saying "The problem with electric cars is that they don't fit the equipment for our emissions test"
If you give a pre-written test, you basically need to curve it because you don't really know how well the test aligns with your expectations of the class's understanding of the material. So you say "typically, about 20% of the class doesn't understand the course at all, 20% have a very good grasp..." etc, and then fit the written test results to the expected curve.
A one-on-one communication with a student allows you to directly gauge their understanding, allowing you to assign a grade. It doesn't matter how well or poorly the rest of the class understands the material, this student, sitting in front of me, has a "B" level understanding.
I disagree. The point of curves is to differentiate the students. Students who don’t understand do and the material should be failed, it that doesn’t tell us who should get A vs B vs C
"Do they proficiently understand the topic?" Should be met with a trinary answer: YES, NO, WITHDRAW.
We should not encourage stack ranking (or a curve) as a way to stratify students because they joined the class with too many smarter people. Go read up about stack ranking, and why it's a horrific practice.
In a "grade on a curve" situation, it benefits you to drop the class if there's too many people you think smarter or more experienced than you. And it benefits you if you're smarter than them. And, thats a REALLY UGLY way to approach education.
In the end I guess it comes down to what the point of undergrad is. Imo we have completely left the "learn stuff" era and are in the "differentiate yourself to employers" era. That means that stack ranking is largely the point, even if it has many downsides.
I feel like a 5-minute conversation, not a pre-determined list of oral questions, on most non-hard science topics would be enough to evaluate a student. You could even have an LLM grade it as a second opinion.
We had an exam that was either pass or fail, no grades. The professor and censor would kick you out with a "PASS" in five minutes if they felt that you had sufficient grasp of the topic. Some found it a bit offensive, I just loved how efficient it was. I happened to hang around on the first day of their planned three days examination and the professor just popped his head out the door and asked if I was ready now (a day and a half early), sure... I guess. Five minutes later "PASS. NEXT!"
I am starting a private high school and this is exactly how we plan to assess students.
Not the LLM part though.
Not only do you get a good sense of what they know, but you get a chance to give real individual feedback about what they need to focus on next.
Of course towards the end in order to get a qualification that a university will accept as entrance we will need to teach the kids to pass the standardised tests too.
Furthermore oral examinations do not scale. We had to do them during covid times and it was gruesome, spending full days in meetings and losing a full week with only 70-80 students.
An AI could generate relevant questions, follow-up questions during the oral, and evaluate the student at the end. The teacher could then review the whole conversarion at a glance and check/adjust the AI evaluation, which shouldn't take more time than grading an exam and be equally scalable.
I've actually tested this with a VR app (with support for mobile also), and it actually works quite well.
> An AI could generate relevant questions, follow-up questions during the oral, and evaluate the student at the end.
I dunno about that level of automation. Maybe the proctors can use it to generate a big bag of questions beforehand, that they manually review, but generating them on the fly is just asking for trouble.
Administering the preset questions and responding to the student is a much better fit.
So as a student, I'm forced to converse with an ai and take it seriously, when in reality I reject them all? The whole original post is an example of this being terrible and not a solution for anything.
It's bad enough that these things don't do a good job, don't basically function for the task, but what's as bad or worse is being forced to pretend you buy into it all like having to pretend to believe in some religion just to not get killed by everyone else.
I'm not a student and so far I have been free to judge all these ais as bad on a variety of fronts, and not use them (directly anyway, of course they are used behind the scenes in ways that affect me which I can't control). This would be hell.
The main technological advances already happened relatively early via printing, first through the printing presses and then later through desktop publication. Other advances are more peripheral such as better indexing via computers.
Is there any value to those lectures, nowadays? I can see a PhD lecture on the fundamentals of virtually any academic topic, for free and at my leisure, on youtube and similar sites. Waking up at 8am to attend an in-person Geology 117 lecture with 300 other freshmen is probably as outdated as a fax machine. Perhaps the next step is, education that isn't based on personal instruction isn't rare enough to be worth paying for.
The value of lecture is to see what you are expected to know for the exam. Khan academy isn’t necessarily covering the same things in the same order or level of detail as your professor.
As a PhD candidate (post classes) I honestly use those lectures a lot. No, I do not find a massive classroom better than an online lecture. Actually worse. The advantage of a classroom over a video lecture is the ability to interrupt and ask questions. To have the expert in front of you. In this day and age, sitting in a classroom and not participating is an entire waste of money and you're doing yourself a disservice by not participating.
I actively remind my students this when I teach, who honestly are less active post pandemic. It is something I worry about. And though my account is "anonymous", if you ever show up to my lectures, not as a student but just sitting in (I don't take attendance, idk if you walk off the street and want to sit in), you too are welcome to participate.
(Fwiw, there aren't really "PhD level lectures" so much as graduate lectures. Generally you stop taking classes at the end of year 2 in your PhD, which is equivalent to a masters. The distinction of the PhD is that you are learning to learn on your own. This does add extra context to why grad students ask for higher pay, especially at the non-conditional and candidate levels, as they're actually taking less resources from the school. E.g. I pay the school $10k/term (big round) for research credits, which requires no involvement from anyone. There no longer is the "student" part to my "51% student, 49% employee" split)
>And though my account is "anonymous", if you ever show up to my lectures, not as a student but just sitting in (I don't take attendance, idk if you walk off the street and want to sit in), you too are welcome to participate.
Good on you for allowing auditing. I had only one teacher in college who allowed people to audit my classes and he even fought the administration to give me credit for the class after I sheepishly approached him and asked if he would mind letting me taking the exam to make sure I understood the material. One of the most insightful classes I took my entire time in college
Ah, I meant the lecturer is a PhD, which was supposed to be juxtaposed with the 1xx level course to indicate that it's probably a waste of time on both sides. I'm sure there are huge lectures that aren't useful, but Geology 117 (which I really took) was obviously a waste of money when I spent a pittance per credit hour on it in the 90s; charging someone for it today just seems like highway robbery.
The value is the interaction. I try to have a discussion about the topic. You can read all about it on the website, but here's where I can ask you questions to see if you get it and we can discuss and clarify.
(Now, I grant you, in a room of 300, interaction is severely limited. I'm lucky my classes tend to have fewer than 20 people.)
If I give a lecture that you could have just watched a video of, I fell short.
Lots of people pay to go in to their weekly workout class where they just do exercises they already know how to do at home.
A well designed class session can be (can be) similar in that it provides a space and guidance for you to do 60 minutes of improving your brain, that let's be honest, you would otherwise half-ass at home. (Of course, the classroom is only part of what you're paying for, there are also assignments, feedback, etc.)
Absolutely. Even in the largest lectures, people can ask clarifying questions, live, from the subject matter expert. Likewise, the professor can look at the students' body language and realize that they've lost a significant portion of the class, and stop, slow down, and maybe prompt the class for questions.
There is an art to a good lecture, and it's definitely harder the larger the class, but good lectures cannot be replaced by recordings.
There are many reasons to attend classes (roughly in order of importance)
1. Real life lecturer that can be asked questions
2. Peers that you can meet and discuss both study-related and unrelated topics.
3. Language. Not relevant for intros, but advanced topics can be explained in more than one way, and as a student you not necessarily will be able to map between them.
4. Schedule. Self-pacing 6 different classes for 12 weeks may be harder than it looks. Instead, you can just attend classes.
5. Advanced topics. Online resources, especially videos become scarcer as you advance in your studies, more so if you are not in hot topic (e.g., CS). I still have some notes from classes I took as undergrad as a best source for the learned topics. Good luck finding video lectures about diagrammatic approach to real time dynamics of open spin systems.
Same goes the other way around, sometimes attending class is just not worth it, even without tons of online courses. I learned my E&M by just reading the official book during the lecture time (see 4.).
The bigger schools like Harvard often have that for the first year classes, especially classes that are required for all or many majors. The class itself, and the lectures, will have many students -- but there's usually ample availability of grad student TAs for more individualized interaction. A grad student is usually perfectly capable of teaching a freshman class.
Smaller schools and upper level classes generally just have fewer people.
Edit: Keep in mind that teaching is something the professor usually just puts up with, to get the resources and credibility to do the thing they are really passionate about (research or writing).
One someone is standing in front of the room talking it doesn't really matter if there is one person in the room or 1 billion. sure you need a PA system and other technology to support a large crowd, but otherwise it doesn't matter. Most students would prefer to be in a large room with someone who is a great speaker vs a small room with less. If you do have questions - that is what the TA is for and you get to see the TA at least weekly.
Note that the above only applies if the class is supposed to be a lecture. If it is supposed to be interactive then small classes are better. However many of the basic undergrad 1st year courses have been done so often the professor should already answer all possible questions if you keep listening. As you advance that becomes less and less true of course.
Also they are the 'weeder' classes as the staff put it to me. Basically everyone wants to go to college. Many can get in. But not everyone is cut out for it. Those classes are usually decently challenging but mostly lectures. They cut out the people who should not be there or are not ready for it. They are also decently big because almost all of the curriculum paths require these core classes. Your second year will be 20-30 people. By the time you are in your 4th year most classes are 10-15 people usually mixed graduate/undergraduate people.
That's only true of lecture and your hypothesis about answering all potential questions is out of whack with how humans actually learn. Someone asking a question might just be a matter of not hearing something repeated enough times or maybe they just need to hear it phrased in a different way. Even with large classes, there are usually a bunch of TAs to try and provide more one-on-one help and often labs that exist solely to encourage questions to the TAs and reinforce material in a smaller setting.
I went to a top 15 cs department and every single undergrad cs class except the senior project had 150+ people. They also had discussion sections though but usually there were 2 TAs running all the discussions so I don’t think enough to run oral exams.
Lectures absolutely are attended by 200-300 students at a time in parts of the EU.
The professor hate teaching those classes, students hate attending them and as a result half the people will drop out before they get to experience the real classes.
You can have multiple sections of the same course. At the university I went to, there were about ~150 CS students per year. The very largest classes were 50 students each, with 3 sections of the same course, sometimes taught by the same professor, sometimes taught by different professors. As you progressed through the program, the number of "every CS major takes this course" courses reduced, and most classes were 20-something students.
Similar in Finland. Courses like math and physics which were required were common for more than one program. Thus even if programs were small these first classes ended up being massive. And there were actually multiple of those classes even then between multiple Departments...
One thing I don't understand is that despite automating more and more of the work away, we seem to have fewer and fewer humans available for anything that's left. I have to scan my own groceries, I can't ever talk to an actual human at any business, I order my food from a machine.
Didn't you know? Everyone is engaged. What did you think the point of all those engagement metrics were? To get everybody engaged. I like it better like this, before, almost no one was engaged.
The boomers are retiring, and the US economy is strong. Here in EU boomers are also retiring, but many countries still have high youth unemployment, and there is no real worker shortage even if businesses like to complain about one.
Get used to it. The emerging model is a virtual vending machine - if what you want isn't there, tough. If the machine breaks, well, sorry about your dollar, maybe it'll work next week.
If you want to talk to a human, start by getting rich. That costs extra.
I've wondered about this myself. It's not just grocery scanning-- my parents used to get a bunch of bills in the mail every month, and send out a bunch of checks to pay them. It's stunning to think about how many people were needed at every step of that pipeline to make the whole thing work. All of that's automated now, and you can find similar things in many industries.
That should have freed up lots of people to handle the special cases that fall through the cracks of the automation, but they're nowhere to be found.
As you may have heard, in the US at least, there’s record low unemployment. There’s literally not enough people to do those jobs. It’s complex why…
1. Demographic changes means more retirees, and smaller workforce. Literally less people to go around.
2. With low unemployment, everyone’s doing something else. Something harder to automate, or otherwise more valuable. It’s not like people are sitting around unemployed (except retirees).
What are they doing? It’s hard to know without combing through data in this moment, but people are surely avoiding minimum wage and low skill customer service jobs if alternatives exist in their area. Anecdotally, I hear people don’t like working in anything customer facing due to rude customers. I know someone in TX working at Amazon warehouses at an hourly rate much higher than retail workers rate. They used to work at a gym in rural California.
Until we get rising low-skill immigration to cover more of the low-wage work, expect service work to be understaffed. Or rising birth rates, or a drop in white collar workers.
It's not just that people are going for "better" jobs to avoid the lowest of low tier schlock jobs. It's that a dedicated chunk of those jobs refuse to hire because the employers are trying to downsize that part of their overall workforce and thus reduce costs. An example of this has been the rail shortage, where despite needing twice as many workers as they actually have in order to keep things from collapsing the rail companies are ignoring the collapse and trying to cut fifty or even hundred car trains down to single man crews.
You also cannot trust the Department Of Labor's official statistics by themselves since there's multiple ways that the U.S. government covers up actual unemployment numbers. For example stay at home parents aren't included in the unemployment statistic even if they worked previous to that situation. People who have been unemployed for more than a year are not included. People on disability but who are in an employable state are not included. Gig workers such as GrubHub drivers are not included. Certain methods of self-employment such as traveling musicians are are not included. They pulled this trick a decade ago and they're doing it again to try and prevent another panic. The one difference this time is that instead of people losing a single full-time job like they did in 2008, many are now working multiple part-times and thus inflating the employment statistic slightly since technically they're never unemployed.
Then there's the gap that happens for employees. If someone has a part-time job and gets scheduled from noon to nine for closing back to back for four days and then gets nothing for the next three days that means they're not getting paid the equivalent of a full-time job despite possibly working just as many if not more hours. When you have lulls like this you need a second part-time job to fill them in, leading to things like ten percent of the current counted national workforce having at least two jobs. But many part-time jobs in retail and services want to monopolize the person's time since scheduling is based on the employer's availability and not the employee's, meaning that sometimes the employee will have to purposely dedicate days of their week to waiting in case either job calls them in. If the person doesn't have a free day and are called in while working one of their two jobs, they may be fired from the other job. When enough places and people have an incongruent schedule like this places run short staffed and the person is still wasting a day or maybe more idle. Then there's the other half of that coin of scheduling abuse, the supposed part-time job that uses that flexible scheduling to transform into a full-time job. I know that Wal-Mart will schedule you for no more than thirty hours a week and will stack your shifts back to back, sometimes with as little as eight hours in-between, just so they have enough leftover hours in your schedule that they can call you back to cover someone without pushing you into full-time or paying overtime. This makes working for Wal-Mart hell because you're rushed and then all but told to be on standby, giving you very little time for things like other jobs, appointments, household chores, and even self care. The end result of being yanked between two part-times or being held hostage by a full-time masquerading as a part-time is that eventually the potential employee gets cynical and stops applying for that kind of work entirely.
> People who have been unemployed for more than a year are not included.
Yep, let's just exclude all the people who tell the real story here.
Every gen-z in my family (and neighbors!) also have 18-21 year-olds who are doing weird shit like working for two days up to a max of two weeks, then abandoning the job. I don't know what this is about ("working" once per quarter to get the work credits for Social Security?) or how this is captured by the stats, but they are by no means employed.
Honestly I could probably fit into that group you're describing on technicality. I usually do informal contract or commission work that I have to self report come tax season. When things slow down in the fall or right around New Year's I often pick up seasonal work or just reply to random help ads to get a few hundred dollars to cover the shortfall. I'm in a grey area already, which makes what content creators like livestreamers do even more of a mystery to me in regards to whether they're considered self-employed or doing the digital equivalent of street busking.
> 1. Demographic changes means more retirees, and smaller workforce. Literally less people to go around.
A small point, but I think what you meant to say here is that there's a greater ratio of retired to employed people than there used to be, which is somewhat different than what you stated.
That said, I think this idea that we necessarily need some incredible amount of low-skill immigration to cover some drastic understaffing is bullshit propaganda put out to achieve bad wages and treatment of workers through basic supply and demand (drastically increasing the supply of labor). Technical innovations such as self-checkout lines and great security cameras and other tools of the modern age mean that you need fewer workers to do the same work.
As I think others have pointed out, the unemployment rate figures can be massaged in a lot of ways to be misleading. Actual wages should be the stat that tells you whether or not the workforce is too small. If wages are dropping or stagnant versus inflation, then claims of low unemployment should be considered laughable because otherwise wages would be increasing.
Scaling is an issue for sure--just having the time in the week is impossible without distributing the work to TAs, and administering oral exams is tough for TAs.
But you can do an oral exam along the lines of "explain to me how your project works" and that'll catch people who don't understand it regardless of whether or not they know the question in advance.
You're right about oral examinations, but written _proctored_ exams would also solve a lot of these problems, particularly if you require it to be handwritten. Having take home essays be handwritten would also help deter using AI by making it much more inconvenient to use compared to just doing the assignment in the first place.
I went to high school (secondary school) back when it was pretty normal to forbid students from typing assignments because "not everyone has a computer". I happen to be personally responsible, via my terrible handwriting, for at least two teachers dropping that policy. At least one other just gave me a specific exception in the name of "accessibility".
A student a year ahead of me had atrocious handwriting. They allowed him to type up his AP tests after the answer period and included the typewritten sheets along with his handwritten booklet (so the graders could confirm he hadn’t altered his answers).
They did for me as well (dysgraphia is the technical word). They made me practice handwriting after class and I think all the times I resultingly burst into tears because my wrist hurt made them take pity on me.
For how much schools talk about how useful handwriting is at memorisation and such - I seldom see anybody in the working world pulling out some paper and handwriting notes. Schools seem to care more about teaching you to pass closed-book tests than they do about teaching you useful skills like how to create high quality, organised, and searchable notes.
That's a workaround for requiring homework to be handwritten, but I think it's not a big deal since there's already a workaround of just writing whatever ChatGPT spits out. That is why I mentioned it only as a deterrent against using ChatGPT for plagiarism. Such a solution doesn't compromise proctored exams at all.
Literally 100% of my degree's classification was determined in eight, three-hour exams taken in a big hall with everyone else and in handwritten response form.
Going back to handwritten exams might be a solution. It is known to scale well and there is still enough expertise running them on this level left.
Having locked down thin-clients does not scale well enough to these sizes. Specially when there is usually multiple concurrent exams running at same time. On small scale it could be solution. A screen, keyboard and mouse with locked down thin client and simple editor.
Unfortunately the tried-and-true multiple choice exam complete with Scantron answer sheets remains fairly resistant to cheating with AI and is fully scalable to classes of any size. The only hope for these poor students is that AI can help their professors write better test questions.
It doesn’t matter if students know what to expect.
An oral exam isn’t the same as reading a written exam out loud. There are a set of learning outcomes defined in the syllabus. The examiner uses the learning outcomes to ask probing questions that start a conversation - emphasis on conversation. A conversation can’t be faked. A simple follow up question can reveal a student only has a shallow understanding of material. Or, the student could remember everything they’ve been told, but fail to make connections between learning outcomes. You can’t cram for an oral exam. You have to digest the course material and think about what things mean in context.
After all, students know what to expect on standardized tests. Some still do better than others :-).
It can. An examiner questions are gonna be parsed via covert Voice recognition,processed by ai and replies played back via a e. g. small headset(a magnet inside an ear cavity, accompanied by an induction loop wire around the neck(works fine,google "микронаушник магнит" in russian, it's popular here) , and it's not as hard as you may think.
Most oral exams I've been to have been either a) talk about a report you've written, or b) draw 1 of x (say 8) subjects that have been covered throughout the year, and do a presentation on it, usually with 20-30 minutes preparation. All subjects have of course been published at least 2 weeks prior to the exam.
So I don't see a problem with telling "what to expect", but I do see a problem with 1 professor and one censor examining 300 students for for 30 minutes each.
Students should have the right to be shown exactly where and why texts are flagged. Much of the snake oil will disappear pretty quickly if the solutions are required to provide any form of proof.
It's would be funny, if it also wasn't so sad, that the point is that the students need to prove that they did the work, while shitty software can flag their work as being plagiarism or AI generated while providing no proof.
Sounds like a nightmare is about to start for kids, parents, schools, teachers and so on. Would be easily solved by going back to oral examinations, paper and pen in class essays...
Or better yet, get rid of all HOME-work. Do it all inclass.
We train children and up to adulthood that encroaching on time for self is acceptable, and its absolutely not. I remember asshole teachers giving out essay homeworks during christmas breaks and similar, because "we needed to be busy".
No wonder why wage theft is the biggest retail theft we have - those utter lack of boundaries are eroded the moment we set foot in schools.
at university level that's not really going to work. you have lectures and assignments and maybe meetings to discuss assignments or present work. there are no more classes like in school. at least there weren't in my university.
you are also missing one important aspect: work is usually 40 hours a week.
full time study should be 40 hours too. but at least where i come from, you generally have less than 20 hours of lectures and TA meetings, etc. so you are absolutely expected to spend the other 20 hours in your room or in the library working on assignments. that is not encroaching on your time.
there is only a problem if you spend more time on assignments causing you to work/study more than 40 hours in a week
Yep the old "blue book" exams I had to sit. You get the exam questions on a piece of paoer, and a small booklet of blank paper in which to write your answers. Called a "blue book" because the cover sheet on the booklet was a sky blue color.
Or you can just trust your students. Most upper-division courses I took at university gave take-home exams. "Here's the test, come back with it in a couple days, don't talk about your answers with other students".
Being kicked out of the science school and into the business school is a pretty huge punishment that late in the game, for negligible reward: "Oooh, I aced my Particle Physics final".
It was just easier to do the test honestly, and lick your wounds later.
Yeah, I remember in a freshman lab that I was TAing for I spotted an assignment that was clearly a bad photocopy of an email (complete with email address at the bottom), which wasn't the student's email. So I went and found a name that matched up with that email, which was another student, and at that point I felt the need to look through all the assignments looking for this particular copy's calling card, a strange rounding error on one of the tables. Ended up finding about a dozen cases of cheating on that prelab, alone. A prelab worth, generously, half of a percent of their overall grade. Let the professor know, finished grading the rest.
Next week I had a dozen more missing assignments than usual, went to the professor asking "hey, why did a quarter of the class not turn in their assignments this week?", and they replied that the students had been served an ultimatum: Drop engineering, go transfer somewhere else in shame, or get hit with an academic dishonesty inquiry, which could carry much stiffer sentences.
They all took the former. The theory was that students cheat because they're either unwilling or unable to engage with the material, which means they're not cut out for engineering.
The entire idea of "spit out knowledge in 1 hour without references" is such a terrible test of one's capability in the real world that it should not be done at all.
"The more things change, the more they stay the same."
When I was a student decades ago it was when they first introduced anti-plagiarism software. We all had to submit a printed out copy of our book reports along with a floppy disk or cd-rom of our report in the latest Microsoft Word format. Almost the whole class was flagged for plagiarism. Of course the school takes plagiarism very seriously so we all had to have a one-on-one with the principal. The real truth is no one copied each others work and it was the anti-plagiarism software flagging essays because students quoted parts of the book and expanded on that. In hindsight as an adult I think the excessive principal one-on-ones was the teachers way of shoving it to him and his new software.
Yeah I remember TurnItIn. It sucked hard. And of course all Uni staff put 100% trust in it to be completely infallible and not be a piece of crap bit of software.
Mathematics programs in Europe still use oral examinations, though they are increasingly rare in the United States. I'm not European, but I have always preferred oral examinations over written examinations because I can actually communicate my understanding of the problem to the professor and outline my thought process towards a solution, something that is much harder to do in writing, especially since you cannot get any hints or feedback on your attempts during a written examination -- you're on your own.
They need to be sued into pain. They are part of the problem and are hurting people. People are going to cheat, we know this, it is an incentive problem. But there is no excuse for false positives
As per usual the problem isn't the technology or the tools, it's the dumb meatbag using them.
This teacher should have thought "hmm, well if I'm going to pull up students on this then it's probably important I do my own research to make sure that I know that the score means and how reliable it is" but nope.
"LLMs are bad cause people will use them to generate spam and fake news!" LLMs aren't the bad actor there...it's the human doing the spamming.
There is a fine line, though. For example the "guns don't kill people argument" is pretty moot, gun control saves lives. If someone grabs an LLM, tells it to "generate text telling young impressionable teens to kill themselves" and spam it on Twitter...well the human is doing bad stuff but they're still enabled by the LLM. However it's easy to enact something like gun control on a physical object. Software? Well, damn that's gonna a problem - especially when the metric by which these LLMs "do better" is how closely the text resembles a hooman.
Has anyone field-tested strategies to fighting allegations like this and would like to share them?
Something I remember from a previous discussion around this topic was writing everything in e.g., Google Docs, where there are captured versions every few seconds. The idea being that the version log shows that one did not copy-and-paste larger chunks of the text.
Due to the nature of the art process it's not uncommon for artists to stream live on youtube/twitch with some music and get a decent number of viewers. Sadly the same wouldn't apply for writing I think.
Yeah, but what if they have a bag of chips, and inside that bag of chips, is a small tablet or phone, in which they ask chat gpt to give them the essay, and then they transcribe it? They might even eat some of those chips to obfuscate that they're getting the information from the bag.
At some level, there has to be some trust, or a system (e.g., in person proctored exams) would have to be used instead.
I bet students would start to print out the plagiarized essays and retype them. If the plagiarists made an effort to make minor changes and rearrange things as they went, no screen recorder or edit history could distinguish it from normal essay writing.
I would try to pass the teacher's message through an AI checker, then answer "Sorry, I can't accept your comment because it was 34% written by an AI, and I only obey to my teacher, not ChatGPT". I don't know if that would work, but that's very clearly the kind of thing I would have done when I was a teenager.
Of course it is. For example, you can make it a game: how can you write in a way that could never be confused with AI? As a bonus, this is almost guaranteed to mean that you'll have a unique style.
Writers have been putting challenges to themselves since writing has existed. Minimize number of words, write without using certain letters, etc. It's fun.
This person (furry?) just chooses to give up at the slightest obstacle.
All of this in addition to making sure that they are writing correctly about the subject and including the topics being required of them by the professor. I can agree this is a likely approach by some people but I can also see how it's just kicking the can down the road; they are helping nobody but themselves.
This social media post can also be seen as an attempt to bring attention to the problem. That is not necessarily the same thing as "giving up at the slightest obstacle". It might instead be seen that you disagree with their choice of the front at which to fight.
Correct, but the people who should adapt are the institutions and professors.
This is not, I think, and unreasonable requirement either, we just need a few professors to be hit with large lawsuits arguing torturous interference with employment (since what most people are in college for is a better job anyway).
If such a lawsuit was successful, wouldn't all professors have the incentive to just pass everyone with top grades, lest someone decide to sue them for "interfering with employment"?
I would sue the teacher and school if this happened to me… total BS.
Also plagarism and having other people do your work, or even take exams for you is as old as time. How is AI any different? Some trust is in order here. The students are the ones paying to be educated, if they cheat they are only cheating themselves anyways.
>if they cheat they are only cheating themselves anyways.
I've heard people say this a lot. It's a very old-school liberal arts perspective that doesn't fit with the modern world. In practice, for most students, education is a third-place benefit of university behind 1. the certificate and 2. networking.
People who see education as a valuable credential care.
And yes, there are a lot of people who would like to outsource questions of competence to, "Do you have the credentials that I am looking for?" Any cynicism about the competence of people who do this is fully justified. But conversely, it isn't the worst approach to take to evaluating people when you personally lack competence.
As an example, I would never hire a programmer based on credentials. But it is often the first and only thing that I look for in a random doctor visit.
If I go to my doctor, or counsellor, or engineer, or mechanic, or... the list goes on... I care that they actually got an education rather than cheated their way through it.
These are all things I can't personally judge competence in, because I'm not competent in medicine, psychology, engineering, etc. The most reasonable way for me to judge competence is whether an accredited school vouches for them via a degree.
Do you want to compete with someone for a job that cheated to get an education and got the job over you? By allowing this to continue, it will just add to the further decline of society.
If they're at another firm... Yes! My company of people who actually know the stuff the certificates say they know will run circles around them in the market.
But if we're talking someone the company just hired who is now my responsibility... Not so much.
I perceive a widespread, cynical belief that the primary purposes of higher education are things like signaling, credentials, and networking. Students who accept this belief are whom I refer to as tragically misled. It's not unthinkable that they're in the majority. On the other hand, I think a fructifying education is up for grabs at virtually any college.
I told my kids that a large state university is like a tiny elite academy, hidden in plain sight within a giant party school.
Don't get me wrong, I know many people had bad college experiences, and that higher education is ripe for reform.
As I recall from my kid’s college search, the majority of US college students attend state schools or community colleges. While there are very many private colleges in number, they are each usually much smaller than state schools in enrollment. Private schools are also vastly outnumbered by community colleges, even though community colleges are on average smaller in enrollment..
It's a curious choice to call the liberal arts "they are only cheating themselves" attitude old-school. One could debate whether the certificate-motivated or culture-motivated type of university came first.
I suspect that advanced hunter-gatherer and early agricultural societies viewed education more as purely motivated by culture: basic useful life skills can be passed on informally within communities or families (so formal education is not strictly necessary for survival), and typical tasks would unlikely to be a risk to others if done incorrectly (making certificates valueless).
However, the modern Western university could arguably originate from the Church (and Jewish institutions before that), in which the education would surely have been considered necessary to correctly interpret texts. In that context, the certificate is a reassurance to the general, uneducated public that they are being led by qualified spiritual leaders (the clergy).
Once you get to the Renaissance and towards the Enlightenment era, it seems as if the increasingly secular focus of education yet again reduces the necessity of certificates and makes broad, 'polymathematic' science and culture more important. The certificates only become important in secular life again once the apprenticeship system breaks down in the late 19th century.
Finally, the idea of a 'liberal arts' course seems to be a reaction to keep the spirit of the Enlightenment alive despite ubiquitous industrialisation, so in this light is both an old school and a new school of though, depending on how you look at it!
These are my thoughts from a very British perspective, so I would be curious to know whether that fits with views from other cultures, such as East Asian or predominantly Catholic ones.
I find the “liberal arts” thing weird. First, as a scientist with an engineering degree, I definitely feel if you cheat through school you’re only cheating yourself. In the hard sciences a lack of knowledge and ability is pretty easy to suss out. The degree gets you the interview, the network might open a door or two, but the education is how you get the job. And once you’ve got it, if you didn’t learn anything, you will be fired fast.
It’s more likely in a liberal art you can fluff your way through life living a continuous con. But heres the thing - they still are cheating themselves. Because the only thing a certificate and a network based on lies gives you is hollow employment built on lies. The education is what gives you a career and a life worth living.
So, maybe that’s old school, but I’ve rarely found old school wisdom to be wrong. It has, after all, withstood the filter of time.
Unfortunately it’s not true for most positions in the tech industry, including the highest paying ones. All of these are game-able. Spend a month running through leetcode questions and learning some jargon, and it won’t matter whether you cheated your way through school, you’ll know enough to pass the interview.
Even on the job people can often get away with not providing much value if they have a bad/apathetic/misaligned manager.
It isn't all or nothing. Computer Science degree, and I regularly use the principles I learned in my CS classes, but the required English classes? Not so much. So some "cheating" wouldn't be cheating myself.
Yes it does but it could be worse. Right now not everyone does this, some students do their work honestly but they may be forced to compete and use AI. It's quite a lose lose situation if you ask me.
Yes, but it's just one of a multitude of ways someone could cheat at work, including lying flat out about holding degrees (which in most cases isn't even illegal), or simply faking their effort in the job, and not really doing it.
If people are going to cheat themselves out of an education, and then cheat at a job... ultimately they will be 'found out' for not doing their job properly. Unless there is absolutely no objective way to evaluate, measure, or observe job performance, in which case the job is probably nonsense anyway and cheating doesn't matter.
The world is filled with (some would say entirely composed of) people who are cheating society, usually in much much worse ways.
Why would I care if I work with people who cheated at college? I work for a business that constantly cheats me, my coworkers, its customers, its suppliers, the government, and innocent people with no formal affiliation to the business at all. This is what businesses exist to do. The only people a business theoretically exists NOT to cheat is its investors, and their empirical record there is mixed at best.
I'm not in any position to evaluate if a doctor is doing a good job or not (almost nobody is, not even doctors themselves), but in principle I am not opposed to their cheating if it doesn't affect my "care" (i.e. my family doctor cheating on some exam about the esoterica of podiatry or some other specialty they would normally just refer me for).
That's a waste of your time and your money. When you consult with your lawyer they're going to point out the Code of Student Conduct you agreed to that stipulates how automated tools are being used to detect plagiarism - tools that have been in place and in use for over ten years at this point.
If anything, the new generation of AI tools should reduce the false positives. As another commenter here noted, even humans weren't perfect at this task. We've never been able to reduce the false positives to zero.
Meanwhile, you owe your lawyer $350 for your initial consultation.
I'm skeptical. It might not make sense financially, but falsely accusing someone of something that wasn't done isn't absolved by using automated tools. It still requires a person to interpret the results and assess a penalty.
Does the Code of Student Conduct say that plagiarism is bad? Or does it say that some secret program will assess the paper and be the sole judge of plagiarism? I'm guessing it's not the latter.
Personally, I would stand my ground and tell the professor straight up, "I'm not writing it again, I've written it once." Leave them with the understanding of the hell to be paid for accusing me falsely.
When I didn't do the work, I accepted the bad grades -- I didn't cheat. When I do the work, they will accept it, or we'll hash it out to the end of the earth.
This is the answer... it is a matter of principle, and is worth a fight, on principle. If they can do it to you, they will do it to somebody else that might not have the ability to stop them that you do.
Ironic you wrote so authoritatively while completely missing the topic being discussed: this isn't AI to detect plagiarism, this is AI to supposedly detect AI output.
That isn't done by looking for plagiarism, they're building snake oil models trained on GPT output.
Imagine an automated plagiarism detector that functioned by decapitating the student, putting their brain in a blender, and examining the results. Could any contractual release that the student signed make this ok?
You can disclaim negligence, but you can’t disclaim malice or recklessness. You might find that this contract is not worth very much, depending on the facts of exactly how stupid this software is and how hard the school tried to find out whether it was stupid or not.
But if their justification for assigning the score is a factual claim that I broke the rules, they'd best be ready to justify that claim or they're committing defamation.
In the real world, defamation must include communication to a third party. Schools do not communicate grades to third parties (unless you explicitly instruct them to).
If anything, leading a bad product review would be closer to "defamation" than this example.
I am an Eagle Scout and was awarded it in 1995. As part of that process, I had to write up a detailed document outlining my project including how it was conceived, the work, and the outcome.
It was rejected on my initial submission because the reviewers felt I it wasn’t written by a sophomore in HS.
My parents rewrote the entire thing in what they described as “baby talk” and it was then accepted. The irony is not lost on me to this day.
This is nothing new; it’s been happening forever and will continue to happen, we just have a new reason to drive a race to the bottom.
If they don't trust an Eagle candidate's attestation of authorship, they probably shouldn't have let (or been the ones to let) the candidate progress that far.
It's not handled at the troop level. The Eagle review is at district and council level. It's common for candidates to receive "assistance" to help along the process and the review boards are on the lookout for that.
The main reason I am not an Eagle Scout is because I took two classes at a winter camp for merit badges, did everything from signup and turned in every bit of work, and then at the end of the class they told me they had no record of me being in it. I did all the work for every other part of that award, but I flat out refused to redo those badges again. I've never jumped through hoops to make up for the failures of lazy leadership, and I never will.
Well, you're an Eagle Scout in everything but name, and honestly that might be better -- you get the personal satisfaction, but you don't do the thing all of us find extremely tedious about Eagle Scouts, which is constantly mentioning (in your 40s) how you're an Eagle Scout.
doing work and not having it recognized but in fact rejected because someone somewhere lost their notes is not the path to personal satisfaction.
a lot of merit badge work is based on trust. it should not be to hard to ask a few questions about the work done to be able to judge if it was really done.
most of this should be in the control of the scout masters. if they can't trust their scouts to be honest about this and work out a compromise then their troop has a real problem
This was at a winter camp. My scoutmaster was not there. It was me and two other kids from my troop along with plenty of others from other troops. I (a) signed up for the classes, (b) paid for the classes, (c) completed all the work for the classes. According to them, instead of trying to figure out why I was attending and turning in the assigned work, they just tossed it. I don't respect truly lazy people, and that was truly lazy. So anyway, I made my case, they rejected it, and I decided I had enough. I didn't need their useless external validation, especially since I had already attended every bit of advanced and leadership training within my grasp due to my work and character. Eagle Scout was icing on the cake, it wasn't the cake.
The council-level Eagle Board of Reviewers likely didn't know the individual scouts and were simply applying their individual knowledge about the content reviewed compared to the whole and made, in this case, poor assumptions about the quality of writing they should expect.
As it relates to academia today, this is now happening at a larger scale with folks lacking more trust, not having to think critically about the individual, and applying a similar outcome based on this. This all goes back to critical thought, understanding your audience, and using tools/knowledge in a way it shouldn't be used; it's just now being used more often and with less control than ever.
To address your specific comment that those folks may not have been qualified to assess scouts attempting to obtain Eagle, I cannot speak to that, but at the face of it, it would absolutely seem they were definitely not exposed to enough quality writing nor did they have the wherewithal to make decisions at the level they were tasked. Working in the business world, I am continually faced with a wide swath of college-educated adults who simply are unable to write coherently, so none of it surprises me today, but it certainly did when I was 15-16 and thought adults were more capable than actually were/are.
I've started referring to this phenomena as "Goodhart's Hell." We're living in a world with quickly increasing complexity. Variance in samples is much larger because our reach in what we sample is larger (we talk to people far from our homes, with different cultures, interpretations of words, language patterns, behaviors, ethics, etc). But we've responded to this only by simplifying our evaluations. We've created more tools and metrics that can help guide us, but we don't use them as guides but answers. We're using mathematics/technology not as a tool or language, but a black box. A black box with high accuracy too! Noise/uncertainty has increased but so has our confidence in our answers. It is entirely unwarranted. In essence, we are just hacking metrics and over-fitting our models to increase laziness and comfort. Our ever pursuit of meritocracy ironically takes us further from achieving the goals because we are unwilling to embrace the noise that is inherent to the system, believing that the noise is a lie, believing we can accurately measure everything to absolute (or even meaningful) precision. We have not only abandoned nuance but have actively turned against it and I do not know why. I think it deserves the hell in its name.
In part it's due to requirements of transparency. If some stranger is going to be evaluating you, a simple way to ensure that you're not being evaluated unfairly is to have the evaluation criteria be very simple, almost to the point of being mechanistic. If you're part of a small enough community it's likely that your evaluator, their overseer, and your peers all know you, so it's very difficult for unfairness to creep in unnoticed. If your community is gigantic everyone starts to be strangers to everyone else, so people can't intuitively know whether someone is being treated unfairly or not. When everyone knows each other it's much more difficult to game the system, because people can tell when you've been lazying around.
I don't disagree with this response. It's actually a good one. But I think we need to be really aware of the trade-offs here. My problem is that people use these metrics as if they are hard proof of competence. But the truth is you have to pick your noise: nepotism or lazy evaluation. Either way, you aren't getting rid of noise.
I actually think scale is part of the problem here. In small communities there is more natural nuance and correction as there's high accountability. But with scale accountability often disappear as you disappear into the crowd. And you'll always be able to point at winners and losers to whichever noise you favor. The problem is we're not looking at things holistically and ensuring that our metrics are aligned with our desired outcomes. Instead we often pick a metric because it reasonably looks like it should align with the desired outcome (not proven, just intuited) and then just tell anyone that disagrees that it is obvious instead of acknowledging the downsides. Every metric has limitations. Ignoring them is the problem.
But, they don't have any knowledge. They don't know how many kids got parents' help and weren't caught, or how many of the kids they "caught" really did know how to write at that level.
wow, computers writing essays that cannot be distinguished from human written essays and the ramifications of this have been going on for… ever. i dont know how to say it nicely… youre just blatantly wrong.
Right. I don't know what to call it, but I'm annoyed by the near pathological need for some people to respond to every piece of news with "same as it ever was".
> This is nothing new; it’s been happening forever and will continue to happen, we just have a new reason to drive a race to the bottom.
The joke about the pinewood derby was that you could always tell the one kid who actually built his own car (which was the whole point, after all). Because every other car had been carefully engineered, carved, machined, sanded, painted, lacquered, and weighted by a dad.
Sounds like you had the honor of being in the only scout troupe that at least tried to keep this kind of thing from happening. And your singular false positive seems to have somehow projected itself onto the world.
If the point of pinewood derby is not to at least provide an opportunity (not necessity) for kids and parents to work on a project together, I don't know what is.
I had a similar experience in the 5th grade. We had a creative writing assignment (write x many pages), and I wrote a short story about "space spiders" that spun giant webs between planets and moons to catch something. I can't recall.
The only thing more mundane and unimaginative than that two pages was my teacher, who accused me of plagiarizing it.
and i bet the teacher could not tell you which story you were supposed to have this plagiarized from.
in a way it's a compliment that your teacher thought the story was so good, that they didn't believe it was your own.
i have had this happen in sports. i was not very sportive and tended to pick easier activities when there was a choice. there was this school sports day where the whole school was doing activities outside, and one of those activities was running for half an hour, one hour or two hours. usually i'd pick the half hour to be done with it, but this one day, my brother, a few friends and me decided to run in a group and do the two hours.
doing things in a group can be a lot easier as everyone encouraged each other. we also didn't race but it was a relaxed slow jog.
at the end we would get a pass indicating that we had completed the activity. when i showed that to my teacher, he didn't believe me.
there was no consequence of that, but if necessary i would have had three witnesses.
in hindsight the disappointment was that the teacher did not understand or recognize my capacity. i don't think he knew that i had been biking for hundreds or even thousands of km during school holidays.
> I am an Eagle Scout [...] I hate the world we live in.
Isn't there a merit badge or something in Optimistic Can-Do American Spirit? :)
> My parents rewrote the entire thing in what they described as “baby talk” and it was then accepted. The irony is not lost on me to this day.
If you could do it over, would you let yourself be pressured into ironically submitting something that your parents wrote?
Also, do you think whomever rejected your initial writeup did so because they didn't think you did it yourself? Or more like they thought you did it, but they wanted a more stereotypical youth style, maybe because they thought it was more the intent of the exercise? Or maybe for optics reasons, so that it looked like the image people have of a Scout, when upper officials and donors read it?
In my country's equivalent to the UK sixth form or later US high school years (ages ~16 to ~18), I once forgot an English as a foreign language essay assignment until late in the evening before the deadline.
Being a somewhat lazy and absent-minded teenager, by then I had also completely forgotten what exactly the assignment was or what I was supposed to be writing about. So I wrote some kind of a bleeding heart piece about the prevalence of violence and war in the world or whatnot.
I was quite pleased with it. Of course it was probably naive teenager stuff, exactly the kind you'd think is pretty smart when you're in high school, but I felt good about it at the time. I also felt the English was pretty good, which of course was the point of the assignment. And I pulled it off in some kind of a flow state I conjured up at the last moment.
The teacher didn't think I had written it. She asked me if I had got the text from some IT firm or something. (This was around the dotcom boom, and I probably seemed like an obvious nerd. Those local dotcom firms had the image of being at the forefront of internationalization and anglicization. I still don't know why she'd think IT firms would make a point of writing high school English essays for nerds.)
It was my mistake, of course, and I understand her suspicion because I turned in something that didn't conform to the specific assignment at all. It's different than submitting work that is actually what was asked for and being rejected for doing it too well. But I did honestly feel a bit bitter about it nevertheless. And possibly a little proud.
When I was in college, they were going to put carpet in the science labs, and a few of us thought it was a really stupid, dangerous idea.
We started a petition and asked for signatures at the on-campus cafe.
After a few days, the administrators asked us into their office for a chat. They asked us stupid questions and tried to talk us out of our petition. When it didn't work, they eventually caved and didn't do the carpet that year.
They waited for the next year. Then a couple years later, they put back the tile.
I don't know why they were so scared of our petition, but I'm sure this is another situation that could benefit from one. It sends a very clear message about the situation to people very high up in the food chain. And I'm sure it could get a ton of signatures.
Have you ever interacted with a bureaucracy? They are an LLM assembled from human flesh: Random behavior, incomprehensible rationales, and an unstoppable and undirectable force of iron will.
It's a power thing. Authorities make arbitrary changes just because they can, and exercising their power feels good. They ignore the reason people object to those changes and interpret the objections as challenges to their power and direct personal insults.
Basicly, "I'm in charge" outweighs "this is dangerous".
This decision was not optimized for the operation of the chemistry lab. It was optimized to game the university's procurement process. It was a brilliant idea, if your objective was to exhaust your allocated facilities maintenance budget on something that you knew would have a very limited lifespan. That means you would have to request an even larger budget in just a year or two, in order to replace the destroyed carpet. Then your department gets both a new floor and a larger budget.
If you submit a word document, is there some way to detect if the words were typed out, how long the document was open etc? Basically trying to detect if the student pasted the text? Obviously students would get around this simply by typing it out word for word, but could work here.
No, there's no mechanism in the Word file format itself to store this information. There's the file system metadata (Created/Accessed/Modified time) which tells you for how long the file was open before being saved. However, file system metadata is likely to be lost when you submit the document.
Ultimately, you could not prevent this timer from being spoofed unless you installed rootkits on the students' computers, and unfortunately (for the security and privacy of students) rootkits are what is often used.
If you use Google Docs, the edit history would pretty clearly the progress and different drafts. I guess eventually somebody will prompt ChatGPT in the right way to create realistic drafts and edits as well.
Watch me Place my tablet with ChatGPT next to my laptop and type the essay off of that into Google Docs on the laptop. Whatever contrived solution you come up with, students will find ingenious solutions around them.
That's a valid way of testing a student's knowledge. I do think that tests somewhat different skills and knowledge than the 20+ hours I would spend on large undergraduate essays. Most critically, the longer format probably is better at training students to get published in academic journals.
As a professor at a community college, I am very hesitant to just declare something is AI generated unless is it very clear. If I suspect something is AI generated I run it through 3 different checkers that I found online and if they don't all agree then I don't call it plagiarism. Normally the writing is so vague that it isn't more than a C anyway.
It's frustrating that I can't know for certain who's going to win the 2024 election, but that doesn't mean I'd be justified in making a prediction based on a magic 8 ball.
AI detectors don't work.
If there aren't any other tools and methods that do detect AI, that genuinely stinks, it's a difficult problem. I definitely sympathize. But all the sympathizing in the world won't change the fact that AI detectors don't work. There are lots of unsolved problems in the world, and they don't justify putting weight behind non-solutions.
I don't know how to cure Alzheimer's disease, but I know that a TV healer can't cure it, and if doctors started prescribing TV healers it would be medical malpractice. Similarly, I don't know how to tell if an essay was AI written... but neither do the AI detectors, and it is unethical for professors to rely on them to make decisions about plagiarism.
When you have an unsolved problem and someone proposes a bad solution with bad consequences, what you end up with is the same unsolved problem, still unsolved, plus a newly created additional problem on top of it from your bad solution. You started with the problem of rampant plagiarism and cheating, now you have both a problem with rampant plagiarism and cheating, and a new problem with making baseless accusations of cheating against potentially honest students with no real evidence.
>3 different checkers that I found online and if they don't all agree then I don't call it plagiarism
How do you verify these "different checkers" aren't all going to the same api or using the same software? How do you know they aren't using a similar heuristic method? What is the false positive rate of each? Have you seen the research that shows text from non-native speakers are more often falsely detected as being AI[1]?
They don't work. If three services that don't work agree, it doesn't mean they are right. It just means whatever spurious metrics they use happen to align in that case.
"I know their predictions are statistically quite bad, so I used a magic 8 ball three times to make sure there wasn't any doubt."
----
How much allure for these products is there that a college professor can be on HN, know that the tools don't work, and still use them anyway?
And how are students supposed to combat that kind of misuse if even educating professors about the problem isn't enough to make them stop trusting the detectors? And for a professor to then to brag about it on HN that they're being responsible as if they expect to be praised for it?
It's just so frustrating to read... What do you do if you're a student in that situation, do you have to sit your professor down and not only show that the tools have problems but also explain the concept of correlated error?
I kind of feel like the only way this changes is with legal challenges to colleges until they eventually ban the tools from being used in grading. Public pushback isn't going to change it, schools aren't going to care about that. And if educating about the problems isn't going to change behavior, I can't expect students to walk every one of their professors through a stats class. I don't know what if any legal rights at all students actually have in these situations, I'm guessing most colleges have boilerplate that if they decide to randomly fail you tough luck.
But at the very least, @jccalhoun you should stop using these programs yourself; you have the individual ability to make your own classroom better, and you should.
This is apparently unpopular but my take on this is that if teachers want students to write something on which AI can get high scores, then whatever they're teaching is obsolete and will be replaced by AIs anyway. Of course, not mentioning AI as co-author remains cheating. But I think the best course of action is to simply allow students to use AI while mentioning it (and the prompts they've used).
I'm in philosophy and so far no AI has produced a text of interest to me. If it did, then I'd deal with it like with any arguments by colleagues.
My boss told me AI isn't quite ready for production yet, but I should expect in just a few years that I'm expected to use an AI for most of my coding tasks. AI can write some very good code, but sometimes it does some "why did you do it that way" things which might or might not be a valid shortcut.
> if teachers want students to write something on which AI can get high scores, then whatever they're teaching is obsolete
Doing trivial things is often the first step in learning how to do non-trivial things. No one has ever needed a five paragraph essay. It's a completely artificial construct that, hopefully, teaches people to structure an essay.
Electronic calculators have existed for decades but we're still teaching kids arithmetic.
My argument is not "if a machine can do it then it's not worth a human's time to do it". Reading provides advantages that doing arithmetic by hand does not.
What is the actual problem with AI generated text in and of itself?
If it was completely AI generated without any human intervention, it likely would have been fairly generic / poor writing anyway. At least for the time being, AI is a new tool at our disposal but it's still just a tool, like a calculator or a hammer.
Students should be judged on the merit of the work even if some of the work is AI generated.
The biggest problem I see is that learning usually requires effort/work and AI requires students to do less effort/work to achieve the same output. So the worry is that the quality of education received will decrease.
Example: no one cares that you wrote 5 pages on To Kill a Mocking Bird. They care that you read the book and thought critically about it. AI allows students to skip the reading and critical thinking portion, which is the most important part.
The message from the teacher looks like it was written by a bot. Compare notes with other students. If two students got exactly the same text, the teacher is outsourcing their own job to a program. Publicize that.
I don't think retaliation is going to benefit anyone here. If they wanted to be confrontational, I think a constructive approach where they have a conversation with the teacher about false positives in these tools would be better.
Take your money elsewhere. Education should be a market. If you feel that "retaliation" is your only option, then you are mostly likely in a monopolized and non-functioning market. If you get good results, it will be due to luck, and not to anything that you've paid for.
The student is paying a lot of money here and the contracted party was just caught not doing their job. Working with them so the teacher is not caught when they use bullshit programs wrong in the future should really not be the goal.
If I pay someone to check my car, and one day they tell me the windows were too dirty so they couldn't do their usual inspection from the outside... I don't clean the windows so they can continue doing their "inspection" without opening the car. I go elsewhere and report them as the risk to life and property that they are.
I have always spelled really well (almost won a spelling bee) and as soon as personal computers became a thing, I realized how useless a "superpower" it was. And now it might become an actual liability as nearly perfect spelling/grammar is an easy attribute to hang "possibly AI-generated" on.
If perfect spelling/grammar was so easy to have the moment "personal computers became a thing", then why do my eyes bleed trying to read what people write online every day? We've had near-flawless spellcheck for years now. I think maybe it's not as much of a useless skill as you think it might be...
I emphasize with the student here, but passing it through an ai detector yourself before submitting and keeping a version history seems like an easy way to combat any false claims.
Comical. Back in the day, when I had to write research papers I would go to Wikipedia by copying the citations verbatim, outline the page, and then essentially rewrite the whole thing in my voice but keeping the exact quotes from the original article since they were cited.
At that point, 70% of the hard work is done and you're just regurgitating. Liberal arts is long dead.
To bypass the obnoxious prompt HR wheenies use to filter job applicants, duh.
As for feeling cheated, yeah I do feel cheated considering the astronomical tuition for BS make work prerequisite liberal arts classes. But the income i make nowadays covers it entirely.
And I still learned something since I had to rewrite the Wikipedia article, but believe me there isn't anything to be learned grinding away doing regurgitation research.
For the record, college liberal arts don't teach you how to think, they teach you WHAT to think and if you deviate from that you get a reduced grade.
I wouldn't have qualms cheating my way into a degree program either if the opportunity presented itself. In world of cheats, thiefs, and late stage capitalism the honest man is the fool.
I am an educator at a UK university. The essay is rapidly ceasing to be an appropriate way to assess students knowledge and critical thinking.
We regularly organise in person face to face practical exams for our entire several hundred strong year group of undergraduates. It is possible to do assessment properly if the will is there.
When I was a student at a uk university, essays were, I think, written in examinations where one can’t easily use AI, as were most of the other things one would need for a degree. There was a small coursework component of my course and I think others. But I’m still a bit surprised — surely if you are getting a coursework essay from a history or English student or similar, the AI will be a worse writer and won’t have sufficiently detailed knowledge to write about the topic, so either the essay won’t really make any sense if you think about it (which you’d hope examiners would be able to tell) or the student will have done their own research and gotten the AI to form the students opinions into the essay, but they will then likely synthesise a much worse essay than the one the student could have written.
Doing practicals seems good, and I think proctored examinations can add objectivity too.
I agree - out of the box, a LLM will write an extremely poor essay. But that is not how to use the LLM. WHat you do, is give the topic to the LLM, and say, "with this topic, what are 20 titles that I could have for essays." Then, you pick the best title, and you then say, "I am writing an essay with the title <insert title>, what should the central argument for the essay be?" You then ask for a list of bullet points for the central theme of each paragraph to elaborate on that argument, then you ask it to write each paragraph individually, then you put the pieces together, and ask it to proof read it's own work and finally you add in some references of your own. Within 1/2 an hour, you have a passable essay. And if you then iterate on it with the LLM you can fairly soon have a half decent one, especially if it is a well trodden area (presumably with lots of training data that the LLM draws on).
I tried to get students to critique things, but even then you can put in the text to critique to a LLM with a long enough context, and the LLM will kick off with a passable critique, if you iterate with it enough.
So, even though they'll never get a top mark, they will still be able to get through the assessment. So I don't set essays any more.
This kind of thing infuriates me so much. Not only are the detectors not accurate, but the damned thing gave it a 27% score that it was written by an AI which means it actually predicted it WASN'T written as AI and this teacher is threatening their success over that. It's absolutely absurd.
Right?! What's the maximum acceptable level of AI-ness, anyway? If I have an AI proofread my paper and suggest a few alternative turns of phrase which sound better to my ear but earn me a 7% AI score, is that okay?
It feels more and more like we need to rethink what scoring schoolwork means.
Worksheets and exams can show how familiar you are with the material, or they can be gamed or useless in some other way. But the best they can ever do is show familiarity.
What I really want as a consumer of someone's grade or academic history is how well they WORK in the subject area, along with how well they work with other people, not just a score that may correlate with regurgitation of facts.
I feel cheated by my school years, because I was led to believe that regurgitation of facts is very important. I also feel like this previous comment will be very emotional for some people.
Colleges are no different at the undergraduate level in many, if not most, courses. Identify what the professor wants to see or hear, repeat it back.
Lab work is probably the biggest exception, in that you tend to need to apply some practical knowledge, but even then I found TAs were very willing to hold your hand through it if you at least demonstrated you had reviewed and memorized any prep materials.
> I feel cheated by my school years, because I was led to believe that regurgitation of facts is very important.
Which school is that?
I guess that's probably true of primary or middle / lower secondary school, but I don't feel like regurgitation of facts was particularly endorsed in my experience at upper secondary level, let alone at university level.
There might of course be quite a bit of variation between countries, schools and even individual teachers in this.
> but I don't feel like regurgitation of facts was particularly endorsed in my experience at upper secondary level,
In my high school experience, most classes could be aced through rote regurgitation of facts.
Math class required another step - identification of which category of problem you have, and rote application of one of a few pre-calc/calc algorithms.
That was my experience as well, and totally ruined my first year at university. The lecturers expected I'd been taught application and derivation of proofs, but in reality I'd only been taught the proofs required to pass the high school exams.
> I guess that's probably true of primary or middle / lower secondary school, but I don't feel like regurgitation of facts was particularly endorsed in my experience at upper secondary level, let alone at university level.
The problem is that by the time someone gets to graduate level courses, their mode of thinking has been set.
One of the best professors I ever had took a radically different approach to grading than I've seen anywhere else but history books: he gave you a grade based on his perception of your understanding of the material.
He graded assignments and exams, out of a courtesy to those who needed a numeric style of feedback, but there was no set way to translate your total scores on assignments into your final grade in the course.
Exams would be, in a word, brutal, in that no one was expected to ace them, or even complete them. What he wanted to see was that you'd be able to apply the course material to novel situations. He'd meet with every student during the course of longer term projects, just to get some insight into their work.
Ultimately, it wouldn't have scaled well beyond the 50-odd of us were in his course, but it was certainly AI proof.
At the time, I thoroughly enjoyed it because it drove the kids who were hyper-fixated on grades and point grubbing up a wall. Looking back, it was the best teaching style I ever experienced.
> One of the best professors I ever had took a radically different approach to grading than I've seen anywhere else but history books: he gave you a grade based on his perception of your understanding of the material.
This is fine--until the professor doesn't like you, personally.
You would have to give students much more flexibility to drop courses (basically up to almost the end of class) to avoid the social failure scenarios.
I like your direction of thought, but I think we should make sure we don’t over correlate education with vocational utility. School isn’t always about work, and not all education should be about work or jobs or whatever.
That’s why I really like that you included how well they work with others.
Personally, I never felt like school was about memorizing facts (maybe some history classes…), but I also had two parents with advanced degrees who repeatedly told me that school was about learning to think and learning to learn (and learning to time manage). I feel like the vocational focus a lot of people have on education really hurts some fields of study - the humanities gets a lot of hate both on HN and in the media.
Ideally, classes with projects and essays should push people to think and show understanding of the subject not facts, like you said. The infinitely mocked “why did the author say the curtains were green” tropes actually show this - understanding why authors use certain imagery requires understanding intent, historic context, deeper understanding, etc. especially when applied to new works. I think people like cut and dry fact memorization and objective grading, and this doesn’t fit that cleanly.
Maybe there's a better word than 'work'. When I say
> how well they WORK in the subject area
I don't mean 'perform in a paid job', I mean 'take productive steps'. So even if you're just noodling around having fun with a musical instrument, you can be working, if you're improving your control and understanding. Messing around without a productive aim can still be very productive.
Grades in university have done absolutely nothing for me. Good grades nor bad grades. What I got out of schooling was quite literally an education – access to perspectives and paradigms, learning methodologies, some expertise in a couple of fields, confidence in working with others. You know... learning.
There's not a lot of value in using AI for targeted tasks like a school essay. Nor is there much value in getting a grade for a targeted task like an essay (not only would the grade be high-noise and low-signal, it also has a tiny or non-existent practical effect).
I wonder if these AI tools will to some extent highlight the meaningless fluff (crust?) that vestigially hangs around. If our current incarnation of task-oriented AI is really good at some particular thing, and we humans readily hand that thing over to AI without regret/loss, what are the chances that that thing really mattered in the first place?
(Note that this is not to hypothesize that all things AI does are non-important.)
I personally believe that we have a weak grasp on the things we do that don't matter. We do a lot of things because we have always done them, and some of these have not been adequately questioned in years/decades. So any indication of what's important or not is very helpful.
(I'm aware my circumstances are unique – maybe for some grades in school have had a significant influence in their lives – I personally have not witnessed this, in myself or others.)
265 comments
[ 3.2 ms ] story [ 304 ms ] threadChatGPT could have the exact same set of questions to ask. All students are independently taking the same exam at the same time, responding conversationally. ChatGPT asks all of them the same question, and then probes based on weaknesses in the responses. Grades the answer to that question, then moves on to the next.
Everyone got equal questions. Is there going to be inconsistency in how minor wording trips different followups? Sure. But it isn't clear that this randomness is worse than the subjective differences in how humans grade an essay response to a question.
If you give a pre-written test, you basically need to curve it because you don't really know how well the test aligns with your expectations of the class's understanding of the material. So you say "typically, about 20% of the class doesn't understand the course at all, 20% have a very good grasp..." etc, and then fit the written test results to the expected curve.
A one-on-one communication with a student allows you to directly gauge their understanding, allowing you to assign a grade. It doesn't matter how well or poorly the rest of the class understands the material, this student, sitting in front of me, has a "B" level understanding.
We should not encourage stack ranking (or a curve) as a way to stratify students because they joined the class with too many smarter people. Go read up about stack ranking, and why it's a horrific practice.
In a "grade on a curve" situation, it benefits you to drop the class if there's too many people you think smarter or more experienced than you. And it benefits you if you're smarter than them. And, thats a REALLY UGLY way to approach education.
Not the LLM part though.
Not only do you get a good sense of what they know, but you get a chance to give real individual feedback about what they need to focus on next.
Of course towards the end in order to get a qualification that a university will accept as entrance we will need to teach the kids to pass the standardised tests too.
Fight fire with fire.
I've actually tested this with a VR app (with support for mobile also), and it actually works quite well.
I dunno about that level of automation. Maybe the proctors can use it to generate a big bag of questions beforehand, that they manually review, but generating them on the fly is just asking for trouble.
Administering the preset questions and responding to the student is a much better fit.
It's bad enough that these things don't do a good job, don't basically function for the task, but what's as bad or worse is being forced to pretend you buy into it all like having to pretend to believe in some religion just to not get killed by everyone else.
I'm not a student and so far I have been free to judge all these ais as bad on a variety of fronts, and not use them (directly anyway, of course they are used behind the scenes in ways that affect me which I can't control). This would be hell.
https://news.ycombinator.com/item?id=37768282
There haven’t been technological advances in education that have overwhelmed the student teacher ratio in terms of effectiveness.
On some level we just have to accept this.
I actively remind my students this when I teach, who honestly are less active post pandemic. It is something I worry about. And though my account is "anonymous", if you ever show up to my lectures, not as a student but just sitting in (I don't take attendance, idk if you walk off the street and want to sit in), you too are welcome to participate.
(Fwiw, there aren't really "PhD level lectures" so much as graduate lectures. Generally you stop taking classes at the end of year 2 in your PhD, which is equivalent to a masters. The distinction of the PhD is that you are learning to learn on your own. This does add extra context to why grad students ask for higher pay, especially at the non-conditional and candidate levels, as they're actually taking less resources from the school. E.g. I pay the school $10k/term (big round) for research credits, which requires no involvement from anyone. There no longer is the "student" part to my "51% student, 49% employee" split)
Good on you for allowing auditing. I had only one teacher in college who allowed people to audit my classes and he even fought the administration to give me credit for the class after I sheepishly approached him and asked if he would mind letting me taking the exam to make sure I understood the material. One of the most insightful classes I took my entire time in college
(Now, I grant you, in a room of 300, interaction is severely limited. I'm lucky my classes tend to have fewer than 20 people.)
If I give a lecture that you could have just watched a video of, I fell short.
It's tough to get right. I'm still working on it.
A well designed class session can be (can be) similar in that it provides a space and guidance for you to do 60 minutes of improving your brain, that let's be honest, you would otherwise half-ass at home. (Of course, the classroom is only part of what you're paying for, there are also assignments, feedback, etc.)
Absolutely. Even in the largest lectures, people can ask clarifying questions, live, from the subject matter expert. Likewise, the professor can look at the students' body language and realize that they've lost a significant portion of the class, and stop, slow down, and maybe prompt the class for questions.
There is an art to a good lecture, and it's definitely harder the larger the class, but good lectures cannot be replaced by recordings.
1. Real life lecturer that can be asked questions
2. Peers that you can meet and discuss both study-related and unrelated topics.
3. Language. Not relevant for intros, but advanced topics can be explained in more than one way, and as a student you not necessarily will be able to map between them.
4. Schedule. Self-pacing 6 different classes for 12 weeks may be harder than it looks. Instead, you can just attend classes.
5. Advanced topics. Online resources, especially videos become scarcer as you advance in your studies, more so if you are not in hot topic (e.g., CS). I still have some notes from classes I took as undergrad as a best source for the learned topics. Good luck finding video lectures about diagrammatic approach to real time dynamics of open spin systems.
Same goes the other way around, sometimes attending class is just not worth it, even without tons of online courses. I learned my E&M by just reading the official book during the lecture time (see 4.).
I've studied in the UK and EU, and I don't think I've ever been in a class larger than 40.
Smaller schools and upper level classes generally just have fewer people.
Edit: Keep in mind that teaching is something the professor usually just puts up with, to get the resources and credibility to do the thing they are really passionate about (research or writing).
Note that the above only applies if the class is supposed to be a lecture. If it is supposed to be interactive then small classes are better. However many of the basic undergrad 1st year courses have been done so often the professor should already answer all possible questions if you keep listening. As you advance that becomes less and less true of course.
The professor hate teaching those classes, students hate attending them and as a result half the people will drop out before they get to experience the real classes.
Well that's easy: An AI will administer the oral exam and transcribe the oral answers from each person simultaneously, and possibly even grade it!
Where is everyone?
And it is going to get far, far worse.
Get used to it. The emerging model is a virtual vending machine - if what you want isn't there, tough. If the machine breaks, well, sorry about your dollar, maybe it'll work next week.
If you want to talk to a human, start by getting rich. That costs extra.
That should have freed up lots of people to handle the special cases that fall through the cracks of the automation, but they're nowhere to be found.
1. Demographic changes means more retirees, and smaller workforce. Literally less people to go around.
2. With low unemployment, everyone’s doing something else. Something harder to automate, or otherwise more valuable. It’s not like people are sitting around unemployed (except retirees).
What are they doing? It’s hard to know without combing through data in this moment, but people are surely avoiding minimum wage and low skill customer service jobs if alternatives exist in their area. Anecdotally, I hear people don’t like working in anything customer facing due to rude customers. I know someone in TX working at Amazon warehouses at an hourly rate much higher than retail workers rate. They used to work at a gym in rural California.
Until we get rising low-skill immigration to cover more of the low-wage work, expect service work to be understaffed. Or rising birth rates, or a drop in white collar workers.
You also cannot trust the Department Of Labor's official statistics by themselves since there's multiple ways that the U.S. government covers up actual unemployment numbers. For example stay at home parents aren't included in the unemployment statistic even if they worked previous to that situation. People who have been unemployed for more than a year are not included. People on disability but who are in an employable state are not included. Gig workers such as GrubHub drivers are not included. Certain methods of self-employment such as traveling musicians are are not included. They pulled this trick a decade ago and they're doing it again to try and prevent another panic. The one difference this time is that instead of people losing a single full-time job like they did in 2008, many are now working multiple part-times and thus inflating the employment statistic slightly since technically they're never unemployed.
Then there's the gap that happens for employees. If someone has a part-time job and gets scheduled from noon to nine for closing back to back for four days and then gets nothing for the next three days that means they're not getting paid the equivalent of a full-time job despite possibly working just as many if not more hours. When you have lulls like this you need a second part-time job to fill them in, leading to things like ten percent of the current counted national workforce having at least two jobs. But many part-time jobs in retail and services want to monopolize the person's time since scheduling is based on the employer's availability and not the employee's, meaning that sometimes the employee will have to purposely dedicate days of their week to waiting in case either job calls them in. If the person doesn't have a free day and are called in while working one of their two jobs, they may be fired from the other job. When enough places and people have an incongruent schedule like this places run short staffed and the person is still wasting a day or maybe more idle. Then there's the other half of that coin of scheduling abuse, the supposed part-time job that uses that flexible scheduling to transform into a full-time job. I know that Wal-Mart will schedule you for no more than thirty hours a week and will stack your shifts back to back, sometimes with as little as eight hours in-between, just so they have enough leftover hours in your schedule that they can call you back to cover someone without pushing you into full-time or paying overtime. This makes working for Wal-Mart hell because you're rushed and then all but told to be on standby, giving you very little time for things like other jobs, appointments, household chores, and even self care. The end result of being yanked between two part-times or being held hostage by a full-time masquerading as a part-time is that eventually the potential employee gets cynical and stops applying for that kind of work entirely.
Yep, let's just exclude all the people who tell the real story here.
Every gen-z in my family (and neighbors!) also have 18-21 year-olds who are doing weird shit like working for two days up to a max of two weeks, then abandoning the job. I don't know what this is about ("working" once per quarter to get the work credits for Social Security?) or how this is captured by the stats, but they are by no means employed.
A small point, but I think what you meant to say here is that there's a greater ratio of retired to employed people than there used to be, which is somewhat different than what you stated.
That said, I think this idea that we necessarily need some incredible amount of low-skill immigration to cover some drastic understaffing is bullshit propaganda put out to achieve bad wages and treatment of workers through basic supply and demand (drastically increasing the supply of labor). Technical innovations such as self-checkout lines and great security cameras and other tools of the modern age mean that you need fewer workers to do the same work.
As I think others have pointed out, the unemployment rate figures can be massaged in a lot of ways to be misleading. Actual wages should be the stat that tells you whether or not the workforce is too small. If wages are dropping or stagnant versus inflation, then claims of low unemployment should be considered laughable because otherwise wages would be increasing.
But you can do an oral exam along the lines of "explain to me how your project works" and that'll catch people who don't understand it regardless of whether or not they know the question in advance.
For how much schools talk about how useful handwriting is at memorisation and such - I seldom see anybody in the working world pulling out some paper and handwriting notes. Schools seem to care more about teaching you to pass closed-book tests than they do about teaching you useful skills like how to create high quality, organised, and searchable notes.
Having locked down thin-clients does not scale well enough to these sizes. Specially when there is usually multiple concurrent exams running at same time. On small scale it could be solution. A screen, keyboard and mouse with locked down thin client and simple editor.
An oral exam isn’t the same as reading a written exam out loud. There are a set of learning outcomes defined in the syllabus. The examiner uses the learning outcomes to ask probing questions that start a conversation - emphasis on conversation. A conversation can’t be faked. A simple follow up question can reveal a student only has a shallow understanding of material. Or, the student could remember everything they’ve been told, but fail to make connections between learning outcomes. You can’t cram for an oral exam. You have to digest the course material and think about what things mean in context.
After all, students know what to expect on standardized tests. Some still do better than others :-).
It can. An examiner questions are gonna be parsed via covert Voice recognition,processed by ai and replies played back via a e. g. small headset(a magnet inside an ear cavity, accompanied by an induction loop wire around the neck(works fine,google "микронаушник магнит" in russian, it's popular here) , and it's not as hard as you may think.
So I don't see a problem with telling "what to expect", but I do see a problem with 1 professor and one censor examining 300 students for for 30 minutes each.
It's would be funny, if it also wasn't so sad, that the point is that the students need to prove that they did the work, while shitty software can flag their work as being plagiarism or AI generated while providing no proof.
We train children and up to adulthood that encroaching on time for self is acceptable, and its absolutely not. I remember asshole teachers giving out essay homeworks during christmas breaks and similar, because "we needed to be busy".
No wonder why wage theft is the biggest retail theft we have - those utter lack of boundaries are eroded the moment we set foot in schools.
https://www.cbsnews.com/news/owed-employers-face-little-acco...
you are also missing one important aspect: work is usually 40 hours a week. full time study should be 40 hours too. but at least where i come from, you generally have less than 20 hours of lectures and TA meetings, etc. so you are absolutely expected to spend the other 20 hours in your room or in the library working on assignments. that is not encroaching on your time.
there is only a problem if you spend more time on assignments causing you to work/study more than 40 hours in a week
It was just easier to do the test honestly, and lick your wounds later.
Next week I had a dozen more missing assignments than usual, went to the professor asking "hey, why did a quarter of the class not turn in their assignments this week?", and they replied that the students had been served an ultimatum: Drop engineering, go transfer somewhere else in shame, or get hit with an academic dishonesty inquiry, which could carry much stiffer sentences.
They all took the former. The theory was that students cheat because they're either unwilling or unable to engage with the material, which means they're not cut out for engineering.
When I was a student decades ago it was when they first introduced anti-plagiarism software. We all had to submit a printed out copy of our book reports along with a floppy disk or cd-rom of our report in the latest Microsoft Word format. Almost the whole class was flagged for plagiarism. Of course the school takes plagiarism very seriously so we all had to have a one-on-one with the principal. The real truth is no one copied each others work and it was the anti-plagiarism software flagging essays because students quoted parts of the book and expanded on that. In hindsight as an adult I think the excessive principal one-on-ones was the teachers way of shoving it to him and his new software.
This teacher should have thought "hmm, well if I'm going to pull up students on this then it's probably important I do my own research to make sure that I know that the score means and how reliable it is" but nope.
"LLMs are bad cause people will use them to generate spam and fake news!" LLMs aren't the bad actor there...it's the human doing the spamming.
There is a fine line, though. For example the "guns don't kill people argument" is pretty moot, gun control saves lives. If someone grabs an LLM, tells it to "generate text telling young impressionable teens to kill themselves" and spam it on Twitter...well the human is doing bad stuff but they're still enabled by the LLM. However it's easy to enact something like gun control on a physical object. Software? Well, damn that's gonna a problem - especially when the metric by which these LLMs "do better" is how closely the text resembles a hooman.
Something I remember from a previous discussion around this topic was writing everything in e.g., Google Docs, where there are captured versions every few seconds. The idea being that the version log shows that one did not copy-and-paste larger chunks of the text.
Most laptops can handle this pretty effortlessly with their encode blocks.
At some level, there has to be some trust, or a system (e.g., in person proctored exams) would have to be used instead.
No it most certainly is not.
Writers have been putting challenges to themselves since writing has existed. Minimize number of words, write without using certain letters, etc. It's fun.
This person (furry?) just chooses to give up at the slightest obstacle.
Why are talking like that to me? Are you responding to someone else?
I literally cannot comprehend how my suggestion that playing with words might be fun could be interpreted as bro-y.
EDIT: Lol I just skimmed your comment history and it's just garbage like this.
Really? You suggest experimentation with fun styles, and you're surprised by a response in a fun style? Bro...
This social media post can also be seen as an attempt to bring attention to the problem. That is not necessarily the same thing as "giving up at the slightest obstacle". It might instead be seen that you disagree with their choice of the front at which to fight.
One of the important adulthood things that many people fail to learn is that they should pick their battles.
This is not, I think, and unreasonable requirement either, we just need a few professors to be hit with large lawsuits arguing torturous interference with employment (since what most people are in college for is a better job anyway).
Fuck, I have to spell out every detail.
Also plagarism and having other people do your work, or even take exams for you is as old as time. How is AI any different? Some trust is in order here. The students are the ones paying to be educated, if they cheat they are only cheating themselves anyways.
I've heard people say this a lot. It's a very old-school liberal arts perspective that doesn't fit with the modern world. In practice, for most students, education is a third-place benefit of university behind 1. the certificate and 2. networking.
The rest of us might too, if we expect "I have a degree in X from Y university" to imply some level of competence.
And yes, there are a lot of people who would like to outsource questions of competence to, "Do you have the credentials that I am looking for?" Any cynicism about the competence of people who do this is fully justified. But conversely, it isn't the worst approach to take to evaluating people when you personally lack competence.
As an example, I would never hire a programmer based on credentials. But it is often the first and only thing that I look for in a random doctor visit.
These are all things I can't personally judge competence in, because I'm not competent in medicine, psychology, engineering, etc. The most reasonable way for me to judge competence is whether an accredited school vouches for them via a degree.
If they're at another firm... Yes! My company of people who actually know the stuff the certificates say they know will run circles around them in the market.
But if we're talking someone the company just hired who is now my responsibility... Not so much.
I told my kids that a large state university is like a tiny elite academy, hidden in plain sight within a giant party school.
Don't get me wrong, I know many people had bad college experiences, and that higher education is ripe for reform.
I suspect that advanced hunter-gatherer and early agricultural societies viewed education more as purely motivated by culture: basic useful life skills can be passed on informally within communities or families (so formal education is not strictly necessary for survival), and typical tasks would unlikely to be a risk to others if done incorrectly (making certificates valueless).
However, the modern Western university could arguably originate from the Church (and Jewish institutions before that), in which the education would surely have been considered necessary to correctly interpret texts. In that context, the certificate is a reassurance to the general, uneducated public that they are being led by qualified spiritual leaders (the clergy).
Once you get to the Renaissance and towards the Enlightenment era, it seems as if the increasingly secular focus of education yet again reduces the necessity of certificates and makes broad, 'polymathematic' science and culture more important. The certificates only become important in secular life again once the apprenticeship system breaks down in the late 19th century.
Finally, the idea of a 'liberal arts' course seems to be a reaction to keep the spirit of the Enlightenment alive despite ubiquitous industrialisation, so in this light is both an old school and a new school of though, depending on how you look at it!
These are my thoughts from a very British perspective, so I would be curious to know whether that fits with views from other cultures, such as East Asian or predominantly Catholic ones.
It’s more likely in a liberal art you can fluff your way through life living a continuous con. But heres the thing - they still are cheating themselves. Because the only thing a certificate and a network based on lies gives you is hollow employment built on lies. The education is what gives you a career and a life worth living.
So, maybe that’s old school, but I’ve rarely found old school wisdom to be wrong. It has, after all, withstood the filter of time.
What a great way to respond. I couldn’t agree more.
Even on the job people can often get away with not providing much value if they have a bad/apathetic/misaligned manager.
The problem is the cheater will then enter into society and tell people "I'm a graduate of $prestigious_university".
At that point they are cheating society.
Would you want to work with people who cheated? Would you want your doctor to be someone who cheated?
If people are going to cheat themselves out of an education, and then cheat at a job... ultimately they will be 'found out' for not doing their job properly. Unless there is absolutely no objective way to evaluate, measure, or observe job performance, in which case the job is probably nonsense anyway and cheating doesn't matter.
Why would I care if I work with people who cheated at college? I work for a business that constantly cheats me, my coworkers, its customers, its suppliers, the government, and innocent people with no formal affiliation to the business at all. This is what businesses exist to do. The only people a business theoretically exists NOT to cheat is its investors, and their empirical record there is mixed at best.
I'm not in any position to evaluate if a doctor is doing a good job or not (almost nobody is, not even doctors themselves), but in principle I am not opposed to their cheating if it doesn't affect my "care" (i.e. my family doctor cheating on some exam about the esoterica of podiatry or some other specialty they would normally just refer me for).
If anything, the new generation of AI tools should reduce the false positives. As another commenter here noted, even humans weren't perfect at this task. We've never been able to reduce the false positives to zero.
Meanwhile, you owe your lawyer $350 for your initial consultation.
Does the Code of Student Conduct say that plagiarism is bad? Or does it say that some secret program will assess the paper and be the sole judge of plagiarism? I'm guessing it's not the latter.
Personally, I would stand my ground and tell the professor straight up, "I'm not writing it again, I've written it once." Leave them with the understanding of the hell to be paid for accusing me falsely.
When I didn't do the work, I accepted the bad grades -- I didn't cheat. When I do the work, they will accept it, or we'll hash it out to the end of the earth.
That isn't done by looking for plagiarism, they're building snake oil models trained on GPT output.
You can disclaim negligence, but you can’t disclaim malice or recklessness. You might find that this contract is not worth very much, depending on the facts of exactly how stupid this software is and how hard the school tried to find out whether it was stupid or not.
Anecdotes I've heard put the new "AI detecting AI-written"'s accuracy at worse than a coin flip.
A school can assign whatever score they want to an assignment, just as you can leave whatever review you want to a product.
In the real world, defamation must include communication to a third party. Schools do not communicate grades to third parties (unless you explicitly instruct them to).
If anything, leading a bad product review would be closer to "defamation" than this example.
If a class is graded on a bell curve then kids who cheat are cheating their classmates who don't.
It was rejected on my initial submission because the reviewers felt I it wasn’t written by a sophomore in HS.
My parents rewrote the entire thing in what they described as “baby talk” and it was then accepted. The irony is not lost on me to this day.
This is nothing new; it’s been happening forever and will continue to happen, we just have a new reason to drive a race to the bottom.
I hate the world we live in.
If they don't trust an Eagle candidate's attestation of authorship, they probably shouldn't have let (or been the ones to let) the candidate progress that far.
But, as the parent of an Eagle scout, I think my point still stands. Either they can, or can't, trust the Eagle to be honest.
IMHO, that trust should be a necessary condition for awarding Eagle.
And if the council isn't capable of making that call, then they're perhaps unqualified for deciding whether or not to award Eagle rank.
(I realize my position is a bit simplistic. I paired it down for brevity.)
a lot of merit badge work is based on trust. it should not be to hard to ask a few questions about the work done to be able to judge if it was really done.
most of this should be in the control of the scout masters. if they can't trust their scouts to be honest about this and work out a compromise then their troop has a real problem
As it relates to academia today, this is now happening at a larger scale with folks lacking more trust, not having to think critically about the individual, and applying a similar outcome based on this. This all goes back to critical thought, understanding your audience, and using tools/knowledge in a way it shouldn't be used; it's just now being used more often and with less control than ever.
To address your specific comment that those folks may not have been qualified to assess scouts attempting to obtain Eagle, I cannot speak to that, but at the face of it, it would absolutely seem they were definitely not exposed to enough quality writing nor did they have the wherewithal to make decisions at the level they were tasked. Working in the business world, I am continually faced with a wide swath of college-educated adults who simply are unable to write coherently, so none of it surprises me today, but it certainly did when I was 15-16 and thought adults were more capable than actually were/are.
I still hate the world.
I actually think scale is part of the problem here. In small communities there is more natural nuance and correction as there's high accountability. But with scale accountability often disappear as you disappear into the crowd. And you'll always be able to point at winners and losers to whichever noise you favor. The problem is we're not looking at things holistically and ensuring that our metrics are aligned with our desired outcomes. Instead we often pick a metric because it reasonably looks like it should align with the desired outcome (not proven, just intuited) and then just tell anyone that disagrees that it is obvious instead of acknowledging the downsides. Every metric has limitations. Ignoring them is the problem.
A scout is _____
(∩ ⌣̀_⌣́)
The joke about the pinewood derby was that you could always tell the one kid who actually built his own car (which was the whole point, after all). Because every other car had been carefully engineered, carved, machined, sanded, painted, lacquered, and weighted by a dad.
Sounds like you had the honor of being in the only scout troupe that at least tried to keep this kind of thing from happening. And your singular false positive seems to have somehow projected itself onto the world.
The only thing more mundane and unimaginative than that two pages was my teacher, who accused me of plagiarizing it.
in a way it's a compliment that your teacher thought the story was so good, that they didn't believe it was your own.
i have had this happen in sports. i was not very sportive and tended to pick easier activities when there was a choice. there was this school sports day where the whole school was doing activities outside, and one of those activities was running for half an hour, one hour or two hours. usually i'd pick the half hour to be done with it, but this one day, my brother, a few friends and me decided to run in a group and do the two hours.
doing things in a group can be a lot easier as everyone encouraged each other. we also didn't race but it was a relaxed slow jog.
at the end we would get a pass indicating that we had completed the activity. when i showed that to my teacher, he didn't believe me.
there was no consequence of that, but if necessary i would have had three witnesses.
in hindsight the disappointment was that the teacher did not understand or recognize my capacity. i don't think he knew that i had been biking for hundreds or even thousands of km during school holidays.
Isn't there a merit badge or something in Optimistic Can-Do American Spirit? :)
> My parents rewrote the entire thing in what they described as “baby talk” and it was then accepted. The irony is not lost on me to this day.
If you could do it over, would you let yourself be pressured into ironically submitting something that your parents wrote?
Also, do you think whomever rejected your initial writeup did so because they didn't think you did it yourself? Or more like they thought you did it, but they wanted a more stereotypical youth style, maybe because they thought it was more the intent of the exercise? Or maybe for optics reasons, so that it looked like the image people have of a Scout, when upper officials and donors read it?
In my country's equivalent to the UK sixth form or later US high school years (ages ~16 to ~18), I once forgot an English as a foreign language essay assignment until late in the evening before the deadline.
Being a somewhat lazy and absent-minded teenager, by then I had also completely forgotten what exactly the assignment was or what I was supposed to be writing about. So I wrote some kind of a bleeding heart piece about the prevalence of violence and war in the world or whatnot.
I was quite pleased with it. Of course it was probably naive teenager stuff, exactly the kind you'd think is pretty smart when you're in high school, but I felt good about it at the time. I also felt the English was pretty good, which of course was the point of the assignment. And I pulled it off in some kind of a flow state I conjured up at the last moment.
The teacher didn't think I had written it. She asked me if I had got the text from some IT firm or something. (This was around the dotcom boom, and I probably seemed like an obvious nerd. Those local dotcom firms had the image of being at the forefront of internationalization and anglicization. I still don't know why she'd think IT firms would make a point of writing high school English essays for nerds.)
It was my mistake, of course, and I understand her suspicion because I turned in something that didn't conform to the specific assignment at all. It's different than submitting work that is actually what was asked for and being rejected for doing it too well. But I did honestly feel a bit bitter about it nevertheless. And possibly a little proud.
We started a petition and asked for signatures at the on-campus cafe.
After a few days, the administrators asked us into their office for a chat. They asked us stupid questions and tried to talk us out of our petition. When it didn't work, they eventually caved and didn't do the carpet that year.
They waited for the next year. Then a couple years later, they put back the tile.
I don't know why they were so scared of our petition, but I'm sure this is another situation that could benefit from one. It sends a very clear message about the situation to people very high up in the food chain. And I'm sure it could get a ton of signatures.
I mean, why change the flooring type at all?
Why was the carpet so important to them?
Basicly, "I'm in charge" outweighs "this is dangerous".
Ultimately, you could not prevent this timer from being spoofed unless you installed rootkits on the students' computers, and unfortunately (for the security and privacy of students) rootkits are what is often used.
Just in-person exams, right?
AI detectors don't work.
If there aren't any other tools and methods that do detect AI, that genuinely stinks, it's a difficult problem. I definitely sympathize. But all the sympathizing in the world won't change the fact that AI detectors don't work. There are lots of unsolved problems in the world, and they don't justify putting weight behind non-solutions.
I don't know how to cure Alzheimer's disease, but I know that a TV healer can't cure it, and if doctors started prescribing TV healers it would be medical malpractice. Similarly, I don't know how to tell if an essay was AI written... but neither do the AI detectors, and it is unethical for professors to rely on them to make decisions about plagiarism.
When you have an unsolved problem and someone proposes a bad solution with bad consequences, what you end up with is the same unsolved problem, still unsolved, plus a newly created additional problem on top of it from your bad solution. You started with the problem of rampant plagiarism and cheating, now you have both a problem with rampant plagiarism and cheating, and a new problem with making baseless accusations of cheating against potentially honest students with no real evidence.
How do you verify these "different checkers" aren't all going to the same api or using the same software? How do you know they aren't using a similar heuristic method? What is the false positive rate of each? Have you seen the research that shows text from non-native speakers are more often falsely detected as being AI[1]?
[1] https://twitter.com/emollick/status/1705057915625697712
"They don't predict the future."
"I know their predictions are statistically quite bad, so I used a magic 8 ball three times to make sure there wasn't any doubt."
----
How much allure for these products is there that a college professor can be on HN, know that the tools don't work, and still use them anyway?
And how are students supposed to combat that kind of misuse if even educating professors about the problem isn't enough to make them stop trusting the detectors? And for a professor to then to brag about it on HN that they're being responsible as if they expect to be praised for it?
It's just so frustrating to read... What do you do if you're a student in that situation, do you have to sit your professor down and not only show that the tools have problems but also explain the concept of correlated error?
I kind of feel like the only way this changes is with legal challenges to colleges until they eventually ban the tools from being used in grading. Public pushback isn't going to change it, schools aren't going to care about that. And if educating about the problems isn't going to change behavior, I can't expect students to walk every one of their professors through a stats class. I don't know what if any legal rights at all students actually have in these situations, I'm guessing most colleges have boilerplate that if they decide to randomly fail you tough luck.
But at the very least, @jccalhoun you should stop using these programs yourself; you have the individual ability to make your own classroom better, and you should.
jfc education is dead
I'm in philosophy and so far no AI has produced a text of interest to me. If it did, then I'd deal with it like with any arguments by colleagues.
This teacher is failing the test.
Time will tell.
Doing trivial things is often the first step in learning how to do non-trivial things. No one has ever needed a five paragraph essay. It's a completely artificial construct that, hopefully, teaches people to structure an essay.
Electronic calculators have existed for decades but we're still teaching kids arithmetic.
I remain unconvinced that it's a worthwhile use of time.
Also, I'd still want a foundation of critical thinking skills to be built inside the brain of the next generation, not outsourced to AI.
If it was completely AI generated without any human intervention, it likely would have been fairly generic / poor writing anyway. At least for the time being, AI is a new tool at our disposal but it's still just a tool, like a calculator or a hammer.
Students should be judged on the merit of the work even if some of the work is AI generated.
Example: no one cares that you wrote 5 pages on To Kill a Mocking Bird. They care that you read the book and thought critically about it. AI allows students to skip the reading and critical thinking portion, which is the most important part.
You're absolutely right, and any participants in this market should be looking to maximize their grade per dollar spent.
(What do you think they should be maximizing?)
What happens when you have to maximize a function with more than one variable or order?
If I pay someone to check my car, and one day they tell me the windows were too dirty so they couldn't do their usual inspection from the outside... I don't clean the windows so they can continue doing their "inspection" without opening the car. I go elsewhere and report them as the risk to life and property that they are.
For one thing, one AI “detector” will not necessarily return the same result as another.
At that point, 70% of the hard work is done and you're just regurgitating. Liberal arts is long dead.
To clear a prerequisite requirement for the degree program which I considered a waste of time and money — the forced prerequisite that is.
Why pay to go to school if you're just regurgitating a Wikipedia page? Wikipedia's free.
As for feeling cheated, yeah I do feel cheated considering the astronomical tuition for BS make work prerequisite liberal arts classes. But the income i make nowadays covers it entirely.
And I still learned something since I had to rewrite the Wikipedia article, but believe me there isn't anything to be learned grinding away doing regurgitation research.
For the record, college liberal arts don't teach you how to think, they teach you WHAT to think and if you deviate from that you get a reduced grade.
I wouldn't have qualms cheating my way into a degree program either if the opportunity presented itself. In world of cheats, thiefs, and late stage capitalism the honest man is the fool.
We regularly organise in person face to face practical exams for our entire several hundred strong year group of undergraduates. It is possible to do assessment properly if the will is there.
Doing practicals seems good, and I think proctored examinations can add objectivity too.
I tried to get students to critique things, but even then you can put in the text to critique to a LLM with a long enough context, and the LLM will kick off with a passable critique, if you iterate with it enough.
So, even though they'll never get a top mark, they will still be able to get through the assessment. So I don't set essays any more.
Worksheets and exams can show how familiar you are with the material, or they can be gamed or useless in some other way. But the best they can ever do is show familiarity.
What I really want as a consumer of someone's grade or academic history is how well they WORK in the subject area, along with how well they work with other people, not just a score that may correlate with regurgitation of facts.
I feel cheated by my school years, because I was led to believe that regurgitation of facts is very important. I also feel like this previous comment will be very emotional for some people.
Lab work is probably the biggest exception, in that you tend to need to apply some practical knowledge, but even then I found TAs were very willing to hold your hand through it if you at least demonstrated you had reviewed and memorized any prep materials.
Which school is that?
I guess that's probably true of primary or middle / lower secondary school, but I don't feel like regurgitation of facts was particularly endorsed in my experience at upper secondary level, let alone at university level.
There might of course be quite a bit of variation between countries, schools and even individual teachers in this.
In my high school experience, most classes could be aced through rote regurgitation of facts.
Math class required another step - identification of which category of problem you have, and rote application of one of a few pre-calc/calc algorithms.
There were definitely people in my uni course who got worse grades, but who had a better understanding of the material.
The problem is that by the time someone gets to graduate level courses, their mode of thinking has been set.
He graded assignments and exams, out of a courtesy to those who needed a numeric style of feedback, but there was no set way to translate your total scores on assignments into your final grade in the course.
Exams would be, in a word, brutal, in that no one was expected to ace them, or even complete them. What he wanted to see was that you'd be able to apply the course material to novel situations. He'd meet with every student during the course of longer term projects, just to get some insight into their work.
Ultimately, it wouldn't have scaled well beyond the 50-odd of us were in his course, but it was certainly AI proof.
At the time, I thoroughly enjoyed it because it drove the kids who were hyper-fixated on grades and point grubbing up a wall. Looking back, it was the best teaching style I ever experienced.
This is fine--until the professor doesn't like you, personally.
You would have to give students much more flexibility to drop courses (basically up to almost the end of class) to avoid the social failure scenarios.
That’s why I really like that you included how well they work with others.
Personally, I never felt like school was about memorizing facts (maybe some history classes…), but I also had two parents with advanced degrees who repeatedly told me that school was about learning to think and learning to learn (and learning to time manage). I feel like the vocational focus a lot of people have on education really hurts some fields of study - the humanities gets a lot of hate both on HN and in the media.
Ideally, classes with projects and essays should push people to think and show understanding of the subject not facts, like you said. The infinitely mocked “why did the author say the curtains were green” tropes actually show this - understanding why authors use certain imagery requires understanding intent, historic context, deeper understanding, etc. especially when applied to new works. I think people like cut and dry fact memorization and objective grading, and this doesn’t fit that cleanly.
> how well they WORK in the subject area
I don't mean 'perform in a paid job', I mean 'take productive steps'. So even if you're just noodling around having fun with a musical instrument, you can be working, if you're improving your control and understanding. Messing around without a productive aim can still be very productive.
There's not a lot of value in using AI for targeted tasks like a school essay. Nor is there much value in getting a grade for a targeted task like an essay (not only would the grade be high-noise and low-signal, it also has a tiny or non-existent practical effect).
I wonder if these AI tools will to some extent highlight the meaningless fluff (crust?) that vestigially hangs around. If our current incarnation of task-oriented AI is really good at some particular thing, and we humans readily hand that thing over to AI without regret/loss, what are the chances that that thing really mattered in the first place?
(Note that this is not to hypothesize that all things AI does are non-important.)
I personally believe that we have a weak grasp on the things we do that don't matter. We do a lot of things because we have always done them, and some of these have not been adequately questioned in years/decades. So any indication of what's important or not is very helpful.
(I'm aware my circumstances are unique – maybe for some grades in school have had a significant influence in their lives – I personally have not witnessed this, in myself or others.)
I think the marker could also do with some improvement of their writing style.