Very interesting write up, would be curious to know more about what an Open Source Strategies course entails, as far as I can remember I never had anything like that on offer at my university.
I'm a university professor, and the amount of students who seem to be in need of LLM as a crutch is growing really exponentially.
We are still in a place where the oldest students did their first year completely without LLMs. But younger students have used LLMs throughout their studies, and I fear that in the future, we will see full generations of students completely incapable of working without LLM assistance.
One helper here is fear. You can be failed for formal errors at university, and it means we were scared shitless of making them and payed close attention.
If people know "at university you can't use LLM, you are forced to think by yourself" they will adjust, albeit by trial of fire.
I think there's an argument that growing up in an educational system unable to teach you how to not rely on LLM would for all intents and purposes permanently nerf you compared to more fortunate peers. Critical thought is a skill we continue to practice until the very end
Interesting write up! I’ve thought about how university exams are done effectively nowadays. I took my degree in CS almost 20 years ago, and being a user of LLMS - I can’t really see how any of my old exams would work today if students would be allowed LLMs.
> 3. I allow students to discuss among themselves [during an exam] if it is on topic.
Makes me wonder if they should also get a diploma together then, saying "may not have the tested knowledge if not accompanied by $other_student"
I know of some companies that support hiring people as a team (either all or none get hired and they're meant to then work together well), so it wouldn't necessarily be a problem if they wish to be a team like that
Quite a thoughtful way to adapt exams to wave of new tools for students and learn on the way.
I wished other universities adapt so quickly too (and have such a mindful attitude to students e.g. try to understand them, be upfront with expectations, learning from students etc).
Majority of professors are stressed and treat students as idiots... at least that was the case decade a go!
rare here: well written and insightful, I would take this course. I'm curious about why he penalized chatbot mistakes more, at first glance sounds like just discouraging their use but the hole setup indicates genuine desire to let it be a possibility. In my mind the rule should be "same penalty and extra super cookies for catching chatbot mistakes"
I wrote this before to another comment like yours:
I thought this part of penalizing mistakes made with the help of LLMs more was quite ingenious.
If you have this great resource available to you (an LLM) you better show that you read and checked its output. If there's something in the LLM output you do not understand or check to be true, you better remove it.
If you do not use LLMs and just misunderstood something, you will have a (flawed) justification for why you wrote this. If there's something flawed in an LLM answer, the likelihood that you do not have any justification except for "the LLM said so" is quite high and should thus be penalized higher.
One shows a misunderstanding, the other doesn't necessarily show any understanding at all.
Only 2 students actually used an LLM in his exam, one well and one poorly so I'm not sure there is much you can draw from this experience.
In my experience LLMs can significantly speed up the process of solving exam questions. They can surface relevant material I don't know about, they can remember how other similar problems are solved a lot better than I can and they can check for any mistakes in my answer. Yes when you get into very niche areas they start to fail (and often in a misleading way) but if you run through practise papers at all you can tell this and either avoid using the LLM or do some fine tuning on past papers.
Something that I think many students, indeed many people, struggle with is the question "why should I know anything?"
For most of us--myself included--once you graduate from college, the answer is: "enough to not get fired". This is far less than most curriculums ask you to know, and every year, "enough to not get fired" is a lower and lower bar. With LLMs, it's practically on the floor for 90% of full-time jobs.
That is why I propose exactly the opposite regimen from this course, although I admire the writer's free thinking. Return to tradition, with a twist. Closed-book exams, no note sheets, all handwritten. Add a verbal examination, even though it massively increases examination time. No homework assignments, which encourage "completionist mindset", where the turning-in of the assignment feels more real than understanding the assignment. Publish problem sets thousands of problems large with worked-out-solutions to remove the incentive to cheat.
"Memorization is a prerequisite for creativity" -- paraphrase of an HN comment about a fondly remembered physics professor who made the students memorize every equation in the class. In the age of the LLM, I suspect this is triply true.
But "enough to not get fired" is not an answer to a question "why should I know anything?". To be honest, it's not clear if the rest of your post tries to answer the initial question of why you should know anything or the implied question of how much should I really know.
You didn't answer why the student should memorize anything, except the hand-waving "Memorization is a prerequisite for creativity".
Students had very good reason to question the education system when they were asked to memorize things that were safe to forget once they graduated from school. And when most functional adults admitted they forgot what they had learned in school. It was an issue before LLM, and triply so now.
By the way, I now am 100% agree with "Memorization is a prerequisite for creativity." However, if you asked me to try to convince the 16-year-old me I would throw my hands up.
But suppose you think strictly in utilitarian terms: what effort should I invest for what $$$ return. I have two things to say to you:
First: what a meaningless life you're living.
Second: you realize that if you don't learn anything because you have LLMs, and I learn everything because it's interesting, when you and I are competing, I'll have LLMs as well...? We'll be using the same tools, but I'll be able to reason and you won't.
I think the people who struggle with the question "Why should I know anything?" aren't going to learn anything anyway. You need curiosity to learn, or at least to learn a lot and well, and if you have curiosity you're not asking why you should learn anything.
Something often left out is the dependence on LLM’s. Students today assume LLM’s will always be available, at a price they (or their companies) can afford.
What happens if LLM’s suddenly change their cost to be 1000 USD per user per month? What if it is 1000 USD per request? Will new students and new professionals still be able to complete their jobs?
> The third chatbot-using student had a very complex setup where he would use one LLM, then ask another unrelated LLM for confirmation. He had walls of text that were barely readable. When glancing at his screen, I immediately spotted a mistake (a chatbot explaining that "Sepia Search is a compass for the whole Fediverse"). I asked if he understood the problem with that specific sentence. He did not. Then I asked him questions for which I had seen the solution printed in his LLM output. He could not answer even though he had the answer on his screen.
Is it possible, and this is an interesting one to me, that this is the smartest kid in the class? I think maybe.
That guy who is playing with the latest tech, and forcing it to do the job (badly), and could care less about university or the course he's on. There's a time and a place where that guy is the one you want working for you. Maybe he's not the number 1 student, but I think there should be some room for this to be the Chaotic Neutral pick.
One potential answer is that this tests more heavily for the ability to memorise, as opposed to understanding. My last exams were over ten years ago and I was always good at them because I have a good medium-term memory for names and numbers. But it's not clearly useful to test for this, as most names and numbers can just be looked up.
When I was studying at university there was a rumour that one of the dons had scraped through their fourth-year exams despite barely attending lectures, because he had a photographic memory and just so happened to leaf through a book containing a required proof, the night before the exam. That gave him enough points despite not necessarily understanding what he was writing.
Obviously very few students have that sort of memory, but it's not necessarily fair to give advantage to those like me who can simply remember things more easily.
I had some take home exams in Physics that you could use internet, books, anything except other people (but that was honor code based). Those were some of the hardest exams I ever took in my life. Pages and pages of mathematical derivations. An LLM with how they can do a pretty good job at constructing mathematics, would actually have solved that issue pretty well.
> This approach still works, why do something else?
One issue is that the time provided to mark each piece of work continues to decrease. Sometimes you are only getting 15 minutes for 20 pages, and management believe that you can mark back-to-back from 9-5 with a half hour lunch. The only thing keeping people sane is the students that fail to submit, or submit something obviously sub-par. So where possible, even for designing exams, you try to limit text altogether. Multiple choice, drawing lines, a basic diagram, a calculation, etc.
Some students have terrible handwriting. I wouldn't be against the use of a dumb terminal in an exam room/hall. Maybe in the background it could be syncing the text and backing it up.
> Unless you're specifically testing a student's ability to Google, they don't need access to it.
I've been the person testing students, and I don't always remember everything. Sometimes it is good enough for the students to demonstrate that they understand the topic enough to know where to find the correct information based on a good intuition.
I am returning to this model in my classes: pen in paper quizzes, no digital devices. I also do seven equally weighted quizzes to deescalate them individually. I have reduced project/programming weight from 60-80% of my grade to 50% because it is not possible to tell if the students actually did the work.
I teach at MSc level. My students are scattered around the country and world. This makes hand-written exams tricky. Luckily, the nature of the questions they are asked to solve in the essay I give them following their coursework are that chatbots produce appalling bad submissions.
I dont know what you majored. But when I was a CS major maybe 50% of my grade came from projects. We wrote a compiler from scratch, wrote something that resembled a SQL engine from scratch, and wrote sizeable portions of an operating system. In my sophomore year we spent at least 20 hours a week on various projects a week.
We could use any resource we coulc find as long as we didn't submit anything we didn't write ourselves. This meant stackoverflow and online documentations.
There is no way you can test a student's ability to implement a large, complex system with thousands of lines of code in a three hour exam. There is just no way. I am not against closed book paper exams, I just wish the people touting them as the solution can be more realistic about what they can and cannot do.
As a professor who had to run Subversion for students (a bit before Git, et al), it's a nightmare to put the infrastructure together, keep it reliable under spiky loads (there is always a crush at the deadline), be customer support for students who manage to do something weird or lose their password, etc. You wind up spending a non-trivial amount of time being sysadmin for the class on top of your teaching duties. Being able to say "Put it on GitHub" short circuits all of that. It sucks, but it makes life a huge amount easier for the professor.
From the students point of view, sure, it sucks that nobody mentioned that Git could be used independently (or jj or Mercurial or ...) However, Github is going to be better than what 99.9% of all professors will put together or be able to use. Sure, you can use Git by itself, but then it needs to go somewhere that the professor can look at it, get submitted to automated testing, etc. That's not a trivial step. My students were happy that I had the "Best Homework Submission System" (said about Subversion of all things ...) because everybody else used the dumbass university enterprise thing that was completely useless (not going to mention its name because it deserves to die in the blazing fires of the lowest circle of Hell). However, it wasn't straightforward for me to put that together. And the probability of getting a professor with my motivation and skill is pretty low.
Agree about the possibility of infra nightmare, especially in the "SVN era" -- but in 2026, it's pretty straightforward to run a gitlab instance (takes about an hour to set up, most of which is DNS and TLS stuff, ime) for a course and set up actions, or use other submission infra like CMU autolab. I do this.
Agree with your comment about probability, motivation, and skill.
If anything, this reinforces the idea that chatbots don't fundamentally change education... they just amplify whatever incentives and structures already exist
I am not so sure about that. I think the difference this time is that AI usage is fairly mainstream. People are using it for all kinds of things, including studies. Using it to quickly get done with school/study-work, is a no-brainer (pun intended).
> Mistakes made by chatbots will be considered more important than honest human mistakes, resulting in the loss of more points.
>I thought this was fair. You can use chatbots, but you will be held accountable for it.
So you're held more accountable for the output actually? I'd be interested in how many students would choose to use LLMs if faults weren't penalized more.
He describes mostly a process where the exam itself,
or rather testing the knowledge of a student, is not
so important.
I think not all exams can occur like that. In some cases
you just have to test one's knowledge about a specific
topic, and knowing facts is a very, very easy way to
test this. I would agree that just focusing on facts
these days is overrated, but I would still reason that
it is not a useless metric still. So, when the author
describes "bring your own exam questions", it more means
that the exam itself is not so relevant, which is fine -
but saying that university exams are now useless in the
age of autosolving chatbots, is simply wrong. It just
means that the exam itself is not important; that in
itself does not automatically mean that ALL exams or
exam styles are useless. Also, it depends on what you
test. For instance, testing solving math questions -
yes, chatbots can solve this, but can a student solve
the same without needing a chatbot? How about practical
skills? Ok, 3D printing will dominate, but the ability
to craft something with your own hands, that is still a
skill that may be useful, at the least to some extent.
I feel that the whole discussion about chatbots dumbs
down a lot. Skills have not become irrelevant just
because chatbots exist.
> I was completely flabbergasted because, to me, discussing "What questions did you have?" was always part of the collaboration between students
When I was a student, professors maintained a public archive of past exams. The reason was obvious: next time the questions would be different, and memorizing past answers wouldn't help you if you don't understand the core ideas being taught. Then I took part in an exchange program and went to some shit-tier uni and I realized that collaboration was explicitly forbidden because professors would usually ask questions along "what was on slide 54". My favorite part was when professor said "I can't publish the slides online because they're stolen from another professor but you can buy them in the faculcy's shop".
My uni maintained a giant presence on Facebook - we'd share a lot of information, and the most popular group was "easy courses" for students who wanted to graduate but couldn't afford a difficult elective course.
The exchange uni had none of that. Literally no community, no collaboration, nothing. It's astonishing.
BTW regarding the stream of consciousness - I distinctly remember taking an exam and doing my best to force my brain to think about the exam questions, rather than porn I had been watching the previous day.
I'm back in school part time for a bachelor's, and have recently had a class where I had a professor who really understood how to implement LLM's into the class.
Our written assignments were a lot of "have an LLM generate a business proposal, then annotate it yourself"
The final exam was a 30 minute meeting where we just talked as peers, kinda like a cultural job interview. Sure there's lots of potential for bias there, but I think it's better than just blindly passing students using LLM's for the final exam.
> I realized that my students are so afraid of cheating that they mostly don’t collaborate before their exams! At least not as much as what we were doing.
This is radically different from the world that's been described to me. Even 20 years ago cheating was endemic and I've only heard of it getting worse.
What a wonderful teacher! I wish all teachers were like him.
Regarding the collaboration before the exam, it's really strange. In our generation, asking or exchanging questions was perfectly normal. I got an almost perfect score in physics thanks to that. I guess the elegant solution was still in me, but I might not have been able to come up with it in such a stressful situation. 'Almost' because the professor deducted one point from my score for being absent too often :)
However, oral exams in Europe are quite different from those at US universities. In an oral exam, the professor can interact with the student to see if they truly understand the subject, regardless of the written text. Allowing a chatbot during a written exam today would be defying the very purpose of the exam.
58 comments
[ 5.8 ms ] story [ 71.5 ms ] threadI think this is changing rapidly.
I'm a university professor, and the amount of students who seem to be in need of LLM as a crutch is growing really exponentially.
We are still in a place where the oldest students did their first year completely without LLMs. But younger students have used LLMs throughout their studies, and I fear that in the future, we will see full generations of students completely incapable of working without LLM assistance.
Or geometrically?
If people know "at university you can't use LLM, you are forced to think by yourself" they will adjust, albeit by trial of fire.
I think there's an argument that growing up in an educational system unable to teach you how to not rely on LLM would for all intents and purposes permanently nerf you compared to more fortunate peers. Critical thought is a skill we continue to practice until the very end
Makes me wonder if they should also get a diploma together then, saying "may not have the tested knowledge if not accompanied by $other_student"
I know of some companies that support hiring people as a team (either all or none get hired and they're meant to then work together well), so it wouldn't necessarily be a problem if they wish to be a team like that
I wished other universities adapt so quickly too (and have such a mindful attitude to students e.g. try to understand them, be upfront with expectations, learning from students etc).
Majority of professors are stressed and treat students as idiots... at least that was the case decade a go!
I thought this part of penalizing mistakes made with the help of LLMs more was quite ingenious.
If you have this great resource available to you (an LLM) you better show that you read and checked its output. If there's something in the LLM output you do not understand or check to be true, you better remove it.
If you do not use LLMs and just misunderstood something, you will have a (flawed) justification for why you wrote this. If there's something flawed in an LLM answer, the likelihood that you do not have any justification except for "the LLM said so" is quite high and should thus be penalized higher.
One shows a misunderstanding, the other doesn't necessarily show any understanding at all.
In my experience LLMs can significantly speed up the process of solving exam questions. They can surface relevant material I don't know about, they can remember how other similar problems are solved a lot better than I can and they can check for any mistakes in my answer. Yes when you get into very niche areas they start to fail (and often in a misleading way) but if you run through practise papers at all you can tell this and either avoid using the LLM or do some fine tuning on past papers.
For most of us--myself included--once you graduate from college, the answer is: "enough to not get fired". This is far less than most curriculums ask you to know, and every year, "enough to not get fired" is a lower and lower bar. With LLMs, it's practically on the floor for 90% of full-time jobs.
That is why I propose exactly the opposite regimen from this course, although I admire the writer's free thinking. Return to tradition, with a twist. Closed-book exams, no note sheets, all handwritten. Add a verbal examination, even though it massively increases examination time. No homework assignments, which encourage "completionist mindset", where the turning-in of the assignment feels more real than understanding the assignment. Publish problem sets thousands of problems large with worked-out-solutions to remove the incentive to cheat.
"Memorization is a prerequisite for creativity" -- paraphrase of an HN comment about a fondly remembered physics professor who made the students memorize every equation in the class. In the age of the LLM, I suspect this is triply true.
Students had very good reason to question the education system when they were asked to memorize things that were safe to forget once they graduated from school. And when most functional adults admitted they forgot what they had learned in school. It was an issue before LLM, and triply so now.
By the way, I now am 100% agree with "Memorization is a prerequisite for creativity." However, if you asked me to try to convince the 16-year-old me I would throw my hands up.
The obvious answer is "Because it's interesting."
But suppose you think strictly in utilitarian terms: what effort should I invest for what $$$ return. I have two things to say to you:
First: what a meaningless life you're living.
Second: you realize that if you don't learn anything because you have LLMs, and I learn everything because it's interesting, when you and I are competing, I'll have LLMs as well...? We'll be using the same tools, but I'll be able to reason and you won't.
I think the people who struggle with the question "Why should I know anything?" aren't going to learn anything anyway. You need curiosity to learn, or at least to learn a lot and well, and if you have curiosity you're not asking why you should learn anything.
What happens if LLM’s suddenly change their cost to be 1000 USD per user per month? What if it is 1000 USD per request? Will new students and new professionals still be able to complete their jobs?
I would not survive without homework. I needed that extra push in school. Otherwise, I would have been doing something else.
They should be encouraged to read and review the LLM output so they can critically understand it and take ownership of it.
Is it possible, and this is an interesting one to me, that this is the smartest kid in the class? I think maybe.
That guy who is playing with the latest tech, and forcing it to do the job (badly), and could care less about university or the course he's on. There's a time and a place where that guy is the one you want working for you. Maybe he's not the number 1 student, but I think there should be some room for this to be the Chaotic Neutral pick.
https://www.youtube.com/watch?v=JcQPAZP7-sE
LLM reasoning models are very good at searching well documented problems. =3
10 years ago, we wrote exams by hand with whatever we understood (in our heads.)
No colleagues, no laptops, no internet, no LLMs.
This approach still works, why do something else? Unless you're specifically testing a student's ability to Google, they don't need access to it.
When I was studying at university there was a rumour that one of the dons had scraped through their fourth-year exams despite barely attending lectures, because he had a photographic memory and just so happened to leaf through a book containing a required proof, the night before the exam. That gave him enough points despite not necessarily understanding what he was writing.
Obviously very few students have that sort of memory, but it's not necessarily fair to give advantage to those like me who can simply remember things more easily.
One issue is that the time provided to mark each piece of work continues to decrease. Sometimes you are only getting 15 minutes for 20 pages, and management believe that you can mark back-to-back from 9-5 with a half hour lunch. The only thing keeping people sane is the students that fail to submit, or submit something obviously sub-par. So where possible, even for designing exams, you try to limit text altogether. Multiple choice, drawing lines, a basic diagram, a calculation, etc.
Some students have terrible handwriting. I wouldn't be against the use of a dumb terminal in an exam room/hall. Maybe in the background it could be syncing the text and backing it up.
> Unless you're specifically testing a student's ability to Google, they don't need access to it.
I've been the person testing students, and I don't always remember everything. Sometimes it is good enough for the students to demonstrate that they understand the topic enough to know where to find the correct information based on a good intuition.
You did, but the best exam I had was open book bring anything. 25 and some change years ago even.
I've also had another professor do the "you can bring one A4 sheet with whatever notes you want to make on it."
It's not like writing prose And there is no syntax highlighting, no compile errors
We could use any resource we coulc find as long as we didn't submit anything we didn't write ourselves. This meant stackoverflow and online documentations.
There is no way you can test a student's ability to implement a large, complex system with thousands of lines of code in a three hour exam. There is just no way. I am not against closed book paper exams, I just wish the people touting them as the solution can be more realistic about what they can and cannot do.
I'm sympathetic to both sides here.
As a professor who had to run Subversion for students (a bit before Git, et al), it's a nightmare to put the infrastructure together, keep it reliable under spiky loads (there is always a crush at the deadline), be customer support for students who manage to do something weird or lose their password, etc. You wind up spending a non-trivial amount of time being sysadmin for the class on top of your teaching duties. Being able to say "Put it on GitHub" short circuits all of that. It sucks, but it makes life a huge amount easier for the professor.
From the students point of view, sure, it sucks that nobody mentioned that Git could be used independently (or jj or Mercurial or ...) However, Github is going to be better than what 99.9% of all professors will put together or be able to use. Sure, you can use Git by itself, but then it needs to go somewhere that the professor can look at it, get submitted to automated testing, etc. That's not a trivial step. My students were happy that I had the "Best Homework Submission System" (said about Subversion of all things ...) because everybody else used the dumbass university enterprise thing that was completely useless (not going to mention its name because it deserves to die in the blazing fires of the lowest circle of Hell). However, it wasn't straightforward for me to put that together. And the probability of getting a professor with my motivation and skill is pretty low.
Agree with your comment about probability, motivation, and skill.
>I thought this was fair. You can use chatbots, but you will be held accountable for it.
So you're held more accountable for the output actually? I'd be interested in how many students would choose to use LLMs if faults weren't penalized more.
I think not all exams can occur like that. In some cases you just have to test one's knowledge about a specific topic, and knowing facts is a very, very easy way to test this. I would agree that just focusing on facts these days is overrated, but I would still reason that it is not a useless metric still. So, when the author describes "bring your own exam questions", it more means that the exam itself is not so relevant, which is fine - but saying that university exams are now useless in the age of autosolving chatbots, is simply wrong. It just means that the exam itself is not important; that in itself does not automatically mean that ALL exams or exam styles are useless. Also, it depends on what you test. For instance, testing solving math questions - yes, chatbots can solve this, but can a student solve the same without needing a chatbot? How about practical skills? Ok, 3D printing will dominate, but the ability to craft something with your own hands, that is still a skill that may be useful, at the least to some extent.
I feel that the whole discussion about chatbots dumbs down a lot. Skills have not become irrelevant just because chatbots exist.
When I was a student, professors maintained a public archive of past exams. The reason was obvious: next time the questions would be different, and memorizing past answers wouldn't help you if you don't understand the core ideas being taught. Then I took part in an exchange program and went to some shit-tier uni and I realized that collaboration was explicitly forbidden because professors would usually ask questions along "what was on slide 54". My favorite part was when professor said "I can't publish the slides online because they're stolen from another professor but you can buy them in the faculcy's shop".
My uni maintained a giant presence on Facebook - we'd share a lot of information, and the most popular group was "easy courses" for students who wanted to graduate but couldn't afford a difficult elective course.
The exchange uni had none of that. Literally no community, no collaboration, nothing. It's astonishing.
BTW regarding the stream of consciousness - I distinctly remember taking an exam and doing my best to force my brain to think about the exam questions, rather than porn I had been watching the previous day.
Our written assignments were a lot of "have an LLM generate a business proposal, then annotate it yourself"
The final exam was a 30 minute meeting where we just talked as peers, kinda like a cultural job interview. Sure there's lots of potential for bias there, but I think it's better than just blindly passing students using LLM's for the final exam.
This is radically different from the world that's been described to me. Even 20 years ago cheating was endemic and I've only heard of it getting worse.
Regarding the collaboration before the exam, it's really strange. In our generation, asking or exchanging questions was perfectly normal. I got an almost perfect score in physics thanks to that. I guess the elegant solution was still in me, but I might not have been able to come up with it in such a stressful situation. 'Almost' because the professor deducted one point from my score for being absent too often :)
However, oral exams in Europe are quite different from those at US universities. In an oral exam, the professor can interact with the student to see if they truly understand the subject, regardless of the written text. Allowing a chatbot during a written exam today would be defying the very purpose of the exam.