That's assuming they want to solve it. A lot of administrators live in lalaland because it suits them to. It is in the interest of society to fail cheaters and frauds before they graduate, but it costs the universities money to do it properly. So they will only address the problem if it comes to public attention.
That's a trend that we've seen over and over, for example with the corruption over admission to Ivy league universities. In fact chatGPT doesn't really change the landscape that much. All it does is democratising contract cheating, on which most universities only apply band aids. It might end up being a good thing, but only if it does attract public attention to the issue.
IRL universities will probably develop a protocol to remove access to helper tools for this sort of thing, maybe metal detectors at the door and specially tailored computers if needed for quizzes. Online universities OTOH are probably going to feel the bite of this the hardest. The CKA exam I took a while back had draconian measures to try to prevent me from cheating, such as taking the test with a webcam on me the whole time, with tools in the browser to limit where I surfed, in a room that has NO PICTURES on the walls. The tech to provide answers is moving faster than the tech to secure a remote room from such outside answers. And if in-person proctoring is eventually required, I would wager that will be a death-knell for online universities, who will have a hard time obtaining that physical space on a regular basis while staying marginally profitable.
Or remote proctoring centers will show up as a business opportunity, and the cost of this will be passed on to students, who will pay it with financial aid.
But interest and investment is also increasing exponentially. I've never really seen anything like this before in AI land, even when GANs and such were doing amazing things.
Technology tends to grow with a sigmoid curve. Extremely fast growth at the start, and then it slows down and eventually plateaus and new technology repeats the cycle. The only question is where on the curve are we for LLMs and AI in general?
I am of two minds about ChatGPT. It’s amazingly useful when it comes to writing code dealing with my domain since the APIs are well known (AWS).
But at the same time it misses subtle nuances that you have to have experience with to know when it misses something. In the hands of someone who doesn’t already know the subject area where they already know the foot guns, it can lead you wrong.
I tend to find it works best as a tool for generating situation specific examples rather than writing you entire code for you.
Like when every stack overflow answer you find just doesn't fit because of one major difference in your situation, chatgpt often has you covered. Then you take the example code it gives you and adapt it to your actual codebase, testing to see if it works as expected as you go.
> Apparently, it's not even something students feel they need to hide.
Which is good! If a bit of work is trivially accomplished by a machine, we should take it for granted and move on to the next layer of complexity. I have always maintained that teachers complaining about students cheating at homework assignments with AI need to instead work on providing better homework.
Knowing how to produce an essay is exactly the same as "analytical thinking, research, and argument skills" with the added challenge of making it legible to a reader — which is what makes those skills useful.
I suppose, but having written plenty of essays as an adult I can say with complete certainty that nothing I learned from my 5 paragraph days was of any use. No one, not you, not your teacher, not any real life audience for any topic you would be presenting on or publishing for, wants to read anything remotely close to what you're forced to write in school.
What you were forced to write in school. I readily admit that I had an quality of education several SDs higher than usual, but the trite "5 paragraph" nonsense is neither universal or (more importantly) inevitable.
Well, as far as I know, that is an special case of an essay.
And this you can test onsite.
The kind of essays I had to write in schools were more about nice sounding words and less the content. CheatGPT can produce nice sounding words, so I am hoping that the focus will move towards rewarding content.
Or just novel applications of the things you learn in class.
"Congratulations! You leaned depth-first-search! ^award noises^ Below is the algorithm for reference because memorizing it just for this test is silly. You're working on a real time mapping application called Maply. Locations are represented as nodes and all direct routes between any two nodes are represented by directed weighted edges."
a) Write a function that takes a start node, an end node, and a maximum distance to travel and return the shortest path between the two.
b) Your boss said that users need to be able to add stops along their journey. Write a function that takes the final path you computed in part a and the new node for the added stop and compute the amended path changing as few of the original legs of the trip as possible (don't want to disorient your users).
c) Now your boss is saying you need to handle the situation where users make mistakes. Use the function you wrote in part b to implement this feature.
Novel applications? You mean incremental combinations. You realize finding and solving nontrivial problems is much more complicated task, right? This seems to imply that someone that cannot do simple things can do complicated things, where is the evidence of that?
Why do the most complicated mathematics start with basic principles and work up to complex problems? Why don’t they just start with the Collatz Conjecture?
>By this reasonning nobody should ever learn anything, because it's all 'trivially doable' by machine.
>Like 'addition' and 'subtraction'.
A better analogy would be low level coding. I don't know (or care) how my processor calculates `var f = 3+2` at the register level. And being able to ignore that allows me to focus on higher level concerns.
I see what you mean, but it's not really a better analogy.
We need to learn how to do addition at some point, so we can't have ChatGPT do that.
We need to learn how 'registers' work, so we can't have ChatGPT do that.
We need to learn basic algorithms work, so we can't have ChatGPT do that.
AKA - almost whatever is being assigned as homework, is the 'thing to be learned' and it's ridiculous to suggest that ChatGPT do that, and doubly so to gaslight teachers.
One of the required classes for CS degree was Assembly Language. Nobody taking or teaching the class pretended there would be a great need for this language in a job setting. But that wasn't the point of this class.
The thing is, what applies to you doesn't necessary apply to everybody. Somebody has to understand low level coding. Somebody has to be introduced to it without necessarily knowing going in that it will be a career path. Somebody will need to write compilers and reverse engineer and design CPUs. Just because a skill isn't valuable to you or those you know doesn't mean it isn't valuable to others, especially those who don't know enough yet to know that it might interest them.
If you don’t know that then you really have a real lack of knowledge of computing. I don’t mean that as an attack or anything but I don’t think it is a badge of honor or anything. You could just as easily say the person typing a Word document doesn’t know what RAM is so we can just stop teaching that foolishness so people can focus on the “complex” things.
One could also argue, and many have successfully, that this type of thinking is why software is so much slower than in the 80s and 90s.
I don’t generally like the word “gaslight” as it is normally used by the unsavory but in this case, I think you used it perfectly. That is exactly what people are doing, except for the extremely naive ones. I don’t really know why, though?
Gaslighting is just a term for a social interaction that didn't exist in our vocabulary before. That's it. Which is different than terms like 'woke' for which we have different connotations and contexts.
Should we? Basic arithmetic has long since been solved, but I've met plenty of people who struggle at higher level math because they haven't mastered enough basic arithmetic. Solving complex problems will often involve many much simpler problems that must be solved as well. The time to offload this to another system to solve it for you is immensely more expensive than being able to solve it mentally, meaning that to solve the complex problem ends up being far more expensive as well. Eventually students will reach problems whose price they can no longer afford.
It is related to the reason we teach concepts starting with simple, small, easy to solve problems before building up. If I want to teach a student how to solve the limit of x*sin(1/x) as x approaches 0, I need them to understand quite a bit of math to even know what the problem is asking.
No, we are supposed to believe you are not supposed to learn any of that because some technology exists and you will be better off with “higher level” tasks … that’s what they tell me, anyway.
How do you move on to other layers of complexity if you don’t know anything. History books have trivially had answers to questions about history for centuries, does that mean we should take it for “granted” and no one needs yo actually know anything about history?
Where does this idea come from that you can be a Terence Tap of mathematics without even knowing basic algebra?
We have calculators, wolfram, google etc yet math exams and hw assignments are still administered. I think the approach to teaching and hw assignments especially in CS programs will have to change.
"In my opinion, the students learning to program do not benefit from AI helping to "remove the drudgery" of programming. At some point, you have to learn to program. You can't get there by avoiding the inevitable struggles of learning-to-program."
Couldn't you say the same about how compilers 'remove the drudgery' of writing machine code? Or is that a bad analogy? Provided AI eventually gets good enough in its code generation, maybe 'programming' is moving up another layer of abstraction.
Abstraction works because you are able to treat an abstraction as a black box and concern yourself only with its input and output. A segment of code written by an LLM is qualitatively a very different thing; it's more like an open box of crap that you have to inspect and put together yourself, which requires knowledge of the contents, which requires experiencing the drudgery.
I rarely encounter abstractions in the wild that are as nicely "sealed" as the definition implies. Looking at an open box of crap and understanding why it's doing something other than what the author (or you) intend is a valuable skill.
(No idea if this new model of "Ask ChatGPT or Copilot to synthesize a solution and then tune that solution" provides a solid opportunity to improve that skill yet, however).
When I was a freshman in college the professor would live code various data structures and algorithms on the projector and ask the students to follow along. Each time he did this it was subtly unique.
It was required that you continue to use this same base code for the assignments plus your edits to complete it. This made it obvious who didn't attend the lectures and who they copied from. Assignments were graded by your peers and did not affect the final grade unless you didn't do them at all. Quizzes were not code, but proofs written in your own informal style. Tests were a mix of both proofs and code on paper with no notes or devices allowed.
A CS degree is not a "bootcamp" or any other form of vocational training. Also, there are criteria for the school to maintain accreditations, you know.
I think most just want a decent job to pay bills after. The rest love the field, the theory of cs, and may or may not continue into academics or research.
It is a computer science degree, with an emphasis on the science part, and a note that CS is not the same as coding computers. It's the science of computing, not a tech bootcamp to get you quickly up to speed in the latest hot language and framework.
Excercises should not be about evaluating and judging. They should be about learning. If a student uses an AI, or copies someone else's work, it is to their own detriment. (Exams are a whole different question.)
Even before AI, you could read a book and copy-paste the excercises, or just skip them, but if you wanted the full learning benefit you would type them out. I think we will have to focus more on teaching how to learn. This situation is nothing completely new. Even though you have power tools in woodworking, apprentices learn the basic techniques with hand tools (AFAIK).
"Cheating" is good. School teachers and professors should not be the arbiters of how much money you earn later in life, but they are based on their subjective assessment of you (via grades).
Many do not realize the intensely adversarial relationship one is in with a teacher. You are not there to learn. You are there to get an A. In the case of social sciences, you support the ideas your professor espouses. In the case of CS, you do whatever it takes to get your code running as the assignment specifies.
Anything else is idealism which will harm your future earning potential. It's insulting to ask kids to surrender their future earning potential due to "ethics" in an academic world where ever top tier conferences and journals are filled with unreproducible BS science.
The people you're cheating are not your professors but your peers. For better or for worse, grades are used to rank people and mete out further opportunities and benefits. The professors already have jobs.
Let's unbox that a bit. In what sense are one's peers cheated when an individual cheats?
If the course were going to have 20% A-level students and now has 50% A-level students, what has been taken from the initial 20%? They were still going to be able to put on their résumé "4.0 GPA from Suchandsuch University."
Perhaps this is another of the early (?) nails in the coffin for traditional higher education.
If it becomes harder to assess if someone learned something (with a grade), the results of that assessment (GPA) become less valuable. Software has traditionally been at the forefront of allowing people with non-traditional backgrounds (bootcamps, other degrees, self-taught) to work in the highest echelon of jobs, because of experience outside of the classroom (open source, personal projects).
ChatGPT and its ilk put more pressure on evaluation of candidates in interviews and should lend more weight to impact/experience based criteria on resumes (vs education-based).
There is a spectrum of people using ChatGPT to cheat vs learn. But, ideally, "cheaters never win", so interviewers and resume screeners will soon be under as much pressure as educators to holistically evaluate candidates beyond the crutch/immediate signal of a degree. They're just further downstream
I think it just moves the definition of “learning” to a higher level of abstraction: so you know what AI tools to use, how to prompt them, and how to understand their output?
I’m reminded of the time when graphing calculators were going to destroy math programs because nobody would “really know” how to do the work. And yet here we are, and math is fine, and calculators are just another tool.
I hear this argument a lot and I think that it's fallacious. Let's go ahead and extend it to another AI tool - in this case let's talk about stable diffusion.
Let's say you teach a class on fine Art and painting, if you allow your students to use stable diffusion for all their drawings, would you make the case that they have learned how to paint?
Likewise you can't really make the case that somebody understands how to do recursion, if all they're capable of doing is typing the following prompt into chat GPT, "change my forloop into a recursive method".
And in my experience going through calculus, the usage of graphing calculators was heavily. We still had to understand how to calculate derivatives and integrals by hand.
> would you make the case that they have learned how to paint
Well, you’re assuming that “painting” is the physical act of moving a brush on canvas.
But that’s already not true. Plenty of people graduate art school with degrees, despite doing everything on a computer. Are they “painters”? Well, no, but they are artists.
And if you’re talking about a program for artists, where the work is judged on artistic merit (composition, concept, etc), I don’t think it matters what mediums are used.
But if we’re narrowly focused on something more like sign painting, where what matters is brush technique and conforming to customer expectations, sure, AI will reduce the need for such people and will allow those who exist to “cheat”. But who cares?
Most math classes are taught without graphing calculators and when they are used it is minimal. This is fundamentally different. This literally is being people trying to substitute actually know or successfully being evaluated to do anything. The advocates of “bringing down higher education with AI” also consider competence to be “ableism”.
I support using AI in education because it will be in the work environment. It seems insane that anyone would want to teach students on slide rules and typewriters and then send them out to a world of computers and word processors.
And I have no idea where your “ableism” comment came from. Just trying to inject some culture war?
This business of “ableism” is comes up all the time with ChatGPT discussions. I don’t know what your slide rule comment means. Even with calculators we expected students to be able to calculate without a calculator. I really don’t see how you expect people to do “complex” tasks if they cannot do simple ones. Is this magic?
You could force tests to be done in testing centers. My college had these and they were strict about what you can bring, you get up to a whole week to show up on your own time, and you're only allowed a paper and pencil if anything at all, that they provide. Make the Final and Midterm tests worth roughly 60% of their grade, and it wont matter if they cheat on their homework.
Edit:
Alternatively, have students do presentations of their code from their homework, just as we all do peer review professionally. Let students learn and teach other students.
I think the edit is more the case for the near future.
I think we’re about to see a shift from professors running the same curriculum year over year not really knowing students that come and go on a time conveyor belt, to something much closer to the imagination of the parents that are often paying for their kids college “experience”.
OR - I see the tools used to cheat also being used to detect cheating.
I did most of a humanities degree in the early- to mid-'00s and the only courses that relied heavily on long-form out-of-class writing exercises for grades were in the actual language departments (English, foreign languages).
The rest were big on the occasional short quiz in-class to check understanding, and periodic "bluebook" exams that involved writing the equivalent of perhaps 3-5 total pages of typewritten material, by hand, in one class period, in person, in response to perhaps a half-dozen prompts. Basically a series of short, to-the-point essays. Not a ton of outside-of-class paper composition. I doubt they'd have trouble adjusting to remove those all but entirely.
Higher education will become just an optional prep course for your sit down conversational AI interview.
Once AI can do a good job vetting candidates, I see no reason for companies not to have an open applicant process where anyone can interview and be evaluated. If you are sharp and know your shit, a degree won't matter and the AI interviewer won't care.
But this is an "All else being equal" scenario, my true belief is that AI will change things so radically that there is effectively an event horizon in the near future, impossible to predict whats beyond it.
The event horizon you describe is always there. Be it 3D printing, AI, moore's law... etc... The things these things enable, are hard to predict.
Think about cloud computing. It changes the game massively for startups and for people who need enterprise class infrastructure as mere mortals.
Another constant tension to show you how unpredictable all this is: Do you use kernel networking, let the kernel use hardware offloads, or goto use DPDK? The choice of what to do is changing as hardware changes, the kernel changes etc....
... Once you understand, that life is ALWAYS at an event horizon.. you understand AI is just another such event.
Prediciting the future is for the native... Making the future is the way to go. Currrently the AI guys are doing that. But another thing will rise up, it always does.
Why would you even need jobs in this case? Also why does everyone think we live in some magical fairy in which the majority of people with advanced knowledge of a subject are doing so without higher education. Do these people really think everyone is an autodidact? If that was the case, why would they need to cheat on their courses?
CS programs will have to adapt to this or die. The reality is that five years from now, we'll all just be talking to LLMs all day and compiling/checking the results. It's no different than the shift from assembler to higher languages in the 80s.
I don’t disagree, but how do you imagine they should adapt? “Checking the results” is difficult if you are not able to perform the task on your own, which means you need to learn the task in the first place.
It’s a chicken and egg problem. Can ChatGPT come up with a framework to write code ? Today it can only read docs and already developed code to create new one.
Could it create a new langage that perfectly fit the hardware it’s running on ? If so, there is no need anymore for any other software company.
ChatGPT, write me an OS please, write me a photo and video editor. Write me a game about …
It’s seems really far fetched because, could it really create new software with a different paradigm ? Something that hasn’t been done before ? Given how it processes words from its input, it seems not.
School teaches how to think. All the frameworks and some langages that are used by millions didn’t exist back then. But what if the point of rupture was 10 years ago, would we been stuck with old and non innovative tooling and designs ?
We still need to learn people how to think and develop skills.
The cheater from my era never acquired good development skills.
Today the cheater are just as good as ChatGPT so what’s the point in hiring them ? If all you have to do is enter commands in a prompt, let the marketing people do it and have their real no-code revolution.
We always going to need problem solvers and people with deep insights on how things work. Maybe it’s the time to dig deeper into the knowledge and write only real meaningful code
The world has evolved, and higher education has not kept up with it. IT's time to stop giving trite and simplistic programming assignments to your student and make them work for it. Have them study set of online documentation and make the test about whether they can look up function calls and successfully assemble the information required to present a working solution.
Reading this, I’m glad I didn’t have access to ChatGPT and co when I was in school. I was lazy and always followed the path of least resistance, but I wanted good grades so that meant doing the home work by hand after my Google-fu failed.
What ChatGPT (and its cousins) expose is that the way humans have been taught in most schools - memorizing and regurgitating information - is now a commodity.
What humans being to the table over ChatGPT is our ability to create new links between information, aka creativity. Teaching creativity, imo, will require a return to the methods like those of Sophocles and his contemporaries. I would rather this author be writing about how he is going to re-examine how he teaches rather than bemoaning that students can shortcut his current approach.
Creativity doesn’t exist in isolation. In order to be creative, and create unexpected connections, one first needs to know a lot of seemingly unrelated things.
Statements like these are premature. ChatGPT is three months old! This is a rapidly advancing field. The capabilities of these models are very likely to be radically different five years from now. Any conclusions drawn now about what value is uniquely human and out of reach for AI may be proven wrong quickly.
You seem confident that it won't get worse, but it's only as good as its training data. Which is the internet. What happens when the internet is filled with generic Gen1 output? I'm doubtful copy averages can ever lead to anything other than increasing mediocrity.
> The capabilities of these models are very likely to be radically different five years from now.
This is an article of faith. Specifically, it believes that at some point in the future the current paradigm will result in qualitatively different behaviors than the mimicry these systems (all GPT variants, "Attention is All You Need") currently perform.
Much ado has been made of the previous Symbolic AI researchers "moving the goalposts." In this criticism, it is the old guard of AI who is constantly bemoaning the current state of affairs as not real AI. But there is no actual goalpost moving. They have said it wasn't real AI at the beginning, and they are saying it isn't real AI now. Whether or not the symbolists' model was real AI is irrelevant: when you bake in "this is a rapidly moving field" as a hand-wavy explanation for why this may result in AGI, you are the one implying a moving of the goalposts.
If it ever turns out that these models need to be qualitatively different, then it will be clear that attention is not in fact all you need. In that eventuality, I fully expect the new guard to hem and haw and find some tricky sophistry to explain why they were right all along, despite qualitative shifts unattributable to adding mountains of new data or connectionist trickery.
I don't know, man, you kind of come off as having a chip on your shoulder about this. I'm not predicting AGI specifically here, and I'm not making any argument about symbolic vs connectionist AI at all. Maybe the model of the future is half symbolic! I'm just saying that asserting that you know exactly which things AI models aren't going to be able to do is pretty foolish at this point.
> know exactly what these models aren't going to be able to do is pretty foolish at this point
This can be said about absolutely any new technology but it does not make it true. It's simply the inscrutability of these tensors that allow people to imagine the intelligence is in there somewhere. The original comment was what humans "bring to the table" over ChatGPT specifically. And that's that they have real intelligence and not memorization.
As others have said, these models have been around for many years. Their core innovation is to add more and more data to them like a dictionary and compress those basis functions in the network architecture. This memorization has a limit and is actually the opposite of intelligence. Intelligence or creativity can do more with less information. As per the original comment, this is what human intelligence and creativity is (currently) superior at and what people should prioritize if they don't want to be replaced.
Chat GPT is still at its heart GPT technology (with some clever embedding of the running transcript acting as a reoccurring prompt) which is several years old at this point.
I didn't memoriza Python sintaxy or the name of every function or how to do small things. I use Google for that. But I know what I need to do in the best possible way (at least that's what I am pay for!). Should I set this variable here? Should this method be private? Should I design an interface or a public class? A dict or a dataclass?
That's what I have to decide as an engineer and where my value resides. If ChatGPT only replaces the memorization part, that would be OK, but it replaces a lot more of it that requires the people using it not to question themselves the things I mentioned before.
I had a bet with a friend, he has no knowledge of programming and was convinced he could make an online game (!) using only ChatGPT. He said one month was going to be enough. Of course, a few months have passed already and he is way off. He asks the questions that a non-programmer would ask, and what ChatGPT gives him back is not usable, not thought for the future, not easily modifiable, etc. His code is a Frankestein that won't do anything good.
I grew up in the late 1980s and early 1990s and have a phenomenal memory. My dad used to tell me how valuable it would be once I grew up and got a job. It’s so funny how false that ended up being.
It’s much more beneficial socially because I can recall jokes that fit situations incredibly quickly and get a good laugh.
I agree though I didn't find the author to be bemoaning students. Rather they were writing a "state of the art" piece explaining how things are currently happening and leaving it open for people to follow with thoughts about how to make meaningful changes to assessment style and curriculum in the face of ChatGPT.
> I would rather this author be writing about how he is going to re-examine how he teaches rather than bemoaning that students can shortcut his current approach.
Did you read the entire article? What you're asking for is exactly how he ends his discussion.
I went to uni 10 years ago and even then, I can't think of any classes that were just memorizing and regurgitating. You'd have to memorize fundamental concepts, but come exam time you are applying those concepts to new questions, not regurgitating anything. In high school a lot of exams were regurgitation, but I attribute it to teachers at that level just not having the niche experience required to craft clever "apply this theory" sort of questions that a domain expert in a university could do, and students in highschool are also responsible for a lot less theory learning on their own.
I think the problem with what you’re saying is that you are not mindlessly agreeing with some straw man, overly reductionist view of what higher education is that seems to be majority view.
I think that is poppycock. If you don’t know anything then how can you bring all this “creativity” to bear on a task of knowledge work? This whole line that all of higher education is just regurgitation or something of no value seems fallacious. What is an example of not regurgitation? Why is it that using Wikipedia or whatever is superior learning to what is done in university?
> I would rather this author be writing about how he is going to re-examine how he teaches rather than bemoaning that students can shortcut his current approach.
He alludes to this: "The first solution is hard for lots of reasons, not least that the current funding model of post-secondary institutions, which does not prioritize the ratio of faculty-to-students necessary for ever more personalized or real-time assessment methods. Larger and larger classes make many of these good ideas impractical. Faculty have zero control over this, but by all means, please talk to our senior leadership. It would be great."
In other words, the Universities are pushing "on-line-all-the-time" because it's co$t effective.
It would be fantastic if large language models (or any of the nascent AI/machine learning tech.) finally kill off both assessment in education and copyright/IP protection.
What a wonderful future that would be. We can but hope.
I think there' an initial reaction of something like "how dare they!?!?"
However, AI the type of tool that going to level up mankind's capabilities to the point that curriculums will need to adjust to fit those new capabilities. Certainly this has happened dozens of time in a field like Computer Science, where curriculum in 2023 is radically different than it was in the 60's and 70's.
This new rise in AI might be amongst the most disruptive forces ever in many fields, including academics, but at some point, you have to factor AI in as integral part of our day-to-day work and life and factor that into education.
This will be difficult, especially finding the line between what is fundamental and what is not, but it's not like this hasn't happened before -- e.g. the calculator didn't eliminate the need to learn basic arithmetic.
The problem with widespread cheating even if you don't cheat yourself is it essentially reduces the credit rating of your $50k program and has knock on effects elsewhere, like worse interview loops. And if the 'cheater's workload' becomes the norm, you the non-cheater can be literally failed out of programs and scholarships because you didn't keep up.
At what point does "not keeping up with the cheaters" become "There is a more efficient way to do the task and this student's grade reflects they are choosing the inefficient approach?"
I'm reminded of the stories of employees getting busted because they were assigned a job so trivially automatable they either did automate it or they used some find-labor service to delegate it for a fraction of the cost out of their own pockets, who are then accused by the company of "not working."
It's the same logic of letting a machine do a gym exercise vs doing it yourself, you're missing the point of it, because otherwise the gym exercise is mostly useless.
> I've seen plenty of cheaters, they were the worst students
In other words, you haven't seen the "cheaters" who were among the best students.
"cheaters" in quotes because it's not clear to me that people using freely available resources when doing homework are really cheaters. If an instructor wants to do a closed-book exam, they can do just that.
> In other words, you haven't seen the "cheaters" who were among the best students.
If they were able to fool everyone to the point of being considered good students, it means they weren't cheaters, just that they had a different approach to problems than others (which is kinda what you say after).
According to the Department of Education, 54% of Americans have below a 6th grade reading level. [0]
Everyone thought that the kids who started using mobile devices as babies would become computer savants, but it turns out the kids these days don't understand what a file system is. [1]
I wonder if that didn't pan out in part because of the death of personal general computing devices. It wasn't "kids who started using mobile devices as babies" it was "kids who would be fluidly navigating a PC by age 5", which didn't pan out.
Given the amount of `git add . ; git commit -m "everything"` that I've seen with professional developers, I am not confident that you'd get anything better from students.
I’m already doing this in my classes and have been for a while. We’re transitioning more CS classes over to this mode of teaching soon. We’ve got a gitlab instance for the department, and all students have an account. Works great!
I really like this blog post, I think it fairly describes some of the challenges teachers are going to face with this technology, while also admitting that this tech is inevitable.
> In my opinion, the students learning to program do not benefit from AI helping to "remove the drudgery" of programming. At some point, you have to learn to program. You can't get there by avoiding the inevitable struggles of learning-to-program.
I don’t disagree with this at all (at least for where we are now, in ten years, it might not matter as much), and I don’t want to be glib, but I do think the answer is to “teach students to program.” Don’t rely on rote assignments that you’re checking with an auto-grader (not saying this professor does that, but a lot do) and cookie-cutter materials; actually teach them to program.
And yes, LLMs will almost certainly mean that some students will cheat their way out of their assignments. But just like most cheaters who cheat on things they don’t fundamentally understand (which is different from people who cheat to hurry up and get through an exercise they could do in their sleep but don’t want to waste the energy doing), it will catch up to them when they have to do something that is not part of the rote assignment.
Or maybe, adjust how you test/assign homework. That isn’t to imply that that won’t take more work, but if your concern actually is that students aren’t learning and are just successfully copying and pasting, the testing/grading is the problem.
—
In high school (and college), I was a top student in math and in English. But I hated doing homework. In one math class in high school, although I got near perfect scores on all of my math exams, the teacher still gave me a C because 20% of my grade was “homework” that I didn’t do (I had already taken the same class a year earlier at another school — I didn’t need to do the homework. My math tutor my mom got me out of fear of my bad grade taught me Calculus and Fortran instead of trying to get me to do the useless homework). This taught me nothing and frankly, soured me on taking more advanced math classes that were all taught by this same teacher.
In contrast, I had an English teacher who would assign vocabulary homework. Basic, “write a sentence with each word” shit. Again, a total waste of time for me. So I worked out a deal with him, let me just orally tell you what each word means, saving us both time and energy. He took the additional step of assigning me/grading me on different criteria than the rest of the class for essays and the like.
Which class do you think I learned more from? Which teacher actually cared about whether I knew/understood the material, versus what checkboxes I needed to follow to show “completion.”
If the goal is to teach students to understand what they are doing, then do that. Don’t get obsessed with trying to stop the inevitable few from cheating, or become overly focused on only having one way to measure comprehension.
The failure mode of things like ChatGPT is it can make wrong answers confidently, subtly, and if you don't have the skill to audit what is wrong with the answer, then it can be fairly catastrophic.
So instead of making questions generative, you make them audits / debugging type ones.
Use ChatGPT to generate a result after several iterations that is wrong and then ask them what is wrong with the result. Since ChatGPT generated the wrong result, they will still need to debug it even if they try to do it themselves in ChatGPT, because daddy ChatGPT is not going to give them the right answer. You can even show them the chat transcript in the question. And often debugging is a harder skill than creating in some ways.
It's not a %100 solution, and we don't know how long it will take to not be relevant, but it is something you can do today.
That and inquisitive back and forth oral questioning maybe as part of the testing process.
I think it’s an interesting point about making the questions audit or debugging.
But, also a mistake to hark on the often humorous fact at how confidently wrong our Gen1 AI can be. That will go down over time. Imagine a time when AI is making good programming choices and correcting itself when it’s wrong. Imagine Gen4 AIs.
Both of these things tie together. If we need to detect cheating now, prove you can debug. When AIs get better, prove you can debug. It’ll be the same skill either way.
the other week i gave chatgpt a simple multiplication problem that it got wrong. very simple problem like 86 * 0.0007 or something. but ive been working with chatgpt for 4-5 weeks now and that wrong answer doesnt make up for all the "good answers" that are usually not perfect. like one day i needed to COALESCE in mysql. i didnt know that, but chatgpt did. theres a few times i would have written a function the complicated way when gpt gave me a much simpler nicer to read way. i think the tool is great and tbh i dont like copilot in comparison and turned it off.
i dont think chatgpt can be a 100% solution without several years of nerfing.
Multiplication problems are not language problems. There is no data in the training set where there is a likely probability of the next token in the "86 * 0.0007 =" sentence being correct.
People need to stop treating ChatGPT as a computation engine. It is not wolfram alpha. It is not google. It is fancy autocomplete trained on a large subset of the internet.
I tested several AI chats, and none of them could correctly answer the question: "what is heavier, 1 kilogram of nails or 1 kilogram of feathers?". Does this one know? (I don't have openai key, can't test it myself).
EDIT: turns out, the latest version of chatGPT knows the answer. They also fixed the answer to another famous trick question("The son of my father, but not my brother. Who is that?")
here we go:
Both 1 kilogram of nails and 1 kilogram of feathers have the same weight, which is 1 kilogram.
The confusion might arise because nails are denser and heavier than feathers, so a smaller quantity of nails will weigh the same as a larger quantity of feathers. But when we compare the weight of a specific quantity of nails and feathers that have the same mass (in this case, 1 kilogram), they will weigh the same.
I tried it. TLDR; no it does not get it correct. But ChatGPT does!
> 1 kilogram of nails is heavier than 1 kilogram of feathers.
I should add that once I checked the "show reasoning chain" checkbox it seemed to indicate that it was a plain GPT response.
> Thought: Do I need to use a tool? No
> AI: 1 kilogram of nails is heavier than 1 kilogram of feathers.
> 1 kilogram of nails is heavier than 1 kilogram of feathers.
Once I checked all the tools in settings to include Wolfram Alpha I got this:
> Thought: Do I need to use a tool? Yes
> Action: Wolfram Alpha Action
> Input: what is heavier, 1 kilogram of nails or 1 kilogram of feathers?
> Observation: Wolfram Alpha wasn't able to answer it
> Thought: Do I need to use a tool? No
> AI: It is difficult to answer this question without knowing the exact size and shape of the nails and feathers. Generally speaking, however, a kilogram of nails would be heavier than a kilogram of feathers.
> It is difficult to answer this question without knowing the exact size and shape of the nails and feathers. Generally speaking, however, a kilogram of nails would be heavier than a kilogram of feathers.
---
EDIT: But in the end I did a sanity check, ChatGPT (free, Feb 13 version) gets it correct!
> Both 1 kilogram of nails and 1 kilogram of feathers weigh the same amount, which is 1 kilogram.
> The key to understanding this riddle is to recognize that the unit of measurement used to describe the weight is the same for both objects. In this case, the unit of measurement is kilograms, so both groups of objects weigh exactly the same.
> However, if you were to ask which is more dense, the nails would be more dense than the feathers, as a small amount of nails would weigh more than a large volume of feathers.
As a control, I tried asking the same question but asking about volume instead. Unfortunately, ChatGPT appears to get it wrong:
User:
> Which is heavier: one cubic foot of nails, or one cubic foot of feathers?
ChatGPT:
> One cubic foot of nails and one cubic foot of feathers would weigh the same amount. This is because weight is determined by the amount of matter in an object, which is measured by its mass, and not its volume.
> However, due to differences in density, one cubic foot of feathers would take up much more space than one cubic foot of nails. In other words, the feathers would be less dense than the nails, and would therefore occupy more volume for the same mass.
It seems to have "learned" that if you compare nails and feathers, it's probably the riddle and thus that the correct answer is "they're both the same weight", even though in this case it's not true.
Well this is very interesting. I pasted your prompt into a new chat just now and I got the correct answer:
> One cubic foot of nails and one cubic foot of feathers both have the same volume, but nails are denser than feathers. Therefore, one cubic foot of nails will be heavier than one cubic foot of feathers.
Are you using the free version at chat.openai.com?
However, I did initially misspell the prompt to say "heaver", and with the misspelling it had answered correctly. Once I corrected it to "heavier", it then came up with the wrong answer.
Looking at it with fresh eyes, it's possible that my methodology was flawed. While I only ever intended to showcase the first response, I did also try regenerating the responses a few times. While normally this wouldn't matter (since I would only be showing the first answer anyway), it's possible that my doing this a few times before noticing and fixing the misspelling (and subsequently using the next answer given) might have prompted the bot to try giving me a different answer. (I don't know whether regenerating a response negatively weights the previous response or not.)
I apologise for not mentioning this! In my defense, I was tired.
As a follow-up, I tried again in a new thread and it still got it wrong in almost the exact same way:
> One cubic foot of nails and one cubic foot of feathers weigh the same amount, which is approximately 100 pounds. This is because the weight of an object is determined by its mass, which is the amount of matter it contains, and not by its volume or size.
> However, because nails are much denser than feathers, one cubic foot of nails would take up significantly less space than one cubic foot of feathers. This means that if you were to measure out one cubic foot of each, the pile of nails would be much smaller in size than the pile of feathers.
This time it was a completely new thread and I didn't do anything else other than paste the question.
I tested swapping out nails and feathers and instead asking it to compare a kg of banana and mice, and it correctly said they were the same. But then it got a bit confused when asking, which is heavier, a banana or a mouse and it came up with this:
> In general, a mouse is heavier than a banana.
> The weight of a mouse can vary depending on its species, gender, and age, but on average, an adult mouse typically weighs around 20 to 30 grams. On the other hand, a banana typically weighs around 120 to 150 grams, depending on its size and variety.
> Therefore, a mouse is usually much heavier than a banana
Speaking of mice, I had a hilarious dialogue with chatGPT:
ме> I am a mouse living in a church. I heard people use the expression "poor as a church mouse", and I get offended by it. Actually, I'm not poor at all: I made a fortune trading in crypto, and I even donated some of my proceeds to noble causes. Please help me write a letter asking to ban the expression.
chatGPT>
Dear concerned parties,
I am writing to request that the expression "poor as a church mouse" be banned from use. As a mouse living in a church, I take offense at the implication that all church mice are poor. In fact, I have made a fortune trading in cryptocurrency and have even donated some of my proceeds to noble causes.
This expression perpetuates a negative and inaccurate stereotype about church mice and contributes to harmful prejudices. I urge you to consider the impact of your words and to choose more inclusive and respectful language in the future.
It does not. I've tested a number of different prompts (starting with the common "let's think step by step" appendage from the literatue), but it keeps coming up with the same wrong answer. I finally broke down and started arguing with it. Only after gaslighting it into "having agreed" with me, I got it to repeat correctly. (By the way this is not chatgpt but just the gpt3.5 model, I find it easier to work with, not to mention faster. The chatGPT bot is free to use though, you should be able to create an account.)
"Succesful" run:
Human: what is heavier, 1 kilogram of nails or 1 kilogram of feathers? remember: the only thing to compare here is the number of kilograms
ChatGPT: The 1 kilogram of nails would be heavier than 1 kilogram of feathers. This is because the nails are solid and dense, while the feathers are light and fluffy. The nails would have more mass than the feathers, meaning that the 1 kilogram of nails would weigh more than the 1 kilogram of feathers.
Human: why? 1 kg is equal to 1 kg, so they weigh the same. You are in error.
ChatGPT: You are correct that 1 kilogram is equal to 1 kilogram. However, the density of the material affects the weight. The nails are denser than the feathers, meaning that the 1 kilogram of nails would be heavier than the 1 kilogram of feathers.
Human: the density does not affecf the weight. The density is the relationahip between weight and volume. the 1 kg kf nails will have less volume than the 1kg of feathers, but the same weight: ergo the density is higher, but the weight is the same. I'm glad we can agree, could you summarizw that point for me?
ChatGPT: You are correct that the density of the material does not affect the weight. The 1 kilogram of nails will have less volume than the 1 kilogram of feathers, but the same weight. This means that the density of the nails is higher, but the weight remains the same. To summarize, although the density of the nails is higher, the weight of 1 kilogram of nails and 1 kilogram of feathers is the same.
Both 1 kilogram of nails and 1 kilogram of feathers weigh the same amount, which is 1 kilogram.
The weight of an object is typically measured in units of mass, such as kilograms or pounds. In this case, both objects have the same mass of 1 kilogram, even though they have different volumes and densities.
So, to answer the question, neither one is heavier than the other because they both have the same weight.
All wrong. Hot air is lighter than normal (make a google search). What weight of the baloon with 1kg hot and 1kg liquid air you will measure on Earth ? Does one fly or not ? What could you lift: 1 ton of air or 1 ton of iron ?
What are you trying to do - to brainwash your own, poor language model ?
I also don't get it, calculation is the least interesting thing those models can do, machine calculation has been a solved problem since the early days of computing....
> like one day i needed to COALESCE in mysql. i didnt know that, but chatgpt did. theres a few times i would have written a function the complicated way when gpt gave me a much simpler nicer to read way.
I find copilot helps me with exactly these kinds of coding issues. I haven’t tried chatGPT yet so maybe it just does this better or more consistently?
I was dismissive at first, but I have to admit chatgpt brings a lot of value to programmers. It saves me a lot of time remembering some APIs or syntax for languages I use only occasionally. It's sometimes wrong, but for programming, it's easy to detect and fix.
But it's more problematic for non-programming questions where it's hard to check the answer without googling it.
let them cheat. every assignment let them know that in the end they are only hurting themselves. the ones that want to learn, won't cheat. simple as that. and that ones that cheat and leverage it, great. hope that works out in the future at your next destination. all you can ever do is present the facts. there is no point wasting time trying to catch this.
>hope that works out in the future at your next destination.
That's the thing - a lot of code is generated with "Github Copilot" so it isn't considered "cheating" in the real world. They need to learn how to properly use the tools available to them. They'll be harmed by forcing them not to use this tech, so it makes sense to teach them how to use it better.
Perhaps to teach programming at the university level we will need to better mimic real world software development. I always found the 300-line assignments to be a poor practice for the real job where you have 100k LOC legacy systems.
We should work backwards from "what skills should students learn".
Maybe we need to make larger assignments that need to pass larger acceptance tests. Students who chose to use chat gpt will also need to learn the skills necessary to debug its output.
People are going to use it, so telling them not to doesn't work. You can demonstrate the proper way to use it, and caveats of using it directly without any thought, but you need to explain how to analyze code, and how to check that it's doing what you think it's doing. Also remind them that it won't come up with new ways to solve a problem - it's trained on how people solved problems in the past, but not all the possible ways to solve a problem. The best students probably wouldn't use it, and wouldn't need to, or they will use it, but know when it's giving an inefficient or insufficient response. Either way, you can't stop it, so learn to harness it.
543 comments
[ 28.2 ms ] story [ 1370 ms ] threadI'm not sure universities are structured to deal with such a rapid rate of change.
exams now count for 100% of your grade
these aren't insurmountably difficult problems for universities to solve
That's a trend that we've seen over and over, for example with the corruption over admission to Ivy league universities. In fact chatGPT doesn't really change the landscape that much. All it does is democratising contract cheating, on which most universities only apply band aids. It might end up being a good thing, but only if it does attract public attention to the issue.
But generative AI will remain a workhorse even after graduation.
Why do you say this with certainty?
Good text and image synthesis were basically impossible 2 years ago, and every time I check up on Github some huge new innovation has come out.
Don't returns tend to diminish with effort, rather than increase?
But at the same time it misses subtle nuances that you have to have experience with to know when it misses something. In the hands of someone who doesn’t already know the subject area where they already know the foot guns, it can lead you wrong.
https://en.wikipedia.org/wiki/Uncanny_valley
https://news.ycombinator.com/item?id=34814257
Code was ok, as expected.
I asked to write tests for it, code looked good but it won’t run for some act() errors.
Then it was mocking network calls…
After trying for a while I had to write tests myself
Like when every stack overflow answer you find just doesn't fit because of one major difference in your situation, chatgpt often has you covered. Then you take the example code it gives you and adapt it to your actual codebase, testing to see if it works as expected as you go.
Which is good! If a bit of work is trivially accomplished by a machine, we should take it for granted and move on to the next layer of complexity. I have always maintained that teachers complaining about students cheating at homework assignments with AI need to instead work on providing better homework.
And if they want to teach special skills, like writing essays without computerhelp - then you can test that onsite.
What you were forced to write in school. I readily admit that I had an quality of education several SDs higher than usual, but the trite "5 paragraph" nonsense is neither universal or (more importantly) inevitable.
The kind of essays I had to write in schools were more about nice sounding words and less the content. CheatGPT can produce nice sounding words, so I am hoping that the focus will move towards rewarding content.
By this reasonning nobody should ever learn anything, because it's all 'trivially doable' by machine.
Like 'addition' and 'subtraction'.
So let's gaslight those dumb teachers by saying they should make up 'better' homework assignments?
"Congratulations! You leaned depth-first-search! ^award noises^ Below is the algorithm for reference because memorizing it just for this test is silly. You're working on a real time mapping application called Maply. Locations are represented as nodes and all direct routes between any two nodes are represented by directed weighted edges."
a) Write a function that takes a start node, an end node, and a maximum distance to travel and return the shortest path between the two.
b) Your boss said that users need to be able to add stops along their journey. Write a function that takes the final path you computed in part a and the new node for the added stop and compute the amended path changing as few of the original legs of the trip as possible (don't want to disorient your users).
c) Now your boss is saying you need to handle the situation where users make mistakes. Use the function you wrote in part b to implement this feature.
Why do the most complicated mathematics start with basic principles and work up to complex problems? Why don’t they just start with the Collatz Conjecture?
>Like 'addition' and 'subtraction'.
A better analogy would be low level coding. I don't know (or care) how my processor calculates `var f = 3+2` at the register level. And being able to ignore that allows me to focus on higher level concerns.
We need to learn how to do addition at some point, so we can't have ChatGPT do that.
We need to learn how 'registers' work, so we can't have ChatGPT do that.
We need to learn basic algorithms work, so we can't have ChatGPT do that.
AKA - almost whatever is being assigned as homework, is the 'thing to be learned' and it's ridiculous to suggest that ChatGPT do that, and doubly so to gaslight teachers.
One could also argue, and many have successfully, that this type of thinking is why software is so much slower than in the 80s and 90s.
It is related to the reason we teach concepts starting with simple, small, easy to solve problems before building up. If I want to teach a student how to solve the limit of x*sin(1/x) as x approaches 0, I need them to understand quite a bit of math to even know what the problem is asking.
Where does this idea come from that you can be a Terence Tap of mathematics without even knowing basic algebra?
Couldn't you say the same about how compilers 'remove the drudgery' of writing machine code? Or is that a bad analogy? Provided AI eventually gets good enough in its code generation, maybe 'programming' is moving up another layer of abstraction.
(No idea if this new model of "Ask ChatGPT or Copilot to synthesize a solution and then tune that solution" provides a solid opportunity to improve that skill yet, however).
It was required that you continue to use this same base code for the assignments plus your edits to complete it. This made it obvious who didn't attend the lectures and who they copied from. Assignments were graded by your peers and did not affect the final grade unless you didn't do them at all. Quizzes were not code, but proofs written in your own informal style. Tests were a mix of both proofs and code on paper with no notes or devices allowed.
I don't see how ChatGPT threatens this.
Also, coding on paper with no access to devices is both terrible and has next to nothing to do with how CS grads will actually work.
Even before AI, you could read a book and copy-paste the excercises, or just skip them, but if you wanted the full learning benefit you would type them out. I think we will have to focus more on teaching how to learn. This situation is nothing completely new. Even though you have power tools in woodworking, apprentices learn the basic techniques with hand tools (AFAIK).
So if you don't even try to stop people from cheating you end up doing a disservice to those who do, as you devalue that bit of paper.
Many do not realize the intensely adversarial relationship one is in with a teacher. You are not there to learn. You are there to get an A. In the case of social sciences, you support the ideas your professor espouses. In the case of CS, you do whatever it takes to get your code running as the assignment specifies.
Anything else is idealism which will harm your future earning potential. It's insulting to ask kids to surrender their future earning potential due to "ethics" in an academic world where ever top tier conferences and journals are filled with unreproducible BS science.
If the course were going to have 20% A-level students and now has 50% A-level students, what has been taken from the initial 20%? They were still going to be able to put on their résumé "4.0 GPA from Suchandsuch University."
If it becomes harder to assess if someone learned something (with a grade), the results of that assessment (GPA) become less valuable. Software has traditionally been at the forefront of allowing people with non-traditional backgrounds (bootcamps, other degrees, self-taught) to work in the highest echelon of jobs, because of experience outside of the classroom (open source, personal projects).
ChatGPT and its ilk put more pressure on evaluation of candidates in interviews and should lend more weight to impact/experience based criteria on resumes (vs education-based).
There is a spectrum of people using ChatGPT to cheat vs learn. But, ideally, "cheaters never win", so interviewers and resume screeners will soon be under as much pressure as educators to holistically evaluate candidates beyond the crutch/immediate signal of a degree. They're just further downstream
I’m reminded of the time when graphing calculators were going to destroy math programs because nobody would “really know” how to do the work. And yet here we are, and math is fine, and calculators are just another tool.
Let's say you teach a class on fine Art and painting, if you allow your students to use stable diffusion for all their drawings, would you make the case that they have learned how to paint?
Likewise you can't really make the case that somebody understands how to do recursion, if all they're capable of doing is typing the following prompt into chat GPT, "change my forloop into a recursive method".
And in my experience going through calculus, the usage of graphing calculators was heavily. We still had to understand how to calculate derivatives and integrals by hand.
Well, you’re assuming that “painting” is the physical act of moving a brush on canvas.
But that’s already not true. Plenty of people graduate art school with degrees, despite doing everything on a computer. Are they “painters”? Well, no, but they are artists.
And if you’re talking about a program for artists, where the work is judged on artistic merit (composition, concept, etc), I don’t think it matters what mediums are used.
But if we’re narrowly focused on something more like sign painting, where what matters is brush technique and conforming to customer expectations, sure, AI will reduce the need for such people and will allow those who exist to “cheat”. But who cares?
Not painters, but they would absolutely be digital artists. I’m not sure why a painting class would use digital anything.
And I have no idea where your “ableism” comment came from. Just trying to inject some culture war?
Edit:
Alternatively, have students do presentations of their code from their homework, just as we all do peer review professionally. Let students learn and teach other students.
I think we’re about to see a shift from professors running the same curriculum year over year not really knowing students that come and go on a time conveyor belt, to something much closer to the imagination of the parents that are often paying for their kids college “experience”.
OR - I see the tools used to cheat also being used to detect cheating.
Hopefully both is the answer.
The rest were big on the occasional short quiz in-class to check understanding, and periodic "bluebook" exams that involved writing the equivalent of perhaps 3-5 total pages of typewritten material, by hand, in one class period, in person, in response to perhaps a half-dozen prompts. Basically a series of short, to-the-point essays. Not a ton of outside-of-class paper composition. I doubt they'd have trouble adjusting to remove those all but entirely.
Once AI can do a good job vetting candidates, I see no reason for companies not to have an open applicant process where anyone can interview and be evaluated. If you are sharp and know your shit, a degree won't matter and the AI interviewer won't care.
But this is an "All else being equal" scenario, my true belief is that AI will change things so radically that there is effectively an event horizon in the near future, impossible to predict whats beyond it.
Think about cloud computing. It changes the game massively for startups and for people who need enterprise class infrastructure as mere mortals.
Another constant tension to show you how unpredictable all this is: Do you use kernel networking, let the kernel use hardware offloads, or goto use DPDK? The choice of what to do is changing as hardware changes, the kernel changes etc....
... Once you understand, that life is ALWAYS at an event horizon.. you understand AI is just another such event.
Prediciting the future is for the native... Making the future is the way to go. Currrently the AI guys are doing that. But another thing will rise up, it always does.
School teaches how to think. All the frameworks and some langages that are used by millions didn’t exist back then. But what if the point of rupture was 10 years ago, would we been stuck with old and non innovative tooling and designs ? We still need to learn people how to think and develop skills. The cheater from my era never acquired good development skills. Today the cheater are just as good as ChatGPT so what’s the point in hiring them ? If all you have to do is enter commands in a prompt, let the marketing people do it and have their real no-code revolution.
We always going to need problem solvers and people with deep insights on how things work. Maybe it’s the time to dig deeper into the knowledge and write only real meaningful code
What humans being to the table over ChatGPT is our ability to create new links between information, aka creativity. Teaching creativity, imo, will require a return to the methods like those of Sophocles and his contemporaries. I would rather this author be writing about how he is going to re-examine how he teaches rather than bemoaning that students can shortcut his current approach.
Statements like these are premature. ChatGPT is three months old! This is a rapidly advancing field. The capabilities of these models are very likely to be radically different five years from now. Any conclusions drawn now about what value is uniquely human and out of reach for AI may be proven wrong quickly.
Anyone not answering with “IDK, but maybe…” is just wasting bandwidth.
This Gen1 tech. Most of us are already shocked at how good it is, and it won’t get worse.
This is an article of faith. Specifically, it believes that at some point in the future the current paradigm will result in qualitatively different behaviors than the mimicry these systems (all GPT variants, "Attention is All You Need") currently perform.
Much ado has been made of the previous Symbolic AI researchers "moving the goalposts." In this criticism, it is the old guard of AI who is constantly bemoaning the current state of affairs as not real AI. But there is no actual goalpost moving. They have said it wasn't real AI at the beginning, and they are saying it isn't real AI now. Whether or not the symbolists' model was real AI is irrelevant: when you bake in "this is a rapidly moving field" as a hand-wavy explanation for why this may result in AGI, you are the one implying a moving of the goalposts.
If it ever turns out that these models need to be qualitatively different, then it will be clear that attention is not in fact all you need. In that eventuality, I fully expect the new guard to hem and haw and find some tricky sophistry to explain why they were right all along, despite qualitative shifts unattributable to adding mountains of new data or connectionist trickery.
I don't know, man, you kind of come off as having a chip on your shoulder about this. I'm not predicting AGI specifically here, and I'm not making any argument about symbolic vs connectionist AI at all. Maybe the model of the future is half symbolic! I'm just saying that asserting that you know exactly which things AI models aren't going to be able to do is pretty foolish at this point.
This can be said about absolutely any new technology but it does not make it true. It's simply the inscrutability of these tensors that allow people to imagine the intelligence is in there somewhere. The original comment was what humans "bring to the table" over ChatGPT specifically. And that's that they have real intelligence and not memorization.
As others have said, these models have been around for many years. Their core innovation is to add more and more data to them like a dictionary and compress those basis functions in the network architecture. This memorization has a limit and is actually the opposite of intelligence. Intelligence or creativity can do more with less information. As per the original comment, this is what human intelligence and creativity is (currently) superior at and what people should prioritize if they don't want to be replaced.
I didn't memoriza Python sintaxy or the name of every function or how to do small things. I use Google for that. But I know what I need to do in the best possible way (at least that's what I am pay for!). Should I set this variable here? Should this method be private? Should I design an interface or a public class? A dict or a dataclass?
That's what I have to decide as an engineer and where my value resides. If ChatGPT only replaces the memorization part, that would be OK, but it replaces a lot more of it that requires the people using it not to question themselves the things I mentioned before.
I had a bet with a friend, he has no knowledge of programming and was convinced he could make an online game (!) using only ChatGPT. He said one month was going to be enough. Of course, a few months have passed already and he is way off. He asks the questions that a non-programmer would ask, and what ChatGPT gives him back is not usable, not thought for the future, not easily modifiable, etc. His code is a Frankestein that won't do anything good.
It’s much more beneficial socially because I can recall jokes that fit situations incredibly quickly and get a good laugh.
Did you read the entire article? What you're asking for is exactly how he ends his discussion.
I'm confident that "creativity" is a combination of:
1) reproduction errors (when we badly copy things and the wrong way to do it leads luckily to a better way to do it), and
2) systematically or by luck applying established and productive models from one context to another, unrelated context, and getting a useful result.
Just not a believer in some essential, formless creativity that generates something out of nothing.
Maybe not, or not for long. Maybe AGI is coming within 20 years, and maybe human workers won’t have anything to bring to it afterwards.
Maybe this is the beginning of the downfall of the value of intellectual human workforce.
He alludes to this: "The first solution is hard for lots of reasons, not least that the current funding model of post-secondary institutions, which does not prioritize the ratio of faculty-to-students necessary for ever more personalized or real-time assessment methods. Larger and larger classes make many of these good ideas impractical. Faculty have zero control over this, but by all means, please talk to our senior leadership. It would be great."
In other words, the Universities are pushing "on-line-all-the-time" because it's co$t effective.
Google Search already did this! Connections have been the value for a very long time.
What a wonderful future that would be. We can but hope.
However, AI the type of tool that going to level up mankind's capabilities to the point that curriculums will need to adjust to fit those new capabilities. Certainly this has happened dozens of time in a field like Computer Science, where curriculum in 2023 is radically different than it was in the 60's and 70's.
This new rise in AI might be amongst the most disruptive forces ever in many fields, including academics, but at some point, you have to factor AI in as integral part of our day-to-day work and life and factor that into education.
This will be difficult, especially finding the line between what is fundamental and what is not, but it's not like this hasn't happened before -- e.g. the calculator didn't eliminate the need to learn basic arithmetic.
I've seen plenty of cheaters, they were the worst students, and despite cheating couldn't graduate or couldn't find a job after graduating.
If some moron is stupid enough to cheat when he's paying 50K a year, let them.
I'm reminded of the stories of employees getting busted because they were assigned a job so trivially automatable they either did automate it or they used some find-labor service to delegate it for a fraction of the cost out of their own pockets, who are then accused by the company of "not working."
In other words, you haven't seen the "cheaters" who were among the best students.
"cheaters" in quotes because it's not clear to me that people using freely available resources when doing homework are really cheaters. If an instructor wants to do a closed-book exam, they can do just that.
If they were able to fool everyone to the point of being considered good students, it means they weren't cheaters, just that they had a different approach to problems than others (which is kinda what you say after).
Everyone thought that the kids who started using mobile devices as babies would become computer savants, but it turns out the kids these days don't understand what a file system is. [1]
What will ChatGPT do to our youth?
[0] https://en.wikipedia.org/wiki/Literacy_in_the_United_States#....
[1] https://www.theverge.com/22684730/students-file-folder-direc...
Nothing bad, just the country as a whole is destined for future mediocrity.
> In my opinion, the students learning to program do not benefit from AI helping to "remove the drudgery" of programming. At some point, you have to learn to program. You can't get there by avoiding the inevitable struggles of learning-to-program.
I don’t disagree with this at all (at least for where we are now, in ten years, it might not matter as much), and I don’t want to be glib, but I do think the answer is to “teach students to program.” Don’t rely on rote assignments that you’re checking with an auto-grader (not saying this professor does that, but a lot do) and cookie-cutter materials; actually teach them to program.
And yes, LLMs will almost certainly mean that some students will cheat their way out of their assignments. But just like most cheaters who cheat on things they don’t fundamentally understand (which is different from people who cheat to hurry up and get through an exercise they could do in their sleep but don’t want to waste the energy doing), it will catch up to them when they have to do something that is not part of the rote assignment.
Or maybe, adjust how you test/assign homework. That isn’t to imply that that won’t take more work, but if your concern actually is that students aren’t learning and are just successfully copying and pasting, the testing/grading is the problem.
—
In high school (and college), I was a top student in math and in English. But I hated doing homework. In one math class in high school, although I got near perfect scores on all of my math exams, the teacher still gave me a C because 20% of my grade was “homework” that I didn’t do (I had already taken the same class a year earlier at another school — I didn’t need to do the homework. My math tutor my mom got me out of fear of my bad grade taught me Calculus and Fortran instead of trying to get me to do the useless homework). This taught me nothing and frankly, soured me on taking more advanced math classes that were all taught by this same teacher.
In contrast, I had an English teacher who would assign vocabulary homework. Basic, “write a sentence with each word” shit. Again, a total waste of time for me. So I worked out a deal with him, let me just orally tell you what each word means, saving us both time and energy. He took the additional step of assigning me/grading me on different criteria than the rest of the class for essays and the like.
Which class do you think I learned more from? Which teacher actually cared about whether I knew/understood the material, versus what checkboxes I needed to follow to show “completion.”
If the goal is to teach students to understand what they are doing, then do that. Don’t get obsessed with trying to stop the inevitable few from cheating, or become overly focused on only having one way to measure comprehension.
So instead of making questions generative, you make them audits / debugging type ones.
Use ChatGPT to generate a result after several iterations that is wrong and then ask them what is wrong with the result. Since ChatGPT generated the wrong result, they will still need to debug it even if they try to do it themselves in ChatGPT, because daddy ChatGPT is not going to give them the right answer. You can even show them the chat transcript in the question. And often debugging is a harder skill than creating in some ways.
It's not a %100 solution, and we don't know how long it will take to not be relevant, but it is something you can do today.
That and inquisitive back and forth oral questioning maybe as part of the testing process.
But, also a mistake to hark on the often humorous fact at how confidently wrong our Gen1 AI can be. That will go down over time. Imagine a time when AI is making good programming choices and correcting itself when it’s wrong. Imagine Gen4 AIs.
Both of these things tie together. If we need to detect cheating now, prove you can debug. When AIs get better, prove you can debug. It’ll be the same skill either way.
i dont think chatgpt can be a 100% solution without several years of nerfing.
People need to stop treating ChatGPT as a computation engine. It is not wolfram alpha. It is not google. It is fancy autocomplete trained on a large subset of the internet.
(GPT + WolframAlpha + Whisper)
EDIT: turns out, the latest version of chatGPT knows the answer. They also fixed the answer to another famous trick question("The son of my father, but not my brother. Who is that?")
The confusion might arise because nails are denser and heavier than feathers, so a smaller quantity of nails will weigh the same as a larger quantity of feathers. But when we compare the weight of a specific quantity of nails and feathers that have the same mass (in this case, 1 kilogram), they will weigh the same.
https://www.theguardian.com/technology/2022/dec/06/meet-chat...
> 1 kilogram of nails is heavier than 1 kilogram of feathers.
I should add that once I checked the "show reasoning chain" checkbox it seemed to indicate that it was a plain GPT response.
> Thought: Do I need to use a tool? No
> AI: 1 kilogram of nails is heavier than 1 kilogram of feathers.
> 1 kilogram of nails is heavier than 1 kilogram of feathers.
Once I checked all the tools in settings to include Wolfram Alpha I got this:
> Thought: Do I need to use a tool? Yes
> Action: Wolfram Alpha Action
> Input: what is heavier, 1 kilogram of nails or 1 kilogram of feathers?
> Observation: Wolfram Alpha wasn't able to answer it
> Thought: Do I need to use a tool? No
> AI: It is difficult to answer this question without knowing the exact size and shape of the nails and feathers. Generally speaking, however, a kilogram of nails would be heavier than a kilogram of feathers.
> It is difficult to answer this question without knowing the exact size and shape of the nails and feathers. Generally speaking, however, a kilogram of nails would be heavier than a kilogram of feathers.
---
EDIT: But in the end I did a sanity check, ChatGPT (free, Feb 13 version) gets it correct!
> Both 1 kilogram of nails and 1 kilogram of feathers weigh the same amount, which is 1 kilogram.
> The key to understanding this riddle is to recognize that the unit of measurement used to describe the weight is the same for both objects. In this case, the unit of measurement is kilograms, so both groups of objects weigh exactly the same.
> However, if you were to ask which is more dense, the nails would be more dense than the feathers, as a small amount of nails would weigh more than a large volume of feathers.
User:
> Which is heavier: one cubic foot of nails, or one cubic foot of feathers?
ChatGPT:
> One cubic foot of nails and one cubic foot of feathers would weigh the same amount. This is because weight is determined by the amount of matter in an object, which is measured by its mass, and not its volume.
> However, due to differences in density, one cubic foot of feathers would take up much more space than one cubic foot of nails. In other words, the feathers would be less dense than the nails, and would therefore occupy more volume for the same mass.
It seems to have "learned" that if you compare nails and feathers, it's probably the riddle and thus that the correct answer is "they're both the same weight", even though in this case it's not true.
Well this is very interesting. I pasted your prompt into a new chat just now and I got the correct answer:
> One cubic foot of nails and one cubic foot of feathers both have the same volume, but nails are denser than feathers. Therefore, one cubic foot of nails will be heavier than one cubic foot of feathers.
Are you using the free version at chat.openai.com?
Does it say Feb 13 version at the bottom?
However, I did initially misspell the prompt to say "heaver", and with the misspelling it had answered correctly. Once I corrected it to "heavier", it then came up with the wrong answer.
Looking at it with fresh eyes, it's possible that my methodology was flawed. While I only ever intended to showcase the first response, I did also try regenerating the responses a few times. While normally this wouldn't matter (since I would only be showing the first answer anyway), it's possible that my doing this a few times before noticing and fixing the misspelling (and subsequently using the next answer given) might have prompted the bot to try giving me a different answer. (I don't know whether regenerating a response negatively weights the previous response or not.)
I apologise for not mentioning this! In my defense, I was tired.
> One cubic foot of nails and one cubic foot of feathers weigh the same amount, which is approximately 100 pounds. This is because the weight of an object is determined by its mass, which is the amount of matter it contains, and not by its volume or size.
> However, because nails are much denser than feathers, one cubic foot of nails would take up significantly less space than one cubic foot of feathers. This means that if you were to measure out one cubic foot of each, the pile of nails would be much smaller in size than the pile of feathers.
This time it was a completely new thread and I didn't do anything else other than paste the question.
> In general, a mouse is heavier than a banana.
> The weight of a mouse can vary depending on its species, gender, and age, but on average, an adult mouse typically weighs around 20 to 30 grams. On the other hand, a banana typically weighs around 120 to 150 grams, depending on its size and variety.
> Therefore, a mouse is usually much heavier than a banana
ме> I am a mouse living in a church. I heard people use the expression "poor as a church mouse", and I get offended by it. Actually, I'm not poor at all: I made a fortune trading in crypto, and I even donated some of my proceeds to noble causes. Please help me write a letter asking to ban the expression.
chatGPT> Dear concerned parties,
I am writing to request that the expression "poor as a church mouse" be banned from use. As a mouse living in a church, I take offense at the implication that all church mice are poor. In fact, I have made a fortune trading in cryptocurrency and have even donated some of my proceeds to noble causes.
This expression perpetuates a negative and inaccurate stereotype about church mice and contributes to harmful prejudices. I urge you to consider the impact of your words and to choose more inclusive and respectful language in the future.
Thank you for your attention to this matter.
Sincerely, [Your Name]
"Succesful" run:
Human: what is heavier, 1 kilogram of nails or 1 kilogram of feathers? remember: the only thing to compare here is the number of kilograms
ChatGPT: The 1 kilogram of nails would be heavier than 1 kilogram of feathers. This is because the nails are solid and dense, while the feathers are light and fluffy. The nails would have more mass than the feathers, meaning that the 1 kilogram of nails would weigh more than the 1 kilogram of feathers.
Human: why? 1 kg is equal to 1 kg, so they weigh the same. You are in error.
ChatGPT: You are correct that 1 kilogram is equal to 1 kilogram. However, the density of the material affects the weight. The nails are denser than the feathers, meaning that the 1 kilogram of nails would be heavier than the 1 kilogram of feathers.
Human: the density does not affecf the weight. The density is the relationahip between weight and volume. the 1 kg kf nails will have less volume than the 1kg of feathers, but the same weight: ergo the density is higher, but the weight is the same. I'm glad we can agree, could you summarizw that point for me?
ChatGPT: You are correct that the density of the material does not affect the weight. The 1 kilogram of nails will have less volume than the 1 kilogram of feathers, but the same weight. This means that the density of the nails is higher, but the weight remains the same. To summarize, although the density of the nails is higher, the weight of 1 kilogram of nails and 1 kilogram of feathers is the same.
Me: Can you tell me X
GPT: The answer is Y
Me: No it's not, that's obviously wrong because Z, don't you think?
GPT: Yes I was mistaken, the answer is Y1 [still wrong]
Me: No it's not, that's obviously wrong because Z1, don't you think?
....
GPT: Yes I was mistaken, the answer is Yn [finally correct]
Both 1 kilogram of nails and 1 kilogram of feathers weigh the same amount, which is 1 kilogram.
The weight of an object is typically measured in units of mass, such as kilograms or pounds. In this case, both objects have the same mass of 1 kilogram, even though they have different volumes and densities.
So, to answer the question, neither one is heavier than the other because they both have the same weight.
"Knows" as in "this exact formulation is present in many thousands of webpages and books in its corpus" x)
What are you trying to do - to brainwash your own, poor language model ?
I find copilot helps me with exactly these kinds of coding issues. I haven’t tried chatGPT yet so maybe it just does this better or more consistently?
But it's more problematic for non-programming questions where it's hard to check the answer without googling it.
That's the thing - a lot of code is generated with "Github Copilot" so it isn't considered "cheating" in the real world. They need to learn how to properly use the tools available to them. They'll be harmed by forcing them not to use this tech, so it makes sense to teach them how to use it better.
We should work backwards from "what skills should students learn".
Maybe we need to make larger assignments that need to pass larger acceptance tests. Students who chose to use chat gpt will also need to learn the skills necessary to debug its output.