Straight up assembly language, preferably Motorola MC680x0 or some other contemporary that was meant to be coded by hand; it's the only way, man! Plus students will gain some appreciation for how microprocessors actually work.
In the same vein, just have them write code out on paper during exams and coding competitions like the way we did it back in my day. It's not as if they're not going to need that skill anyway when they get whiteboarded during interview loops when even the most lenient employers get exasperated at "CoPilot-only" graduates.
I guess CS programs are going to need to grow up and teach real topics like ethics, technical communications, effective testing, and large-scale systems design instead of how to write simple loops, which is something I learned at the age of 4 from “Gortek and the Microchips”.
You can test loop-writing in an oral-exam, too. Thus your proposed solution for GPT-3 writing ethics essays is so good, that it fixes the original problem.
Just add a requirement where the student has to explain how the code works.
To be honest there should be a class called language model querying because for simple tasks it does well but eventually you will have to debug the output (not just copilot but GPT3 and dall-e).
But then you would have a foundation to figure out the new models. This is the essential problem when teaching any field because the field itself advances faster than one can effectively teach you.
That's why getting an Internship is always a good idea.
Much like reverse engineering, it's easier to come up with an explanation when you have someone else's working solution to examine than it is to come up with a working solution from scratch on your own. So that doesn't really help.
but maybe, they'll end up learning anyway? one thing i realized when i went back to studying online is that i was no longer the student who needed to prove something to someone else, and the typical test annoyed me. I get that this article is about a teacher trying to find out how much their students have learned a subject, but maybe that needs to change?
Article author here. Turns out language models already can explain how code works, pretty convincingly. I gave a prompt to OpenAI, got the solution, and then asked it to explain the solution (possibly easier to read in my tweet: everything in green below is from OpenAI - https://twitter.com/emeryberger/status/1560618825208320001). I wrote the comment asking a question ("# How did you solve this problem? Explain how you got each digit from each number.") and Tab completed the rest:
# I solved this problem by using the % operator to get each digit from the number. I declared and initialized a sum variable to keep track of the sum of the digits of the number. I set a temp variable equal to the input number. I then used a while loop to loop through each digit of the number. In the loop, I used the % operator to get the remainder of temp divided by 10. This gave me the last digit of the number. I added this digit to the sum variable. I then divided temp by 10 to remove the last digit of the number. I repeated these steps until all of the digits of the number were added to the sum variable. Finally, I printed the value of the sum variable.
This is why I love the disable javascript plugin for Chrome. One click and terrible sites like these are tamed. uBlock origin is also useful to block sticky headers.
I get that copilot is AI and pretty cool, but students could look up a Fibonacci program on Google before it existed. What's more these algorithms are written in books. If students wanted to cheat on your "write depth first search" assignment then they already were.
I think the big change here is how much easier it is? Students, like everyone else, are lazy. Previously just doing the assignment was typically less work than cheating, and Copilot makes that much less common.
Type the assignment into google and press enter vs type it as a comment and press tab?
Seems roughly the same amount of effort, and since Copilot is not free anymore most students won't bother paying for access on any kind of scale that matters. Whoever was going to cheat will cheat regardless.
Besides it's one of those "you won't have a calculator with you everyday" fallacies. If you can solve problems with Copilot in class you can also do it at your job later on.
Not really, for small snippets, sure. But jobs require getting something done. Sure you might be able to tab complete 10 pieces, but at some point you have to tie them together to get a working program and those 10 pieces will have different assumptions and require some real understanding to integrate.
I understand what you're getting at, but feel like you're splitting hairs here since somebody who goes to look it up on Stack overflow is just going to copy paste... not type it out line by line like it's a page of BASIC from BYTE magazine.
This reminds me of when my school teacher insisted that real engineers didn't use calculators. They'd calculate their sines and cosines from a table.
3 years later, the education ministry decided that was BS and let everyone bring calculators to math exams. Surprisingly, math scores didn't go up much.
Cheating is getting easier by the day, but the bottom line is that if the student is cheating, they're probably not absorbing the coursework (or if they are cheating and absorbing the coursework, there might be something wrong with the coursework).
The most effective policy might not be to try and mechanically prevent cheating, but to explain that, this is information that you're going to need if you actually want to go into the field and if you're using copilot here you are damaging yourself in the long run.
Maybe it's not fair to give as good grades to students who are cheating on assignments as those who are actually doing the assignments, but at the end of the day is the purpose of the course to measure the student or to teach the student?
> "...this is information that you're going to need if you actually want to go into the field..."
That has never, ever stopped anyone motivated to cheat. Even here on highly-educated HN, you get posters with open disdain for what they learned in college.
> "... at the end of the day is the purpose of the course to measure the student or to teach the student?"
Both. Otherwise, the college diploma really does become the meaningless scrap of sheepskin that its detractors claim it is.
There's more to this: it used to be that a college degree carried with it something of a soft guarantee that the student could achieve a baseline level of work on their own. This acted as a useful filter for employers. If cheating runs rampant that filter becomes meaningless, and once the student has successfully cheated their way into the work force, their incompetence can (depending on field) do very real damage to the lives of others.
Because of this, there will always be considerable pressure to detect cheating and remove offending parties from the program, be that the students copying answers from the internet, or the institutions failing to detect the problem before handing out a degree.
A college degree used to signal competence. Now it signals you have the ability to show up and avoid doing something to get kicked out for 4 years.
I feel we'd probably get the same value to society if we just made grades 9-12 optional. It would remove the people that don't want to be there, improve the quality of instruction, and college would actually be a meaningful achievement so we no longer have requirements for entry level jobs to require a Master's degree.
> The most effective policy might not be to try and mechanically prevent cheating, but to explain that, this is information that you're going to need if you actually want to go into the field and if you're using copilot here you are damaging yourself in the long run.
I don't teach CS, but I teach an online course in a STEM field. I give out a similar message ("you're only hurting yourself in the long run").
My anecdotal evidence is that my warning does nothing--I have trap questions on various quizzes/exams and the frequency of cheating hasn't changed with or without the warning.
My suspicion is that the students who cheat feel this way: "This is just some bullshit hoop that I have to jump through, so it's okay if I cheat. I'll figure out the important stuff when the time comes."
I mean, they're delusional, but I understand the mindset for cheating.
edit: amusing story.
Faculty members in my department are required to take an on-line course on how to handle hazardous waste. There are a few hours worth of videos to watch and a test that you need to pass at the end.
Last year, one of the faculty members took the test, compiled the answers and emailed them around to everyone else (to save them the time for this "bullshit task.").
I said to him, "Isn't this precisely the kind of shit that makes every faculty member angry when the students do it?"
> I mean, they're delusional, but I understand the mindset for cheating.
Parental (other similar) pressure can indeed be quite strong. Absent of that pressure they wouldn't be there in the first place, so perhaps it is not so much being delusional as a rational response to their environment? If they cheat their way through the parents are happy and then they can return to the life they otherwise would have lived. The time lost is unfortunate in some respects, but at the same time if you're cheating, the time investment likely isn't that great and an acceptable cost given the circumstances.
> the kind of shit that makes every faculty member angry
I find it curious that the customer using the product in an unintended way would be upsetting to the vendor. What drives such emotions? In my businesses, I couldn't care less how the customer uses my product. If they're happy, I'm happy.
That certainly would be true within the public education (primary, secondary) system, where the government is the customer. Hence why attendance is mandated. However, typically college level students are initiating and fulfilling the transaction, thus they are the customer. They offer up money in exchange for keeping arbitrary social pressures at bay. In rare cases they offer money in exchange for learning things.
It is possible for the customer to also be the product. Especially in the age of salable data, that is becoming more and more common. However, when that is the case there is incentives given to the customers to shape them into what selling them as a product requires. In context here, that would mean something like giving discounts to those who don't cheat, which I am not familiar with any college doing, so... That brings us back to why would a vendor get emotional about the customer not using the product as intended?
> I find it curious that the customer using the product in an unintended way would be upsetting to the vendor
Because your customer, in so doing, is stealing from your other customers and undermining your reputation as a vendor.
Plus, you're disheartened because this interaction is a waste of time, regardless of the fact you get paid: if money was all you cared about, you wouldn't be in education.
> Because your customer, in so doing, is stealing from your other customers
That's always going to be the case, though. Imagine you built software that helped businesses find customers. Using it the intended way they can realize a small number of new customer each day distributed across your customer base. Now, someone finds a new way to use your software in a manner that you never envisioned and they're attracting all the customers, taking from those who used the software as expected.
But, really, who cares? You just pivot to the embrace the new way and carry on with life. That's your market now. It's fun and all to want to be the elevator operator of old, but at some point you have to realize that nobody cares about your nostalgia. Markets change.
> Plus, you're disheartened because this interaction is a waste of time
But it is not. The students are only there to abade social pressures and cheating their way through gives them what they want out of the deal. Very few students care about the academics.
> if money was all you cared about, you wouldn't be in education.
If you don't care about money, why are you marketing the social need so hard? This is like a car manufacturer marketing that their cars are great gateway vehicles when committing crimes and then lamenting that criminals are using their vehicles to get away...
Before the ridiculous, albeit successful, "If you don't go to college you will end up flipping burgers" marketing campaign, college only attracted a small number of students who were serious about learning and all was well with the world. Colleges still spend an inordinate amount of time justifying why the cost is worthwhile to keep up the social need illusion.
It's not hard to revert to the natural state. Fact of the matter is that you (not you personally, perhaps, but the group) don't want to.
You picked a very specific example of customer behaviour, with aspects of a zero sum game, that is by no means "always going to be the case".
And yes it's a waste of time! If you want a fake credential then it's much cheaper and easier to just lie on your CV. It's not like many employers will check your certificates, at least not in software. I'd really rather people did that than waste my time first trying to teach them and later handling their academic misconduct cases.
If I'm marketing social need, this is the first time I've heard of it ;-)
It is, but an acceptable cost to keep the social pressures at bay. There are a lot of things in life that are ultimately wastes of time, but worthwhile to maintain a civil society. Such is life. If you are happy to escape to the deep woods away from all others where every moment of your time is purposeful, good on you.
> If you want a fake credential then it's much cheaper and easier to just lie on your CV.
What would that accomplish? The business world couldn't care less about your past achievements. They don't care that you performed in a play, rode a horse, swam in the ocean, or went to college. There is no value in even putting a legitimate degree accomplishment on your CV, let alone faking it. If anything, it detracts from your standing as it shows that you're so useless you couldn't come up with anything more relevant to share about yourself.
Instead, it is parental pressure that puts one into these schools when they otherwise shouldn't be. And those parents will turn up at your graduation. Once you've thrown your cap in the air, so to speak, nobody will ever think about it ever again. Getting to that one moment in time is the so called hoops that one is cheating for.
> I find it curious that the customer using the product in an unintended way would be upsetting to the vendor. What drives such emotions? In my businesses, I couldn't care less how the customer uses my product. If they're happy, I'm happy.
There's a lot more going on here than maybe you realize.
First, the students are not your only customers. The administration is also one of your customers. If you make the course TOO difficult and fail everyone, your "administration customer" will find a way to get rid of you.
Conversely, if you make the course too easy and everyone gets to cheat and everyone gets an "A", then many students who pass your class may not pass various professional certification exams that are more rigorously controlled (i.e., no cheating). If that happens, then the school may lose accreditation for programs that you are associated with (i.e., if all the nursing majors fail their certification exam, your nursing program may get nuked). Then you get nuked.
Third, whether you like it or not, your relationships with your students is something you have to deal with. Students who don't cheat have a tendency to get really upset if they find out you turn a blind eye to cheating. Happy cheating customers leads to unhappy honest customers.
Fourth, if your fellow faculty members feel like you're going easy on the students to get favorable reviews (or because you're lazy or because you hate confrontation), they can make life unpleasant for you, too. They also aren't too happy if they teach upper level courses and you send a bunch of garbage students their way.
It seems that you're not managing your customers well. If, in any other business, you needed to adjust the behaviour of some customers to keep other customers happy you would provide incentives, like a price discount. You don't need to fail the cheaters, you just need to make it more appealing to not cheat to keep the non-cheaters happy. How are you doing that? Is there a financial discount to those who have shown to not cheat (or conversely a higher rate for those who wish to cheat)?
If not, you might want to try a new business. You may not be cut out for the one you're in. The business world is not very forgiving, nor should it be. Those who can't adapt need to perish.
> It seems that you're not managing your customers well. If, in any other business, you needed to adjust the behaviour of some customers to keep other customers happy you would provide incentives, like a price discount
Spoken like someone who hasn't done the job and is confident in his ignorance.
Yes, that is what "seems" implies. It suggests that there is gaps in the information and that one should come back with more details. There were also directed questions included to help guide one to where information was lacking.
Curious that a primitive emotional response has transpired instead. Given the greater context of discussion also about emotions, is there something about the job that attracts this type of behaviour?
Some courses really are required bullshit to some students. A lot of professors don't realize this, but if you teach a required course, it is simple math to say that some of your students just don't need to learn what you are teaching. Sure, a lot of students do need it but don't think they do, but a good fraction of the class does not need to learn the content you are teaching.
The only remaining question is how to engage these people. I have seen good approaches to that problem and really, really terrible approaches, and it seems that most professors go for the terrible approaches. Unfortunately, engaging people who don't want to be there is a lot harder than just having a bunch of required tests.
These students are not all wrong that some courses are BS hoops to jump through. Meet them in the middle. Teach them something.
I know a lot of what I teach is not really relevant to students who aren't chemistry majors--I even say this UP FRONT at the beginning of the course. I tell the students that they should think of the course as a way of assessing whether they can think logically and critically about "weird things." Because life is full of weird things that need logical and critical analysis.
In general, they appreciate my candor.
That being said, in my experience, the students who cheat often delude themselves into thinking that they understand the material--and they really don't.*
Then they seem shocked when they take a certification exam (where they have to leave phones at the door and are closely monitored) and they bomb the exam.*
* I'm speaking in generalities here. I'm sure there are some cheaters who are being "smart" about their cheating.
In the time I was at school, I heard of a lot more cheating than would be justified solely by people conserving time. I have recently seen studies that argue that 50-80% of students cheat at some point, and something like 10-20% cheat in every course they take. Personally, when I conserved time, I just accepted the bad grades - it was easier than trying to cheat and not get caught. The OP was talking about students cheating because they are not engaged with the material. I think that's only part of the problem with cheating: there are lots of other factors, like pressure to get a high GPA.
One of my psychology professors had an alternative: The course I took was required for psych majors, but was also interesting-sounding and worked as an elective for non-psych majors, so at the start of the semester she announced to every one of her students that psych majors should switch to the 3-days-a-week one to get the in-depth knowledge they'll need for the rest of their major, and non-psych majors should switch to the 2-days-a-week one to get the watered-down version. The two classes would explicitly be held to different standards, because this way the non-psych majors wouldn't have to keep up with the psych majors. And the psych majors were warned that if they tried to take the easy out, they'd fall behind later on.
Adjunct instructor here. Before I came into the program, cheating was on the rise. When given programming assignments, students would go to Geeks for geeks and download solutions, some of them mostly correct, and submit those for credit. One remedy was to try to make programming assignments harder and harder. Eventually, they got so hard that we were asking people new to computer science to implement a full arbitrary length integer calculator using nothing but a single tape Turing machine (in Java, not BF).
Eventually, some of us came to the realization that you can never prove that somebody is cheating. People have been known to hire tutors to do their assignments for them. There is just no end.
As a result, we evaluate students on four dimensions. Programming assignments,* homework assignments, but also class discussions and group participation. Those last two count for a small, but non-trivial percentage of the grade, and are usually enough to separate and identify those who understand what they are doing, from those who are just "following along" solutions that they find on the internet.
* one addition to programming assignments includes an analysis write up. Tell me in human words what is happening. Why do you see that effect. What is the running time. And separately, comment your code to tell me how it works. Those two parts count significantly towards the grade.
Any other professors here, what have you found that works?
> one addition to programming assignments includes an analysis write up.
when i was interviewing candidates this was the fastest way to filter those who understood and those who didnt.
ask someone to talk you through how to solve some real-life problem. to look at a stack trace and describe what they see, or to run a profile and trace through the cause of a hotspot/contention. show them an issue and have them live debug to root-cause and fix. it's okay if they use the internet, SO, etc. -- that's how we all do it. see what they have to look up! just listening to the amount of depth someone can verbally communicate during novel problem solving (including additional questions they ask you) turns a 60min interview that wastes time into a 10min one that tells you if you can move on.
a favorite one of mine was to ask the candidate to describe in as much detail as possible what happens between a keystroke typed into a google search box and the search results appearing. the diversity of replies is facinating. some will tell you "google returns results from its database", others will ask you if you want them to first describe how the keyboard works at an electrical level through the USB or bluetooth driver stack.
"Never memorize something that you can look up." --someone smart
Ugh I had someone ask me to debug a verbally communicated error message in an interview.
Like they picked some random port configuration issue that had stumped them for days in the past and thought cool let’s remove the internet, the command line, and ask people to solve it on the spot.
I offered plenty of plans for how I would go about debugging, but I didn’t know the one simple trick.
I agree with trying to use challenges that are closer to the real work, but it’s really hard to do that without over-testing domain knowledge.
For instance, I ask candidates to do some asynchronous control flow. These are all candidates with js listed as their best language and I offer to let them look things up or to show them the promise apis they need, but a certain percent just refuse to engage with the problem because they don’t have the domain knowledge and seem to feel they’re being tested unfairly.
The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks.
> Ugh I had someone ask me to debug a verbally communicated error message in an interview.
i dont mean that the problem is only verbally communicated. i mean that the debugging process the candidate does is verbally communicated.
in the scenario you're describing you'd be sitting at that machine with access to the internet on another machine. like, you know, in real life.
> The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks.
async stuff in js is pretty good, but it's also easy to go too deep on it with some bizzarely poorly architected code. many js devs still fail to grasp all the implications of closures, or how to avoid memory leaks, or how to work with the GC rather than against it.
That's a great way to evaluate someone! Individualized and personal, uses concrete problems, no artificial restrictions.
It's a shame our mass education systems cannot use it. They simply cannot apply such a humane method to hundreds or thousands of students. They are reduced to applying their bullshit test questions because it's the best they can come up with at their scale.
> but also class discussions and group participation
I really, really hated those parts of school, second only to group projects. I do appreciate why teachers want that, and I hear you about how it can surface those who are cheating, but I do wonder if it's causing the class to suffer for the alleged misdeeds of a few
School assignments are so different from the dynamics of a real workplace that I find this point very moot. Always hated them with a passion, but otoh I always enjoyed team math and programming competitions, situations where you have to do stuff that's actually hard, work with people you actually respect and trust them to do their job.
For me at least, the dynamics of "School Group Project" are basically completely different and separate from "Work team". I happen to do well in both, but I do not enjoy both: the motivations and structure and dynamics and goals and timelines of "School Group Project" are so much more artificial and ultimately pointless, and obviously to all from the start.
Same things with discussions - I happen to be an engaged student, usually front row, hand always up, discussing with instructor and team and colleagues and everybody. I like to be engaged and figure things out together. But I'll never be half the developer than my colleague who barely speaks a word unless asked. He's friendly, meek, polite, and excellent team player and developer - he just does not initiate conversations, especially in group settings. I can imagine he'd have a nice big 0 in that category if that was a grade criteria.
Which is not to criticize the professors who try to use group discussions; just to point out, this is not a solved problem at scale any more than interviewing/hiring. There's too much humanity and too few absolutes :)
In my experience, yes, since much like school the team gets the team's grade if you're trying to ship a product; there can be room for individual merits if the company has bonuses or a separate reward structure, but my experience is that expecting everyone on a team to pull their weight equally is a fool's errand (even if it's for something quite reasonable like a family emergency, or other "good reason")
In school it never bothered me having freeloaders get the same grade for my work, so long as they stay out of my way and don't "help." There's a very famous mechanic's poster: "repair: $5, if you watch: $20, if you help: $1000"
When teachers require recording of timelapses, we will have AI which will be able to generate text and a video of a user typing it. IMHO, they should start to require timelapses right now.
That requires either trusting that students haven't edited out the evidence of cheating, which is pointless, or forcing them to use specific, proprietary software to record the video, which is unacceptable.
Timelapse can be recorded online, just like a security camera with upload to cloud does, so it will be impossible to edit later. Cheating will be visible because of sudden changes in text of the program.
That is, as I said, unacceptable. Among other things, it means students can't use their editor of choice, and students with disabilities who use assistive technologies will be much more likely to be flagged as cheaters. It also won't stop anyone from cheating by doing the exercise with Copilot and typing it up a second time for the time lapse.
I do want to say that, those criteria exclude neurodivergent folk (who for many reasons may have never even been diagnosed).
Honestly I think moving away from grades is going to be the way forward. Part of this is also making university, etc cheaper so you’re not financially incentivized to not take a class one, or multiple more times.
Mass-produced education is like a train, once you fall off you'll get hurt trying to get back on. I've come to the realization that given the inhumane pace, difficult and poorly-taught curriculum, and ambiguous/out-of-scope homework assignments in a top-tier university's CS program, the majority of my university classes were either not worth taking, or I'd be better off self-studying if I found an intrinsically rewarding project (definitely less panic attacks and psychological trauma, sometimes even similar or deeper learning). Perhaps I'd have a better time in a less prestigious university that doesn't see its purpose as weeding out and breaking lesser students.
I think during education yes, during execution of the skills learned in education - not really but also depends.
A lot of people hate how tech interviews are done, right? Cause they have nothing to do with the job the majority of the time. Those are places where I can see improvements.
However, some jobs do genuinely need you to interact with a diverse set of people all the time, for those you need to make sure for the sake of your company & the candidate that they match the criteria. Other jobs, you need only maintain strong relations with small groups, and the requirements for those are a lot easier to hit - even the most introverted people can do well in smaller groups that don't change a lot.
I do really think a lot of evaluations that happen during education though makes a confrontational relationship with learning. You're constantly judged about your _ability to learn_and that has far reaching impacts beyond your first twoish decades of your life. You get scared of making mistakes, you get scared of taking risks, etc.
So, evaluate when it's important for the function & safety of the job, and the candidate. Skip evaluation when its evaluation for the sake of evaluation.
Evaluation is vital but the current model used in education today is not evaluation, it is punishment. Failed to get the answers right? You are punished. You lose points. Your GPA plummets. You could even fail the class and have to take it again which means the punishment is not only social but economical.
There's real life consequences to this kind of evaluation. Huge consequences. Students cannot afford to make mistakes. There's huge pressure and anxiety before and during a test because the stakes are so high. Failing at this stuff can cost someone their future: future jobs, future career, even the student loans that enable them to study in the first place may only be afforded to students who get good grades.
Well if you have to meet certain criteria to get a title then "punishment" is exactly what I'd expect to see.
Isn't that the whole point of all this? You can't call yourself a med doctor if you have no idea about being one, if you don't know about the key knowledge areas.
Therefore you're not a doctor and don't get to have that title.
> Part of this is also making university, etc cheaper so you’re not financially incentivized to not take a class one, or multiple more times.
I'd rather people had better opportunities in life that didn't depend on education. There's a lot of people who hate school and are just going through the motions because they believe it's the only path to success in life.
It would prevent people from simply going and downloading solutions from the internet. A lot of easier assignments are fairly well treaded and already have solutions floating around.
Cheating on programming assignments has been rampant forever at every undergrad institution I have experience with and somewhat present among graduate students. My experience spans about 25 years in that space now.
When I taught an introductory class, I gave open-book exams with no laptops or phones allowed. About 1/3 of the class was unable to write a syntactically correct for loop in Python despite our textbook being an introductory Python-based book chock full of examples. It was pretty clear that a subset of students were either working together on project assignments or out-and-out having someone else do the assignments for them. I mainly compensated for this by having a large part of the grade being a 1-1 meeting with me to talk me through the code. That and the exam had the effect of actually making cheating somewhat less worthwhile. But this approach simply doesn't scale these days - my max class was around 32-33 students and the time I spent meeting with students was insane. I haven't taught in a few years and understand that class sizes of 150+ are not uncommon. I could have never used the same approach with that many students. I probably would have doubled-down on exams and made exam length darn near impossible to finish without actually knowing the material well enough to do without referring to the textbook.
I was teaching more from a practice-based viewpoint so mostly I came up with "weird" projects that mirrored problems I spend time on (data cleaning, using existing libraries to do neat little things, and also having students pick a personal project to implement that I helped them scope appropriately).
We bitch and moan about interview whiteboarding but given grade inflation its kind of hard to trust university credentials. Grade inflation was kind of disheartening in that the worst students didn't really get a poor grade. But I also didn't have any problems with the top 1/3 of the class getting a very high grade - these students were motivated, understood the material, and often impressed me with where they got in a single semester.
I recently worked at a large state university. I remember a conversation with an instructor in a master-level operating systems course. One of the assignments was to implement a simple filesystem. The amount of cheating was insane. He started academic dishonesty proceedings against many of the students but the department pressured him to "work it out."
He had to basically interview each student individually and ask for an explanation of the code. Most could not explain what their code was doing (because they didn't write it).
They were all given an opportunity to resubmit original work. Many could not do it, because they simply didn't understand the basics of programming, or the language they were using, or how computers even worked.
That is just awful - I have similar stories. To me, it seems so unfortunate that students are just in this desperate grind to get all A's.
But I'm on the other side and don't have the stress of "my whole life depends on looking like an all-star" at university. Like, what do you do if you leave school with such a severe lack of actual skill?
OTOH, the last year I taught, every student in the program I was an adjunct for graduated with a job offer in hand. Maybe it all works out fine.
The majority of CS Masters students in the US don't have a BS in CS, or have one from a non top tier overseas university. Even more so than undergrad, Masters students in CS are there for the credentials, so I would expect cheating to be worse.
In reality there's only a small percentage who are genuine creators. The rest are copycats also because the opportunity to copy exists.
The other side is real world pressure for good grades and when you're young you want to party and experience life. You finished that assignment by copy pasting it. It's better than not finishing it.
However you get that anywhere in life. Why is, sorry, was Stackoverflow so popular?
Why is religion still a thing?
People want easy answers to hard questions.
I teach data structures and algorithms among other subjects.
My assignments are similar to everyone else's (CLRS, JeffE, Leetcode) yet different enough that simply copy and pasting will not work.
All I ask is that they comment their code, analyze complexity and cite sources.
I've been using copilot for the last year in my live coding sessions. It saves so much time (especially in comments). If students want to use Copilot I do not
see a problem. It is just another tool.
Copilot is near useless without decent programming knowledge.
80% of time it is fantastic.
About 10% of time Copilot gives you subpar snippets. O(n2) instead of O(n) etc.
Then 10% of time it just gives you wrong snippets.
Just like GPT-3, once you go beyond a paragraph it loses context.
I think grading on multiple dimensions makes sense, though scaling is a challenge.
As for analysis write-up: turns out, language models can also explain what the code does really well, in convincing vernacular. They generate reasonable comments.
As an illustrative example, I just typed in a little test into Copilot. I only typed the first line and the text that starts with "a paragraph explaining...", and typed a "#" for each comment. Copilot filled in the rest.
def compute_sum_recursively(n):
"""A paragraph explaining what this function does and why, and explaining its running time:
This function computes the sum of the first n natural numbers.
The running time of this function is O(n).
"""
# Base case: n = 0
if n == 0:
# The sum of the first 0 natural numbers is 0
return 0
else:
# Recursive case: n > 0
# The sum of the first n natural numbers is n + the sum of the first n-1 natural numbers
return n + compute_sum_recursively(n-1)
yeah Copilot really changes nothing here, all these basic questions can be found with a quick Google search. The reason Copilot is so good at these types of questions is lazy professors assigning the same basic stuff so there is a ton of training data.
instead of writing an article crying about it, the professor could try making some unique questions to test knowledge of the underlying concepts
True story. I got my CS degree pre-internet. I did well with programming. Some of my friends / peers were less talented. In one of my 300 level courses the assignment was more difficult than usual. I naturally figued it out. They struggled.
Evetually I gave a copy of my code / solution to Friends Group A and also to another Group B. They didn't know each other. I said, "Be careful! If you copy, disguise it."* Deadline comes. Everyone hands in their work.
The following week, we go to lecture. Prof walks in and writes a list of names on the board. I knew each name. I knew what was happening. I waited for the shoe to drop (i.e., my name on the list). The shoe never dropped**
I believe they were all given Ds. Not sure why they didn't fail (F).
* In retrospect, this was a stupid on my part. If they knew how to alter enough it's likely they'd be able to write it themselves. That is, I all but suggested they walk on water.
** Also in retrospect, the TAs + prof had to realize I was the source. The class was big but not that big. And if only some completed the assignment correctly, the source had to be not difficult to identify. I'm not sure why I was never pulled aside and spoken to. Thank gawd.
If someone is just looking to fill a course requirement and has no interest in being a programmer, my response is "meh, whatever".
If someone is cheating and actually expects to land a decent CS job, my response is "good luck on the leetcode questions and the 5-7 hours of questions you'll have to answer between the intro call, panels, and an offer".
And if someone magically manages to get all that way without actually learning the material they will get absolutely crushed at work. Meetings where you have to give your professional opinion will induce fits of anxiety. Making changes to your code based on feedback and not having any idea what the feedback means. Even just the amount of work alone that needs to be done at most tech places crushes great engineers. Without the knowledge it will just stack up on you even further.
You underestimate the human ability to get away with incompetence. Unfortunately many of those cheaters manage to get in roles where they can hide their inadequacies, either by slacking of their colleagues that do actual work or by sucking up to the right managers.
Oh it definitely happens, I'm not saying it doesn't. But in those examples you mention I would say there are more people who are bad at their job than just the IC who "faked it".
> is the purpose of the course to measure the student or to teach the student?
The very basis of the scientific method is measurement. If you can not measure your learning progress, how are you to know if your training methods are adequate or failing?
I really don't understand this idea that testing and measuring students is a problem. Even the dreaded "teaching to the test", sounds like a good idea. If doing so would somehow exclude important learning, that just identifies an area where the testing needs to be improved.
> this is information that you're going to need if you actually want to go into the field and if you're using copilot here you are damaging yourself in the long run
When you say into the field, do you mean academia or industry?
Because the things copilot does for you are absolutely not the things you need to know yourself to be in industry. They're, for the most part, things we tried to put in libraries (or more ideally language standard libraries for a lot of things).
The fact that is solves a lot of interview questions just means our interview process was absolutely garbage.
And I'm skeptical of the academic side of this as well. It sounds like professors from the early 60s being annoyed that students have compilers (another tool that saves a ton of work and repetition). Y'all are forgetting that this is the ultimate lazy man's field. This isn't the first time the basics have been swept away and replaced with something easier to work with (and hopefully won't be the last).
> They're, for the most part, things we tried to put in libraries.
We need some people who can do things like making those libraries. It also seems plausible that the people who have at least some of the knowledge and judgement to do that effectively will, on average, be more effective on more mundane tasks as well.
Larry Wall is one who wrote (somewhat drolly) about the virtues of laziness, but there was nothing lazy about what he did.
> We need some people who can do things like making those libraries.
Do we? For things as simple as copilot tends to put out? Why?
Do you also believe we need to keep people around that do other automated things? Plowing fields by hand? Hand compiling higher level languages (as the first lisp compiler was bootstrapped)?
I mean, keep the information around. Don't go burning textbooks on subjects just because we automated something. But, what exactly is the value proposition of having students do these things?
> It also seems plausible that the people who have at least some of the knowledge and judgement to do that effectively will, on average, be more effective on more mundane tasks as well.
And your claim is also that this is the only way to get the requisite knowledge and judgement? Have students take in a string from stdin with a format that changes every semester, munge it around, and do things with it instead.
> Larry Wall is one who wrote (somewhat drolly) about the virtues of laziness, but there was nothing lazy about what he did.
As someone who has done string munging in C, I'm not entirely convinced that creating perl isn't an effort saving defense mechanism (only half joking).
I find this a very puzzling reply, and it may be that I misunderstood to what you are referring with the "they're" in "They're, for the most part, things we tried to put in libraries (or more ideally language standard libraries for a lot of things.)" It might refer to "the things Copilot does for you" or alternatively "the things you need to know yourself to be in industry."
The thing is, regardless of which way you meant it, we need some people who can make the sort of libraries we need in part precisely because automation such as Copilot is no substitute (at least not yet.)
This observation does not (and is not intended to) endorse current methods of instruction or hiring; on the contrary, it supports spcebar's view that riding your way to a degree, certificate or entry-level position on the back of Copilot is not doing yourself any favors.
The point about Larry Wall is that we don't get labor-saving tools without someone making an effort.
> I find this a very puzzling reply, and it may be that I misunderstood to what you are referring with the "they're" in "They're, for the most part, things we tried to put in libraries (or more ideally language standard libraries for a lot of things.)" It might refer to "the things Copilot does for you" or alternatively "the things you need to know yourself to be in industry."
I do see how that could be ambiguous. That's on me. I was referring to "the things Copilot does for you". Generally speaking trivial (or near trivial) algorithms.
> The thing is, regardless of which way you meant it, we need some people who can make the sort of libraries we need in part precisely because automation such as Copilot is no substitute
Fine. Some people might need to be able to implement and maintain libraries filled with generic algorithms, especially language maintainers. That's still a very different claim from the original "this is information that you're going to need if you actually want to go into the field". That claim implies that it's a universal requirement, whereas the reality is the vast majority probably don't need that knowledge.
Implementing these things is tedious, and results in a bunch of code that has to be maintained vs using something out of the standard library. For example, sort or max functions, which are at the intersection of what I see copilot generate a lot and intro to CS classes. Even without copilot, that's not really a skill that the average practitioner needs to have ready at all times, in fact I'd probably block a PR that implemented either of those things in my projects, it results in extra code that needs to be maintained and can be broken by a typo or something silly.
> The point about Larry Wall is that we don't get labor-saving tools without someone making an effort.
My comment about perl was sarcastic, and probably not helpful. That, also, is my bad.
To be clear, I take it that you are saying that a) everything Copilot is currently capable of can be found in libraries, and consequently b) learning how to do those things oneself is a waste of time, so c) it does not matter if people entering industry as software developers cannot do that themselves.
My point is that even if this is the case for a majority of such people, we still need the people who make all the library contents that are beyond Coplilot's capabilities, and we both seem to agree that its capabilities are limited.
The thing is, a world in which a lot of people can be productive software developers, without even being capable of writing the sort of algorithms Copilot is capable of, is highly dependent on the people who design and write the libraries that implement not only those algorithms, but also a great deal else that is beyond Copilot's capabilities. The industry may not need everyone to be able to do that, but then it is entirely dependent on those who can.
One can certainly argue that there is no point in an education that stops at the ability to reproduce what Copilot does, but that would be something of a straw man, as education does not typically stop there. I agree that it makes a poor hiring benchmark, regardless of the role being filled.
> To be clear, I take it that you are saying that...
I'm going to answer these individually because the answers are all different.
> a) everything Copilot is currently capable of can be found in libraries,
The vast majority, perhaps not 100%.
> b) learning how to do those things oneself is a waste of time
It's a waste of time if the pupil doesn't actually want to do it. If they do want to do it, there's probably quite a bit of value to be had. It's personal value for the student, not as a field.
Sort of like how I got quite a bit of value out of reading the old ITS documentation I found on github once, but I don't think I'd recommend it as part of the standard CS curriculum.
> c) it does not matter if people entering industry as software developers cannot do that themselves.
Correct. Assuming cannot means "without googling".
> My point is that even if this is the case for a majority of such people, we still need the people who make all the library contents that are beyond Coplilot's capabilities, and we both seem to agree that its capabilities are limited.
Sure, but that's a relatively small number of people. My point is that if using these tasks to measure aptitude is causing you problems because copilot exists, it's perfectly fine to just use something else.
> The thing is, a world in which a lot of people can be productive software developers, without even being capable of writing the sort of algorithms Copilot is capable of, is highly dependent on the people who design and write the libraries that implement not only those algorithms, but also a great deal else that is beyond Copilot's capabilities. The industry may not need everyone to be able to do that, but then it is entirely dependent on those who can.
Fair, but that's already the situation we're in with programming languages. We're entirely dependent on those as a field, and the vast majority of practitioners wouldn't be able to create a compiler or be anywhere near competent in language design. I would put the large frameworks (e.g. spring) in the same category.
These trivial algorithms become like the opcodes for a particular processor. Someone has to know them, but basically everybody not working on a compiler can ignore it (unless it particularly tickles your fancy).
I do have one small issue with your wording, however. Specifically the use of the word "capable" in this bit:
> without even being capable of writing the sort of algorithms Copilot is capable of
The people going into the field definitely need to be capable of implementing these sorts of algorithms. You're going to fail at so much of software if you aren't capable of something that trivial. It just doesn't need to be taught. These algorithms can be looked up if they're ever required. The ones that come up frequently will be naturally memorized and the others won't. This is a field where you have to learn new things, often without any sort of available expert in the subject, likely this will happen with an entire language or how to use a particular library.
I'm comfortable with copilot the same way I am comfortable with a calculator. Sure, if you have to, you should be able to do a logarithm with a slide rule, but I'm ok with not teaching university students how to use a slide rule. And if a professor assigned a bunch of homework under the assumption that students were going to use a sliderule, and the students all used calculators, I'd tell them to just drop that particular lesson.
We have reached a point of mutual understanding and considerable agreement, but I feel we have circled around to the original problem.
In your last paragraph, you consider the case of calculator vs. slide rule, but one can, I think, make a more apposite case with calculator vs. learning arithmetic (and even if you don't agree that it is more apposite, the point can still be made!) By your logic, we should not be assigning any arithmetic problems that can be solved with a calculator.
But why stop there? Even current calculators can do much more than basic arithmetic, and their capabilities pale in comparison to those of general-purpose computers. I do not believe you can teach mathematics in a way that begins at a point beyond what has been automated, and before long that may be true for computing as well. This has not led to the demise of mathematics education, and neither will it for computing, for the same reason in both cases: anyone who needs to use these tools to get through the introductory classes will wash out later - which brings us right back to spcebar's original comment about only hurting themselves in the long run.
> I think, make a more apposite case with calculator vs. learning arithmetic (and even if you don't agree that it is more apposite, the point can still be made!) By your logic, we should not be assigning any arithmetic problems that can be solved with a calculator.
I think that's an excellent point, and one whose nuances shouldn't be un-examined.
I guess the difference to me is that arithmetic feels more fundamental. I may be using fundamental in a weird way here, but I don't mean it in the sense of something that you build on. I mean it in the sense that arithmetic itself is the idea that's trying to be taught. Whereas a sorting algorithm has both a explanation as a semi-platonic ideal and as an implementation and they're very different things. You can fairly effectively wield the implementation only using the idea in your head, mostly regardless of your understanding of the implementation. In fact, it's generally considered good practice to hide an implementation from your eventual users.
I'm not so convinced that you could do the same with arithmetic, although I'm open to arguments to the contrary.
Now that I'm thinking about it, I think this is the dividing line between math and computing in my head. The ability to separate out the real world part from the ideal and operate the former using only the latter.
Perhaps I'm mistaken, but I would be very surprised if a mathematician could operate without understanding arithmetic, but I've seen quite competent programmers construct APIs just fine without having been taught sorting algorithms.
“Coping with google… students can just search for the answers by typing in ‘depth first search code python’. Buckle up and hunker down for one slippery slope of an article!”
The good students will take the time to learn and understand (even if they didn't write the code). Other students, as they always have, will look for short cuts.
Not everyone who take CS1* courses are CS (or similar eng) majors. We keep crying "Everyone should learn to code" and now that's made easier than ever some of continue to complain. Huh?
Full disclosure: I started to use CoPilot a couple of months ago. It does not remove me from the process. It does allow me to shift more brain cycles from the mundane to the more difficult. My general feeling is, I have yet to get close to taking full advantage of what it can offer me.
I love the choice of an AI-generated photo for the header, and the hidden message of its poor quality if you look at it longer than a glance.
There's an emerging divide between those with experience with AI tools and people who tried them a few times and got hyped and blogged a maximalist view of consequences.
The solution doesn't seem that hard to me. Put less weight on homework grades and more weight on projects and tests. If they cheat the homework they'll bomb on tasks that can't be solved with Copilot.
Yes, I mention this in the original article ("Well, how about we just weigh grades on exams more, and have students take their tests either using pen and paper or locked-down computers?").
These examples of producing known algorithms in response to function signatures make it seem to me like Copilot is an intelligent, vendoring "package manager", not something that solves problems by itself.
The perceived productivity boost that people get from it could be interpreted as a condemnation of how verbose a lot of coding still is even in known areas. Lots of brain cycles wasted on doing things suboptimally that we can already do optimally.
The difference is that package management hides packages somewhere deep and you're not supposed to care about code quality. If it popular and works, good enough. Copilot puts terrible code snippets right before your eyes and you'll spend more time making it better and fixing bugs rather than writing that code from scratch. I tried copilot and for me it's not ready yet. I guess my issue is that I don't like average code and copilot was trained on that code, so it's as good as average coder and that bar is not high. Or I'm too picky and high-minded, doesn't matter, the result is the same.
Extra nitpick regarding generated code: the `quicksort_random_pivot` implementation isn't what people commonly mean by "quicksort", as in: it's not in-place. This means that some properties like space complexity are also going to be different than what's usually expected of quicksort.
My proudest moment was in an embedded systems course, the teacher asked for a function to generate fibonacci linearly in assembly, there was a reaalllllly long pause, and I barked the answer across the classroom. I vividly remember some stunned, disgusted glances.
I've always thought that the ability to communicate to other humans about programs and to be able to synthesize a conceptual understanding from looking at code are highly valuable and sometimes-overlooked in the workplace. Assignments to write a function don't really demonstrate either skill.
Maybe more assignments like:
- "Here is a function that is supposed to generate a fibonacci sequence, but it is not correct, explain why"
- Here is a sort function. What is the name of the algorithm implemented by this sort function?
The reality of any career in CS is going to involve a lot of maintenance/understanding of existing code any systems. Copilot's failure to replace all human programmers isn't because of it's inability to cough up complex code, it's because it can't communicate with the product owner, debug a program amidst a time-sensitive incident, or explain its work.
Agreed. I just tried this by writing three sort functions named "some_sort", "another_sort", and "yet_another_sort". I started comments after each with "# The algorithm used in some_sort is known as: " and Copilot correctly completed them in all three cases (bubble, insertion, and merge).
In computer class back in 9th grade there were one or two exams about Excel, where we were to use excel to compute some of the answers for questions. The fact that "memorization questions" (if that's the right English term?) about several functions and concepts were on the same exam sheet and that the Excel help function was right there was something that seemingly slipped the teacher's mind.
Although he was an older physics teacher who only really did computer class ("Informatik") because there was nobody else available or (more) qualified to do the job.
When later on there were classes on programming, he made us use TurboPascal instead of something less cluttered and actually useful in modern economy like Python.
That same teacher later on performed some show experiments about induction and electromagnetic forces in physics class where he let a magnet fall through a glass tube with several copper coils around it. So far so good, but he then tried to readjust one of the coils one the glass tube, which predictably shattered in his hands, nearly severing some fingers and actually severing some tendons, which IIRC were later replaced with other tendons from his legs or something. Needless to say, not the sharpest tool in the shed.
My other physics teacher during 10th-12th grade physics classes was a severely esoteric nutjob, preferring to hold long rants about "science" being able to diagnose diseases in people by somehow shining a laser at a drop of their blood and interpreting the reflection/refraction because there was a "connection" between the person and their blood (or something like that, I honestly tuned out his ravings after a while). Apparently his wife is/was some kind of homeopathic "healer".
All that is to say, computer and physics class teachers for me were "very fun" and "useful", since one did not have a clue that their students were "cheating" and both were not very in tune with modern and/or scientific methods.
Guess how I learned query/replace in Emacs in 1985?
My friend who copied my Pascal programming assignment is now a director at a $1B hedge fund. I'm the founder of a series of scrappy, ramen noodle startups. I guess it's the "Senator Blutarsky" effect:
I love their take at the end which is to not expand the already crazy anti-cheating apparatus and just give students more interesting problems and let them use all the information at their disposal. I wish more professors took this attitude. Having every student implement the exact same famous algorithms is really silly when you take a step back. You don’t gain any appreciation for them because they’re unmotivated and you lack all the foundational
knowledge. The proofs/explanations are all totally bespoke and don’t generalize.
You really win as a professor when you design a project that naturally causes you to reach for the knowledge you’re trying to teach. One of the best was my networking class where the project had us secretly rediscover tcp by trying to invent reliable transmission over UDP and then once we sufficiently cut our teeth the second half of the course was implementing a real working version on top of the professor provided userspace ip stack.
This is no different from finding the solutions on wikipedia, or any of a thousand other websites. People that want to cheat will do so, and they might even be correct in a completely selfish sense.
Could you just use another teaching language? Does copilot work as well with e.g. Racket, Raku or Pascal, which all seem like decent CS101 languages to me?
> Here’s an approach that’ll work for sure: use some, let’s call them alternative, programming languages that Copilot doesn’t really know. (...) Sadly, I have news: Copilot’s love for programming languages knows no bounds! Racket! Haskell! ML! (...) Copilot is a ravenous beast: if any code in any language found its way into a GitHub repo, it’s already swallowed it up and is hungry for more, nom nom nom.
Not sure how true this is in practice - I only used Copilot once, with Python - but if it is that might invalidate this concept
1. The latest and greatest Codex is now twice as good on its own benchmark suite than the original version published a year ago.
2. It's just as good on Python as it is on JS, Scala, C++, Swift, TypeScript... and other languages are not too far behind. It's not bad at bash of all things.
462 comments
[ 3.0 ms ] story [ 329 ms ] threadIn the same vein, just have them write code out on paper during exams and coding competitions like the way we did it back in my day. It's not as if they're not going to need that skill anyway when they get whiteboarded during interview loops when even the most lenient employers get exasperated at "CoPilot-only" graduates.
</tongue-in-cheek>
You can test loop-writing in an oral-exam, too. Thus your proposed solution for GPT-3 writing ethics essays is so good, that it fixes the original problem.
Sorry, what's the point of that? I don't think it would make any difference for cheating?
The blue book is a standard. https://www.amazon.com/BookFactory-Exam-Blue-Ruled-Format/dp...
https://www.thoughtco.com/blue-book-1856928
In the classes that I had that used blue books for exams, the exam book was free and given out by the proctor.
To be honest there should be a class called language model querying because for simple tasks it does well but eventually you will have to debug the output (not just copilot but GPT3 and dall-e).
There's an AI for that!
Do you think the skills taught in that class would be relevant for very long?
That's why getting an Internship is always a good idea.
I don't think this is generally true, especially for algorithms and smaller programs: Googling 'fibonacci python' takes 3 seconds.
Seems roughly the same amount of effort, and since Copilot is not free anymore most students won't bother paying for access on any kind of scale that matters. Whoever was going to cheat will cheat regardless.
Besides it's one of those "you won't have a calculator with you everyday" fallacies. If you can solve problems with Copilot in class you can also do it at your job later on.
Copilot is free for students: https://docs.github.com/en/billing/managing-billing-for-gith...
Most book algorithms are in pseudocode so you need to know how to write a loop or a function in the language you are using.
And just the act of typing it line-by-line will focus attention and help cement concepts in the brain (assuming one really wants to learn).
Hitting TAB in Copilot doesn't do any of those things.
3 years later, the education ministry decided that was BS and let everyone bring calculators to math exams. Surprisingly, math scores didn't go up much.
A calculator actually helped me understand physics a lot better, especially things like slopes.
The most effective policy might not be to try and mechanically prevent cheating, but to explain that, this is information that you're going to need if you actually want to go into the field and if you're using copilot here you are damaging yourself in the long run.
Maybe it's not fair to give as good grades to students who are cheating on assignments as those who are actually doing the assignments, but at the end of the day is the purpose of the course to measure the student or to teach the student?
That has never, ever stopped anyone motivated to cheat. Even here on highly-educated HN, you get posters with open disdain for what they learned in college.
> "... at the end of the day is the purpose of the course to measure the student or to teach the student?"
Both. Otherwise, the college diploma really does become the meaningless scrap of sheepskin that its detractors claim it is.
Because of this, there will always be considerable pressure to detect cheating and remove offending parties from the program, be that the students copying answers from the internet, or the institutions failing to detect the problem before handing out a degree.
I feel we'd probably get the same value to society if we just made grades 9-12 optional. It would remove the people that don't want to be there, improve the quality of instruction, and college would actually be a meaningful achievement so we no longer have requirements for entry level jobs to require a Master's degree.
I don't teach CS, but I teach an online course in a STEM field. I give out a similar message ("you're only hurting yourself in the long run").
My anecdotal evidence is that my warning does nothing--I have trap questions on various quizzes/exams and the frequency of cheating hasn't changed with or without the warning.
My suspicion is that the students who cheat feel this way: "This is just some bullshit hoop that I have to jump through, so it's okay if I cheat. I'll figure out the important stuff when the time comes."
I mean, they're delusional, but I understand the mindset for cheating.
edit: amusing story.
Faculty members in my department are required to take an on-line course on how to handle hazardous waste. There are a few hours worth of videos to watch and a test that you need to pass at the end.
Last year, one of the faculty members took the test, compiled the answers and emailed them around to everyone else (to save them the time for this "bullshit task.").
I said to him, "Isn't this precisely the kind of shit that makes every faculty member angry when the students do it?"
That being said, it was a bullshit test.
Parental (other similar) pressure can indeed be quite strong. Absent of that pressure they wouldn't be there in the first place, so perhaps it is not so much being delusional as a rational response to their environment? If they cheat their way through the parents are happy and then they can return to the life they otherwise would have lived. The time lost is unfortunate in some respects, but at the same time if you're cheating, the time investment likely isn't that great and an acceptable cost given the circumstances.
> the kind of shit that makes every faculty member angry
I find it curious that the customer using the product in an unintended way would be upsetting to the vendor. What drives such emotions? In my businesses, I couldn't care less how the customer uses my product. If they're happy, I'm happy.
It is possible for the customer to also be the product. Especially in the age of salable data, that is becoming more and more common. However, when that is the case there is incentives given to the customers to shape them into what selling them as a product requires. In context here, that would mean something like giving discounts to those who don't cheat, which I am not familiar with any college doing, so... That brings us back to why would a vendor get emotional about the customer not using the product as intended?
Because your customer, in so doing, is stealing from your other customers and undermining your reputation as a vendor.
Plus, you're disheartened because this interaction is a waste of time, regardless of the fact you get paid: if money was all you cared about, you wouldn't be in education.
That's always going to be the case, though. Imagine you built software that helped businesses find customers. Using it the intended way they can realize a small number of new customer each day distributed across your customer base. Now, someone finds a new way to use your software in a manner that you never envisioned and they're attracting all the customers, taking from those who used the software as expected.
But, really, who cares? You just pivot to the embrace the new way and carry on with life. That's your market now. It's fun and all to want to be the elevator operator of old, but at some point you have to realize that nobody cares about your nostalgia. Markets change.
> Plus, you're disheartened because this interaction is a waste of time
But it is not. The students are only there to abade social pressures and cheating their way through gives them what they want out of the deal. Very few students care about the academics.
> if money was all you cared about, you wouldn't be in education.
If you don't care about money, why are you marketing the social need so hard? This is like a car manufacturer marketing that their cars are great gateway vehicles when committing crimes and then lamenting that criminals are using their vehicles to get away...
Before the ridiculous, albeit successful, "If you don't go to college you will end up flipping burgers" marketing campaign, college only attracted a small number of students who were serious about learning and all was well with the world. Colleges still spend an inordinate amount of time justifying why the cost is worthwhile to keep up the social need illusion.
It's not hard to revert to the natural state. Fact of the matter is that you (not you personally, perhaps, but the group) don't want to.
And yes it's a waste of time! If you want a fake credential then it's much cheaper and easier to just lie on your CV. It's not like many employers will check your certificates, at least not in software. I'd really rather people did that than waste my time first trying to teach them and later handling their academic misconduct cases.
If I'm marketing social need, this is the first time I've heard of it ;-)
It is, but an acceptable cost to keep the social pressures at bay. There are a lot of things in life that are ultimately wastes of time, but worthwhile to maintain a civil society. Such is life. If you are happy to escape to the deep woods away from all others where every moment of your time is purposeful, good on you.
> If you want a fake credential then it's much cheaper and easier to just lie on your CV.
What would that accomplish? The business world couldn't care less about your past achievements. They don't care that you performed in a play, rode a horse, swam in the ocean, or went to college. There is no value in even putting a legitimate degree accomplishment on your CV, let alone faking it. If anything, it detracts from your standing as it shows that you're so useless you couldn't come up with anything more relevant to share about yourself.
Instead, it is parental pressure that puts one into these schools when they otherwise shouldn't be. And those parents will turn up at your graduation. Once you've thrown your cap in the air, so to speak, nobody will ever think about it ever again. Getting to that one moment in time is the so called hoops that one is cheating for.
There's a lot more going on here than maybe you realize.
First, the students are not your only customers. The administration is also one of your customers. If you make the course TOO difficult and fail everyone, your "administration customer" will find a way to get rid of you.
Conversely, if you make the course too easy and everyone gets to cheat and everyone gets an "A", then many students who pass your class may not pass various professional certification exams that are more rigorously controlled (i.e., no cheating). If that happens, then the school may lose accreditation for programs that you are associated with (i.e., if all the nursing majors fail their certification exam, your nursing program may get nuked). Then you get nuked.
Third, whether you like it or not, your relationships with your students is something you have to deal with. Students who don't cheat have a tendency to get really upset if they find out you turn a blind eye to cheating. Happy cheating customers leads to unhappy honest customers.
Fourth, if your fellow faculty members feel like you're going easy on the students to get favorable reviews (or because you're lazy or because you hate confrontation), they can make life unpleasant for you, too. They also aren't too happy if they teach upper level courses and you send a bunch of garbage students their way.
If not, you might want to try a new business. You may not be cut out for the one you're in. The business world is not very forgiving, nor should it be. Those who can't adapt need to perish.
Spoken like someone who hasn't done the job and is confident in his ignorance.
Curious that a primitive emotional response has transpired instead. Given the greater context of discussion also about emotions, is there something about the job that attracts this type of behaviour?
The only remaining question is how to engage these people. I have seen good approaches to that problem and really, really terrible approaches, and it seems that most professors go for the terrible approaches. Unfortunately, engaging people who don't want to be there is a lot harder than just having a bunch of required tests.
These students are not all wrong that some courses are BS hoops to jump through. Meet them in the middle. Teach them something.
In general, they appreciate my candor.
That being said, in my experience, the students who cheat often delude themselves into thinking that they understand the material--and they really don't.*
Then they seem shocked when they take a certification exam (where they have to leave phones at the door and are closely monitored) and they bomb the exam.*
* I'm speaking in generalities here. I'm sure there are some cheaters who are being "smart" about their cheating.
Eventually, some of us came to the realization that you can never prove that somebody is cheating. People have been known to hire tutors to do their assignments for them. There is just no end.
As a result, we evaluate students on four dimensions. Programming assignments,* homework assignments, but also class discussions and group participation. Those last two count for a small, but non-trivial percentage of the grade, and are usually enough to separate and identify those who understand what they are doing, from those who are just "following along" solutions that they find on the internet.
* one addition to programming assignments includes an analysis write up. Tell me in human words what is happening. Why do you see that effect. What is the running time. And separately, comment your code to tell me how it works. Those two parts count significantly towards the grade.
Any other professors here, what have you found that works?
when i was interviewing candidates this was the fastest way to filter those who understood and those who didnt.
ask someone to talk you through how to solve some real-life problem. to look at a stack trace and describe what they see, or to run a profile and trace through the cause of a hotspot/contention. show them an issue and have them live debug to root-cause and fix. it's okay if they use the internet, SO, etc. -- that's how we all do it. see what they have to look up! just listening to the amount of depth someone can verbally communicate during novel problem solving (including additional questions they ask you) turns a 60min interview that wastes time into a 10min one that tells you if you can move on.
a favorite one of mine was to ask the candidate to describe in as much detail as possible what happens between a keystroke typed into a google search box and the search results appearing. the diversity of replies is facinating. some will tell you "google returns results from its database", others will ask you if you want them to first describe how the keyboard works at an electrical level through the USB or bluetooth driver stack.
"Never memorize something that you can look up." --someone smart
Like they picked some random port configuration issue that had stumped them for days in the past and thought cool let’s remove the internet, the command line, and ask people to solve it on the spot.
I offered plenty of plans for how I would go about debugging, but I didn’t know the one simple trick.
I agree with trying to use challenges that are closer to the real work, but it’s really hard to do that without over-testing domain knowledge.
For instance, I ask candidates to do some asynchronous control flow. These are all candidates with js listed as their best language and I offer to let them look things up or to show them the promise apis they need, but a certain percent just refuse to engage with the problem because they don’t have the domain knowledge and seem to feel they’re being tested unfairly.
The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks.
i dont mean that the problem is only verbally communicated. i mean that the debugging process the candidate does is verbally communicated.
in the scenario you're describing you'd be sitting at that machine with access to the internet on another machine. like, you know, in real life.
> The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks.
async stuff in js is pretty good, but it's also easy to go too deep on it with some bizzarely poorly architected code. many js devs still fail to grasp all the implications of closures, or how to avoid memory leaks, or how to work with the GC rather than against it.
It's a shame our mass education systems cannot use it. They simply cannot apply such a humane method to hundreds or thousands of students. They are reduced to applying their bullshit test questions because it's the best they can come up with at their scale.
I really, really hated those parts of school, second only to group projects. I do appreciate why teachers want that, and I hear you about how it can surface those who are cheating, but I do wonder if it's causing the class to suffer for the alleged misdeeds of a few
No person an island.
And if they are, I don't want to be on a work team with them.
For me at least, the dynamics of "School Group Project" are basically completely different and separate from "Work team". I happen to do well in both, but I do not enjoy both: the motivations and structure and dynamics and goals and timelines of "School Group Project" are so much more artificial and ultimately pointless, and obviously to all from the start.
Same things with discussions - I happen to be an engaged student, usually front row, hand always up, discussing with instructor and team and colleagues and everybody. I like to be engaged and figure things out together. But I'll never be half the developer than my colleague who barely speaks a word unless asked. He's friendly, meek, polite, and excellent team player and developer - he just does not initiate conversations, especially in group settings. I can imagine he'd have a nice big 0 in that category if that was a grade criteria.
Which is not to criticize the professors who try to use group discussions; just to point out, this is not a solved problem at scale any more than interviewing/hiring. There's too much humanity and too few absolutes :)
In school it never bothered me having freeloaders get the same grade for my work, so long as they stay out of my way and don't "help." There's a very famous mechanic's poster: "repair: $5, if you watch: $20, if you help: $1000"
https://www.youtube.com/watch?v=YCutBaSt6tM
https://www.youtube.com/watch?v=DmaKc0I1VAo
https://www.youtube.com/watch?v=N14T_4gUH-c
Honestly I think moving away from grades is going to be the way forward. Part of this is also making university, etc cheaper so you’re not financially incentivized to not take a class one, or multiple more times.
A lot of people hate how tech interviews are done, right? Cause they have nothing to do with the job the majority of the time. Those are places where I can see improvements.
However, some jobs do genuinely need you to interact with a diverse set of people all the time, for those you need to make sure for the sake of your company & the candidate that they match the criteria. Other jobs, you need only maintain strong relations with small groups, and the requirements for those are a lot easier to hit - even the most introverted people can do well in smaller groups that don't change a lot.
I do really think a lot of evaluations that happen during education though makes a confrontational relationship with learning. You're constantly judged about your _ability to learn_and that has far reaching impacts beyond your first twoish decades of your life. You get scared of making mistakes, you get scared of taking risks, etc.
So, evaluate when it's important for the function & safety of the job, and the candidate. Skip evaluation when its evaluation for the sake of evaluation.
There's real life consequences to this kind of evaluation. Huge consequences. Students cannot afford to make mistakes. There's huge pressure and anxiety before and during a test because the stakes are so high. Failing at this stuff can cost someone their future: future jobs, future career, even the student loans that enable them to study in the first place may only be afforded to students who get good grades.
Is it any wonder people cheat?
Isn't that the whole point of all this? You can't call yourself a med doctor if you have no idea about being one, if you don't know about the key knowledge areas.
Therefore you're not a doctor and don't get to have that title.
It's about filtering out those who can't
I'd rather people had better opportunities in life that didn't depend on education. There's a lot of people who hate school and are just going through the motions because they believe it's the only path to success in life.
How is that going to stop cheating? Surely that will just make people more inclined to cheat, since they have less chance of solving it on their own.
Cheating on programming assignments has been rampant forever at every undergrad institution I have experience with and somewhat present among graduate students. My experience spans about 25 years in that space now.
When I taught an introductory class, I gave open-book exams with no laptops or phones allowed. About 1/3 of the class was unable to write a syntactically correct for loop in Python despite our textbook being an introductory Python-based book chock full of examples. It was pretty clear that a subset of students were either working together on project assignments or out-and-out having someone else do the assignments for them. I mainly compensated for this by having a large part of the grade being a 1-1 meeting with me to talk me through the code. That and the exam had the effect of actually making cheating somewhat less worthwhile. But this approach simply doesn't scale these days - my max class was around 32-33 students and the time I spent meeting with students was insane. I haven't taught in a few years and understand that class sizes of 150+ are not uncommon. I could have never used the same approach with that many students. I probably would have doubled-down on exams and made exam length darn near impossible to finish without actually knowing the material well enough to do without referring to the textbook.
I was teaching more from a practice-based viewpoint so mostly I came up with "weird" projects that mirrored problems I spend time on (data cleaning, using existing libraries to do neat little things, and also having students pick a personal project to implement that I helped them scope appropriately).
We bitch and moan about interview whiteboarding but given grade inflation its kind of hard to trust university credentials. Grade inflation was kind of disheartening in that the worst students didn't really get a poor grade. But I also didn't have any problems with the top 1/3 of the class getting a very high grade - these students were motivated, understood the material, and often impressed me with where they got in a single semester.
He had to basically interview each student individually and ask for an explanation of the code. Most could not explain what their code was doing (because they didn't write it).
They were all given an opportunity to resubmit original work. Many could not do it, because they simply didn't understand the basics of programming, or the language they were using, or how computers even worked.
These were Masters students.
But I'm on the other side and don't have the stress of "my whole life depends on looking like an all-star" at university. Like, what do you do if you leave school with such a severe lack of actual skill?
OTOH, the last year I taught, every student in the program I was an adjunct for graduated with a job offer in hand. Maybe it all works out fine.
The other side is real world pressure for good grades and when you're young you want to party and experience life. You finished that assignment by copy pasting it. It's better than not finishing it. However you get that anywhere in life. Why is, sorry, was Stackoverflow so popular? Why is religion still a thing? People want easy answers to hard questions.
My assignments are similar to everyone else's (CLRS, JeffE, Leetcode) yet different enough that simply copy and pasting will not work.
All I ask is that they comment their code, analyze complexity and cite sources.
I've been using copilot for the last year in my live coding sessions. It saves so much time (especially in comments). If students want to use Copilot I do not see a problem. It is just another tool.
Copilot is near useless without decent programming knowledge. 80% of time it is fantastic. About 10% of time Copilot gives you subpar snippets. O(n2) instead of O(n) etc. Then 10% of time it just gives you wrong snippets.
Just like GPT-3, once you go beyond a paragraph it loses context.
As for analysis write-up: turns out, language models can also explain what the code does really well, in convincing vernacular. They generate reasonable comments.
https://twitter.com/emeryberger/status/1560618825208320001?s...
As an illustrative example, I just typed in a little test into Copilot. I only typed the first line and the text that starts with "a paragraph explaining...", and typed a "#" for each comment. Copilot filled in the rest.
With grading like that cheating on the homework is basically pointless, so few did it.
instead of writing an article crying about it, the professor could try making some unique questions to test knowledge of the underlying concepts
Evetually I gave a copy of my code / solution to Friends Group A and also to another Group B. They didn't know each other. I said, "Be careful! If you copy, disguise it."* Deadline comes. Everyone hands in their work.
The following week, we go to lecture. Prof walks in and writes a list of names on the board. I knew each name. I knew what was happening. I waited for the shoe to drop (i.e., my name on the list). The shoe never dropped**
I believe they were all given Ds. Not sure why they didn't fail (F).
* In retrospect, this was a stupid on my part. If they knew how to alter enough it's likely they'd be able to write it themselves. That is, I all but suggested they walk on water.
** Also in retrospect, the TAs + prof had to realize I was the source. The class was big but not that big. And if only some completed the assignment correctly, the source had to be not difficult to identify. I'm not sure why I was never pulled aside and spoken to. Thank gawd.
If someone is cheating and actually expects to land a decent CS job, my response is "good luck on the leetcode questions and the 5-7 hours of questions you'll have to answer between the intro call, panels, and an offer".
And if someone magically manages to get all that way without actually learning the material they will get absolutely crushed at work. Meetings where you have to give your professional opinion will induce fits of anxiety. Making changes to your code based on feedback and not having any idea what the feedback means. Even just the amount of work alone that needs to be done at most tech places crushes great engineers. Without the knowledge it will just stack up on you even further.
The very basis of the scientific method is measurement. If you can not measure your learning progress, how are you to know if your training methods are adequate or failing?
I really don't understand this idea that testing and measuring students is a problem. Even the dreaded "teaching to the test", sounds like a good idea. If doing so would somehow exclude important learning, that just identifies an area where the testing needs to be improved.
When you say into the field, do you mean academia or industry?
Because the things copilot does for you are absolutely not the things you need to know yourself to be in industry. They're, for the most part, things we tried to put in libraries (or more ideally language standard libraries for a lot of things).
The fact that is solves a lot of interview questions just means our interview process was absolutely garbage.
And I'm skeptical of the academic side of this as well. It sounds like professors from the early 60s being annoyed that students have compilers (another tool that saves a ton of work and repetition). Y'all are forgetting that this is the ultimate lazy man's field. This isn't the first time the basics have been swept away and replaced with something easier to work with (and hopefully won't be the last).
We need some people who can do things like making those libraries. It also seems plausible that the people who have at least some of the knowledge and judgement to do that effectively will, on average, be more effective on more mundane tasks as well.
Larry Wall is one who wrote (somewhat drolly) about the virtues of laziness, but there was nothing lazy about what he did.
Do we? For things as simple as copilot tends to put out? Why?
Do you also believe we need to keep people around that do other automated things? Plowing fields by hand? Hand compiling higher level languages (as the first lisp compiler was bootstrapped)?
I mean, keep the information around. Don't go burning textbooks on subjects just because we automated something. But, what exactly is the value proposition of having students do these things?
> It also seems plausible that the people who have at least some of the knowledge and judgement to do that effectively will, on average, be more effective on more mundane tasks as well.
And your claim is also that this is the only way to get the requisite knowledge and judgement? Have students take in a string from stdin with a format that changes every semester, munge it around, and do things with it instead.
> Larry Wall is one who wrote (somewhat drolly) about the virtues of laziness, but there was nothing lazy about what he did.
As someone who has done string munging in C, I'm not entirely convinced that creating perl isn't an effort saving defense mechanism (only half joking).
The thing is, regardless of which way you meant it, we need some people who can make the sort of libraries we need in part precisely because automation such as Copilot is no substitute (at least not yet.)
This observation does not (and is not intended to) endorse current methods of instruction or hiring; on the contrary, it supports spcebar's view that riding your way to a degree, certificate or entry-level position on the back of Copilot is not doing yourself any favors.
The point about Larry Wall is that we don't get labor-saving tools without someone making an effort.
I do see how that could be ambiguous. That's on me. I was referring to "the things Copilot does for you". Generally speaking trivial (or near trivial) algorithms.
> The thing is, regardless of which way you meant it, we need some people who can make the sort of libraries we need in part precisely because automation such as Copilot is no substitute
Fine. Some people might need to be able to implement and maintain libraries filled with generic algorithms, especially language maintainers. That's still a very different claim from the original "this is information that you're going to need if you actually want to go into the field". That claim implies that it's a universal requirement, whereas the reality is the vast majority probably don't need that knowledge.
Implementing these things is tedious, and results in a bunch of code that has to be maintained vs using something out of the standard library. For example, sort or max functions, which are at the intersection of what I see copilot generate a lot and intro to CS classes. Even without copilot, that's not really a skill that the average practitioner needs to have ready at all times, in fact I'd probably block a PR that implemented either of those things in my projects, it results in extra code that needs to be maintained and can be broken by a typo or something silly.
> The point about Larry Wall is that we don't get labor-saving tools without someone making an effort.
My comment about perl was sarcastic, and probably not helpful. That, also, is my bad.
My point is that even if this is the case for a majority of such people, we still need the people who make all the library contents that are beyond Coplilot's capabilities, and we both seem to agree that its capabilities are limited.
The thing is, a world in which a lot of people can be productive software developers, without even being capable of writing the sort of algorithms Copilot is capable of, is highly dependent on the people who design and write the libraries that implement not only those algorithms, but also a great deal else that is beyond Copilot's capabilities. The industry may not need everyone to be able to do that, but then it is entirely dependent on those who can.
One can certainly argue that there is no point in an education that stops at the ability to reproduce what Copilot does, but that would be something of a straw man, as education does not typically stop there. I agree that it makes a poor hiring benchmark, regardless of the role being filled.
I'm going to answer these individually because the answers are all different.
> a) everything Copilot is currently capable of can be found in libraries,
The vast majority, perhaps not 100%.
> b) learning how to do those things oneself is a waste of time
It's a waste of time if the pupil doesn't actually want to do it. If they do want to do it, there's probably quite a bit of value to be had. It's personal value for the student, not as a field.
Sort of like how I got quite a bit of value out of reading the old ITS documentation I found on github once, but I don't think I'd recommend it as part of the standard CS curriculum.
> c) it does not matter if people entering industry as software developers cannot do that themselves.
Correct. Assuming cannot means "without googling".
> My point is that even if this is the case for a majority of such people, we still need the people who make all the library contents that are beyond Coplilot's capabilities, and we both seem to agree that its capabilities are limited.
Sure, but that's a relatively small number of people. My point is that if using these tasks to measure aptitude is causing you problems because copilot exists, it's perfectly fine to just use something else.
> The thing is, a world in which a lot of people can be productive software developers, without even being capable of writing the sort of algorithms Copilot is capable of, is highly dependent on the people who design and write the libraries that implement not only those algorithms, but also a great deal else that is beyond Copilot's capabilities. The industry may not need everyone to be able to do that, but then it is entirely dependent on those who can.
Fair, but that's already the situation we're in with programming languages. We're entirely dependent on those as a field, and the vast majority of practitioners wouldn't be able to create a compiler or be anywhere near competent in language design. I would put the large frameworks (e.g. spring) in the same category.
These trivial algorithms become like the opcodes for a particular processor. Someone has to know them, but basically everybody not working on a compiler can ignore it (unless it particularly tickles your fancy).
I do have one small issue with your wording, however. Specifically the use of the word "capable" in this bit:
> without even being capable of writing the sort of algorithms Copilot is capable of
The people going into the field definitely need to be capable of implementing these sorts of algorithms. You're going to fail at so much of software if you aren't capable of something that trivial. It just doesn't need to be taught. These algorithms can be looked up if they're ever required. The ones that come up frequently will be naturally memorized and the others won't. This is a field where you have to learn new things, often without any sort of available expert in the subject, likely this will happen with an entire language or how to use a particular library.
I'm comfortable with copilot the same way I am comfortable with a calculator. Sure, if you have to, you should be able to do a logarithm with a slide rule, but I'm ok with not teaching university students how to use a slide rule. And if a professor assigned a bunch of homework under the assumption that students were going to use a sliderule, and the students all used calculators, I'd tell them to just drop that particular lesson.
In your last paragraph, you consider the case of calculator vs. slide rule, but one can, I think, make a more apposite case with calculator vs. learning arithmetic (and even if you don't agree that it is more apposite, the point can still be made!) By your logic, we should not be assigning any arithmetic problems that can be solved with a calculator.
But why stop there? Even current calculators can do much more than basic arithmetic, and their capabilities pale in comparison to those of general-purpose computers. I do not believe you can teach mathematics in a way that begins at a point beyond what has been automated, and before long that may be true for computing as well. This has not led to the demise of mathematics education, and neither will it for computing, for the same reason in both cases: anyone who needs to use these tools to get through the introductory classes will wash out later - which brings us right back to spcebar's original comment about only hurting themselves in the long run.
I think that's an excellent point, and one whose nuances shouldn't be un-examined.
I guess the difference to me is that arithmetic feels more fundamental. I may be using fundamental in a weird way here, but I don't mean it in the sense of something that you build on. I mean it in the sense that arithmetic itself is the idea that's trying to be taught. Whereas a sorting algorithm has both a explanation as a semi-platonic ideal and as an implementation and they're very different things. You can fairly effectively wield the implementation only using the idea in your head, mostly regardless of your understanding of the implementation. In fact, it's generally considered good practice to hide an implementation from your eventual users.
I'm not so convinced that you could do the same with arithmetic, although I'm open to arguments to the contrary.
Now that I'm thinking about it, I think this is the dividing line between math and computing in my head. The ability to separate out the real world part from the ideal and operate the former using only the latter.
Perhaps I'm mistaken, but I would be very surprised if a mathematician could operate without understanding arithmetic, but I've seen quite competent programmers construct APIs just fine without having been taught sorting algorithms.
They may lose in the long run, but the non cheaters aren't even allowed to run
Not everyone who take CS1* courses are CS (or similar eng) majors. We keep crying "Everyone should learn to code" and now that's made easier than ever some of continue to complain. Huh?
Full disclosure: I started to use CoPilot a couple of months ago. It does not remove me from the process. It does allow me to shift more brain cycles from the mundane to the more difficult. My general feeling is, I have yet to get close to taking full advantage of what it can offer me.
There's an emerging divide between those with experience with AI tools and people who tried them a few times and got hyped and blogged a maximalist view of consequences.
Just pose better questions and more open ended assignments.
The perceived productivity boost that people get from it could be interpreted as a condemnation of how verbose a lot of coding still is even in known areas. Lots of brain cycles wasted on doing things suboptimally that we can already do optimally.
Maybe more assignments like:
- "Here is a function that is supposed to generate a fibonacci sequence, but it is not correct, explain why"
- Here is a sort function. What is the name of the algorithm implemented by this sort function?
The reality of any career in CS is going to involve a lot of maintenance/understanding of existing code any systems. Copilot's failure to replace all human programmers isn't because of it's inability to cough up complex code, it's because it can't communicate with the product owner, debug a program amidst a time-sensitive incident, or explain its work.
Although he was an older physics teacher who only really did computer class ("Informatik") because there was nobody else available or (more) qualified to do the job. When later on there were classes on programming, he made us use TurboPascal instead of something less cluttered and actually useful in modern economy like Python.
That same teacher later on performed some show experiments about induction and electromagnetic forces in physics class where he let a magnet fall through a glass tube with several copper coils around it. So far so good, but he then tried to readjust one of the coils one the glass tube, which predictably shattered in his hands, nearly severing some fingers and actually severing some tendons, which IIRC were later replaced with other tendons from his legs or something. Needless to say, not the sharpest tool in the shed.
My other physics teacher during 10th-12th grade physics classes was a severely esoteric nutjob, preferring to hold long rants about "science" being able to diagnose diseases in people by somehow shining a laser at a drop of their blood and interpreting the reflection/refraction because there was a "connection" between the person and their blood (or something like that, I honestly tuned out his ravings after a while). Apparently his wife is/was some kind of homeopathic "healer".
All that is to say, computer and physics class teachers for me were "very fun" and "useful", since one did not have a clue that their students were "cheating" and both were not very in tune with modern and/or scientific methods.
My friend who copied my Pascal programming assignment is now a director at a $1B hedge fund. I'm the founder of a series of scrappy, ramen noodle startups. I guess it's the "Senator Blutarsky" effect:
https://en.wikipedia.org/wiki/John_%22Bluto%22_Blutarsky
You really win as a professor when you design a project that naturally causes you to reach for the knowledge you’re trying to teach. One of the best was my networking class where the project had us secretly rediscover tcp by trying to invent reliable transmission over UDP and then once we sufficiently cut our teeth the second half of the course was implementing a real working version on top of the professor provided userspace ip stack.
This is another clear indicator that most students today see college as a higher class trade school, not as a place of higher learning.
College may have been a place of higher learning at some time before the last thirty years but I doubt it.
Javascript may be the only one of those four worth teaching as it has different patterns and different ways of solving the same problem.
> Here’s an approach that’ll work for sure: use some, let’s call them alternative, programming languages that Copilot doesn’t really know. (...) Sadly, I have news: Copilot’s love for programming languages knows no bounds! Racket! Haskell! ML! (...) Copilot is a ravenous beast: if any code in any language found its way into a GitHub repo, it’s already swallowed it up and is hungry for more, nom nom nom.
Not sure how true this is in practice - I only used Copilot once, with Python - but if it is that might invalidate this concept
A reader of my article sent me this note, pointing me to some relevant hot-off-the-presses work (reproduced with permission):
-----
A group recently evaluated the performance of Codex (and another model) on 18 programming languages:
https://nuprl.github.io/MultiPL-E/
The high-order bits are:
1. The latest and greatest Codex is now twice as good on its own benchmark suite than the original version published a year ago.
2. It's just as good on Python as it is on JS, Scala, C++, Swift, TypeScript... and other languages are not too far behind. It's not bad at bash of all things.
Paper here: https://arxiv.org/abs/2208.08227