I have never once seen this software meaningfully work in any reasonable manner other than be a massive privacy violation and be a massive waste of time for the GSIs being forced to go through a timestamped log of every time a student blinked.
How about writing open-book tests in ways that are impossible to cheat on if you don't understand the material? You've had almost a year to adapt.
Why outsource student PII to a developing country sweatshop? This all seems absurd to me, it's been much easier to just write a one-liner inline script to hook blur, focus, visibilitychange, and onkeydown and log the userid when the event happens.
>> How about writing open-book tests in ways that are impossible to cheat on if you don't understand the material? You've had almost a year to adapt.
So in a remote test situation, explain how this is possible without use of these software. I can think of many scenarios where a second person can be in the room and doing the exam.
I can also think of many scenarios where group project-based assessments can be completely done by someone else. I've seen people pay others to physically attend as them, pre-COVID, literally hand over their student ID to be physically present and take a test. You can't really stop all of them.
Are you claiming that having open-book tests makes it so that everyone can get an MSc? That’s what it sounds like you’re saying, so I assume that I just don’t know what you’re arguing.
Every system is going to catch some percentage of cheaters and wrongly punish some percentage of innocent people. The pandemic has put us in a bad position where we can’t use some of the more effective systems (in-class tests) for assessing knowledge / preventing cheating, so we are forced to come up with some kind of compromise, and in many subjects, open-book, take-home tests work very well (although they require more work from the professors).
The professors are overworked already. So that's one reason not to. Second, cheating is also fairly easy on open questions. Just get someone to prompt you the answers. That's what the proctoring software is for.
But it's the style of justification: because you can't catch them all, just ignore the problem. That's not ok. Education is supposed to teach you something else than cheating.
What stopped people from having someone else do their take home tests before the pandemic?
On a more constructive note, it seems pretty simple to require some portion of responses to reference the lectures such that only someone who attended the class in this semester would be able to correctly answer, for example "use the method we discussed in the first half of lecture 3, use the third of the four numbers I told you to write down that day as variable y, if you weren't in class that day instead do X" If a test comes back and someone claims to not remember the material from any classes but still got everything right, that warrants scrutiny. If someone currently enrolled in the class is helping a person cheat, you can use standard anti-cheating techniques like comparing answers.
>> What stopped people from having someone else do their take home tests before the pandemic?
Nothing except these tests were not take home prior to COVID.
>> On a more constructive note, it seems pretty simple to require some portion of responses to reference the lectures such that only someone who attended the class in this semester would be able to correctly answer, for example "use the method we discussed in the first half of lecture 3, use the third of the four numbers I told you to write down that day as variable y, if you weren't in class that day instead do X" If a test comes back and someone claims to not remember the material from any classes but still got everything right, that warrants scrutiny. If someone currently enrolled in the class is helping a person cheat, you can use standard anti-cheating techniques like comparing answers.
So you are expecting the student to remember 100% of what they heard in the online classes? Or more likely the student would write down these details and handle it to the cheater to use during the take home exam. Or the cheater would "do X". In most cases the one doing the helping is not a current student, so it doesn't help.
I've passed college classes not attending any classes. Doing assignments, exams, and mid-terms. So would I be penalized for not attending classes?
> Nothing except these tests were not take home prior to COVID.
Take home tests were definitely a thing before covid.
> So you are expecting the student to remember 100% of what they heard in the online classes? Or more likely the student would write down these details and handle it to the cheater to use during the take home exam. Or the cheater would "do X". In most cases the one doing the helping is not a current student, so it doesn't help.
I expect students to take notes for their open book exams. If you're competent enough to identify all the material needed for the cheater to do an open book test, congratulations you have the knowledge to pass the test. If you just record every piece of information possible, congratulations you have spent way more time and effort than it would have taken to just learn the material.
If you don't attend any classes, do X. You're not being penalized for doing that, it just warrants scrutiny. If this is an entry level course that an intelligent person could teach themselves the material, then it probably doesn't matter if you cheated or not, you'll be found out in higher level courses. If this is a high level course, your department would probably have a good idea of how capable you are from past performance. If the guy struggling to get a C in introductory physics gets a perfect score on his quantum final without going to class once, that is super suspicious.
> require some portion of responses to reference the lectures such that only someone who attended the class in this semester would be able to correctly answer, for example "use the method we discussed in the first half of lecture 3, use the third of the four numbers I told you to write down that day as variable y, if you weren't in class that day instead do X"
Then you are measuring class attendance rather than subject matter mastery. If someone has already mastered the year’s material by the third class, why penalize them for skipping the rest of the lectures?
They aren't penalized, they can still do the X alternative that relies only on understanding the material. This is merely a method for determining who to take a closer look at, not what that closer look will reveal.
And yet I had many open book tests/assignments decades ago before there were even personal computers. Technology doesn't solve everything and some things aren't worth trying to solve 100%.
> This all seems absurd to me, it's been much easier to just write a one-liner inline script to hook blur, focus, visibilitychange, and onkeydown and log the userid when the event happens.
If you're curious, this is exactly what Canvas does to detect foul play (although they don't advertise it for that, it just goes into the log)[0][1]. Schools don't think it's good enough, so they spend thousands on these more invasive solutions.
IMO, the problem isn't the software. It's how humans are using the software.
The software itself shouldn't have any control over the student's grades. A person should have to review the flags and actually find some wrongdoing. Not just push 'yes' and walk away.
Except that the software is generating "scores" based on facial movement/expressions. Facial recognition/scoring is known to result in racist outcomes (links in article).
This is an echo of the “guns don’t kill people, people kill people” argument that has been going around for ages.
Let’s say you’re only using the software to flag suspicious behavior, and bringing in humans to make the final decision. What happens when (inevitably) the software disproportionally flags people with dark skin because it is not trained to recognize dark-skinned faces? Or when the software disproportionally flags poor people, or people with families?
It means that those groups of people will be targeted by the (human) bureaucracy and tasked with defending themselves, when they’ve done nothing wrong. Humans will inevitably trust the algorithm, they will use the algorithm’s outputs to justify their own biases, and even investigations come with a cost.
There’s this meme going around that the “algorithm isn’t biased, it’s the data”, but that argument doesn’t really hold water—machine learning systems, by default, learn to recognize correlations, and correlations in the real world collected with real sensors contain biases. ML, by its nature, picks up and encodes those biases, and you must make an effort to remove them—you can’t just throw an ML algorithm at a pile of data.
The point is to not let the algorithm make decisions. The human bureaucracy is suppose to be there to determine the quality of the flags and analyze whether there is any discrimination at play. A company that lacks this human element is negligent and should be held responsible. Unless algorithms can be trialed and held accountable, they shouldn’t be allowed to make decisions.
Also guns don’t kill people, people do. Otherwise explain to me why it would be okay for certain institutions to be armed but not individuals. If guns are the problem, then no one should have them (including the military/police).
Just because you disagree with me, it doesn’t mean that I misunderstood the viewpoint I’m responding to.
> The point is to not let the algorithm make decisions.
And my response is—that’s not enough. It sounds like the algorithm, because it is biased, has the effect of increasing the bias in the whole system. If your response is that humans should work harder to counteract biases in machine systems, well, I think that’s just a way to CYA and assign blame but not a way to solve the problem—humans will remain biased, and they will trust automated systems even when that trust is misplaced.
As an analogy, it’s like a driver in a partially autonomous car. As soon as the automation takes over, the driver stops paying attention to the road. We can make a big fuss and production and talk about how it’s the driver’s fault, and the driver should pay attention, but we’ve placed them in a system where they are discouraged from paying attention, and the system is more dangerous as a consequence.
> Also guns don’t kill people, people do. Otherwise explain to me why it would be okay for certain institutions to be armed but not individuals. If guns are the problem, then no one should have them (including the military/police).
This is a false dilemma / false dichotomy. This argument assumes that EITHER access to guns is to blame OR people are to blame, but not both, but there are obviously other ways to think about the problem.
Any rational way to look at problems will look at multiple contributing factors.
Parent: The software itself shouldn't have any control over the student's grades. A person should have to review the flags and actually find some wrongdoing. Not just push 'yes' and walk away.
You: Let’s say you’re only using the software to flag suspicious behavior, and bringing in humans to make the final decision. What happens when (inevitably) the software disproportionally flags people with dark skin because it is not trained to recognize dark-skinned faces? Or when the software disproportionally flags poor people, or people with families?
Answer: A person should have to review the flags and actually find some wrongdoing.
You: It means that those groups of people will be targeted by the (human) bureaucracy and tasked with defending themselves, when they’ve done nothing wrong
Me: The human bureaucracy is suppose to be there to determine the quality of the flags and analyze whether there is any discrimination at play. A company that lacks this human element is negligent and should be held responsible.
> And my response is—that’s not enough. It sounds like the algorithm, because it is biased, has the effect of increasing the bias in the whole system.
Hence why the humans should be held responsible for not addressing bias in their system. And why the actions of an algorithm should be the responsibility of its creators.
> If your response is that humans should work harder to counteract biases in machine systems, well, I think that’s just a way to CYA and assign blame but not a way to solve the problem—humans will remain biased, and they will trust automated systems even when that trust is misplaced.
So...? What’s your solution? All you’re saying is that humans will remain bias, yeah they will. That’s why we have laws that punish discrimination and bias. If your company creates products (algorithms) that discriminate, you should be held responsible. The human element is not there to “work harder” but to assure that what you’re releasing works properly. If you don’t think increased accountability fixes the problem, please tell us what would be “enough”.
> This argument assumes that EITHER access to guns is to blame OR people are to blame, but not both
No assumption. If you think a cop can have a gun but a criminal can’t then the gun isn’t the problem. If you believe cops can have guns but civilians can’t then the main factor is the person with the gun and not the gun itself. This isn’t an argument against increased restrictions and if you believe no one should have guns (including the government) im all for it. But if you believe someone has the right to have guns while others don’t, im hard pressed to see any other determining factor except who has the gun.
Please make an effort to engage with the comments I make, rather than making guesses about my mental state.
> Me: The human bureaucracy is suppose to be there to determine the quality of the flags and analyze whether there is any discrimination at play. A company that lacks this human element is negligent and should be held responsible.
The human bureaucracy doesn’t do that very well. The human bureaucracy is deeply flawed and has limited skills. We can assign blame to the human bureaucracy for its failings all we want, but if we want to effect change then it’s necessary to include a broader range of factors in out fault analysis.
In other words, “assigning blame” is a low-stakes political game, and “root-cause analysis” is what really matters.
This is like the 737 MAX failures. You can say that it’s the pilot’s responsibility to fly the plane correctly—but the fact is, pilots have a limited amount of skill and focus, and can’t overcome any arbitrary failing of technology. So we rightly attribute the problem to the design of the system, of which the human is only one component.
This grading software is like the 737 MAX—it’s software that, as part of a complete system including non-software components like humans, does a bad job and needs repair. The 737 MAX reports listed something like NINE different root causes.
I don’t understand this absolutist viewpoint that the human bureaucracy is the ONLY thing that you need to protect you from bad software. There are multiple root causes, and the bad software is one of them.
> Hence why the humans should be held responsible for not addressing bias in their system. And why the actions of an algorithm should be the responsibility of its creators.
So you’re saying that there’s a problem with the software, and that we shouldn’t place all the blame on the college administrators? Isn’t that what I’m saying?
> But if you believe someone has the right to have guns while others don’t, im hard pressed to see any other determining factor except who has the gun.
I do believe that not everyone should have the right to own guns, but if you’re interested in arguing with me about it, I won’t engage. If the comparison doesn’t work for you, think of something less emotionally charged like the 737 MAX or the Tesla Autopilot—both are scenarios where we rightly cite the software / automation as a root cause in accidents.
> I don’t understand this absolutist viewpoint that the human bureaucracy is the ONLY thing that you need to protect you from bad software. There are multiple root causes, and the bad software is one of them.
There are multiple intermediate causes, and all of them are the responsibility of the human bureaucracy—including, to the extent it contributes, the selection, use, and failure to correct bad software—and all of them stem from one root cause, to wit, that the bureaucracy faces insufficient consequences for it's failures and thus lacks motivation to do it's job well.
Now, were the analysis being performed on behalf of the bureaucracy because they had decided to do their job, rather than being part of a discussion outside of them, the causes which are intermediate from a global perspective would be root causes, sure. Context matters.
I don't think you meant to do this but you seem to have inadvertently made a very damn good argument for "guns don't kill people people kill people.
People are getting screwed because they are at the long end of an unbroken chain of crap. Crappy organizations buy crappy software and crappy professors take the results seriously. The fact that there exists a crappy tool that flags all the black people as cheaters (or whatever, point is that the false positives are unacceptable common and unacceptably distributed).
Blaming the gun (the software in this case) is tacitly condoning the unbroken chain of half a dozen people/entities that are failing to do the job they are being paid to do. The software developers shouldn't be building crap software. The companies shouldn't be selling crap software. The universities shouldn't be buying crap software. The professors shouldn't be using the results of crap software. To look at that situation and say "yeah the problem here is that this crap software exists" is beyond naive. The problem is that nobody is being accountable for the bad outcome. I'm not asking for a whipping boy or a scapegoat here, the problem is that when nobody can be held fully responsible it seems like nobody even gets held partly responsible.
This argument is fallacious. Your argument assumes that “blaming the gun” is condoning the users, and this is a false dilemma.
There’s really no room for absolutism, where blame is assigned to one source rather than distributed among many contributing factors. Imagine how dangerous air travel would still be if, after an accident, investigators looked for only one cause to blame, and tacitly condoned anything else they came across in their investigations.
Re-read my comment. My point is that fault is distributed sufficiently that accountability seems to evaporate. This is a systemic or organizational problem. Anti-cheating software is just the form this specific instance has taken.
>Your argument assumes that “blaming the gun” is condoning the users
Well until now you weren't throwing even the slightest bit of shade in their direction.
>and this is a false dilemma.
And you've created a false middle ground.
>Imagine how dangerous air travel would still be if, after an accident, investigators looked for only one cause to blame, and tacitly condoned anything else they came across in their investigations.
What's the difference between "dangerous harmful cheating software" and "cheating software that's being shoehorned into use cases in which it was never expected to be used"?
That's why you don't blame the (metaphorical) gun. Everything is just tools.
The FAA doesn't go off half cocked about the evils of grade-2 fasteners because once upon a time an engineer thought a grade-2 would be enough when he should have used a grade-5. I can't believe I have to defend (invasive to the point of being unethical) anti cheating software but these sorts of software tools are just tools and can be used either wisely or poorly. The software doesn't know or care how it's being used. In an industrial setting they can (and are, same underlying tech different companies) be used to design more effective interfaces to reduce errors (which I think we would all agree is a net positive contribution to the world).
I will agree that certain biases are almost certainly baked into the software and that they will disadvantage anyone (and any situation) that isn't considered 'normal' by the software's creators.
If your argument is that they should be paying a person to sit here and watching a class of students while they're doing the tests, I'm not against that. They probably should.
But humans will always attempt to make their own work easier and less time-consuming, and this is a tool for that. Eventually, something like this is going to exist for distance learning. This is unlikely to be the final configuration of that tool, but it's a step on that road, no matter how much people don't like it.
What's needed are proper controls on the usage of the tool. And proper training. And proper oversight.
If your argument is for something else instead of the above, then I don't know what your solution would be. "Don't have school" and "don't worry about cheating" aren't acceptable.
You give people a tool and they’ll use it. Taking a measurement changes the subject being measured, showing a metric changes the judgment of the watcher. You have to be an extremely thoughtful and interested person to be shown a metric and have it not color your opinion if it has no value.
I see online synchronous 1-2 hour exams as a mistaken result of the "let's emulate in-person teaching in a remote setting" mindset.
You don't succeed in remote teaching and learning by trying to make it as close as possible to the in-person setup. You have to treat it as an entirely different problem.
Consider the synchronous exam. It is a perfect method of grading in-person. It is hard to cheat and all people take the same test in the same place. It is as fair as it can get.
In an online setting however, people can face all sort of troubles in a few hour window in their home. Your internet might stop working, neighbor might be making too much noise etc. Everyone takes the exam in a different setting and it is as unequal as it can get. It is also practically impossible to prevent cheating.
Crazy idea in this space. Eye-tracking will eventually be standard on consumer VR headsets that can be had for $300. As someone following the industry closely, I believe that the Oculus Quest 4 or a device like it will certainly have eye-tracking support. Obviously if one were to take their test in a VR environment with their eye-movements tracked it would be almost impossible to cheat. Eyes also provide a good biometric identifier. Mirrors, screens, exploits, etc can be foiled using software locks, HMDs cameras and imu data. Of course I'm not advocating a testing system like this, but it would put a hard stop to cheating on synchronous exams, enforce some level of standardization, and should be accessible to every student in the 1st world. Understandable privacy and ethics issues with something like this. It is probably better to address why people cheat in the first place and make education less reliant on exams. (continuous low-impact variable feedback vs discrete high-impact fixed feedback)
Good luck wearing a VR headset for a full 2 hour exam. I get headaches if I wear one for more than 10 minutes, and the weight is uncomfortable and messes with my hair. More surveillance technology is not the solution.
My wife, who has issues with motion sickness, would absolutely love this approach - she'd be effectively barred from taking exams due to getting headaches and throwing up whenever she attempted it.
Anyways, I'd prefer an approach that led to less invasive face tracking - even if it suffered from a lower detection rate.
> You don't succeed in remote teaching and learning by trying to make it as close as possible to the in-person setup. You have to treat it as an entirely different problem.
I agree at a high level, but practically speaking it's not realistic to expect colleges to completely reinvent their entire teaching systems for a temporary, 1-2 year remote learning period.
Now that we've been dealing with COVID for almost an entire year, it's easy to forget that at one point we thought this would all be over in a matter of weeks or months. The situation was also evolving in real time. Colleges were looking for the most efficient stop-gap solutions, not for ways to completely overhaul their learning experience.
> It is also practically impossible to prevent cheating.
It's a mistake to assume that because we can't eliminate all cheating, we shouldn't bother reducing any cheating.
The advantage of synchronous test taking is that everyone is exposed to the problem set at the same time. The obvious cheat with asynchronous test taking would be for one person to volunteer to take the test early and then send the questions to their peers, all of whom take the test at the last possible minute.
This happens whenever in-person classes offer two time slots for taking a test. The later time slot is always far more packed than the first and comes back with significantly higher scores. Adding the internet and screenshots/camera phones to the equation amplifies this because students can share the exact test, not just what they recall from memory.
> In an online setting however, people can face all sort of troubles in a few hour window in their home. Your internet might stop working, neighbor might be making too much noise etc.
Educators aren't oblivious to this fact. Working with students who have interruptions is just part of the job. Having someone lose internet isn't much different from having someone get a flat tire on the way to the test. It happens, we deal with it, and it's fine.
I also think you're not giving students enough credit. They're not dumb. If noise is a problem, they're going to use headphones. If internet is flakey, they're going to find a better location to take a test.
It's such a weird double standard to see HN champion work from home as unequivocally superior to working in an office, yet whenever the topic of learning from home comes up we get a laundry list of what-about possibilities that might make the experience worse.
Absent COVID, institution should have been (and many have been) embracing remote learning anyway.
This should have been an option all along, with out the need for a pandemic to force their hand.
>>The advantage of synchronous test taking is that everyone is exposed to the problem set at the same time. The obvious cheat with asynchronous test taking would be for one person to volunteer to take the test early and then send the questions to their peers,
This problem has been largely solved for a long time, because as you noted it is generally impossible to give a test to EVERYONE at the same time.
Thus properly written tests will draw a random selection of questions from a larger pool, the ratio between Pool:Questions the better the security. (i.e a 25 question exam using a 50 question pool is not as secure as a 25 question exam using a 200 question pool)
This method is also used for standardized tests given at the same time, as it cuts out the problem of shoulder surfing or other in-person cheating methods.
PragmaticPulp doesn't seem to have a grasp on how college systems work. From what I have seen UW, Oregon State, Seattle Colleges, etc were already employing question banks and timed test windows rather than resorting to poorly functioning half measures like Respondus LockDown Browser and its ilk.
Many colleges were already fully capable of distance learning in multiple forms, whether through correspondence courses (what WGU often pitches, complete the project or test and bypass the class, though some of their certificate partners abuse test takers with Respondus or similar) or online learning with systems like Canvas.
Decent colleges offer a mix of these, I can attest to the quality of these programs at Seattle Colleges (specifically North Seattle College & Seattle Central). There is little value in building a panopticon of surveillance in higher education, especially when these divert resources that would otherwise enable students to better master the subject.
Most colleges offered correspondence courses in the pre-internet era. You did everything by mail. There was no proof that you did the work yourself; it was just a matter of trust. I took a required writing course that way because it never worked out to schedule it as a regular class. I think for some of the classes that had exams you maybe went to a local exam center where a proctor would check your ID and you'd take the exam. I don't know what they did if there wasn't an exam center conveniently nearby.
In a college scenario, perhaps you can say that dealing with problems is the students responsibility, but for k-12 schools it is the responsibility of the school to provide education even to children with no internet or headphones.
> I agree at a high level, but practically speaking it's not realistic to expect colleges to completely reinvent their entire teaching systems for a temporary, 1-2 year remote learning period.
I don't really understand what you mean by this though. I graduated college in 2007, and all throughout my time from 2003-2007 I had half my courses online, including the quizzes and tests.
Online courses and asynchronous testing isn't something new to Covid, colleges have been doing it for well over a decade now.
Yes you had people try and cheat their way through asynchronous testing by having friends take them earlier, but why is that more of a big deal now than it was previously?
> It's such a weird double standard to see HN champion work from home as unequivocally superior to working in an office, yet whenever the topic of learning from home comes up we get a laundry list of what-about possibilities that might make the experience worse.
This seems like a false equivalence; most employers don't use surveillance software to ensure their remote employees keep their microphone and webcam on, continue looking at the screen at all times, etc. One of the benefits of working from home is that your privacy is _increased_ compared to working an office; if everyone needed to allow their boss or HR or whoever to demand microphone and webcam access to me as they worked (and no, this is NOT comparable to Zoom meetings), then of course they wouldn't be praising work as much.
> I also think you're not giving students enough credit. They're not dumb. If noise is a problem, they're going to use headphones. If internet is flakey, they're going to find a better location to take a test.
They aren't, but many are much less privileged than you are making them out to be. My mom teaches students who join her class from their car parked as close as possible to get a weak Wi-Fi signal from their house because they have no quiet places at home.
I TAed this semester and a surprising amount of students have totally terrible internet. One poor kid had to attempt the blackboard exam a dozen times due to connection issues. Even my home network gets saturated with everyone in the neighborhood working from home, which makes zoom calls impossible when lag is spiking, despite having the fastest internet my limited ISP choices offer me.
> Now that we've been dealing with COVID for almost an entire year, it's easy to forget that at one point we thought this would all be over in a matter of weeks or months.
I absolutely agree, but I have to say, I also think it's somewhat bizarre that anyone ever thought that. I know it seemed weird to me at the time.
Where exactly did everyone think the pandemic would go after a few months of lockdown?
> Where exactly did everyone think the pandemic would go after a few months of lockdown?
We thought we'd combine lockdown with tighter quarantines for visitors and excellent test & trace systems with quarantines for people with covid and that we would, like several countries, get covid under control.
When I was in college in 2005 I had classes that could be taken online. When tested we were given a deadline, usually 24 hours after the test had been handed out. We could start whenever we wanted, take breaks, work throughout the day, as long as we had it in before the deadline. It seemed to work fine.
Yeah, take home tests were awesome. The problems on the ones I remember were ridiculously hard (specifically thinking about an algorithms take home that we were given a week for), but that spurred us to do some of our best thinking and learning.
How does one fib proficiency in a role as an individual contributor? I mean after school is through? I have been a software engineer for 14years. I am self taught with only a 2yr degree i completed my 4th year into this career. Can a line in resume(a degree mention) remedy a failed technical screening process?
Personal experience : I have declined several MD CS holders this year alone for positions because their demonstrated abilities were not up to par(and these were not particularly difficult questions).
Yeah, incredible story... Could have lasted, but too bad he was a bit careless in handing out his company's personal credentials to the Chinese team as well.
People cheat for all kinds of reasons. I'd say in many cases it bears little or no relevance to their proficiency as an IC. For example, I've seen people cheat in a Computer History class (compsci elective) presumably because they just want an easy A to push up the GPA. It doesn't necessarily mean they can't code.
Even if they can code, I don't think I would want to hire anyone who would cheat on something so basic just to push up their GPA. What is to prevent them from doing the same thing at work to get a better bonus/raise?
In the end it comes down to trust. We can't trust someone who is willing to cheat to get ahead even if they are a super coder.
We lost power (and internet) 45 seconds before my son's Physics final last spring during remote learning. He had to quickly scramble to take his testing materials outside and used his cell phone to access the exam.
It worked out for him but easily could have been a disaster. We live at the edge of the school boundary and it was a localized outage due to a car hitting a pole, so he was the only student in his class that was affected. Good luck convincing the teacher if you don't have alternate access to the test.
If you’re a straight-A student, it’s probably not an issue. Like cutting up, or seen as an “average” student, your excuses are just that, and they aren’t excused.
I think this is a pretty big miss on where the issues with these sorts of systems are - I'm sure some teachers are phoning it in but the student in the article (Molina) was immediately awarded an F and then had to go through a two month appeal process to get that undone. Once he was talking to a person things moved pretty quickly but the backlog of cases results in a lot of unnecessary stress on students.
My uni had me TA this semester which included proctoring exams over zoom. They gave me zero info what I’m supposed to do or even look for while sitting there in the zoom room. I think I was just there to add some semblance of authority. Total waste of time.
Its stupid easy to cheat in this remote world: write your cheat sheet out on a piece of paper and put it just out of sight of the webcam. No technology can beat that.
They can if they stop enrolling and paying tuition. There are plenty of options online for college-style certifications that most employers in my industry (software engineering) would accept.
I get that other professional training with certifications would be harder to do online, but those tend to be done in community colleges anyway. The most "abusive" shops are universities offering BS degrees to students swimming in debt.
Running essays through searches for similarity is one thing, this de-facto remote polygraph is quite another.
Polygraphs are bullshit and basically select for submissive behavior, which I suppose is what these institutions are looking to reward, but it is conspicuous that nobody called these surveillance schemes what they truly are: degrading.
From the many students I know, I can tell you that cheating is rampant in online learning. The average GPA has gone up 0.3 points at my alma mater. I don't think people understand just how widespread it is.
This software isn't ideal but there's no good solutions here short of major rethink of how these classes, and possible all of the college, is structured.
Certainly I had many take-home exams in college and grad school. (Of course, I had many proctored in-person exams too.)
So one solution is to switch to open book take home exams as much as possible. The other frankly is to have an honor code and, at some point, recognize that some people will cheat but they're mostly hurting themselves.
Scholarships, internships and grad school spots are a scarce resource.
Students who cheat can secure a higher GPA with less effort. That effort can be redirected towards looking for internships and improving grad school applications.
So I fail to see how cheaters are only “hurting themselves”. Morality aside, if you pursue a cheating strategy your pay-off will be much higher when we consider the scarcity of resources available to students.
That's why I wrote "mostly." Some reasonable measures to deter casual cheating may well be worthwhile. At some point though you draw a line where you start hurting honest students more than cheaters. You're not going to prevent it 100% absent draconian measures.
> Students who cheat can secure a higher GPA with less effort. That effort can be redirected towards looking for internships and improving grad school applications.
This is definitely not the reason why most people cheat.
> Morality aside, if you pursue a cheating strategy your pay-off will be much higher
As with any kind of lie, you will have to cheat more and more to compensate for previous cheating, and eventually your little web of lies will come back and bite you in the ass.
The first part certainly is why they cheat; higher grades allow easier admissions to jobs and better schools.
> As with any kind of lie, you will have to cheat more and more to compensate for previous cheating, and eventually your little web of lies will come back and bite you in the ass.
Completely disagree. People cheat to climb the ladder of life without having to put in the work or having the resources required to do bypass that ladder. However, once you climb the ladder high enough, it is much harder to fall back down.
Sure you're not going to be a good doctor if you cheat your way through med school but if you cheat your way through undergrad and get into a better program with better resources, more driven peers and better professors, you are better off regardless.
At least the cheaters I know cheat because a) they’re unable to pass without cheating and/or b) they’re avoiding putting in effort. The jobs are part of the goal, but that’s true for basically everyone that attenda college. I don’t know anyone who cheats and invests all the saved time into grinding Leetcode.
> if you cheat your way through undergrad and get into a better program with better resources, more driven peers and better professors, you are better off regardless.
Until you end up doing a surgery. You can’t cheat the real world. Your lack of knowledge will show eventually, and then it’ll limit your opportunities.
> recognize that some people will cheat but they're mostly hurting themselves.
This assumes that what you learn in university is the useful part, not the grade or the allocation of time that could be used for hackathons, projects, clubs, or internships.
- written exams where students are distributed over a larger area (e.g. the university rents a warehouse for the examination time) so that the COVID-19 spreading risk is nevertheless kept very small.
This has been done in my last exam phase and has worked well. They cranked the AC to 11, which made it quite an unpleasant envirommemt, but I would rather wear a jacket than take online exams
I can remember taking many exams in the athletic field house at my university back in the 1990s. It was pretty standard when you were in a large class.
The desks were so far apart that the current "social distancing" standards would be met. I remember that many courses had multiple variations of the same exam given to students to further reduce cheating.
I'm taking an online masters right now and have taken a few (pre-pandemic) proctored exams. The nearby university offers to proctor any exam for $10. They have a bunch of rooms with desks, and it's no big deal at all.
This proctoring technology thing has gone too far and too fast. It was a knee-jerk reaction, and with a lot of complaints I bet that a lot of it gets dropped.
At least where I'm from (Norway), grade transcripts will show a class grade distribution/histogram in the background of your grade.
That way the viewer / reader will be able to compare your results to your class.
Obviously if 80% of your class got A, that A is not going to look as impressive.
On the other hand, if you're the only student that got A, while the class distribution is monotonically increasing towards Failure, that's going to be a good thing for you.
When I went to college, our department gave two grades on the transcript: percentage mark and class ranking. The ranking is your ranking among all the sections of that class taught that semester.
Cheating is rampant in in-person learning as well. We shouldn't be treating online learning the same as in-person learning. I took distance learning classes back in the early 00's. Some courses required proctored exams, some had open book exams. For proctored exams, I would do them in the library, then the librarian would send the test back to the school. The open book exams weren't really any easier - you were expected to know the material and therefore the tests were more challenging.
In my professional experience, GPA has little correlation with how well someone can do a job. GPA going up or down by 0.3 points really doesn't mean that much to me.
At some point, when GPA is used exclusively to filter-out and rank students instead of as one of the many factors in an application, you get that metrics gamification.
Let's roll this back and ask a different question: Are the students learning the material in a way that they can display mastery outside of an exam setting / on the job?
If so, who cares if cheating numbers / grades have been inflating?
Software people in general like "hacks". We game hiring process by reading books like "Cracking the coding interview". However, the hiring process is broken but that is a different story.
I worked super hard in school, did well and pulled several all nighters. I work at an enterprise now and my algorithm knowledge is useless. I just build enterprise web applications, which is fine.
In software any job you take you will be learning new stuff and new way of doing things. I gained much more from classes that focused on building actual working projects instead of strict algorithms and testing. In our AI class we had to pick an AI algo and implement it in a popular game, one of the best experiences in school.
I just got rid of my exams for my deep learning courses once the pandemic hit. It doesn't make sense to have these traditional exams with remote education.
Boom. That’s it. It’s not cheating that’s the problem—that would be an epitome of a red herring. Emulation of a traditional environment with online learning software is not how it was intended to be used, nor is it practical.
Even when I was in college, the number of traditional courses serving additional content through Moodle/Blackboard was stupid high. I understand honor system and all that jazz. But really, if the ~15-some athletes in a business 200-level section are going to gather in the library to cheat together...then maybe the quiz should have been delivered traditionally. At least then, I wouldn’t have to hear the half-a-class rant from the professor about what losers the cheaters are.
I'm a professor in the humanities so I can write tests that make it harder to cheat than some other fields. I cringe whenever I hear someone using these invasive kinds of software. There has to be a better way than making students install spyware.
> I can write tests that make it harder to cheat than some other fields
This might be my prejudice speaking (I studied math) but isn't this basically because grading is that much more subjective? So it would cut both ways, harder to cheat but also less objective / impartial assuming no cheating.
My university does a lot of oral math exams. After failing 3 times you have the option for an oral test as a last chance and people do indeed fail math in large numbers.
I was on the other side as an academic assistant for a few days (no role in testing) and I can tell you that oral math exams about algebra can be as hard as you want them to be, especially if you have problems with tests and get nervous. You have to take drugs against that.
The prof I had was a bit of a dick but he was really good at measuring how well you understood the concepts and if you ware able to apply them to problems.
Not necessarily. I do a lot of questions that are here is a situation. What concept does this illustrate? Now sure, they can look up the definition of the concepts but they will still need to be able to identify how the concept relates to the situation.
Sure. Have at most ten to twenty students per teacher. Then the teacher can spend a couple of hours per student to assess their understanding. Unfortunately, in education quantity seems to be prioritized over quality. When you have over 50 students, let alone over 200 hundred, good luck grading authentic open-ended projects or papers in just two weeks.
As a result, most tests consists of multiple-choice questions and standard-format open questions. It is unclear to what extent they test students' understanding, and students do become quite proficient test takers, but at least theses tests can be executed given the current constraints of available staff, acceptable rigor, and student expectations.
Unfortunately, you cannot move these type of test to an online setting and expect students not to cheat at all. It is too easy to talk to classmates, look at the study materials, or even on the Internet for some hints to answers or even the actual answers. The only option many institutions saw was to move to a draconian proctoring solution because they just lack the means to roll-out anything else given the constraints they have to work with.
Ideally, they would re-evaluate their choices regarding quality versus quantity, but because they need the large number of students to stay afloat, nothing meaningful will happen.
> Then the teacher can spend a couple of hours per student to assess their understanding.
That sounds as if you're steering close to a whiteboarding interview which a lot of people will claim isn't great.
Take home tests aren't perfect but I had plenty of them in school. And even when I didn't I don't think I had just about any engineering class that didn't assign problem sets that counted for a decent portion of your grade. If you can do take-home problem sets, you can do a take-home exam.
Taking home exams or assignments are great, but they need to be assessed as well. Here too, the number of students is a limiting factor. With too many students, the best a teacher can do is summative assessment and generate a reasonable grade for the student. Personally, I think formative assessment is far superior to summative assessment, but formative assessment takes a lot of effort from the teacher. Often more than is allotted to her for assessement (or the whole course!).
I was thinking more about a larger project or paper with quite a lot of freedom for the students that would take the teacher hours to get familiar with and give constructive formative feedback on. As part of that, an oral section to discuss the project and paper with the student would be great as well. In assessment, I would like the teacher and student to be partners rather than opponents.
If a student understands the material, together they should be able to have a conversation where the student can reflect on her understanding and learning to the extent where the teacher can determine that the student has passed the course. Of course, if there is an intense interaction between the student and teacher throughout the course, the whole idea of a final assessment becomes pretty meaningless.
I know, my ideas are a bit far-fetched and there are a lot of practicalities that are difficult to work out. But one can dream!
> That sounds as if you're steering close to a whiteboarding interview which a lot of people will claim isn't great.
One major difference is that unlike a whiteboarding interview there are presumably clear expectations as to what one is expected to have learned in the class and therefore what topics the discussion will cover, and at what depth.
Another issue with whiteboarding interviews, specific to computer-related subjects, is being asked to write code on the whiteboard, which is a severely unnatural act in all sorts of ways compared to the way one normally writes code. But an in-person assessment in a proofs-based math class, for example, would not have that problem. Similar for a CS (as opposed to software engineering) class.
And in a software engineering class, the concept of "test" is pretty odd, for the same reasons that whiteboarding interviews are; I'd expect longer-term projects to be closer to the right evaluation tool.
We have collaborative notebooks on iko.ai[0] which makes it easy either to pair program/troubleshoot on the same notebook, or for a teacher or instructor to review work.
We haven't added autograding yet as it is specific, but nbgrader[1][2] comes to mind.
And that would somehow be bad? Maybe the sooner current system explodes, the better? After Thatcher, practical schools were closed and there has been pressure to make as many people earn a university degree as possible. Universities are lowering standards and churning out people that have no business studying at a university. Most people go to a university because that's what society, and employers, expect.
How many students do you think Plato, Aristotle, Jesus etc. had? Teach the brightest and the ones with genuine interest in the subject (I don't use the word "passion" because after adoption by corporations it no longer seems to mean anything). The rest will be fine with a high school/secondary school education. Most people retain only a fraction of that knowledge anyway.
Maybe it's time to stop treating (higher) education like an assembly line?
How many students do you have and what's your teaching load? I think that's another primary consideration. Teaching a 3/3 (or god forbid, a 4/4) with 30 students in each makes for a rough grading experience.
I haven't used any essay-checking or exam-taking software yet.
Even so, based on what I've heard from my immediate superior, I'm flagging somewhere between 10-20% of all of the plagiarism and cheating cases in my faculty.
It's not hard to detect (generally good writing mixed with bad on essays / the repetition of similar phrases by different students on exams), but it does take a bit of time to confirm and you need to write a some policy-consistent boilerplate to make it simple to take action.
Some here have suggested that we need to write the equivalent of "open book" exams for the online environment. I'm not sure that would be beneficial. It's still important to exercise your mental faculties. People who can recall information and reason for themselves are still more valuable than those who can cut and paste. Or to put it another way, you need to understand the math in order to wield a calculator well.
In a lot of disciplines, open book doesn’t help you solve problems all that much.
It might supply some of the pieces required to solve problems, but the ability to remember all of those pieces by rote vs. looking them up is really orthogonal to the ability to apply them to solving a novel problem. Indeed the ability to find the requisite information to solve a problem in a timely manner is an important skill in itself and more accurately represents the real world problem solving one is preparing for.
The thing that has always frustrated me about anti-cheating/anti-plagiarism software is that it almost always only hurts people that did something by accident or unintentionally. When I took the one kubernetes certification exam as part of an old job, you used software like this and at one point I leaned too close to the screen and the proctor couldn't see my face and that got me flagged.
People that want to cheat or game the system will find ways, it isn't hard to do. In undergrad we setup a copy of the code checking software that our department used so that we could share code without it getting flagged as copies. I'm sure there are ways to game these systems too if you are motivated enough.
One of the issues is also just academia being so hyper-focused on thinking everyone is there to become an academic which is not the case for the vast majority of people. So these exams and the guidelines for student evaluation is grounded in that expectation instead of the reality.
>> People that want to cheat or game the system will find ways, it isn't hard to do.
As someone who taught a college, that's a flawed statement. Let me propose this to you. You teach Calculus 1000 and there is an end of term exam. Your normal end of term exam is one where everyone sits in the same room, proctors are looking for cheats, things checked, etc. Instead this year you tell your students that this year it will be a take home exam with the following rules: 1) they have 2 hours to do it during the team home peroid; 2) no cheating by the honor system.
Do you think the rate of cheating will be the same? I mean by your logic, it should be.
The idea that an honor code is enough is belied by the fact that software like this catches cheaters, no?
I mean, we're obviously upset about false positives, and we should be. But I'm presuming that some people are caught who were cheating, and without the software they would have cheated and not been caught.
We can suggest that with an honor code in place, maybe some of those cheating students would have not cheated, because... I mean, if they were willing to cheat with software in place, I'm not sure why would have been deterred by an honor code.
I think in about 98-99% of cases, people who claim an honor code prevents cheating are deluding themselves. Yeah, if you don't have any way of catching cheaters, then you can pretend you have a 0% cheating rate. But it's pretend.
I have taught at multiple honor-code institutions (and still do). It does not prevent cheating. However, it shifts focus: I can go about my teaching life starting from an assumption that students are not cheaters—and I'm personally convinced that most aren't.
The flipside is that when you do catch a cheating case, you completely throw the book at them. It's legitimately easier to cheat under an honor system, if that's what you're wanting to do... so my assumption is that if we catch you at it, it's likely part of a pattern, and if we catch you multiple times the pattern is irreformable. It is not uncommon at honor-code institutions to expel students on the second offence (sometimes even on the first).
I do think that cheating is less prevalent at my institution (and my previous institution) than it is in the larger university population.
I perhaps wasn't clear enough but I meant solely when it comes to these type of anti-cheat systems being used. Obviously it would be different for in-person vs remote/take home.
I don't see why we should allow for the possibility that the ratio of cheaters would stay the same with or without the surveillance. That could only be true if the ratio of potential cheaters were so low that the threat of detection introduced by the spyware can't reduce the ratio further.
We may hate the software on ethical grounds, or because it degrades the exam-experience on many levels, or because its use can be considered abusive, but obviously it has an effect in the intended direction.
I had my fair share of uncheatable open book exams, calculators allowed back in college. One I had on Calculus (or was it Linear Algebra?) was way harder than a comparable closed book exam. You really needed to know the subject to pass vs just memorizing a couple of formulas and plugging them in the right place.
Another professor devised an exam that used your student ID as a variable of the first question, and subsequent questions used the previous answers as inputs. Impossible to cheat.
> I had my fair share of uncheatable open book exams, calculators allowed back in college. One I had on Calculus (or was it Linear Algebra?) was way harder than a comparable closed book exam. You really needed to know the subject to pass vs just memorizing a couple of formulas and plugging them in the right place.
We're discussing this in relation in relation to COVID, so no large gathering. This means no open book in person exams, I'm specifically talking about take home exams.
> Another professor devised an exam that used your student ID as a variable of the first question, and subsequent questions used the previous answers as inputs. Impossible to cheat.
If someone can figure out how to cheat on an exam where each answer depends on the previous, and the original seed is unique to each student, wouldn't it show a pretty thorough mastery of the subject matter?
I think the larger point is that it is fairly easy in any subject to design a test that is very hard to cheat on. It is much harder to find the resources in modern education to grade that test since each submission is likely unique.
Tests that are easy to grade (like multiple choice), tend to be tests that are easy to cheat....
One option, if you have a reasonably low student-to-instructor ratio: make it an oral 1-1 exam for each student. A video call with the instructor and the student; you ask questions, they answer. If you have 20-ish students, it will eat up half a work-week or so, which is more work than grading 20 exam papers, but not _that_ much more.
Of course if you have 50 students per instructor this is not going to work...
How do you propose that this should work e.g. in proof-based maths courses? You can't just tweak a theorem to prove by the value of some "unique seed", the theorem might become wrong.
It's true that you usually can't cheat your way through such an exam provided you actually write the answer yourself, but in a take-home situation you can always ask someone else to solve it for you.
You give it a few hundred problem classes, and constraints for possible answers, and it will randomize the class of problems and generate a unique problem as well as calculate the answer. Then you submit the answer as well as your work and it gets corrected.
For things such as proofs, it might give you a problem for which the theorem is needed, then ask you to solve the problem, indicate which theorem you used, and then prove the theorem.
Do you have any example of any software that can generate problems for actual proof-based courses (e.g. abstract algebra)? I'm having a really hard time imagining this, we can't even fully automate theorem proving - how are we supposed to automate theorem generation? And this ignores the fact that you also need to make sure that all the proofs are "of the same difficulty" in terms of fairness.
Ah, no, the theorems would have to be manually selected. But given a high enough number and an automatically generated context it makes cheating much harder.
> had my fair share of uncheatable open book exams, calculators allowed back in college. One I had on Calculus (or was it Linear Algebra?) was way harder than a comparable closed book exam. You really needed to know the subject to pass vs just memorizing a couple of formulas and plugging them in the right place.
My partner recently finished her PhD and has started working as a medical writer. During her academic career, it was beat into her head don’t plagiarize, everything must be yours.
In her first review after a few weeks on the job, her manager says that she takes too long to do her work; they just need her to take what the client says verbatim, fact check it, and then slap it in a document. She was treating her work like an academic assignment and putting in the effort to craft something unique, when they really just need a fact checking typist.
8 years in higher ed, published study on cancer drugs, thousands of mice died, millions of dollars spent on the lab... all so she can transcribe some text and then validate it against the studies.
Unfortunately each faculty member produces about 30 PhD students in their career (one per year). The number of faculty positions has been close to flat since the 1970's. So one in 30 PhD's will get to be faculty member. She probably took the job because she wants/needs a job and the company likes to have the status of PhDs doing the work.
Sure, so what? Anyone smart and dedicated enough to complete a Phd is smart enough to realize this going in. Surely the vast majority of people getting Phds don't expect to ever become academics and have some other plan.
> Anyone smart and dedicated enough to complete a Phd is smart enough to realize this going in.
No, this isn't true at all. People don't complete PhD's because they looked at their options and thought that one was the best. They do it because other people told them they should, and they just never thought about it.
Society treats university as a jobs training program. Observationally it seems like the universities (in Asia in particular, and the US to a lesser degree) encourage you to treat your diploma like a golden ticket to a good job.
> Academia is the worst job training program ever.
Pet peeve of mine. Academia is not a training program for jobs. The role of academia is not to produce business-perfect-candidates.
If business wants trained workers, they should train them. Cutting costs by not training them, then blaming universities for not producing trained workers is disingenuous at best.
Yes... and no. Yes, what you say is true - that isn't the point of an academic degree.
But no, because the way students (and parents) think about it is "go to college so you can get a good job". And many, many employers require a degree or they won't look at the candidate. In the real world, academia is functioning as a job training program.
Or at least as a gateway to the good jobs. But if it's going to be a gateway, but not do any training... that's pretty inefficient.
> One of the issues is also just academia being so hyper-focused on thinking everyone is there to become an academic which is not the case for the vast majority of people
I'm not sure I follow why this is bad. I think academic rigor and being a decent scholar, as well as being able to parse and produce research are good things (and in my mind, those are the corner stones of being an academic). Did misunderstand you?
FWIW, in germany (and likely other european countries) we have a two tier system for higher education, consisting of universities and "applied universities", with the latter focusing on applied skills and the former focusing on research, which in think is sensible.
I don't think the problem is that academic rigor isn't good. I also should have stated I was talking more about US universities in this particular case.
The problem I see is that what you talk about as academic rigor isn't what is taught and evaluated in many of the programs and classes that I've seen or been a part of. A lot of these exams and assessments don't particularly evaluate you on your ability to research and understand knowledge. If I know for example that my physics professor uses a bank of questions then it is much more incentivized for me to memorize that bank of questions vs. understanding the content and working the problems myself. Whereas in say a Discrete Math or Algorithm based class, the final exam/grade is based on a proof you have to write yourself, that encourages (or rather at times forces) you to learn and research like you said.
I also think the issue, and this may just be me looking at from my own experience, a lot of people don't want to be scholars, as you put it. They went to a University based on the unfortunate expectation for some jobs that say you need that diploma as your entry ticket.
in germany (and likely other european countries) we have a two tier system for higher education, consisting of universities and "applied universities", with the latter focusing on applied skills and the former focusing on research
While it isn't codified, we effectively have this in the US as well.
Most of the "brand name" universities, plus the flagship state universities, conduct research and grant various doctoral degrees.
Then we have the middle-tier colleges (state and private) that grant masters (often only professional degrees like nursing, MBAs, etc).
And thousands of Baccalaureate-only and 2-year community colleges.
Also, in the US, "university" generally indicates a post-graduate degree granting institution. And "college" usually refers to a 2-year and Baccalaureate-only school. But, also not codified and there are notable exceptions (ex: The College of William & Mary is a top-notch full university who's name pre-dates the convention).
In the US, "college" means narrow subject matter, and "university" means a wide variety of colleges all together on one campus.
For example, there may be a "College of Engineering" and a "College of Arts and Science" that are part of one university.
It's possible to have a stand-alone college that isn't in a university. A good example is Berklee College of Music, which is narrowly focused on music.
For my Masters degree, most of my courses were absolutely like this. Our assignments were difficult and heavy. There was no expectation that you should complete them without Google or reference material.
In contrast, the "final exams" were either non-existent or extremely easy and account for only 10-20% of the grade, where they existed.
I really liked this approach. Make your assignments very difficult. Alternatively, give difficult 10-15 minute quizzes every other week. IMO, it's a much better way of evaluating students.
This is, IMO, just lazy professors that didn't bother to make exams for the digital space. Instead, they invested money and resources in digital proctors - with all the invasive pains that come with it.
If you can't bother to make a proper exam, don't bother. Rather make the classes pass/fail on project work, or grade the classes on projects / home exams. Trying to bring a 100% replica of the physical exam space into the digital space is just a recipe for disaster.
While some of it may be lazy professors, it may also be that professors don't know how to create exams in this new world, or they're not being given the appropriate resources.
What do you think it would take to bring educators up to this new standard that is very different?
Using this software sounds idiotic. They're trying to apply methods to in-person learning to distance learning. Instead of focusing on detecting cheating, they should be re-evaluating their grading and teaching methodology so that cheating is unproductive. Prioritize in class participation and essay writing. If tests are needed, they can be timed and open book.
Great example of why our educational system isn't all that good.
I took a year of Japanese in college. A big part of the grade was memorizing and performing these dialogs. If you didn't remember the dialog word for word, you'd get points off. I wasted a lot of time trying to memorize those stupid things when I could have been acquiring new vocabulary words or studying Kanji. I took a trip to Japan a couple of years go, and did some brushing up for a couple of months before I left. I learned more in those two months than I did in my entire time at school.
Their "powerful AI engine" is almost certainly just humans. It might have a few off-the-shelf components like face detection but most of what they claim to do is just so easy to outsource that almost all companies do it. If there is any delay between the system observing a suspect behaviour and the student being told to correct it then they are definitely using humans.
An institution using a service like this is a huge red flag. You should take it as an indicator of a low quality administration if not a low quality institution.
As an engineering problem this task is hard. Ryan Calo (Prof of Law, UW) once presented a fascinating bit of research on trying to automate something as simple as fining someone for speeding. Given perfect information how do you build the system? If someone exceeds the speed limit for 1 second, do you fine them? If everyone around the person is exceeding the speed limit do you use the same rules? If someone oscillates between just above and just below the speed limit, how many times do you fine them? If someone exceeds the speed limit and stays there does this result in fewer fines? How do you square the code written with the law as written? The problems are so extensive it may be that application of rules like this require human level judgement. Proctoring an exam may turn out to be an AI-complete problem.
>Ryan Calo (Prof of Law, UW) once presented a fascinating bit of research on trying to automate something as simple as fining someone for speeding. [...]
Most of the problems you've listed only exist because of expectations caused by inconsistent/lenient enforcement by human police officers. People don't seem to have a problem with strictly enforced rules in finance. eg. "your bank account can't below zero without triggering a overdraft".
> People don't seem to have a problem with strictly enforced rules in finance. eg. "your bank account can't below zero without triggering a overdraft".
I think a lot of people have problems with strictly enforced rules in finance. Especially when banks re-order daily transactions in order to maximize overdraft fees.
>I think a lot of people have problems with strictly enforced rules in finance
They have problems with it because the rules cost them money, not because it's hard to understand or rigidly enforced. If the rules just resulted in a "transaction declined" they wouldn't really care either way. For instance, I don't think anyone thinks that it's unreasonable for your debit card to get declined if you don't have enough money.
In the case referenced, there were 3 transactions on one day, 2 of $10 & 1 of $20. User has $20 in account. Do you take the true chronological order of transactions (10/10/20) & charge 1 fee, or do you go highest to lowest (20/10/10) & charge 2 fees?
A bank executive would likely argue why it's 'fair' to charge 2 fees because a business day is the relevant period for a bank (closing at end of day, etc.) & that method of accounting is mentioned on page 85 of the checking account TOS that a user signed.
Many consumers would argue that the bank's relevant period is meaningless, especially in the age of computers. They would also argue that it is unfair to lay out complex rules like this in an opaque way because of the asymmetrical information advantage that a bank has. They wouldn't complain that the rule was enforced per se, they would complain that the rule (which is plainly anti-consumer) exists.
Same is true of this cheating stuff - I think in general, students want a fair platform for grading. They just want that platform to be actually fair.
> They would also argue that it is unfair to lay out complex rules like this in an opaque way because of the asymmetrical information advantage that a bank has. They wouldn't complain that the rule was enforced per se, they would complain that the rule (which is plainly anti-consumer) exists.
Doesn't that justify my point? If we go back to the speeding example, if the rule was that you can't go over the speed limit (within the capabilities of the measuring device), then everyone would drive a little more slowly. The only reason people 5-10 miles above the speed limit is that 5-10 miles over is generally accepted to be "fine". If anything strict enforcement of speed limits reduce the amount of room for abuse by law enforcement (eg. pulling over someone for going 1 mph over).
Also, at the risk of victim blaming, maybe it isn't such a good idea to have your deposits/withdraws lined up on the same day? Deposits can get delayed/withheld, and withdraws can be moved up unexpectedly. Leaving zero days between a deposit and a withdraw is just asking for trouble. I agree that reordering the transactions from largest to smallest is probably greed motivated, but at the same time expecting it to behave differently is optimistic at best and foolish at worst. It's the equivalent of relying on undefined behavior in programming (eg. assuming that reading one byte after the end of an array wouldn't cause a fault).
> Also, at the risk of victim blaming, maybe it isn't such a good idea to have your deposits/withdraws lined up on the same day? Deposits can get delayed/withheld, and withdraws can be moved up unexpectedly. Leaving zero days between a deposit and a withdraw is just asking for trouble.
I'm not sure you understood the example that GP gave; there was no mention of withdrawals. The idea was that you have $20 in your account, you buy something for $10, then later buy something else for $10, and then later buy something for $20, and the bank reverses the order of applying the transaction and says that you made two purchases after your account was empty, so you get charged two overdraft fees.
Or you could just set the speed limit appropriately, at a level that most drivers will naturally conform to.
This optimizes for both expediency and safety, but, sadly, not for revenue, or for readily-available probable cause to pull over essentially anyone the police want to pull over.
> Or you could just set the speed limit appropriately, at a level that most drivers will naturally conform to.
The problem is that such a limit would not be a safe one, you would basically be replacing the limit with a sign that posts the rate that people like to drive at.
There is a real difference, in both actual safety and perceived safety, between divided roads and undivided roads with the same number of lanes. Wider lanes seem (and are) safer than narrower lanes. See https://en.wikipedia.org/wiki/Traffic_calming for more.
Reasonable drivers conform their driving habits to the environment.
Unreasonable drivers should not be allowed to drive.
> There is a real difference, in both actual safety and perceived safety, between divided roads and undivided roads ... Wider lanes seem (and are) safer than narrower lanes.
> [Wiki] Traffic calming can include the following engineering measures ... Narrowing traffic lanes ... Converting one-way streets into two-way streets forces opposing traffic into close proximity, which requires more careful driving
What is happening here? Did the engineers forget that they were supposed to optimize for safety, and instead optimized for getting people to slow down? Does the net effect of this improve safety, or worsen it?
If everyone drives as fast as they can while meeting some perceived safety level, then does that mean these efforts are unlikely to affect safety and merely to slow everyone down? Is the best set of measures the maximally deceptive set, which looks as dangerous as possible while being as safe as possible?
> If everyone drives as fast as they can while meeting some perceived safety level, then does that mean these efforts are unlikely to affect safety and merely to slow everyone down?
No. It means that people are usually wrong when they judge the safety level of their driving speed on a nice clear road that happens to have pedestrians next to it or trying to cross it, but are more accurate at judging safety when they are primed to expect obstacles, steering challenges, other actors moving not in parallel with them on the road itself.
My understanding is that speed limits have been largely unchanged for the past 50+ years. Car safety, on the other hand, has increased by a great deal.
It seems likely to me that speed limits are conservative when it comes to safety.
I suspect it has more to do with the incentives faced by those who set the speed limits.
I don't know exactly how it goes. If it's chosen (or heavily influenced) by politicians, I suspect politicians can win some votes by saying they'll improve safety by lowering speed limits (particularly around schools or other places where children might be); while they'd be less likely to win as many votes by saying they'll raise the limits (exposing them to the risk of their opponents calling them reckless/irresponsible). If it's chosen by non-elected officials, that's less of an issue, but something like it may still be there; or it may be that raising the limit and then there being a fatal accident will damage your career, while lowering the limit and irritating everyone will not damage your career.
The speed limit is (in the USA) supposed to adhere to the 85th percentile rule, that being the speed at which people drive on a free-flowing road with no traffic and no enforcement. Most freeways have speed limits that are probably closer to the 0th percentile than the 85th percentile. I know driving the speed limit on some roads in my neck of the woods would be utterly hazardous.
The problem is that posting a lower limit doesn't really stop most people from driving faster. Then, some people will obey the speed limit and they become hazards. Other people might believe the posted speed limit tells you about the speed you can expect cars to go on the road. It doesn't.
The main safety increase I can see is with respect to commercial trucking. Not all vehicles have the same acceleration and braking ability. But the fact that people just ignore the speed limit is a hard problem.
So, yes, as far as is practical, that's the idea. We just post the speed people are driving at.
>The problem is that such a limit would not be a safe one,
According to who? The minority who wants the limit set elsewhere. Things like "what speed is safe and reasonable" are matters of social consensus. The "experts" can pontificate all they want and the vocal minority can gripe all they want but the median or average person and society at large is always going to be right on matters of social consensus. It's a tautology.
People like to drive at the rate that is safe. Humans are pretty good at measuring risk.
I've driven on many roads in countries that don't have any speed limits. The overwhelming majority are driving a speed that makes sense given the road, driving conditions, etc.
This is getting really far off topic, but the solution is clearly to make speeding fines a "dollars per mile per hour, per hour" system. It should scale continuously both with speed over the limit and with time spent speeding. Programmers who focus on discrete systems are too prone to forget about real analysis. ;)
This is clearly not the solution, unless it is accompanied by changes to speed limits in North America- which itself introduces a bunch of complexity and negative side-effects.
It won't solve speed limits as a civic policy issue, but it does solve the specific thing the question was posed to ask: the morass of trouble created by trying to write discrete rules for continuous phenomena.
So, on a 65mph highway, driving at 66mph for an hour has the same fine as driving at 125mph for one minute? (Minus speedup and slowdown time.) Don't think a linear scale would make sense.
I'm not sure if any scale would make sense, though. The true thing to be fined for should be "driving unsafely", of course, whether that means driving too fast, braking suddenly, swerving, changing lanes without warning, etc. The thing is, the unsafety of all those things depends in very large part on the cars around you and sometimes on the road. (Driving at 100mph on a straight, flat freeway with no cars nearby is safer than weaving between a bunch of 65-70mph cars to maintain a speed of 80mph.)
But it would be really hard to come up with an objective algorithm to calculate how unsafe a rule violation is, and even harder to implement it without a vast array of cameras. So ... they implement an enforceable set of rules even if it's not a good one. (On a related note, it bothers me that driving with more than some arbitrary blood alcohol level is illegal, but driving after e.g. staying awake all night is not, even though the latter is worse[1]. Likely this is partly because checking BAC can be done fairly directly with cheap equipment, while checking how recently someone slept is... I dunno, there might be ways to mostly do that with good equipment, but I assume it's currently impractical.)
> Their "powerful AI engine" is almost certainly just humans.
I wonder how parents feel that an anonymous foreign, most probably male, can watch their daughter in her room working on her exams and she can't opt-out else she'll be treated like a criminal.
Wonder if at some point we'll find a zip file shared among employees with screenshots of students from one of these outsourced proctoring vendors.
> I wonder how parents feel that an anonymous foreign, most probably male, can watch their daughter in her room working on her exams and she can't opt-out else she'll be treated like a criminal.
No legal recourse, and cultural differences. An adult male in Japan propositioning a 13 year old for sex is legal in their current system, whereas that doesn't really fly in America.
I've seen something related actually play out 20 years back. My dad needed to drive through a particular toll booth multiple times one weekend. He figured out that same weekend you could go pretty fast and the ezpass would still register. So he went fast every time.
A week later he got a half dozen letters in the mail all at once:
* warning do not speed
* warning (+fine)
* final warning (+fine)
* ezpass revocation (+fine)
* etc.
I think he ended up arguing that he didn't get the first warning before the others, and they agreed to waive the fines and roll back to the first warning. I.E. human judgement applied after the fines "fixed" the issue.
In any case: toll booths are a workplace, there's people there, go slow.
It's definitely a face detection thing for some of them. My spouse had to deal with all of her black students being hectored nonstop by the software because it had a hard time reading dark skin faces and was constantly interrupting them to accuse them of not looking in the correct direction.
> At the self-described “heart” of the company’s monitoring software is Monitor AI, a “powerful artificial intelligence engine” that collects facial detection data [...] to identify “patterns and anomalies associated with cheating.”
> "... people who have some sort of facial disfigurement have special challenges; they might get flagged because their face has an unexpected geometry.”
So this company is extrapolating "patterns ... associated with cheating" from facial geometry. This is just phrenology laundered through their "powerful artificial intelligence engine" black box. Predicting behavior from the shape of someone's skull is still bullshit pseudoscience, even if the calipers[1] are replaced with a bunch of linear algebra.
It's probably worse. I'd bet that in addition to humans with non-'normal' heads, it would also flag the blind, deaf, sufferers of Tourette syndrome, certain muslim women, bearded men, humans of african descent, humans of asian descent, etc.
> even if the calipers[1] are replaced with a bunch of linear algebra.
I love this line. If your head does not fit the space of eigen-heads, throw an error.
It's clearly just looking for people looking at another computer or otherwise glancing elsewhere before entering answers. It's not phrenology. It may not be effective, either, and I don't want to be put in a position of defending this company. But it shouldn't be interpreted as they analyzing if someone looks like a cheater.
My dad had a pretty severe lazy eye, and I have no doubt eye tracking software would think that he was looking down at his desk constantly.
I stare off into the distance (usually sideways) when I'm thinking.
Those sorts of things are common and barely conscious.
So, what's left for the AI? Trying to detect if people seem nervous? Well, I've got friends who have nervous twitches under normal conditions.
What is normal behavior varies wildly person-to-person. Trying to figure out "is someone glancing somewhere before answering" vs "does someone nod to themselves when they think they got an answer in their head" doesn't sound like something AI can do.
Figuring out "is this person cheating" from a recording of their face sounds pretty close to phrenology. Determining personality from skull-size, determining cheating from facial-motion... they both sound bogus, and like someone's trying to create a correlation that cannot account for the variety of human behaviors and shapes.
I read the article. The specific cases mentioned involved missed instructions, interruptions where the test taker let their seat, or the test taker talking. This seems invasive, an invasion of privacy, and unreasonable. I object to the use of this software. But I really don't think they are doing or even being accused of anything like phrenology. I think it weakens the case against this type of software to make inaccurate accusations like that.
> It's clearly just looking for people looking at another computer or otherwise glancing elsewhere before entering answers
This assumes that the AI only picks up on patterns between eye movement and cheating, and not correlations between unrelated dimensions of data in the dataset that the model was trained on.
Famously, such correlations created AI systems that resulted in disparate impact on legally protected classes in the US.
Also, these systems are usually ill-equipped to handle anomalies in an accurate capacity.
College students are easy to manipulate because they're young and naive and still used to taking orders from control figures like their parents. They need to realize the power they wield and simply refuse to participate in this intrusive monitoring. They don't have to play the game, and the school can't exist without their tuition.
This reminds of what Filtered.AI does. Except that it does it for some sort of automated job interview. When interviewing through them, I had to install a Chrome Plugin that recorded audio, video, and browsing activity at all moments. But I guess they're just trying to fix the software engineer interview process.
Seconded. Who would want to work at a place that installs surveillance-ware on the personal machine of a prospective employee?
If you're technical interview team can't tell that I didn't do technical work myself within about 5 minutes , then there is a lot wrong with your whole organization.
I found myself brewing with rage while reading this article. The approach of closely inspecting a student's words & physical actions while taking a test just feels...dated. Ancient. Anachronistic. Embarrassing.
In the real world of 2020+, almost everyone has a portable Hitchhiker's Guide to the Galaxy in their pocket. Let people use calculators. Heck, let people use Wikipedia! If they copy an incorrect fact from Wikipedia, well, that's their problem and penalize them for that.
Education needs to be considered in the context of the world we live in. If you want to test students, then develop ways of TESTING students - move to live oral exams. Written long-form essays. Develop education techniques that USE the tools we're blessed to have around us instead of fearing them.
I don't have all the answers as I haven't thought about this deeply enough. But I've got to imagine there are ways of testing learning & knowledge that aren't based on fact memorization & regurgitation or performing calculations in your head that can easily be done on a calculator.
Live oral exams can be an effective method of assessment. They can also be confounded by the testee's charisma. (See: any discussion of programming interviews, which are live oral exams.[1])
They do have the advantage of degrading fairly gracefully from in-person examination to remote examination.
They're also tremendously expensive compared to a test given en masse.
[1] Oral exams in a school setting would feel much fairer to people than programming interviews do, mainly because you know what to expect on an exam in a class you just took.
> They can also be confounded by the testee's charisma.
Agreed, but I don't think oral exams are the only type of effective live testing. Nor are live/timed exercises the only way to demonstrate knowledge.
Case studies, short/long form writing and creative exercises are all viable options. The primary concern seems regarding scaling grading and review, which with a little effort and ingenuity is solvable.
My Engineering Thesis had an oral component it was legitimately of the most nerve-wracking things I ever went through. I had to spend about half an hour in a room with 3 professors getting absolutely grilled over everything from basic theoretical knowledge to detailed technical nuances specific to my experiment. At times it felt like they were deliberately trying to trip me up by asking leading questions and trying to steer me in the wrong direction.
It would be very, very hard to bluff your way through something like this with Charisma alone. You have to be damn sure you know your topic in and out. For once off exams before graduation I think it is acceptable but I'd pity students who had to go through a grilling like this every couple of months.
Voice of San Diego has been doing some great reporting. This is yet another example.
Profs who use these things are examples of teachers who just don't care enough to revamp their courses to involve more project, critical-thinking assignments, or exams where cheating won't help. They don't want to do the work to adjust the course for remote learning, and just give the course that they always teach, and do whatever it takes to get as close to the in-person exam that they used to have.
This has been my experience in college so far. My technical college I was at before had better policies for online learning and preventing heating by just making you work more so that cheating would just make it harder toward the end.
What is the hourly rate that you would expect for a professor?
Consider that a tenure track professor (ignoring adjuncts) making $60,000 is expected to teach 3-4 classes, write original peer-reviewed research, and perform "service" (which takes the form of helping the university run itself, external speaking events, and advising.)
Each individual class takes 15-20 hours a week for a course they haven't taught before (which is common). Some classes require managing a group of Teaching Assistants for classes of 500 people. Other universities may have a teacher responsible for as many as 70 students without a TA.
So, that ends up being a 60 hour week. (This is average; I've seen more.) So, without overtime, that ends up at a little more than $22 an hour for a 10 month contract.
(The summers, in which they are either paid a small sum for summer classes, are not included. However, that's also when they are expected to research and write and publish, all technically unpaid. If you consider that time as paid, drop the $22 to $20 an hour.)
So, when you talk about "not caring enough," consider that they do not get stock options for working another 20 hours a week.
Further, a large portion of their job is bringing prestige to the university. That is why their job is based on the output of their research rather than their teaching. As such, in the first 7 years of the job, they are being judged on their publications first and their teaching second. As such, you are upset with a professor because they are only working 60 hours and not dedicating more time to what is a secondary concern for promotion and, for pre-tenured professors, keeping their job.
If you want professors to care, you need more lecturers paid the same amount to teach fewer courses. However, that makes university more expensive. (Or, you end up hiring adjuncts making sub-minimum wage for the same workload, sans research.)
It's a hard problem that looks easy on the outside.
Why are we building exams that are only valid if people don't use google? Will google suddenly disappear when I get a job? Modern exams should require me to use google.
What we need is a way to have every exam question be unique to each student. If a student google the "how to" and then does it, great!
Is it fitting that Vernor Vinge wrote about this in Rainbows End [1]? He was a professor at SDSU.
These companies should actively be put out of business. College is garbage as it is. Now theyre actively suppressing people with lower incomes and lesser resources or abilities.
316 comments
[ 4.5 ms ] story [ 288 ms ] threadHow about writing open-book tests in ways that are impossible to cheat on if you don't understand the material? You've had almost a year to adapt.
Why outsource student PII to a developing country sweatshop? This all seems absurd to me, it's been much easier to just write a one-liner inline script to hook blur, focus, visibilitychange, and onkeydown and log the userid when the event happens.
So in a remote test situation, explain how this is possible without use of these software. I can think of many scenarios where a second person can be in the room and doing the exam.
Are you claiming that having open-book tests makes it so that everyone can get an MSc? That’s what it sounds like you’re saying, so I assume that I just don’t know what you’re arguing.
Every system is going to catch some percentage of cheaters and wrongly punish some percentage of innocent people. The pandemic has put us in a bad position where we can’t use some of the more effective systems (in-class tests) for assessing knowledge / preventing cheating, so we are forced to come up with some kind of compromise, and in many subjects, open-book, take-home tests work very well (although they require more work from the professors).
But it's the style of justification: because you can't catch them all, just ignore the problem. That's not ok. Education is supposed to teach you something else than cheating.
On a more constructive note, it seems pretty simple to require some portion of responses to reference the lectures such that only someone who attended the class in this semester would be able to correctly answer, for example "use the method we discussed in the first half of lecture 3, use the third of the four numbers I told you to write down that day as variable y, if you weren't in class that day instead do X" If a test comes back and someone claims to not remember the material from any classes but still got everything right, that warrants scrutiny. If someone currently enrolled in the class is helping a person cheat, you can use standard anti-cheating techniques like comparing answers.
Nothing except these tests were not take home prior to COVID.
>> On a more constructive note, it seems pretty simple to require some portion of responses to reference the lectures such that only someone who attended the class in this semester would be able to correctly answer, for example "use the method we discussed in the first half of lecture 3, use the third of the four numbers I told you to write down that day as variable y, if you weren't in class that day instead do X" If a test comes back and someone claims to not remember the material from any classes but still got everything right, that warrants scrutiny. If someone currently enrolled in the class is helping a person cheat, you can use standard anti-cheating techniques like comparing answers.
So you are expecting the student to remember 100% of what they heard in the online classes? Or more likely the student would write down these details and handle it to the cheater to use during the take home exam. Or the cheater would "do X". In most cases the one doing the helping is not a current student, so it doesn't help.
I've passed college classes not attending any classes. Doing assignments, exams, and mid-terms. So would I be penalized for not attending classes?
Take home tests were definitely a thing before covid.
> So you are expecting the student to remember 100% of what they heard in the online classes? Or more likely the student would write down these details and handle it to the cheater to use during the take home exam. Or the cheater would "do X". In most cases the one doing the helping is not a current student, so it doesn't help.
I expect students to take notes for their open book exams. If you're competent enough to identify all the material needed for the cheater to do an open book test, congratulations you have the knowledge to pass the test. If you just record every piece of information possible, congratulations you have spent way more time and effort than it would have taken to just learn the material.
If you don't attend any classes, do X. You're not being penalized for doing that, it just warrants scrutiny. If this is an entry level course that an intelligent person could teach themselves the material, then it probably doesn't matter if you cheated or not, you'll be found out in higher level courses. If this is a high level course, your department would probably have a good idea of how capable you are from past performance. If the guy struggling to get a C in introductory physics gets a perfect score on his quantum final without going to class once, that is super suspicious.
Yes, and they work pretty well for some subjects. But not for others.
> ... that is super suspicious.
But suspicion is not enough to take disciplinary measures.
In many cases, there really is no solution that is privacy-preserving, anti-cheating, covid-safe and affordable.
Then you are measuring class attendance rather than subject matter mastery. If someone has already mastered the year’s material by the third class, why penalize them for skipping the rest of the lectures?
If you're curious, this is exactly what Canvas does to detect foul play (although they don't advertise it for that, it just goes into the log)[0][1]. Schools don't think it's good enough, so they spend thousands on these more invasive solutions.
[0]: https://community.canvaslms.com/t5/Instructor-Guide/How-do-I... [1]: https://github.com/instructure/canvas-lms/blob/master/public...
The software itself shouldn't have any control over the student's grades. A person should have to review the flags and actually find some wrongdoing. Not just push 'yes' and walk away.
The software isn't suited to its purpose.
Let’s say you’re only using the software to flag suspicious behavior, and bringing in humans to make the final decision. What happens when (inevitably) the software disproportionally flags people with dark skin because it is not trained to recognize dark-skinned faces? Or when the software disproportionally flags poor people, or people with families?
It means that those groups of people will be targeted by the (human) bureaucracy and tasked with defending themselves, when they’ve done nothing wrong. Humans will inevitably trust the algorithm, they will use the algorithm’s outputs to justify their own biases, and even investigations come with a cost.
There’s this meme going around that the “algorithm isn’t biased, it’s the data”, but that argument doesn’t really hold water—machine learning systems, by default, learn to recognize correlations, and correlations in the real world collected with real sensors contain biases. ML, by its nature, picks up and encodes those biases, and you must make an effort to remove them—you can’t just throw an ML algorithm at a pile of data.
The point is to not let the algorithm make decisions. The human bureaucracy is suppose to be there to determine the quality of the flags and analyze whether there is any discrimination at play. A company that lacks this human element is negligent and should be held responsible. Unless algorithms can be trialed and held accountable, they shouldn’t be allowed to make decisions.
Also guns don’t kill people, people do. Otherwise explain to me why it would be okay for certain institutions to be armed but not individuals. If guns are the problem, then no one should have them (including the military/police).
Just because you disagree with me, it doesn’t mean that I misunderstood the viewpoint I’m responding to.
> The point is to not let the algorithm make decisions.
And my response is—that’s not enough. It sounds like the algorithm, because it is biased, has the effect of increasing the bias in the whole system. If your response is that humans should work harder to counteract biases in machine systems, well, I think that’s just a way to CYA and assign blame but not a way to solve the problem—humans will remain biased, and they will trust automated systems even when that trust is misplaced.
As an analogy, it’s like a driver in a partially autonomous car. As soon as the automation takes over, the driver stops paying attention to the road. We can make a big fuss and production and talk about how it’s the driver’s fault, and the driver should pay attention, but we’ve placed them in a system where they are discouraged from paying attention, and the system is more dangerous as a consequence.
> Also guns don’t kill people, people do. Otherwise explain to me why it would be okay for certain institutions to be armed but not individuals. If guns are the problem, then no one should have them (including the military/police).
This is a false dilemma / false dichotomy. This argument assumes that EITHER access to guns is to blame OR people are to blame, but not both, but there are obviously other ways to think about the problem.
Any rational way to look at problems will look at multiple contributing factors.
Parent: The software itself shouldn't have any control over the student's grades. A person should have to review the flags and actually find some wrongdoing. Not just push 'yes' and walk away.
You: Let’s say you’re only using the software to flag suspicious behavior, and bringing in humans to make the final decision. What happens when (inevitably) the software disproportionally flags people with dark skin because it is not trained to recognize dark-skinned faces? Or when the software disproportionally flags poor people, or people with families?
Answer: A person should have to review the flags and actually find some wrongdoing.
You: It means that those groups of people will be targeted by the (human) bureaucracy and tasked with defending themselves, when they’ve done nothing wrong
Me: The human bureaucracy is suppose to be there to determine the quality of the flags and analyze whether there is any discrimination at play. A company that lacks this human element is negligent and should be held responsible.
> And my response is—that’s not enough. It sounds like the algorithm, because it is biased, has the effect of increasing the bias in the whole system.
Hence why the humans should be held responsible for not addressing bias in their system. And why the actions of an algorithm should be the responsibility of its creators.
> If your response is that humans should work harder to counteract biases in machine systems, well, I think that’s just a way to CYA and assign blame but not a way to solve the problem—humans will remain biased, and they will trust automated systems even when that trust is misplaced.
So...? What’s your solution? All you’re saying is that humans will remain bias, yeah they will. That’s why we have laws that punish discrimination and bias. If your company creates products (algorithms) that discriminate, you should be held responsible. The human element is not there to “work harder” but to assure that what you’re releasing works properly. If you don’t think increased accountability fixes the problem, please tell us what would be “enough”.
> This argument assumes that EITHER access to guns is to blame OR people are to blame, but not both
No assumption. If you think a cop can have a gun but a criminal can’t then the gun isn’t the problem. If you believe cops can have guns but civilians can’t then the main factor is the person with the gun and not the gun itself. This isn’t an argument against increased restrictions and if you believe no one should have guns (including the government) im all for it. But if you believe someone has the right to have guns while others don’t, im hard pressed to see any other determining factor except who has the gun.
Please make an effort to engage with the comments I make, rather than making guesses about my mental state.
> Me: The human bureaucracy is suppose to be there to determine the quality of the flags and analyze whether there is any discrimination at play. A company that lacks this human element is negligent and should be held responsible.
The human bureaucracy doesn’t do that very well. The human bureaucracy is deeply flawed and has limited skills. We can assign blame to the human bureaucracy for its failings all we want, but if we want to effect change then it’s necessary to include a broader range of factors in out fault analysis.
In other words, “assigning blame” is a low-stakes political game, and “root-cause analysis” is what really matters.
This is like the 737 MAX failures. You can say that it’s the pilot’s responsibility to fly the plane correctly—but the fact is, pilots have a limited amount of skill and focus, and can’t overcome any arbitrary failing of technology. So we rightly attribute the problem to the design of the system, of which the human is only one component.
This grading software is like the 737 MAX—it’s software that, as part of a complete system including non-software components like humans, does a bad job and needs repair. The 737 MAX reports listed something like NINE different root causes.
I don’t understand this absolutist viewpoint that the human bureaucracy is the ONLY thing that you need to protect you from bad software. There are multiple root causes, and the bad software is one of them.
> Hence why the humans should be held responsible for not addressing bias in their system. And why the actions of an algorithm should be the responsibility of its creators.
So you’re saying that there’s a problem with the software, and that we shouldn’t place all the blame on the college administrators? Isn’t that what I’m saying?
> But if you believe someone has the right to have guns while others don’t, im hard pressed to see any other determining factor except who has the gun.
I do believe that not everyone should have the right to own guns, but if you’re interested in arguing with me about it, I won’t engage. If the comparison doesn’t work for you, think of something less emotionally charged like the 737 MAX or the Tesla Autopilot—both are scenarios where we rightly cite the software / automation as a root cause in accidents.
There are multiple intermediate causes, and all of them are the responsibility of the human bureaucracy—including, to the extent it contributes, the selection, use, and failure to correct bad software—and all of them stem from one root cause, to wit, that the bureaucracy faces insufficient consequences for it's failures and thus lacks motivation to do it's job well.
Now, were the analysis being performed on behalf of the bureaucracy because they had decided to do their job, rather than being part of a discussion outside of them, the causes which are intermediate from a global perspective would be root causes, sure. Context matters.
People are getting screwed because they are at the long end of an unbroken chain of crap. Crappy organizations buy crappy software and crappy professors take the results seriously. The fact that there exists a crappy tool that flags all the black people as cheaters (or whatever, point is that the false positives are unacceptable common and unacceptably distributed).
Blaming the gun (the software in this case) is tacitly condoning the unbroken chain of half a dozen people/entities that are failing to do the job they are being paid to do. The software developers shouldn't be building crap software. The companies shouldn't be selling crap software. The universities shouldn't be buying crap software. The professors shouldn't be using the results of crap software. To look at that situation and say "yeah the problem here is that this crap software exists" is beyond naive. The problem is that nobody is being accountable for the bad outcome. I'm not asking for a whipping boy or a scapegoat here, the problem is that when nobody can be held fully responsible it seems like nobody even gets held partly responsible.
There’s really no room for absolutism, where blame is assigned to one source rather than distributed among many contributing factors. Imagine how dangerous air travel would still be if, after an accident, investigators looked for only one cause to blame, and tacitly condoned anything else they came across in their investigations.
Re-read my comment. My point is that fault is distributed sufficiently that accountability seems to evaporate. This is a systemic or organizational problem. Anti-cheating software is just the form this specific instance has taken.
>Your argument assumes that “blaming the gun” is condoning the users
Well until now you weren't throwing even the slightest bit of shade in their direction.
>and this is a false dilemma.
And you've created a false middle ground.
>Imagine how dangerous air travel would still be if, after an accident, investigators looked for only one cause to blame, and tacitly condoned anything else they came across in their investigations.
What's the difference between "dangerous harmful cheating software" and "cheating software that's being shoehorned into use cases in which it was never expected to be used"?
That's why you don't blame the (metaphorical) gun. Everything is just tools.
The FAA doesn't go off half cocked about the evils of grade-2 fasteners because once upon a time an engineer thought a grade-2 would be enough when he should have used a grade-5. I can't believe I have to defend (invasive to the point of being unethical) anti cheating software but these sorts of software tools are just tools and can be used either wisely or poorly. The software doesn't know or care how it's being used. In an industrial setting they can (and are, same underlying tech different companies) be used to design more effective interfaces to reduce errors (which I think we would all agree is a net positive contribution to the world).
Stopped reading at that point, cheers.
If your argument is that they should be paying a person to sit here and watching a class of students while they're doing the tests, I'm not against that. They probably should.
But humans will always attempt to make their own work easier and less time-consuming, and this is a tool for that. Eventually, something like this is going to exist for distance learning. This is unlikely to be the final configuration of that tool, but it's a step on that road, no matter how much people don't like it.
What's needed are proper controls on the usage of the tool. And proper training. And proper oversight.
If your argument is for something else instead of the above, then I don't know what your solution would be. "Don't have school" and "don't worry about cheating" aren't acceptable.
You give people a tool and they’ll use it. Taking a measurement changes the subject being measured, showing a metric changes the judgment of the watcher. You have to be an extremely thoughtful and interested person to be shown a metric and have it not color your opinion if it has no value.
You don't succeed in remote teaching and learning by trying to make it as close as possible to the in-person setup. You have to treat it as an entirely different problem.
Consider the synchronous exam. It is a perfect method of grading in-person. It is hard to cheat and all people take the same test in the same place. It is as fair as it can get.
In an online setting however, people can face all sort of troubles in a few hour window in their home. Your internet might stop working, neighbor might be making too much noise etc. Everyone takes the exam in a different setting and it is as unequal as it can get. It is also practically impossible to prevent cheating.
Anyways, I'd prefer an approach that led to less invasive face tracking - even if it suffered from a lower detection rate.
I agree at a high level, but practically speaking it's not realistic to expect colleges to completely reinvent their entire teaching systems for a temporary, 1-2 year remote learning period.
Now that we've been dealing with COVID for almost an entire year, it's easy to forget that at one point we thought this would all be over in a matter of weeks or months. The situation was also evolving in real time. Colleges were looking for the most efficient stop-gap solutions, not for ways to completely overhaul their learning experience.
> It is also practically impossible to prevent cheating.
It's a mistake to assume that because we can't eliminate all cheating, we shouldn't bother reducing any cheating.
The advantage of synchronous test taking is that everyone is exposed to the problem set at the same time. The obvious cheat with asynchronous test taking would be for one person to volunteer to take the test early and then send the questions to their peers, all of whom take the test at the last possible minute.
This happens whenever in-person classes offer two time slots for taking a test. The later time slot is always far more packed than the first and comes back with significantly higher scores. Adding the internet and screenshots/camera phones to the equation amplifies this because students can share the exact test, not just what they recall from memory.
> In an online setting however, people can face all sort of troubles in a few hour window in their home. Your internet might stop working, neighbor might be making too much noise etc.
Educators aren't oblivious to this fact. Working with students who have interruptions is just part of the job. Having someone lose internet isn't much different from having someone get a flat tire on the way to the test. It happens, we deal with it, and it's fine.
I also think you're not giving students enough credit. They're not dumb. If noise is a problem, they're going to use headphones. If internet is flakey, they're going to find a better location to take a test.
It's such a weird double standard to see HN champion work from home as unequivocally superior to working in an office, yet whenever the topic of learning from home comes up we get a laundry list of what-about possibilities that might make the experience worse.
This should have been an option all along, with out the need for a pandemic to force their hand.
>>The advantage of synchronous test taking is that everyone is exposed to the problem set at the same time. The obvious cheat with asynchronous test taking would be for one person to volunteer to take the test early and then send the questions to their peers,
This problem has been largely solved for a long time, because as you noted it is generally impossible to give a test to EVERYONE at the same time.
Thus properly written tests will draw a random selection of questions from a larger pool, the ratio between Pool:Questions the better the security. (i.e a 25 question exam using a 50 question pool is not as secure as a 25 question exam using a 200 question pool)
This method is also used for standardized tests given at the same time, as it cuts out the problem of shoulder surfing or other in-person cheating methods.
Many colleges were already fully capable of distance learning in multiple forms, whether through correspondence courses (what WGU often pitches, complete the project or test and bypass the class, though some of their certificate partners abuse test takers with Respondus or similar) or online learning with systems like Canvas.
Decent colleges offer a mix of these, I can attest to the quality of these programs at Seattle Colleges (specifically North Seattle College & Seattle Central). There is little value in building a panopticon of surveillance in higher education, especially when these divert resources that would otherwise enable students to better master the subject.
I don't really understand what you mean by this though. I graduated college in 2007, and all throughout my time from 2003-2007 I had half my courses online, including the quizzes and tests.
Online courses and asynchronous testing isn't something new to Covid, colleges have been doing it for well over a decade now.
Yes you had people try and cheat their way through asynchronous testing by having friends take them earlier, but why is that more of a big deal now than it was previously?
This seems like a false equivalence; most employers don't use surveillance software to ensure their remote employees keep their microphone and webcam on, continue looking at the screen at all times, etc. One of the benefits of working from home is that your privacy is _increased_ compared to working an office; if everyone needed to allow their boss or HR or whoever to demand microphone and webcam access to me as they worked (and no, this is NOT comparable to Zoom meetings), then of course they wouldn't be praising work as much.
They aren't, but many are much less privileged than you are making them out to be. My mom teaches students who join her class from their car parked as close as possible to get a weak Wi-Fi signal from their house because they have no quiet places at home.
I absolutely agree, but I have to say, I also think it's somewhat bizarre that anyone ever thought that. I know it seemed weird to me at the time.
Where exactly did everyone think the pandemic would go after a few months of lockdown?
We thought we'd combine lockdown with tighter quarantines for visitors and excellent test & trace systems with quarantines for people with covid and that we would, like several countries, get covid under control.
Personal experience : I have declined several MD CS holders this year alone for positions because their demonstrated abilities were not up to par(and these were not particularly difficult questions).
In the end it comes down to trust. We can't trust someone who is willing to cheat to get ahead even if they are a super coder.
- Suramya
It worked out for him but easily could have been a disaster. We live at the edge of the school boundary and it was a localized outage due to a car hitting a pole, so he was the only student in his class that was affected. Good luck convincing the teacher if you don't have alternate access to the test.
Its stupid easy to cheat in this remote world: write your cheat sheet out on a piece of paper and put it just out of sight of the webcam. No technology can beat that.
Or is this just another social pathology that we need to rethink?
I get that other professional training with certifications would be harder to do online, but those tend to be done in community colleges anyway. The most "abusive" shops are universities offering BS degrees to students swimming in debt.
Polygraphs are bullshit and basically select for submissive behavior, which I suppose is what these institutions are looking to reward, but it is conspicuous that nobody called these surveillance schemes what they truly are: degrading.
This software isn't ideal but there's no good solutions here short of major rethink of how these classes, and possible all of the college, is structured.
So one solution is to switch to open book take home exams as much as possible. The other frankly is to have an honor code and, at some point, recognize that some people will cheat but they're mostly hurting themselves.
Students who cheat can secure a higher GPA with less effort. That effort can be redirected towards looking for internships and improving grad school applications.
So I fail to see how cheaters are only “hurting themselves”. Morality aside, if you pursue a cheating strategy your pay-off will be much higher when we consider the scarcity of resources available to students.
This is definitely not the reason why most people cheat.
> Morality aside, if you pursue a cheating strategy your pay-off will be much higher
As with any kind of lie, you will have to cheat more and more to compensate for previous cheating, and eventually your little web of lies will come back and bite you in the ass.
> As with any kind of lie, you will have to cheat more and more to compensate for previous cheating, and eventually your little web of lies will come back and bite you in the ass.
Completely disagree. People cheat to climb the ladder of life without having to put in the work or having the resources required to do bypass that ladder. However, once you climb the ladder high enough, it is much harder to fall back down.
Sure you're not going to be a good doctor if you cheat your way through med school but if you cheat your way through undergrad and get into a better program with better resources, more driven peers and better professors, you are better off regardless.
> if you cheat your way through undergrad and get into a better program with better resources, more driven peers and better professors, you are better off regardless.
Until you end up doing a surgery. You can’t cheat the real world. Your lack of knowledge will show eventually, and then it’ll limit your opportunities.
This assumes that what you learn in university is the useful part, not the grade or the allocation of time that could be used for hackathons, projects, clubs, or internships.
- oral exams via video call
- written exams where students are distributed over a larger area (e.g. the university rents a warehouse for the examination time) so that the COVID-19 spreading risk is nevertheless kept very small.
The desks were so far apart that the current "social distancing" standards would be met. I remember that many courses had multiple variations of the same exam given to students to further reduce cheating.
I'm taking an online masters right now and have taken a few (pre-pandemic) proctored exams. The nearby university offers to proctor any exam for $10. They have a bunch of rooms with desks, and it's no big deal at all.
This proctoring technology thing has gone too far and too fast. It was a knee-jerk reaction, and with a lot of complaints I bet that a lot of it gets dropped.
That way the viewer / reader will be able to compare your results to your class.
Obviously if 80% of your class got A, that A is not going to look as impressive.
On the other hand, if you're the only student that got A, while the class distribution is monotonically increasing towards Failure, that's going to be a good thing for you.
In my professional experience, GPA has little correlation with how well someone can do a job. GPA going up or down by 0.3 points really doesn't mean that much to me.
If so, who cares if cheating numbers / grades have been inflating?
Put it on your resume.
I worked super hard in school, did well and pulled several all nighters. I work at an enterprise now and my algorithm knowledge is useless. I just build enterprise web applications, which is fine.
In software any job you take you will be learning new stuff and new way of doing things. I gained much more from classes that focused on building actual working projects instead of strict algorithms and testing. In our AI class we had to pick an AI algo and implement it in a popular game, one of the best experiences in school.
Even when I was in college, the number of traditional courses serving additional content through Moodle/Blackboard was stupid high. I understand honor system and all that jazz. But really, if the ~15-some athletes in a business 200-level section are going to gather in the library to cheat together...then maybe the quiz should have been delivered traditionally. At least then, I wouldn’t have to hear the half-a-class rant from the professor about what losers the cheaters are.
I without question wrote every bit of this text. It's saying I plagiarized. I feel like plagiarism software isn't quite there yet.
1 thing is for sure though. I certainly don't use enough commas.
This might be my prejudice speaking (I studied math) but isn't this basically because grading is that much more subjective? So it would cut both ways, harder to cheat but also less objective / impartial assuming no cheating.
I was on the other side as an academic assistant for a few days (no role in testing) and I can tell you that oral math exams about algebra can be as hard as you want them to be, especially if you have problems with tests and get nervous. You have to take drugs against that.
The prof I had was a bit of a dick but he was really good at measuring how well you understood the concepts and if you ware able to apply them to problems.
As a result, most tests consists of multiple-choice questions and standard-format open questions. It is unclear to what extent they test students' understanding, and students do become quite proficient test takers, but at least theses tests can be executed given the current constraints of available staff, acceptable rigor, and student expectations.
Unfortunately, you cannot move these type of test to an online setting and expect students not to cheat at all. It is too easy to talk to classmates, look at the study materials, or even on the Internet for some hints to answers or even the actual answers. The only option many institutions saw was to move to a draconian proctoring solution because they just lack the means to roll-out anything else given the constraints they have to work with.
Ideally, they would re-evaluate their choices regarding quality versus quantity, but because they need the large number of students to stay afloat, nothing meaningful will happen.
That sounds as if you're steering close to a whiteboarding interview which a lot of people will claim isn't great.
Take home tests aren't perfect but I had plenty of them in school. And even when I didn't I don't think I had just about any engineering class that didn't assign problem sets that counted for a decent portion of your grade. If you can do take-home problem sets, you can do a take-home exam.
I was thinking more about a larger project or paper with quite a lot of freedom for the students that would take the teacher hours to get familiar with and give constructive formative feedback on. As part of that, an oral section to discuss the project and paper with the student would be great as well. In assessment, I would like the teacher and student to be partners rather than opponents.
If a student understands the material, together they should be able to have a conversation where the student can reflect on her understanding and learning to the extent where the teacher can determine that the student has passed the course. Of course, if there is an intense interaction between the student and teacher throughout the course, the whole idea of a final assessment becomes pretty meaningless.
I know, my ideas are a bit far-fetched and there are a lot of practicalities that are difficult to work out. But one can dream!
One major difference is that unlike a whiteboarding interview there are presumably clear expectations as to what one is expected to have learned in the class and therefore what topics the discussion will cover, and at what depth.
Another issue with whiteboarding interviews, specific to computer-related subjects, is being asked to write code on the whiteboard, which is a severely unnatural act in all sorts of ways compared to the way one normally writes code. But an in-person assessment in a proofs-based math class, for example, would not have that problem. Similar for a CS (as opposed to software engineering) class.
And in a software engineering class, the concept of "test" is pretty odd, for the same reasons that whiteboarding interviews are; I'd expect longer-term projects to be closer to the right evaluation tool.
A merge/code review tool could be used to annotate individual problems. Have you seen this done?
We haven't added autograding yet as it is specific, but nbgrader[1][2] comes to mind.
- [0]: https://iko.ai/docs/notebook/#collaboration
- [1]: https://nbgrader.readthedocs.io
- [2]: https://nbgrader.readthedocs.io/en/stable/command_line_tools...
How many students do you think Plato, Aristotle, Jesus etc. had? Teach the brightest and the ones with genuine interest in the subject (I don't use the word "passion" because after adoption by corporations it no longer seems to mean anything). The rest will be fine with a high school/secondary school education. Most people retain only a fraction of that knowledge anyway.
Maybe it's time to stop treating (higher) education like an assembly line?
Even so, based on what I've heard from my immediate superior, I'm flagging somewhere between 10-20% of all of the plagiarism and cheating cases in my faculty.
It's not hard to detect (generally good writing mixed with bad on essays / the repetition of similar phrases by different students on exams), but it does take a bit of time to confirm and you need to write a some policy-consistent boilerplate to make it simple to take action.
Some here have suggested that we need to write the equivalent of "open book" exams for the online environment. I'm not sure that would be beneficial. It's still important to exercise your mental faculties. People who can recall information and reason for themselves are still more valuable than those who can cut and paste. Or to put it another way, you need to understand the math in order to wield a calculator well.
It might supply some of the pieces required to solve problems, but the ability to remember all of those pieces by rote vs. looking them up is really orthogonal to the ability to apply them to solving a novel problem. Indeed the ability to find the requisite information to solve a problem in a timely manner is an important skill in itself and more accurately represents the real world problem solving one is preparing for.
But there's a point to that.
Democratising education is great for literally every industry except education industry itself.
There should be some way to do that.
People that want to cheat or game the system will find ways, it isn't hard to do. In undergrad we setup a copy of the code checking software that our department used so that we could share code without it getting flagged as copies. I'm sure there are ways to game these systems too if you are motivated enough.
One of the issues is also just academia being so hyper-focused on thinking everyone is there to become an academic which is not the case for the vast majority of people. So these exams and the guidelines for student evaluation is grounded in that expectation instead of the reality.
As someone who taught a college, that's a flawed statement. Let me propose this to you. You teach Calculus 1000 and there is an end of term exam. Your normal end of term exam is one where everyone sits in the same room, proctors are looking for cheats, things checked, etc. Instead this year you tell your students that this year it will be a take home exam with the following rules: 1) they have 2 hours to do it during the team home peroid; 2) no cheating by the honor system.
Do you think the rate of cheating will be the same? I mean by your logic, it should be.
I mean, we're obviously upset about false positives, and we should be. But I'm presuming that some people are caught who were cheating, and without the software they would have cheated and not been caught.
We can suggest that with an honor code in place, maybe some of those cheating students would have not cheated, because... I mean, if they were willing to cheat with software in place, I'm not sure why would have been deterred by an honor code.
I think in about 98-99% of cases, people who claim an honor code prevents cheating are deluding themselves. Yeah, if you don't have any way of catching cheaters, then you can pretend you have a 0% cheating rate. But it's pretend.
P.S. I'm not speaking for my employer at all.
Where would they get reliable training data?
There's a pile of money to be spent on this stuff, and virtually zero accountability. What's that a recipe for?
The flipside is that when you do catch a cheating case, you completely throw the book at them. It's legitimately easier to cheat under an honor system, if that's what you're wanting to do... so my assumption is that if we catch you at it, it's likely part of a pattern, and if we catch you multiple times the pattern is irreformable. It is not uncommon at honor-code institutions to expel students on the second offence (sometimes even on the first).
I do think that cheating is less prevalent at my institution (and my previous institution) than it is in the larger university population.
We may hate the software on ethical grounds, or because it degrades the exam-experience on many levels, or because its use can be considered abusive, but obviously it has an effect in the intended direction.
Another professor devised an exam that used your student ID as a variable of the first question, and subsequent questions used the previous answers as inputs. Impossible to cheat.
We're discussing this in relation in relation to COVID, so no large gathering. This means no open book in person exams, I'm specifically talking about take home exams.
> Another professor devised an exam that used your student ID as a variable of the first question, and subsequent questions used the previous answers as inputs. Impossible to cheat.
Really easy to do so in a take home exam.
I think the larger point is that it is fairly easy in any subject to design a test that is very hard to cheat on. It is much harder to find the resources in modern education to grade that test since each submission is likely unique.
Tests that are easy to grade (like multiple choice), tend to be tests that are easy to cheat....
Please tell me how you would structure such an exam without a proctoring system as described.
Computer science: Solve a complex problem in code. Include a git history. Be ready to defend your program design if I get suspicious.
Econ: Long answer question: Pick 5 concepts that we learned about that you think are most important. Explain them as you would to a ten year old.
Of course if you have 50 students per instructor this is not going to work...
It's true that you usually can't cheat your way through such an exam provided you actually write the answer yourself, but in a take-home situation you can always ask someone else to solve it for you.
For things such as proofs, it might give you a problem for which the theorem is needed, then ask you to solve the problem, indicate which theorem you used, and then prove the theorem.
You can always just pay someone to do it for you.
In her first review after a few weeks on the job, her manager says that she takes too long to do her work; they just need her to take what the client says verbatim, fact check it, and then slap it in a document. She was treating her work like an academic assignment and putting in the effort to craft something unique, when they really just need a fact checking typist.
8 years in higher ed, published study on cancer drugs, thousands of mice died, millions of dollars spent on the lab... all so she can transcribe some text and then validate it against the studies.
Academia is the worst job training program ever.
Academia was never meant to be a jobs training program. The fact that many students treat it as such is not really the fault of the institutions.
No, this isn't true at all. People don't complete PhD's because they looked at their options and thought that one was the best. They do it because other people told them they should, and they just never thought about it.
Pet peeve of mine. Academia is not a training program for jobs. The role of academia is not to produce business-perfect-candidates.
If business wants trained workers, they should train them. Cutting costs by not training them, then blaming universities for not producing trained workers is disingenuous at best.
But no, because the way students (and parents) think about it is "go to college so you can get a good job". And many, many employers require a degree or they won't look at the candidate. In the real world, academia is functioning as a job training program.
Or at least as a gateway to the good jobs. But if it's going to be a gateway, but not do any training... that's pretty inefficient.
I'm not sure I follow why this is bad. I think academic rigor and being a decent scholar, as well as being able to parse and produce research are good things (and in my mind, those are the corner stones of being an academic). Did misunderstand you?
FWIW, in germany (and likely other european countries) we have a two tier system for higher education, consisting of universities and "applied universities", with the latter focusing on applied skills and the former focusing on research, which in think is sensible.
The problem I see is that what you talk about as academic rigor isn't what is taught and evaluated in many of the programs and classes that I've seen or been a part of. A lot of these exams and assessments don't particularly evaluate you on your ability to research and understand knowledge. If I know for example that my physics professor uses a bank of questions then it is much more incentivized for me to memorize that bank of questions vs. understanding the content and working the problems myself. Whereas in say a Discrete Math or Algorithm based class, the final exam/grade is based on a proof you have to write yourself, that encourages (or rather at times forces) you to learn and research like you said.
I also think the issue, and this may just be me looking at from my own experience, a lot of people don't want to be scholars, as you put it. They went to a University based on the unfortunate expectation for some jobs that say you need that diploma as your entry ticket.
While it isn't codified, we effectively have this in the US as well.
Most of the "brand name" universities, plus the flagship state universities, conduct research and grant various doctoral degrees.
Then we have the middle-tier colleges (state and private) that grant masters (often only professional degrees like nursing, MBAs, etc).
And thousands of Baccalaureate-only and 2-year community colleges.
Also, in the US, "university" generally indicates a post-graduate degree granting institution. And "college" usually refers to a 2-year and Baccalaureate-only school. But, also not codified and there are notable exceptions (ex: The College of William & Mary is a top-notch full university who's name pre-dates the convention).
For example, there may be a "College of Engineering" and a "College of Arts and Science" that are part of one university.
It's possible to have a stand-alone college that isn't in a university. A good example is Berklee College of Music, which is narrowly focused on music.
https://www.usnews.com/education/best-colleges/articles/2018...
But, you are correct that subject schools within a larger uni are often called College of Such and Such.
P.s. love the user name.
In contrast, the "final exams" were either non-existent or extremely easy and account for only 10-20% of the grade, where they existed.
I really liked this approach. Make your assignments very difficult. Alternatively, give difficult 10-15 minute quizzes every other week. IMO, it's a much better way of evaluating students.
If you can't bother to make a proper exam, don't bother. Rather make the classes pass/fail on project work, or grade the classes on projects / home exams. Trying to bring a 100% replica of the physical exam space into the digital space is just a recipe for disaster.
What do you think it would take to bring educators up to this new standard that is very different?
Great example of why our educational system isn't all that good.
I took a year of Japanese in college. A big part of the grade was memorizing and performing these dialogs. If you didn't remember the dialog word for word, you'd get points off. I wasted a lot of time trying to memorize those stupid things when I could have been acquiring new vocabulary words or studying Kanji. I took a trip to Japan a couple of years go, and did some brushing up for a couple of months before I left. I learned more in those two months than I did in my entire time at school.
Their "powerful AI engine" is almost certainly just humans. It might have a few off-the-shelf components like face detection but most of what they claim to do is just so easy to outsource that almost all companies do it. If there is any delay between the system observing a suspect behaviour and the student being told to correct it then they are definitely using humans.
An institution using a service like this is a huge red flag. You should take it as an indicator of a low quality administration if not a low quality institution.
As an engineering problem this task is hard. Ryan Calo (Prof of Law, UW) once presented a fascinating bit of research on trying to automate something as simple as fining someone for speeding. Given perfect information how do you build the system? If someone exceeds the speed limit for 1 second, do you fine them? If everyone around the person is exceeding the speed limit do you use the same rules? If someone oscillates between just above and just below the speed limit, how many times do you fine them? If someone exceeds the speed limit and stays there does this result in fewer fines? How do you square the code written with the law as written? The problems are so extensive it may be that application of rules like this require human level judgement. Proctoring an exam may turn out to be an AI-complete problem.
Most of the problems you've listed only exist because of expectations caused by inconsistent/lenient enforcement by human police officers. People don't seem to have a problem with strictly enforced rules in finance. eg. "your bank account can't below zero without triggering a overdraft".
I think a lot of people have problems with strictly enforced rules in finance. Especially when banks re-order daily transactions in order to maximize overdraft fees.
They have problems with it because the rules cost them money, not because it's hard to understand or rigidly enforced. If the rules just resulted in a "transaction declined" they wouldn't really care either way. For instance, I don't think anyone thinks that it's unreasonable for your debit card to get declined if you don't have enough money.
In the case referenced, there were 3 transactions on one day, 2 of $10 & 1 of $20. User has $20 in account. Do you take the true chronological order of transactions (10/10/20) & charge 1 fee, or do you go highest to lowest (20/10/10) & charge 2 fees?
A bank executive would likely argue why it's 'fair' to charge 2 fees because a business day is the relevant period for a bank (closing at end of day, etc.) & that method of accounting is mentioned on page 85 of the checking account TOS that a user signed.
Many consumers would argue that the bank's relevant period is meaningless, especially in the age of computers. They would also argue that it is unfair to lay out complex rules like this in an opaque way because of the asymmetrical information advantage that a bank has. They wouldn't complain that the rule was enforced per se, they would complain that the rule (which is plainly anti-consumer) exists.
Same is true of this cheating stuff - I think in general, students want a fair platform for grading. They just want that platform to be actually fair.
Doesn't that justify my point? If we go back to the speeding example, if the rule was that you can't go over the speed limit (within the capabilities of the measuring device), then everyone would drive a little more slowly. The only reason people 5-10 miles above the speed limit is that 5-10 miles over is generally accepted to be "fine". If anything strict enforcement of speed limits reduce the amount of room for abuse by law enforcement (eg. pulling over someone for going 1 mph over).
Also, at the risk of victim blaming, maybe it isn't such a good idea to have your deposits/withdraws lined up on the same day? Deposits can get delayed/withheld, and withdraws can be moved up unexpectedly. Leaving zero days between a deposit and a withdraw is just asking for trouble. I agree that reordering the transactions from largest to smallest is probably greed motivated, but at the same time expecting it to behave differently is optimistic at best and foolish at worst. It's the equivalent of relying on undefined behavior in programming (eg. assuming that reading one byte after the end of an array wouldn't cause a fault).
I'm not sure you understood the example that GP gave; there was no mention of withdrawals. The idea was that you have $20 in your account, you buy something for $10, then later buy something else for $10, and then later buy something for $20, and the bank reverses the order of applying the transaction and says that you made two purchases after your account was empty, so you get charged two overdraft fees.
This optimizes for both expediency and safety, but, sadly, not for revenue, or for readily-available probable cause to pull over essentially anyone the police want to pull over.
The problem is that such a limit would not be a safe one, you would basically be replacing the limit with a sign that posts the rate that people like to drive at.
Reasonable drivers conform their driving habits to the environment.
Unreasonable drivers should not be allowed to drive.
> [Wiki] Traffic calming can include the following engineering measures ... Narrowing traffic lanes ... Converting one-way streets into two-way streets forces opposing traffic into close proximity, which requires more careful driving
What is happening here? Did the engineers forget that they were supposed to optimize for safety, and instead optimized for getting people to slow down? Does the net effect of this improve safety, or worsen it?
If everyone drives as fast as they can while meeting some perceived safety level, then does that mean these efforts are unlikely to affect safety and merely to slow everyone down? Is the best set of measures the maximally deceptive set, which looks as dangerous as possible while being as safe as possible?
No. It means that people are usually wrong when they judge the safety level of their driving speed on a nice clear road that happens to have pedestrians next to it or trying to cross it, but are more accurate at judging safety when they are primed to expect obstacles, steering challenges, other actors moving not in parallel with them on the road itself.
https://www.ite.org/technical-resources/traffic-calming/
My understanding is that speed limits have been largely unchanged for the past 50+ years. Car safety, on the other hand, has increased by a great deal.
It seems likely to me that speed limits are conservative when it comes to safety.
I don't know exactly how it goes. If it's chosen (or heavily influenced) by politicians, I suspect politicians can win some votes by saying they'll improve safety by lowering speed limits (particularly around schools or other places where children might be); while they'd be less likely to win as many votes by saying they'll raise the limits (exposing them to the risk of their opponents calling them reckless/irresponsible). If it's chosen by non-elected officials, that's less of an issue, but something like it may still be there; or it may be that raising the limit and then there being a fatal accident will damage your career, while lowering the limit and irritating everyone will not damage your career.
The main safety increase I can see is with respect to commercial trucking. Not all vehicles have the same acceleration and braking ability. But the fact that people just ignore the speed limit is a hard problem.
So, yes, as far as is practical, that's the idea. We just post the speed people are driving at.
According to who? The minority who wants the limit set elsewhere. Things like "what speed is safe and reasonable" are matters of social consensus. The "experts" can pontificate all they want and the vocal minority can gripe all they want but the median or average person and society at large is always going to be right on matters of social consensus. It's a tautology.
Turns out that people do drive at a reasonably safe speed, generally.
Source: https://jalopnik.com/utah-raises-some-speed-limits-to-80-mph...
I've driven on many roads in countries that don't have any speed limits. The overwhelming majority are driving a speed that makes sense given the road, driving conditions, etc.
The ticket is merely an incentive to not do it in the future.
I'm not sure if any scale would make sense, though. The true thing to be fined for should be "driving unsafely", of course, whether that means driving too fast, braking suddenly, swerving, changing lanes without warning, etc. The thing is, the unsafety of all those things depends in very large part on the cars around you and sometimes on the road. (Driving at 100mph on a straight, flat freeway with no cars nearby is safer than weaving between a bunch of 65-70mph cars to maintain a speed of 80mph.)
But it would be really hard to come up with an objective algorithm to calculate how unsafe a rule violation is, and even harder to implement it without a vast array of cameras. So ... they implement an enforceable set of rules even if it's not a good one. (On a related note, it bothers me that driving with more than some arbitrary blood alcohol level is illegal, but driving after e.g. staying awake all night is not, even though the latter is worse[1]. Likely this is partly because checking BAC can be done fairly directly with cheap equipment, while checking how recently someone slept is... I dunno, there might be ways to mostly do that with good equipment, but I assume it's currently impractical.)
[1] "Being awake for at least 24 hours is equal to having a blood alcohol content of 0.10%. This is higher than the legal limit (0.08% BAC) in all states." https://www.cdc.gov/sleep/about_sleep/drowsy_driving.html
I wonder how parents feel that an anonymous foreign, most probably male, can watch their daughter in her room working on her exams and she can't opt-out else she'll be treated like a criminal.
Wonder if at some point we'll find a zip file shared among employees with screenshots of students from one of these outsourced proctoring vendors.
Sick, twisted? That's the complaint exactly.
Unlikely? Too many things like this happen to dismiss it.
Something else?
Why does foreign matter?
No or weak legal recourse in the event of wrongdoing being discovered.
Plus the outsourced entity can just go bankrupt and you'll never hear about it again.
A week later he got a half dozen letters in the mail all at once:
* warning do not speed * warning (+fine) * final warning (+fine) * ezpass revocation (+fine) * etc.
I think he ended up arguing that he didn't get the first warning before the others, and they agreed to waive the fines and roll back to the first warning. I.E. human judgement applied after the fines "fixed" the issue.
In any case: toll booths are a workplace, there's people there, go slow.
> "... people who have some sort of facial disfigurement have special challenges; they might get flagged because their face has an unexpected geometry.”
So this company is extrapolating "patterns ... associated with cheating" from facial geometry. This is just phrenology laundered through their "powerful artificial intelligence engine" black box. Predicting behavior from the shape of someone's skull is still bullshit pseudoscience, even if the calipers[1] are replaced with a bunch of linear algebra.
[1] http://antiquescientifica.com/phrenology_calipers_George_Com...
It's probably worse. I'd bet that in addition to humans with non-'normal' heads, it would also flag the blind, deaf, sufferers of Tourette syndrome, certain muslim women, bearded men, humans of african descent, humans of asian descent, etc.
> even if the calipers[1] are replaced with a bunch of linear algebra.
I love this line. If your head does not fit the space of eigen-heads, throw an error.
My dad had a pretty severe lazy eye, and I have no doubt eye tracking software would think that he was looking down at his desk constantly.
I stare off into the distance (usually sideways) when I'm thinking.
Those sorts of things are common and barely conscious.
So, what's left for the AI? Trying to detect if people seem nervous? Well, I've got friends who have nervous twitches under normal conditions.
What is normal behavior varies wildly person-to-person. Trying to figure out "is someone glancing somewhere before answering" vs "does someone nod to themselves when they think they got an answer in their head" doesn't sound like something AI can do.
Figuring out "is this person cheating" from a recording of their face sounds pretty close to phrenology. Determining personality from skull-size, determining cheating from facial-motion... they both sound bogus, and like someone's trying to create a correlation that cannot account for the variety of human behaviors and shapes.
This assumes that the AI only picks up on patterns between eye movement and cheating, and not correlations between unrelated dimensions of data in the dataset that the model was trained on.
Famously, such correlations created AI systems that resulted in disparate impact on legally protected classes in the US.
Also, these systems are usually ill-equipped to handle anomalies in an accurate capacity.
If you're technical interview team can't tell that I didn't do technical work myself within about 5 minutes , then there is a lot wrong with your whole organization.
In the real world of 2020+, almost everyone has a portable Hitchhiker's Guide to the Galaxy in their pocket. Let people use calculators. Heck, let people use Wikipedia! If they copy an incorrect fact from Wikipedia, well, that's their problem and penalize them for that.
Education needs to be considered in the context of the world we live in. If you want to test students, then develop ways of TESTING students - move to live oral exams. Written long-form essays. Develop education techniques that USE the tools we're blessed to have around us instead of fearing them.
I don't have all the answers as I haven't thought about this deeply enough. But I've got to imagine there are ways of testing learning & knowledge that aren't based on fact memorization & regurgitation or performing calculations in your head that can easily be done on a calculator.
Yes. The application of principles, concepts and information is not terribly difficult to test in either a timed or take home setting.
They do have the advantage of degrading fairly gracefully from in-person examination to remote examination.
They're also tremendously expensive compared to a test given en masse.
[1] Oral exams in a school setting would feel much fairer to people than programming interviews do, mainly because you know what to expect on an exam in a class you just took.
Agreed, but I don't think oral exams are the only type of effective live testing. Nor are live/timed exercises the only way to demonstrate knowledge.
Case studies, short/long form writing and creative exercises are all viable options. The primary concern seems regarding scaling grading and review, which with a little effort and ingenuity is solvable.
It would be very, very hard to bluff your way through something like this with Charisma alone. You have to be damn sure you know your topic in and out. For once off exams before graduation I think it is acceptable but I'd pity students who had to go through a grilling like this every couple of months.
The hardest test I ever took was the AI final at CMU. Open book. Books didn't help :(
Caltech had tremendous success with a culture of honesty and take-home exams.
They couldn't care less whether they get it by cheating or by honest work.
The piece of paper is all that matters.
Any controlled studies?
Are college students more ethical and honest than their non college counterparts?
What metric are you using to determine which people are "lesser"? Less educated? Less motivated? Less intelligent?
Profs who use these things are examples of teachers who just don't care enough to revamp their courses to involve more project, critical-thinking assignments, or exams where cheating won't help. They don't want to do the work to adjust the course for remote learning, and just give the course that they always teach, and do whatever it takes to get as close to the in-person exam that they used to have.
Consider that a tenure track professor (ignoring adjuncts) making $60,000 is expected to teach 3-4 classes, write original peer-reviewed research, and perform "service" (which takes the form of helping the university run itself, external speaking events, and advising.)
Each individual class takes 15-20 hours a week for a course they haven't taught before (which is common). Some classes require managing a group of Teaching Assistants for classes of 500 people. Other universities may have a teacher responsible for as many as 70 students without a TA.
So, that ends up being a 60 hour week. (This is average; I've seen more.) So, without overtime, that ends up at a little more than $22 an hour for a 10 month contract.
(The summers, in which they are either paid a small sum for summer classes, are not included. However, that's also when they are expected to research and write and publish, all technically unpaid. If you consider that time as paid, drop the $22 to $20 an hour.)
So, when you talk about "not caring enough," consider that they do not get stock options for working another 20 hours a week.
Further, a large portion of their job is bringing prestige to the university. That is why their job is based on the output of their research rather than their teaching. As such, in the first 7 years of the job, they are being judged on their publications first and their teaching second. As such, you are upset with a professor because they are only working 60 hours and not dedicating more time to what is a secondary concern for promotion and, for pre-tenured professors, keeping their job.
If you want professors to care, you need more lecturers paid the same amount to teach fewer courses. However, that makes university more expensive. (Or, you end up hiring adjuncts making sub-minimum wage for the same workload, sans research.)
It's a hard problem that looks easy on the outside.
What we need is a way to have every exam question be unique to each student. If a student google the "how to" and then does it, great!
Is it fitting that Vernor Vinge wrote about this in Rainbows End [1]? He was a professor at SDSU.
[1] https://en.wikipedia.org/wiki/Rainbows_End