Por que no los dos? The real problem is that academics aren't reading the papers, couldn't the solution here to use both the automated tools, and to pay better attention to what they're consuming? Maybe improve the UX of the plagiarism tools as well.
"Throw it all out" doesn't seem like a real solution to me, more of a flashy headline.
Do you mean not reading the plagiarism checker’s output, or the paper itself? Because the latter isn’t sufficient; you’d have to have read anything the student might’ve (presumably found by googling), and pattern match the plagiarized text. If its not blatant/poorly inserted, there’s not really any tell.
I think the idea that they work is more important than these systems actually working. The hope is that these systems scare students enough that they don’t try to plagerize in the first place.
I don’t think these systems are nearly as egregious as the 5 paragraph essay scoring system they used around 2010 when I was in 9th grade back in 2009. I thought it was fine back then, but now I wonder how that thing had even the appearance of working some of the time.
TBH, this is the function of an awful lot of audit/compliance systems. Their value is mostly not in catching actual fraud but to keep people who might be tempted to cheat on policies "just this one time" or "just a little bit" from doing so.
Heh, in my English class in 9th grade (2005-2006) for one essay our teacher had us try one of those new automatic essay scoring programs. The scores were all over the place, for mine it claimed I didn't even address the essay subject at all. Anyway the teacher was reassured of his skepticism and continued with the practice of manual grading and feedback.
I'd rather be the victim of a computer's bogus score than the victim of a computer's bogus accusation of wrong-doing. On the other hand it's important for everyone to experience the latter (by computer or not) at least once, and at least with plagiarism even if it's true you're not going to go to jail...
Agree and would add that there is no reason that the algorithms need to be secret. Well, there might be some value in keeping the algorithm from the writers, but definitely none in keeping it from the editor/teacher. Also, the algorithm can report on the likelihood of it's own error, which can help teachers to interpret the result.
Coders live in a bit of a utopia. Imagine your code quality being scored by a proprietary black box. That would be intolerable and contrary to the spirit of producing a quality product and developing better coders.
Keeping the algorithms secret for plagiarism detectors is about as dubious. If they are not good enough to reject gaming they are not robust enough to deliver reliable scores. If you don't trust students to see that gaming a plagiarism detector is more effort than honest work, you've admitted the wrong student, and/or you are using a duff plagiarism detector.
At the very tail end of when I was going to college, classes were starting to move to "online assignments" where you type your answer into an online tool which matches it as a string against the expected answer, and it's exactly as intolerable as you imagine.
Teachers like them because they don't have to grade papers, textbook publishers like them because the online code prevents reselling the textbook (or at least, the new customer will need to purchase a key anyway).
This seems to complete the charade that a college degree is just a taxi medallion, an inherently worthless object that only has value because powers conspire to require it as a roadblock to something impot, and then extract money from people who need to get past the roadblock
Are students really comparing their work against the scores generated by these tools to game the scores lower because they're worried about being punished for false positives?
That is, pardon my French, fucked up.
Beyond that it's egregious that any academic institution would allow the success of students/researchers/whatever to hinge on scores generated by a black box algorithm that hasn't been rigorously proven as very close to 100% accurate. There should always be a human in the loop, and if we don't have enough humans for that then I think we have a problem that goes beyond automated plagiarism checkers.
First of all, if your work is submitted through these detection systems, you almost never have the opportunity to see your 'score' and then resubmit. That's just asking for people to cheat the system.
These plagarism checkers are also not really smart enough to differentiate proper citation from actual plagarism. Thier function is to alert the person grading the papers to sections that might be problematic. It will highlight "plagarized" sections, but the professor will ignore it if it's properly quoted and attributed or if it's just a short phrase, etc.
Quite often you can buy the solutions manual to a given textbook. They are on half.com or available in pdf format, a lot of the time.
They won't show you the work, so copying the answer is not sufficient to get credit in practically any course.
"Academic integrity" has ridiculous rules (eg self-plagiarism is not really a thing that anyone should care about) and I'm sure some people would be pissed about that, but if you're doing the work I personally don't see anything wrong with getting immediate feedback on whether you've done it correctly or not. Particularly given that many textbooks do like one or two simple examples and then jump to complex problems in the assignments... being able to check a couple problems to be sure you understand the process is extremely helpful as a learning aid.)
If you want a real-world assessment of whether someone has mastered a concept, schedule an in-class quiz or a test.
> self-plagiarism is not really a thing that anyone should care about
Sure it is. The entire methodology of schooling is to learn through exercise; if you are asked to write a paper, it isn't because the teacher wants the information in the paper. They are wanting you to learn through the act of writing the paper.
If you just turn in a paper you have written for another class, you are not doing that exercise.
Turning in a paper that you wrote for a different class and getting upset that the school doesn't approve is like thinking that sending a picture of you lifting weights 3 years ago is going to make you stronger.
All of this nonsense comes one root evil: giving publicly viewable grades for fundamentally in original work. Grades and homework should be for the student's private benefit. The only public marks should be proctored exams for demonstrating skills, and whatever original work someone creates that has value that stands on its own outside he artificial school environment.
That all depends on what you think school is for. I subscribe to the idea that school is for preparing people for society and provides a "great filter".
Doing a task your superior asks of you and doing it well is arguably the most important thing school teaches you, with time and stress management being a close second. This is what grades at school measures.
People love to talk about the virtue of being "well rounded" but the truth is that knowing bits of Shakespeare or knowing different types of Greek collumns doesn't benefit you as a person. All that information leaves your head pretty much immediately anyway. This is all especially true of high school. "Exams demonstrating skill" are not what matters to the people who look at your grades.
A proctored exam can only really measure a few types of skills, and don't work very well for demonstrating skills learned in many classes.
Take, for example, a class that is about learning to write effectively, which is clearly a valuable skill that adults need no matter what they do in life.
Part of that skill is being able to construct, from scratch, an effective piece of writing in a length of time that is longer than a proctored exam session. You can only measure that skill by having them do that, and if they turn in work they have written prior, you aren't learning anything about their ability to write something effectively on demand.
Research classes are similar; you can't demonstrate that you know how to do research well in a 3 hour exam. Programming classes, too; you can't measure the ability to create a large project in a 3 hour exam.
Self-plagiarism can also be an issue in the real world. Sure, I've cut and pasted an explanation of something into a different white paper or other docs. But there have certainly been times (don't remember the details) where journalists being paid to write a new article have taken a hit because they copied from an earlier piece. It's very situational but self-plagiarism can absolutely be a thing.
You can't just "rent" the software. They're going to try to verify that you work for an edu. That's way more hoops than even a semi-motivated plagiarizer is willing to jump through.
Men, i would con some Venture Capital guy into building this, by giving the ilusion somebody else has already built a version, but im just too lazy. Would hire some students, but those would just copy paste it together from stack overflow.
If you were plagiarizing you could compare your paper with your sources. Might not be a bad idea even if you aren't to make sure you didn't fuck up and cut and paste something from your sources.
I mean, yeah, there should be a real person in the loop. But the issues are arising from when these types of tools aren't being used properly.
It's not a problem when a professor is checking mid-terms from an upper-division class of 30. It is a problem when a grad-student with too little time is grading finals for a freshman class of 500.
Sure, the purveyors of these tools can say: "Use this thing properly! If you don't, bad things will happen." But time and time again, bad things keep happening when people don't use the tool right. After some amount of incidents, the purveyors of the tools should just re-design the dang thing [0] such that it's harder to use it wrong.
[0] Provided that the people that make the tool have any sense of morality and put human learning above dollars.
That's pretty much my experience too. A good paper will use lots of citations, and hence have lots of flags from these services (eg turnitin), but this doesn't mean plagiarism - it means the student's doing the right thing: gathering supporting (or negating) evidence and quoting it for the purposes of argumentation.
I think the problem is the testing methods they may use to verify the service.
They pass it the same text, it comes back as 100% plagiarized. Cool.
They pass it text that is close, but clearly just slightly modified original text. Comes back as plagiarized. Cool.
They pass it text that isn't close at all. Comes back as not plagiarized. Cool.
It works. Right?
Nope. We forgot to check if there are any cases where two different texts will show as being plagiarized despite knowing they've been independently created.
We also didn't check to see if a text that was essentially plagiarized can return a result of not plagiarized.
And I think those checks fall into the halting problem. You only need to find one case to throw the entire system into doubt. But to make sure there are no cases, you have to check every piece against every other piece.
And if you have false positives and false negatives, you can either double check every result. In which case, you might as well not use it. Or you give up on one the case of false negatives. If a plagiarized paper passes the system, oh well. Focus on checking every paper that gets tagged as plagiarized. Double check them and make sure it truly is plagiarized.
"And I think those checks fall into the halting problem."
Metaphorically, if you mean that there isn't going to be a concrete metric that really determines whether two bits of text are "similar" with bright shining line, sure.
If you mean halting problem literally, no, this isn't affected by the halting problem. String similarity evaluation won't be strong enough to invoke the properties that result in a halting problem. For any two strings, you can certainly terminate in a reasonably period of time, and have the "correct" answer as defined by whatever the local meaning of "answer" is. (Barring a particularly perverse implementation of a string similarity algorithm probably explicitly constructed just to screw with my statement here, that for some reason starts executing the string as instructions directly or something.)
Once in undergrad, I got an email from my professor one day instructing me to meet him in his office with all of my notes and materials for a research paper I had recently turned in.
When I arrived, his first words to me "I think we both know why you're here." I didn't.
He explained that he had run my paper through a plagiarism detector, and it had come back as a 100% match with a paper turned into a university in another state. I was rather dumbfounded because I had researched and written the paper entirely on my own.
We quickly came to an impasse, because he had faith in this rather damning score from the plagiarism detector, and I obviously was quite sure I had in fact written the paper myself.
As I left the office and began gearing up for a battle with the Office of Academic Integrity, I got another email from him profusely apologizing and telling me he had run the paper through the same plagiarism detector again and it had come back clean.
I'm really glad I didn't have to fight that fight, but I got awful close. My theory is that there was some problem during the upload, and it just checked an empty document against another empty document and found they matched exactly, but I'll never know.
One time in a take home programming interview problem I was asked to create my own quicksort implementation in python. I had already done this previously in answer to a StackOverflow question, so I just submitted a copy of that implementation.
Later I was told I was disqualified because I had cheated and copied my answer from StackOverflow. I tried to explain that the user I copied it from was indeed me and that I could prove it by logging into SO in front of them but I never heard back.
I wrote and published a book through Createspace and Kindle on Amazon a few years ago. I got an email from Amazon that the Kindle book was flagged as containing material from other sources. This was, in fact, the case given that parts of the book drew heavily on blogs I had published in a number of different places. I emailed back that I had the rights to the material and provided a bunch of links. I never heard back but they went ahead and published.
FFS how many different implementations of quicksort in Python can there possibly be? How would they expect you to make yours unique? Different variable names or something?
I'll never understand the concept of plagiarizing long-established solutions to problems.
It's ridiculous. If you're going to require a take home programming test, at least make it something that isn't solved immediately just by opening SO. Ask a cookie cutter question, get a cookie cutter answer.
Honestly, I think I'd be more impressed by the person who got the answer off of SO. At least that person isn't wasting their time trying to solve silly problems that have already been done.
Or Wikipedia. (OK, you'd have to turn the pseudocode into Python. I'm not even primarily a developer and that would take me about 10 minutes.) Or wherever. There are apparently a few slightly different implementation details to Quicksort but how unique can a Quicksort be?
> FFS how many different implementations of quicksort in Python can there possibly be? How would they expect you to make yours unique? Different variable names or something?
Not even different variable names, as any plagiarism detector which is halfway-competent would have to ignore them because changing variable names is one of the things any halfway-competent plagiarist would do. The arms race has gone to the point where it's impossible to detect mechanically whether an implementation of a standard algorithm has been plagiarized. It's probably impossible for a human to detect it without other context.
And that context comes down to "Has this student shown steady gains or are they suddenly handing in competent code after demonstrating complete incompetence?" which you can't know unless the class is structured such that students show what they know to the graders in person. Which requires a certain student-to-grader ratio. Which is expensive.
The problem is, it isn't at all unlikely that a quicksort implementation created entirely independently w/o plagiarizing will match an implementation somewhere online. When I took algorithms, I would normally do the assignment and compare my implementation with online sources, mostly to compare against. Aside from variable names, ~80% of those assignments matched one of the first few SO answers. I think the real fail here is a company assigning something like quicksort and expecting to get original answers.
> The problem is, it isn't at all unlikely that a quicksort implementation created entirely independently w/o plagiarizing will match an implementation somewhere online.
It's practically guaranteed. Quicksort is quicksort. You can't get creative with it without implementing something which isn't quicksort anymore. That code may well sort a list, but if the assignment is to implement quicksort, you've failed at the task.
> I think the real fail here is a company assigning something like quicksort and expecting to get original answers.
Right. You have to go to IOCCC levels of perversity to be able to turn in something which is quicksort but isn't going to trip a well-trained plagiarism detector.
I remember being asked how I'd go about implementing quicksort in a phone interview. My response was: I wouldn't. Sorting is a solved problem that's available in pretty much any language's standard library (every non assembly language I've used, at least). The interviewer chuckled and something to the effect of "good answer, then can you describe the algorithm to me?", to which I said sure, and did.
Thankfully, I went through college before these automated plagiarism detectors had caught on. They were around, but not as pervasive as today. I mean, if you're going to sort a vector in C++ for a trivial app, there's only so many ways to write:
And, lets face it: most lab assignments requiring a student to write a program are pretty small and trivial (at least mine were 20 years ago). The projects, on the other hand, were definitely more involved and you'd likely see more divergence in solutions there.
I've often re-implemented long established solutions, just to become intimate with the inner workings. Sort of like taking things apart and trying to put them back together as a child. While I may never use that exact knowledge, I may be able to apply the principles elsewhere. I imagine that is the perceived benefit to a student (of which I was never a very good one).
I don't want to work with someone who trusts their own quicksort implementation over one from an OSS library. It's a toxic combination of optimism and egotism that I don't have time for. What other stupid shit are they doing all day long? What does it say about my team if I'm on the asking end? Interviews are short, and information exchange is always limited.
Instead I may ask them to sort a complex object, or cook a piece of data that's organized opposite of what the UI will require. lots of people fail at these tasks. They're my version of fizzbuzz, but they actually represent the work we do.
Maybe modeling a many to one relationship inversely? Like if you want to show all the chips sold in a day but the data comes back with chips sold per customer and then you have to massage that to show how many chips were sold at a location.
Even the bugs, if there are any, are not likely to be original. That's why the idea of increasing software reliability by combining multiple independently-developed implementations of a set of requirements with a voting strategy is not as effective as it would appear under the assumption of uncorrelated errors.
Well, that is self plagiarism. Although I don't agree with the question or how that was handled, it is using your previous work without citing that you had previously written it.
I guess the corollary would be a baker asking a potential apprentice to bake a baguette and send a photograph. Sending a photo from your blog of a baguette you cooked a year ago might still be considered disingenuous, even if it shows capability.
Well, no, because a code snippet is the same every time you write it, whereas an apprentice may, even while following the same recipe, produce variations in the result due to the nature of baking.
Alas they'll attempt to justify it by saying they want your capability now, not X years ago. Maybe it's not entirely unjustified -- I have some old mathematical code on github that I don't think I could explain the technical details of without many hours of review, I wouldn't want to be hired to produce more code of that sort without an understanding that ramp up will be needed to return to my previous level. Given something like quicksort, though, I'd just get the feeling of "dance, monkey, dance".
I just learned this myself, but apparently if you don't cite yourself as the source of previously published information it's considered self-plagiarism.
Oh man. One of my biggest pet peeves is being accused of something I did not do. I would have probably gotten into trouble for heading back to that guy's office and ripping him a new *hole for presuming I am guilty based on a faulty software tool without first verifying the result manually himself. That's just absolutely ridiculous. I mean... that is a really serious accusation!
We trust our systems way too much, when we should first be trusting our own minds, and other people.
I had an interview about a decade ago, before doing a massive amount of prep on toy problems became standard for interviews candidates. The first question was to implement itoa, by far the easiest question of the day, and one presumably designed to warm the candidate up while also directing interviewers whether they should even bother with their harder questions.
I walked to the whiteboard and just wrote the implementation top to bottom without mistakes, then explained the code. I may have been a little lucky that I didn’t have any missteps, such as briefly forgetting to handle negatives or zero, or not immediately realizing that the first idea that pops into your head for pulling digits off will be a little awkward as you want to fill the string in the reverse order. But I’d never done itoa before so I didn’t think to offer any sort of disclaimer about why this was particularly easy, I was just happy with my performance and ready to do the next question.
The interviewers excused themselves and I saw them consulting the person who recommended me down the hall. I was surprised to learn that they were asking if it was possible I’d been supplied a list of interview questions and memorized the answers, to work out whether they even wanted to proceed from there.
I nearly had an incident in a MIPS assembly class. There was a fairly basic data structures assignment that I had submitted only to be called into the professor's office being accused of plagiarism. The program used aggregates all submissions over the lifetime of the class and compares them.
Because of how basic the assignment was, and how rigid assembly is, there's a baseline ~40% match with all assignments, and mine just happened to hit ~50% and that was grounds for plagiarism. The match was for some submission 6 years prior.
At the time I took backups of school projects on save and could replay the history and walk through my design which was enough to convince the professor. There were at least 3 other students roped into this and were probably not as lucky.
My university basically did a viva to get around this. They know they can't prevent you from copying the code, but they _can_ make sure you understand it well enough to walk through it and be able to explain why you did things, and ultimately that's more important.
They also biased _heavily_ in favour of exams (practicals are essentially something you had to do, but are ungraded), where there is no (reasonable) opportunity for plagiarism.
I studied electrical and computer engineering in college. One of the classes was a hardware/software codesign class. There was a graded lab component, i.e. We had to design and build a circuit for the given purpose, hook it up to a bare-bones computer w/ a 68K chip and write assembly to drive the circuit and pass a set of tests.
Each lab was typically booked as 2 3-hour sessions (one session per week). I lived off campus, and the commute was about an hour, in good traffic, so I wanted to minimize the number of times I had to go to campus.
Unlike the students that lived on campus, I was methodically prepared walking into the first session. I'd probably have spent 6-8 hours on a weekend working on the lab. I was meticulous. I designed the circuit using Visio (obviously can't simulate, but I was concerned with layout), so I had precise diagrams of the circuit.
My diagrams matched the layout of the bread board, and my physical implementation was clean (wires were always just as long as they needed to be and never longer). I could take the diagram and literally check off wires as I placed them (and my wiring was clean).
I had my 68K assembly written ahead of time and was generally bug-free, though some debugging was needed from time-to-time.
I was generally in and out of lab in 15 minutes of what was supposed to be a 6 hour session. After a few rounds of this, my TA suspected I might be cheating and he walked up, ripped out a bunch of random wires from my circuit and told me to fix it. I chuckled, said ok, and I pulled out my design, replaced the wires and checked them all off, plugged it back in and it passed. He was dumbfounded. He sort of had a "Wow, a student that's actually prepared for lab for once" kind of look on his face.
He never bothered me again.
I think the TA was doubly surprised because I also had a habit of obviously sleeping during lecture, dead center in the front row.
It sounds like that detector didn't reveal the source that was supposedly plagiarized. Who would be dumb enough to rely on the results of such a tool? Is that a common attribute of modern detectors?
Revealing a paper that another student had written, would violate that student's FERPA rights, and potentially lead to abuse. For instance, if you knew that your papers were all going to be gathered in a centralized database, would you share your innermost thoughts about any sensitive topic?
When I was in college, 1999-2003, they were explicit in that the institute assumed copyright ownership of all class work submitted be it an essay, lab write up, project submission, etc. It was a condition of attending.
That's why I'll never accept automated systems for determining things that need a human with context. "We used our black box and it says you're guilty." - what do you do against that? There's nothing to argue and inevitably you'll end up facing some tribunal where a group of people are going to decide your ultimate guilt. What if they use their own black box?
Yup, my university used Moss and caught a ring of cheaters with it. Funnily enough at my first job I actually worked on a product that was essentially Moss with a slicker UI, and it would also check online sources such as Github. It worked fairly well, essentially we highlight areas of code that look suspect and it is up to the professor to determine whether it is copied or not.
Haven't worked there in almost a year so I'm not sure if the product took off or not.
When teachers rely on the output of these detectors without checking the source material, they are effectively doing what they are accusing their students of doing with Wikipedia, etc.
The end result may be totally convincing, but nobody really knows until they actually look at what they think they are citing. The teachers should operate at least at the lowest level they impose on their students -- Check your work. Don't blindly copy someone else's results.
I once reviewed a journal article that at least a few pages of plagiarized text in it. I noticed only because I thought some sentences sounded like I wrote them, and as it turned out, I did write them. Not surprisingly, I found the paper even worse from a technical standpoint, so I gave it a rather negative review. I tried to be as helpful as possible, even suggesting what they should do instead of what they did. It appears the paper was rejected.
Fast forward 6 to 12 months. I found what seems to be a descendant of the paper published in another journal. While the vast majority of the plagiarism I identified was removed, they added some parts plagiarized from my review. I contacted the journal, who replied that plagiarism detection software didn't find anything. I replied that it wouldn't because a confidential review would not be in their system. Plus, I gave them a list of plagiarized parts; all they had to do was check against the paper. I told them that I did not spend much time checking for plagiarism, so for all I know there's a lot more plagiarism in the paper. The journal representative never did anything about the paper, unfortunately. I now refuse to publish in that journal and discourage others from publishing there as well.
I'm surprised that you didn't reject it for plagiarism. That's what I did when I discovered plagiarism while reviewing a paper. I wasn't an expert in some of the background material, and they referenced a survey paper on that topic, so I went and read that survey paper. To my surprise, the author had lifted entire paragraphs from the survey paper. I rejected it for that. I was actually a sub-reviewer, so I notified the PC member who had asked me to review the paper, and they agreed that yielded an automatic rejection.
Some of these tools use our company SerpApi.com as a data backend.
To be fair, they seem to be doing a good job and they are good a detecting copy/paste from online sources. But, of course, results are to be taken with a grain of salt.
As someone who teaches degree level content to apprentices, I definitely think these tools have value. I know that until very recently my current institution has not bothered with such a tool and I saw with my own eyes that quite obvious (to me) plagiarism was rampant and that the already overburdened assignment markers were not catching these instances. By moving to an automated tool, our students were immediately put on notice that such behaviour would not be acceptable.
That said, I do still hear worrying comments from my colleagues (“what number is too much?” is I’m afraid a common question) - but I can clearly see the change in student behaviour - so I don’t think this is such a bad thing. Yes, markers do still need to watch for plagiarism - and there are of course other types of plagiarism that these tools can’t usually deal with (ghost-written essays being a common talking point these days), but just because a tool may be misused or may be misunderstood, it certainly doesn’t necessarily mean it is all bad.
I don't have any real experience with these tools but for writing (as opposed to some types of code, especially well-defined algorithms), the odds that two pieces of writing are going to be highly similar to each other by chance are pretty tiny.
Sure, the same quotes from another source may be used here and there (and hopefully cited). And there may be some rote definition or other short text lifted from Wikipedia, etc. that maybe doesn't rise to the level of requiring a footnote. But if any sections of text longer than a line or two aren't substantially different, something is going on.
Why don't you skip all this nonsense and just give grades only for exams? This is how colleges were run before they turned into remedial high schools.
If your course needs a big final paper, you don't grade the paper, you grade a defense of the paper, that shows they really know the material behind the paper.
All the work before the final is just practice and grades don't matter.
Michael Crichton (somewhat famously) retyped and submitted a George Orwell essay for an undergrad literature course at Harvard. Orwell got a B- and Crichton decided lit wasn't for him and ended up in anthropology/pre-med.
This is really nothing new. "Ethical problems will never be resolved by a software tool" [1]
The author uses a straw man argument that plagiarism detection software promises to "replace the professor" when in fact, it is just a tool.
Besides, there are highly sophisticated ways of cheating that even a professor would not catch reliably. I wrote my undergrad thesis about plagiarism and it was interesting to learn that there are students who are highly motivated to "beat the system" instead of doing the task properly.
As a founder of an startup that uses AI to paraphrase text (https://quillbot.com), I find that this article is hitting a very valid point. As fluency enhancing tools become more prolific and higher quality, submissions will be more standard and thus there will be more unintentional overlap. Plagiarism detection software will become increasingly less reliable.
Sounds like good territory for implementing a GAN. You end up with one network that aggregates and summarizes, and another that penalizes for lack of originality. Though I suppose theres a good chance the GAN would drift away from human patterns as it trains.
What is sad is that this could be a great tool for students. It would be really nice to have something you can have it check your paper and give you warnings about missing citations etc. The ones I've seen you can only check your work after you submit it for grading. I'm not saying that people don't deliberately plagiarize, but it's also easy to make a mistake when you're under pressure.
Plagarism Detectors for Code, like MOSS, are a major issue. Take a simple getter/setter method. MOSS will mark these as duplicates unless you explicitly go in an say "no, its okay for THIS to be the same". This adds a degree of bias to what is justifiably "expected" and "unique" for student submissions. As someone that uses MOSS, I regularly look at any flags, "eyeball" them, and in only 2 instances did it result in a legitimate integrity violation. Not to mention that I have to hunt for all possible solutions across the internet and add them to the pile to ensure students didn't "copy it from a website".
I think current CS is doing itself a disservice because we want students to collaborate, but discourage them from even looking at each other's code (that is literally how students have spoken to me about it). I don't know if English or History students refuse to look over each other's work, but I doubt it.
Soloway described programmers as having "recurring basic patterns". If you accept his theories, we literally have tiny little rolodexes in our minds and when we encounter a problem we go "hang on, I've done this before" and pull out some code we've worked on in the past. We literally have tiny snippets of code we try to reuse if we can.
Instead, for well known algorithms, just give them the code! My research [1] require them to at least do it once, but either way - what good is it to have a student reinvent how A* search works? What about inserting into a Linked List? There are only a finite number of ways to implement these things. Instead of discouraging StackOverflow, and letting students try to sneak that type of work into their code, control it through the use of additional examples. They are less pressed to copy and paste code from a 3rd party if you've literally given it to them.
We want students to understand how to take something like a programming language and use it to solve a problem, but then we confound the entire process by not training them on the language. Even among SIGCSE, people can't agree what CS1 should be and somehow everyone wants it to be "more than just learning to program" like its a bad thing.
Back to plagiarism detectors for code, I think our current track discourages students to be open and help each other because they fear if they describe how they solved the exact same problem, there goes college. The result is encouraging the loner stereotype instead of reducing it. Hell, paired programming has to be forced on us because we can't "work with other people". I recognize some simply don't like it, however, my point is we've trained ourselves to not work together for so long that we designed a coding paradigm to have "controlled helping someone code".
There are just too many competing tugs. Something has to give. The problem of plagiarism goes away entirely if you make the course grade consist entirely of the final in-person exam. That doesn't mean you have to give up assigning homework and scoring it, but for mysterious reasons probably related in part to modern students and teachers not wanting to do anything unless forced to by the system colleges have moved away from the "final is everything" model.
This is something that I emphasized in my CS1 courses (I haven't taught them in a few years). Homework was 50% divided into 10% for Typing Exercises (my research), 10% for Lecture Exercises (MCQ, TF, Fill in the Blanks), and 30% Programming. Exams accounted for the other 50% (Midterm 20%, Final 30%). The big thing is that if you look at 20% of their grade was simply on drilling. Explicit cheating wouldn't help for them and they are just enough to still impact your grade if you don't do them.
Exams were in class, 2-3 hours long, and with a coding without Internet section. This approach does become more difficult with a 75 minute course (which is what NC State has), simply due to time constraints, so I can agree with you there.
However, your point about "Something has to give" is one of the reasons I argue that we expect too much out of CS1. In other technical skills, like cooking, music, martial arts, dance, etc., the introduction is much more about accumulating the technical skills necessary. Music theory isn't as prominent in an Intro to Singing class; however, in Intro to CS, I don't think we do that. As my course structure should indicate, I think intro courses should be more drilling to build the neuromuscular memory because then at higher levels we can do more critical thinking with the trust that students "know how to execute the plan". I do think that something like Problem Solving should be given its own separate course where its skills can be trained and honed parallel, but separate, from Programming.
This is funny because I've talked to multiple computer scientists who teach classes where all the programming projects are gated by a plagiarism detector and they all insist it has 100% TP and TN rate, 0% FP and FN rate.
I think it is all just a data grab. They HAVE to identify students on their computers, then link with lots and lots of other companies which tie everything together. Look at the turnitin privacy policy.
Honestly, I'm a little disappointed with this article. Plagiarism detectors themselves are neither a crutch nor a problem--they're simply a tool. People's use of them as the grand arbiter on plagiarism as though they were stone tablets delivered to them from Mount Sinai is the problem.
I thought that, despite the title, that's what the author was talking about for the majority of this article, but then they mention things like...
> Only if a text is somehow off, and online searching does not help, should software systems be consulted.
Sorry, what? The only reason text would sound weird is if it's been specifically mangled to defeat plagiarism software (which makes plagiarism software already useful). There's practically no way you, as a student marker, busy professor, or someone reviewing hundreds of academic proposals has the time to slowly wade through each paper you get by googling choice sentences manually--something the plagiarism checker software can do just fine. I'm more-so confused, by the pair of statements that 1) "Software cannot determine plagiarism [...] That decision must be taken by a person" but also 2) "Only if a text is somehow off, and online searching does not help, should software systems be consulted". Isn't the author's whole point that plagiarism software false flags all the time? Isn't this, then, just "hey this sentence sounds funny, time to fail this student on some plagiarism."
If their argument, however, is that you should use plagiarism software for curatable results, then this seems like the opposite order of what should be done. Why waste all your time fruitlessly finessing Google if the software will straight up just find the OG source for you? You're not going to have read every single piece of writing conceivably connected to the essay/etc. you're reviewing (unless it's a field you know much about and also so narrow that you'd never consider using plagiarism software in the first place), so you're bound to be missing actual plagiarism all the time.
The university where I'm studying for my MS uses such a system and, interestingly, has it integrated into their course management system such that I can actually see the reports on my own assignments. I've noticed that on a number of lengthy papers I've turned in it marks nothing at all, which is conspicuous considering that I tend to quote from other literature very heavily.
I'm not sure if the plagiarism service is just not very effective or if the fact that I turn in mostly LaTeX-generated PDFs is causing a problem with their text extraction. I actually suspect the latter, because a couple of times I've turned in MS Word files it's marked one or two random phrases as suspect.
I used to work in a helpdesk at a university. I don't know how many times I had to explain how to interpret the plagiarism scores they would get from the plagiarism program we had. Many times they wanted to trust the simple number in front of them and not do the leg work needed to interpret that score.
I've worked a bit in this space, porting Dr Bloomfield's WCopyfind [0] algo to js [1] for a small non profit (& other proprietary solutions). One of the harder forms of plagiarism over copy-paste issues is that of ideas. It's easy to cross reference phrases in text A to text B, but what if it's reworded in such a way that it's conceptually the same? In some areas of research this is a huge ongoing problem and although there are some small inroads being made, it's still a huge problem. It's also compounded by the issue of language translation. eg. text A (in English) was reworded in text B (in Chinese)
109 comments
[ 5.0 ms ] story [ 175 ms ] thread"Throw it all out" doesn't seem like a real solution to me, more of a flashy headline.
I don’t think these systems are nearly as egregious as the 5 paragraph essay scoring system they used around 2010 when I was in 9th grade back in 2009. I thought it was fine back then, but now I wonder how that thing had even the appearance of working some of the time.
I'd rather be the victim of a computer's bogus score than the victim of a computer's bogus accusation of wrong-doing. On the other hand it's important for everyone to experience the latter (by computer or not) at least once, and at least with plagiarism even if it's true you're not going to go to jail...
Keeping the algorithms secret for plagiarism detectors is about as dubious. If they are not good enough to reject gaming they are not robust enough to deliver reliable scores. If you don't trust students to see that gaming a plagiarism detector is more effort than honest work, you've admitted the wrong student, and/or you are using a duff plagiarism detector.
Teachers like them because they don't have to grade papers, textbook publishers like them because the online code prevents reselling the textbook (or at least, the new customer will need to purchase a key anyway).
That is, pardon my French, fucked up.
Beyond that it's egregious that any academic institution would allow the success of students/researchers/whatever to hinge on scores generated by a black box algorithm that hasn't been rigorously proven as very close to 100% accurate. There should always be a human in the loop, and if we don't have enough humans for that then I think we have a problem that goes beyond automated plagiarism checkers.
First of all, if your work is submitted through these detection systems, you almost never have the opportunity to see your 'score' and then resubmit. That's just asking for people to cheat the system.
These plagarism checkers are also not really smart enough to differentiate proper citation from actual plagarism. Thier function is to alert the person grading the papers to sections that might be problematic. It will highlight "plagarized" sections, but the professor will ignore it if it's properly quoted and attributed or if it's just a short phrase, etc.
They won't show you the work, so copying the answer is not sufficient to get credit in practically any course.
"Academic integrity" has ridiculous rules (eg self-plagiarism is not really a thing that anyone should care about) and I'm sure some people would be pissed about that, but if you're doing the work I personally don't see anything wrong with getting immediate feedback on whether you've done it correctly or not. Particularly given that many textbooks do like one or two simple examples and then jump to complex problems in the assignments... being able to check a couple problems to be sure you understand the process is extremely helpful as a learning aid.)
If you want a real-world assessment of whether someone has mastered a concept, schedule an in-class quiz or a test.
Sure it is. The entire methodology of schooling is to learn through exercise; if you are asked to write a paper, it isn't because the teacher wants the information in the paper. They are wanting you to learn through the act of writing the paper.
If you just turn in a paper you have written for another class, you are not doing that exercise.
Turning in a paper that you wrote for a different class and getting upset that the school doesn't approve is like thinking that sending a picture of you lifting weights 3 years ago is going to make you stronger.
Doing a task your superior asks of you and doing it well is arguably the most important thing school teaches you, with time and stress management being a close second. This is what grades at school measures.
People love to talk about the virtue of being "well rounded" but the truth is that knowing bits of Shakespeare or knowing different types of Greek collumns doesn't benefit you as a person. All that information leaves your head pretty much immediately anyway. This is all especially true of high school. "Exams demonstrating skill" are not what matters to the people who look at your grades.
Take, for example, a class that is about learning to write effectively, which is clearly a valuable skill that adults need no matter what they do in life.
Part of that skill is being able to construct, from scratch, an effective piece of writing in a length of time that is longer than a proctored exam session. You can only measure that skill by having them do that, and if they turn in work they have written prior, you aren't learning anything about their ability to write something effectively on demand.
Research classes are similar; you can't demonstrate that you know how to do research well in a 3 hour exam. Programming classes, too; you can't measure the ability to create a large project in a 3 hour exam.
And you did, only earlier.
"is like thinking"
Funny. Do you also think that writing exactly the same paper many times is making you learn something each time?
I mean, yeah, there should be a real person in the loop. But the issues are arising from when these types of tools aren't being used properly.
It's not a problem when a professor is checking mid-terms from an upper-division class of 30. It is a problem when a grad-student with too little time is grading finals for a freshman class of 500.
Sure, the purveyors of these tools can say: "Use this thing properly! If you don't, bad things will happen." But time and time again, bad things keep happening when people don't use the tool right. After some amount of incidents, the purveyors of the tools should just re-design the dang thing [0] such that it's harder to use it wrong.
[0] Provided that the people that make the tool have any sense of morality and put human learning above dollars.
They pass it the same text, it comes back as 100% plagiarized. Cool.
They pass it text that is close, but clearly just slightly modified original text. Comes back as plagiarized. Cool.
They pass it text that isn't close at all. Comes back as not plagiarized. Cool.
It works. Right?
Nope. We forgot to check if there are any cases where two different texts will show as being plagiarized despite knowing they've been independently created.
We also didn't check to see if a text that was essentially plagiarized can return a result of not plagiarized.
And I think those checks fall into the halting problem. You only need to find one case to throw the entire system into doubt. But to make sure there are no cases, you have to check every piece against every other piece.
And if you have false positives and false negatives, you can either double check every result. In which case, you might as well not use it. Or you give up on one the case of false negatives. If a plagiarized paper passes the system, oh well. Focus on checking every paper that gets tagged as plagiarized. Double check them and make sure it truly is plagiarized.
Metaphorically, if you mean that there isn't going to be a concrete metric that really determines whether two bits of text are "similar" with bright shining line, sure.
If you mean halting problem literally, no, this isn't affected by the halting problem. String similarity evaluation won't be strong enough to invoke the properties that result in a halting problem. For any two strings, you can certainly terminate in a reasonably period of time, and have the "correct" answer as defined by whatever the local meaning of "answer" is. (Barring a particularly perverse implementation of a string similarity algorithm probably explicitly constructed just to screw with my statement here, that for some reason starts executing the string as instructions directly or something.)
When I arrived, his first words to me "I think we both know why you're here." I didn't.
He explained that he had run my paper through a plagiarism detector, and it had come back as a 100% match with a paper turned into a university in another state. I was rather dumbfounded because I had researched and written the paper entirely on my own.
We quickly came to an impasse, because he had faith in this rather damning score from the plagiarism detector, and I obviously was quite sure I had in fact written the paper myself.
As I left the office and began gearing up for a battle with the Office of Academic Integrity, I got another email from him profusely apologizing and telling me he had run the paper through the same plagiarism detector again and it had come back clean.
I'm really glad I didn't have to fight that fight, but I got awful close. My theory is that there was some problem during the upload, and it just checked an empty document against another empty document and found they matched exactly, but I'll never know.
Later I was told I was disqualified because I had cheated and copied my answer from StackOverflow. I tried to explain that the user I copied it from was indeed me and that I could prove it by logging into SO in front of them but I never heard back.
I'll never understand the concept of plagiarizing long-established solutions to problems.
Not even different variable names, as any plagiarism detector which is halfway-competent would have to ignore them because changing variable names is one of the things any halfway-competent plagiarist would do. The arms race has gone to the point where it's impossible to detect mechanically whether an implementation of a standard algorithm has been plagiarized. It's probably impossible for a human to detect it without other context.
And that context comes down to "Has this student shown steady gains or are they suddenly handing in competent code after demonstrating complete incompetence?" which you can't know unless the class is structured such that students show what they know to the graders in person. Which requires a certain student-to-grader ratio. Which is expensive.
It's practically guaranteed. Quicksort is quicksort. You can't get creative with it without implementing something which isn't quicksort anymore. That code may well sort a list, but if the assignment is to implement quicksort, you've failed at the task.
> I think the real fail here is a company assigning something like quicksort and expecting to get original answers.
Right. You have to go to IOCCC levels of perversity to be able to turn in something which is quicksort but isn't going to trip a well-trained plagiarism detector.
Thankfully, I went through college before these automated plagiarism detectors had caught on. They were around, but not as pervasive as today. I mean, if you're going to sort a vector in C++ for a trivial app, there's only so many ways to write:
And, lets face it: most lab assignments requiring a student to write a program are pretty small and trivial (at least mine were 20 years ago). The projects, on the other hand, were definitely more involved and you'd likely see more divergence in solutions there.I don't want to work with someone who trusts their own quicksort implementation over one from an OSS library. It's a toxic combination of optimism and egotism that I don't have time for. What other stupid shit are they doing all day long? What does it say about my team if I'm on the asking end? Interviews are short, and information exchange is always limited.
Instead I may ask them to sort a complex object, or cook a piece of data that's organized opposite of what the UI will require. lots of people fail at these tasks. They're my version of fizzbuzz, but they actually represent the work we do.
What does this mean?
Basically, it's not necessarily OK to plagiarize yourself. Always disclose when you do it.
We trust our systems way too much, when we should first be trusting our own minds, and other people.
I walked to the whiteboard and just wrote the implementation top to bottom without mistakes, then explained the code. I may have been a little lucky that I didn’t have any missteps, such as briefly forgetting to handle negatives or zero, or not immediately realizing that the first idea that pops into your head for pulling digits off will be a little awkward as you want to fill the string in the reverse order. But I’d never done itoa before so I didn’t think to offer any sort of disclaimer about why this was particularly easy, I was just happy with my performance and ready to do the next question.
The interviewers excused themselves and I saw them consulting the person who recommended me down the hall. I was surprised to learn that they were asking if it was possible I’d been supplied a list of interview questions and memorized the answers, to work out whether they even wanted to proceed from there.
Because of how basic the assignment was, and how rigid assembly is, there's a baseline ~40% match with all assignments, and mine just happened to hit ~50% and that was grounds for plagiarism. The match was for some submission 6 years prior.
At the time I took backups of school projects on save and could replay the history and walk through my design which was enough to convince the professor. There were at least 3 other students roped into this and were probably not as lucky.
They also biased _heavily_ in favour of exams (practicals are essentially something you had to do, but are ungraded), where there is no (reasonable) opportunity for plagiarism.
Each lab was typically booked as 2 3-hour sessions (one session per week). I lived off campus, and the commute was about an hour, in good traffic, so I wanted to minimize the number of times I had to go to campus.
Unlike the students that lived on campus, I was methodically prepared walking into the first session. I'd probably have spent 6-8 hours on a weekend working on the lab. I was meticulous. I designed the circuit using Visio (obviously can't simulate, but I was concerned with layout), so I had precise diagrams of the circuit.
My diagrams matched the layout of the bread board, and my physical implementation was clean (wires were always just as long as they needed to be and never longer). I could take the diagram and literally check off wires as I placed them (and my wiring was clean).
I had my 68K assembly written ahead of time and was generally bug-free, though some debugging was needed from time-to-time.
I was generally in and out of lab in 15 minutes of what was supposed to be a 6 hour session. After a few rounds of this, my TA suspected I might be cheating and he walked up, ripped out a bunch of random wires from my circuit and told me to fix it. I chuckled, said ok, and I pulled out my design, replaced the wires and checked them all off, plugged it back in and it passed. He was dumbfounded. He sort of had a "Wow, a student that's actually prepared for lab for once" kind of look on his face.
He never bothered me again.
I think the TA was doubly surprised because I also had a habit of obviously sleeping during lecture, dead center in the front row.
What you described sounds like it might come in handy some time for me personally
IntelliJ has a feature called "Local History" which stores basically is storing undo history, but it stores other things too (like test execution).
https://www.jetbrains.com/help/idea/local-history.html
(I'm lucky enough to be old enough that this wasn't an issue. OTOH, we couldn't just Google or search Stack Overflow from source code examples...)
Clearly they hadn't read the Zen of Python.
Is this the norm?
[0] https://theory.stanford.edu/~aiken/moss/
Haven't worked there in almost a year so I'm not sure if the product took off or not.
The end result may be totally convincing, but nobody really knows until they actually look at what they think they are citing. The teachers should operate at least at the lowest level they impose on their students -- Check your work. Don't blindly copy someone else's results.
Fast forward 6 to 12 months. I found what seems to be a descendant of the paper published in another journal. While the vast majority of the plagiarism I identified was removed, they added some parts plagiarized from my review. I contacted the journal, who replied that plagiarism detection software didn't find anything. I replied that it wouldn't because a confidential review would not be in their system. Plus, I gave them a list of plagiarized parts; all they had to do was check against the paper. I told them that I did not spend much time checking for plagiarism, so for all I know there's a lot more plagiarism in the paper. The journal representative never did anything about the paper, unfortunately. I now refuse to publish in that journal and discourage others from publishing there as well.
I wonder if you could submit DMCA requests for this.
To be fair, they seem to be doing a good job and they are good a detecting copy/paste from online sources. But, of course, results are to be taken with a grain of salt.
That said, I do still hear worrying comments from my colleagues (“what number is too much?” is I’m afraid a common question) - but I can clearly see the change in student behaviour - so I don’t think this is such a bad thing. Yes, markers do still need to watch for plagiarism - and there are of course other types of plagiarism that these tools can’t usually deal with (ghost-written essays being a common talking point these days), but just because a tool may be misused or may be misunderstood, it certainly doesn’t necessarily mean it is all bad.
Sure, the same quotes from another source may be used here and there (and hopefully cited). And there may be some rote definition or other short text lifted from Wikipedia, etc. that maybe doesn't rise to the level of requiring a footnote. But if any sections of text longer than a line or two aren't substantially different, something is going on.
If your course needs a big final paper, you don't grade the paper, you grade a defense of the paper, that shows they really know the material behind the paper.
All the work before the final is just practice and grades don't matter.
Today he'd just be expelled!
The author uses a straw man argument that plagiarism detection software promises to "replace the professor" when in fact, it is just a tool.
Besides, there are highly sophisticated ways of cheating that even a professor would not catch reliably. I wrote my undergrad thesis about plagiarism and it was interesting to learn that there are students who are highly motivated to "beat the system" instead of doing the task properly.
[1] https://www.researchgate.net/publication/248606848_Support_f...
I think current CS is doing itself a disservice because we want students to collaborate, but discourage them from even looking at each other's code (that is literally how students have spoken to me about it). I don't know if English or History students refuse to look over each other's work, but I doubt it.
Soloway described programmers as having "recurring basic patterns". If you accept his theories, we literally have tiny little rolodexes in our minds and when we encounter a problem we go "hang on, I've done this before" and pull out some code we've worked on in the past. We literally have tiny snippets of code we try to reuse if we can.
Instead, for well known algorithms, just give them the code! My research [1] require them to at least do it once, but either way - what good is it to have a student reinvent how A* search works? What about inserting into a Linked List? There are only a finite number of ways to implement these things. Instead of discouraging StackOverflow, and letting students try to sneak that type of work into their code, control it through the use of additional examples. They are less pressed to copy and paste code from a 3rd party if you've literally given it to them.
We want students to understand how to take something like a programming language and use it to solve a problem, but then we confound the entire process by not training them on the language. Even among SIGCSE, people can't agree what CS1 should be and somehow everyone wants it to be "more than just learning to program" like its a bad thing.
Back to plagiarism detectors for code, I think our current track discourages students to be open and help each other because they fear if they describe how they solved the exact same problem, there goes college. The result is encouraging the loner stereotype instead of reducing it. Hell, paired programming has to be forced on us because we can't "work with other people". I recognize some simply don't like it, however, my point is we've trained ourselves to not work together for so long that we designed a coding paradigm to have "controlled helping someone code".
[1] https://research.csc.ncsu.edu/arglab/projects/exercises.html
[2] https://www.sciencedirect.com/science/article/pii/B978093461...
Exams were in class, 2-3 hours long, and with a coding without Internet section. This approach does become more difficult with a 75 minute course (which is what NC State has), simply due to time constraints, so I can agree with you there.
However, your point about "Something has to give" is one of the reasons I argue that we expect too much out of CS1. In other technical skills, like cooking, music, martial arts, dance, etc., the introduction is much more about accumulating the technical skills necessary. Music theory isn't as prominent in an Intro to Singing class; however, in Intro to CS, I don't think we do that. As my course structure should indicate, I think intro courses should be more drilling to build the neuromuscular memory because then at higher levels we can do more critical thinking with the trust that students "know how to execute the plan". I do think that something like Problem Solving should be given its own separate course where its skills can be trained and honed parallel, but separate, from Programming.
I thought that, despite the title, that's what the author was talking about for the majority of this article, but then they mention things like...
> Only if a text is somehow off, and online searching does not help, should software systems be consulted.
Sorry, what? The only reason text would sound weird is if it's been specifically mangled to defeat plagiarism software (which makes plagiarism software already useful). There's practically no way you, as a student marker, busy professor, or someone reviewing hundreds of academic proposals has the time to slowly wade through each paper you get by googling choice sentences manually--something the plagiarism checker software can do just fine. I'm more-so confused, by the pair of statements that 1) "Software cannot determine plagiarism [...] That decision must be taken by a person" but also 2) "Only if a text is somehow off, and online searching does not help, should software systems be consulted". Isn't the author's whole point that plagiarism software false flags all the time? Isn't this, then, just "hey this sentence sounds funny, time to fail this student on some plagiarism."
If their argument, however, is that you should use plagiarism software for curatable results, then this seems like the opposite order of what should be done. Why waste all your time fruitlessly finessing Google if the software will straight up just find the OG source for you? You're not going to have read every single piece of writing conceivably connected to the essay/etc. you're reviewing (unless it's a field you know much about and also so narrow that you'd never consider using plagiarism software in the first place), so you're bound to be missing actual plagiarism all the time.
I'm not sure if the plagiarism service is just not very effective or if the fact that I turn in mostly LaTeX-generated PDFs is causing a problem with their text extraction. I actually suspect the latter, because a couple of times I've turned in MS Word files it's marked one or two random phrases as suspect.
[0] https://plagiarism.bloomfieldmedia.com/software/wcopyfind/
[1] https://github.com/cmroanirgo/pl-copyfind/