worked about 20% of the time/content for me.
It does not seem to perform very well in languages other than english.
However there is a huge market potential for whoever figure this out. (Although I believe this will be nearly impossible)
This doesn’t work. Stack Exchange tried it, and it turns out there is a lot of misleading, superficial content that easily gets a high number of upvotes from newbies who don’t know better.
A nice approach - estimates the randomness in text. I tried with various cases - blog posts, wikipedia, scientific docs, and with a few examples from chat, and it figured it out accurately.
Respect. Even though I disagree with a need for such tools - it doesn’t matter if content was written by a human or by a machine. What matters is whether
it’s easy to read and worthwhile.
Yeah what we really need is a language model that can take some text and remove everything that is void of substance, and run both human and machine writing through it.
Disagree, this is very much needed. The problem with AI-generated content is that it looks superficially worthwhile and plausible, but in fact it often says things that are not correct.
> The problem with AI-generated content is that it looks superficially worthwhile and plausible, but in fact it often says things that are not correct.
This is also a problem with human-generated content.
Sort of. But if I read a five page article by a human, and everything on the first couple pages checks out as correct, I expect the rest of the article to be at least reasonable. If it’s written by AI, it’s entirely possible that it’ll start in on something that doesn’t even make sense. Then I’ll realize that I can’t even trust the part that seemed correct.
But that is true also of lots of content not generated by AI! Fact checking always needs to be done, AI generated or not. But does it matter that it was generated by AI?
What I find fascinating is that aspect of AI-generated content actually captures what humans do all the time, confidently making incorrect statements, "bullshitting" with filler text that only pretends to be meaningful, making illogical statements that contradict what was stated earlier, etc.
> it doesn’t matter if content was written by a human or by a machine
It matters a lot instead. On the top of my mind I can think about few reasons:
- if you've to train a NLP model you'd rather do with data that is not autogenerated as AI generated content is rarely used to train new models (like generating a bounce of dogs pic with dall-e and use as input for image detection may not create a precise model)
- if you pay content creators to generate content and they use ChatGPT that's definitely a breach on contract and also a problem
- many search engines (e.g. Google) already heavily penalize auto generated content
What about false positives? Irresponsible to market this to laypersons as perfect with absolutely no word of caution about false positives. There's been some chatter on Reddit that some content writers were falsely accused of using ChatGPT and lost their clients.
This is an everlasting battle. Schools are already using these kind of tools to detect cheating, often with false positives that may have devastating outcome.
It works both ways, but generation is advantaged in the long run. There has to actually be a statistical difference to detect, and AI outputs without statistical differences from human output are obviously possible, since humans make them all the time.
Sort off, I’m aware that in principle the generator has an advantage and eventually the detector will average out to a coin flip at best.
However some advantages can disappear when you put constraints on the output such as quality and correctness.
So whilst the end result might be less statistically significant in terms of was it human or AI generated it can overall be also less useful to the end user.
"However some advantages can disappear when you put constraints on the output such as quality and correctness."
Only if you suppose that the ideal output is superhuman. In the case of OpenAI et al, that's arguably the case, but those aren't the players that are going to get into an arms race with detection anyway. They want it to be relatively easy to detect AI generated content, because they're not in the plagiarism business, and anti-plagiarism measures will get the public and media off their backs. And nobody who is interested in targeting plagiarism has nearly the funding to build their own LLM on a level that matters.
So if there's an arms race in the near term, I expect it will be with postprocessors instead. These will be much smaller models (i.e. runs in your browser, or at least on a small backend machine) that take the output of ChatGPT and tweak it to fool detectors. They won't care about maximizing quality or accuracy, but will just care about preserving meaning while erasing statistical signs of AI generation.
I don't know if the business case for that will be there. It's there for selling papers, and almost certainly some people will try their hand at these models just for the challenge and/or to prove a point.
Im not sure if that how it actually would work out.
Most humans can’t write say an essay to save their life.
And those who do write very well tend to have their own signature.
Whilst it’s not 100% accurate we’ve managed to fairly successfully attribute a lot of unknown works to specific authors based on their known works.
So if you create a generator that produces output equals to say top 1% of human authors I’m not entirely sure that you can get one that doesn’t have its own signature.
Because whilst as you said most humans produce output that is statistically indistinguishable from most other humans the output that tends to survive selection bias and become known works is quite distinguishable by definition.
So you don’t even need to get to superhuman capability you just need to get to a high enough output quality that it would limit the statistical search space from billions to millions or even thousands.
This may be along the lines of what you’re suggesting, but what if you flipped this around: instead of trying to recognize AI, you recognize the student? You model each student’s quirks so you can tell if they wrote their essay, or if someone else did. Now you don’t care about AI specifically; you just care about whether they wrote what they submitted.
The main failure mode I see here is students dramatically improving and throwing the system off. If someone gets a tutor or goes to writing workshops, you don’t want to accuse them of plagiarism just because they got better. But there may be ways you could deal with that, like having the student submit new samples.
That could work but that is changing the problem and moving the goal posts, a plagiarism detection system that is essentially trained on individual authors would be able to identify any time they skew too far from their rolling average.
I’m not even sure if ML is absolutely necessary for this or not.
I tried with a few texts that ChatGPT generated for me and it says it was entirely generated by a human. It's true, I fixed some things here and there, for cohesion basically, but I think it's almost impossible for these tools to get it right.
Conversely, I wrote a few sentences in simple English off the top of my head (about The Sound of Silence, because they were talking about it on the radio this morning) and it said it was "likely to be written entirely by AI".
False positives in a plagiarism tool are pretty bad IMO. It should definitely skew towards "we can't be certain" rather than "definitely AI".
I can tell you that detecting AI plagiarism can be a challenging task, as AI models like ChatGPT are able to generate text that is similar to existing text. It is important to note that not all text generated by AI models is considered plagiarism, as it may be used in a legal, educational or research context. The best way to evaluate the accuracy of a plagiarism detection website would be to test it using a variety of inputs, and compare its results to those of other plagiarism detection tools.
The problem with such things is that you would need 100% accuracy in most cases for it to be useful.
For example many schools and universities fear that students use ChatGPT for homework. If such a plagiarism checker has false positive results in just a small percentage of cases, the consequences for honest students would be too severe to actually act on the results of the check. But perfect accuracy can never be reached for text classification, so those tool will never be that useful despite being interesting to AI people.
Not sure how a student could prove that the negative, apart from videotaping themselves writing the essay, or only being allowed type it on an airgapped machine owned by the university.
Even before GPT, schools should require students to submit derailed outlines and notes of their work. For one, students need to learn these skills. For two, it helps protect against more plagiarism and cheating.
How is that? Students already are required to submit a lot of evidence with their work such as research notes, plans, lab work etc. to prove they’ve actually done that and if plagiarism detection systems flag any of their work they have to defend it rather than the institution having to investigate and build evidence on its own outside of w/e shoddy plagiarism detection system they bought told them.
> The problem with such things is that you would need 100% accuracy in most cases for it to be useful.
Why? If there is 75% confidence a students report was generated using ChatGPT, then that’s enough to sit down with the student and discuss the content in person and see if they actually know it. A tool such as this could help the teacher having to avoid doing this with every student, and also reinforce to students that if they don’t actually know the material there’s still a chance they get caught.
> the consequences for honest students would be too severe to actually act
Only if acting is immediately accusing them of plagiarism rather than working with the student to ensure it’s really their work.
This is not what is going to happen tho. Flagged content will not be accepted and the student failed for the class, or worse for the year. Expulsion is on the card as the war against generated content becomes bitter.
This is such a short-sighted view. Students who wish to cheat using ChatGPT will immediately start running their text through these tools preemptively, changing some phrases here and there and add spelling errors to ensure they don't get caught. You're left with innocent students being accused of cheating on regular basis.
Maybe the education system will finally learn that asking students to merely recite information that can be found by anyone, anywhere, in less than 10 seconds isn't helpful in judging their understanding of the subject, except in very limited scenarios.
> changing some phrases here and there and add spelling errors to ensure they don't get caught.
The students who will do this already do this though, just with different source material I suppose.
> Maybe the education system will finally learn that asking students to merely recite information that can be found by anyone, anywhere
I think we’ll see an increased amount of face to face assignments like exams or essays written in class only, where students can’t use these tools. It solves the problem entirely and is arguably better - but increases education cost.
This seems to be a reductive response to a confidence based policy approach. There are many systems out there today that provide a confidence score of bots vs humans. I happen to work with a product that uses this exact approach and it's highly effective. So with education scenarios I think that the teacher/professor/administration will use these systems to inspect the entirety of the submissions. From there a baseline value will be derived and from there outliers will appear that may require deeper analysis/interview. Schools are in the position that they can't get it wrong more often than not (they need tuition dollars after all), but a severe detractor will need to be hanging over the heads of students to avoid cheating with it.
I think the other thing we'll see is a nanny state approach by some educational institutions. Falling in a trap of being sold on software to "block" or "monitor" students using these tools. It would be easy to implement on a campus network to a certain extent (correlating student network login to URL accessed and potentially MitM) but the reality is smart students will know better and will use phone hotspots and VPN. The other dark side to consider is that the owners of ChatGPT could provide logs of user accounts and queries to higher education as a service.
At the end of the day my guess is all approaches are going to be tested at some level. But the cat is out of the bag and this is going to generate some very interesting countermeasure solutions/approaches along the way.
At the university level, the problem of cheating is more a problem with our idea of a university. In an ideal world, you go to a university to learn. If someone wants to cheat, ultimately they're just cheating themselves (and, if at a private university or even most public universities, throwing a lot of money away). "Cheating" isn't really the university's problem in that view.
Ofc, in reality universities aren't (or not only, and in most cases not primarily) about learning, but about credentialism. Employers and various social systems outsource to universities the role of verifying that people actually know something, or are the "right" sort of person, or other signals.
I don't know how one goes about fixing that, or if it's even possible, but I'd like to see more acknowledgement of it. Fixing "cheating" feels like the equivalent of looking to clever programming to fix product bugs.
You don't really need any radical changes, you just have to get rid of the idea of graded homework, which I always hated from elementary school onwards.
Larger projects have a role to play in education, but ultimately if you can't pass the proctored exams, you don't pass the course.
Grades are also needed in university to check for prerequisite knowledge for courses. You waste everybody's time and resources if you let people into courses who think or pretend that they have the prerequisite knowledge. People who are cheating aren't only cheating themselves.
I tried this on my recent AI satire article. While ChatGPT detected my story as AI (because I wrote it like a machine), GPTZero gladly says it's human written.
I don’t know which is why this is completely up for debate, Office also offers grammatical corrections and you have websites like Grammarly that do the same often without any “AI” as far as I know.
Like with anything else this is something that would require society to come to a consensus.
And there are various issues with AI assisted or generated content depending on the context.
For example educational institutions may see this as cheating whilst corporate entities might fear legal challenges in relation to copyright and IP.
ChatGPT and other AIs are theoretical machines built on human data.
Can an algorithm write an article about the 2008 financial crisis? Of course! it has read God knows how many Wikipedia articles, books and online discussions about it. But everything the AI knows is because some humans put in the work and documented everything.
If we don't know something, be it a historical fact or a scientific model, we have the ability to go out in the real world and record the data. Can ChatGPT fly to Yemen and interview refugees? Can ChatGPT tear apart the circuitry of a home appliance, reverse engineer it and write a blog post about it? Can ChatGPT go to a lab and make chemical experiments to create new compounds?
No, it can only regurgitate what it knows and even what it doesn't know.
ChatGPT will absolutely steal the job of all the "journalists" that just regurgitate what others have already written. Original research and field reporting will be left to the real journalists for God knows how many decades. If your job consists of actually making new things or collecting real-world data then I think it will be safe for the decades to come.
I tried it on a section of an article I wrote with the help of ChatGPT. It marked some of my sentences as generated by AI while it didn't mark the ones coming from ChatGPT.
Interesting. It only seems to work in English. English texts that it successfully detects as AI generated are classified as human generated if you ask ChatGPT to translate them into a different language (or make the initial prompt in a different language). Perhaps they want to clarify this limitation, either through an FAQ or by stating that it doesn't support it when other languages are supplied in the prompt.
The accuracy just isn’t there, which isn’t surprising. I’m feeding it Star Wars reviews from both ChatGPT and IMDb, and for sure the answers are correlated. The false negative rate is OK, although I’ve hit plenty of them. But man, it sure does think a lot of people were using ChatGPT to write reviews back in the 2000s.
Again, it’s definitely correlated, even strongly correlated, but that’s not good enough for plagiarism detection. You can’t go accusing students of academic dishonesty based on a tool that gets it wrong multiple times on a couple dozen samples.
I thought to make something similar but what scared me away was the idea that some educational institution could use it and decide a student was being dishonest due to a false positive. I wouldn't want to bear responsibility for something like that.
And secondly, this is a never-ending race. Even if it were to be able to detect ChatGPT content with 100% accuracy today, it would just be used to assist in training another model to defeat it.
What I think would be interesting is to add the assignment / questions as input for the tool to consider in its evaluation. That way your tool could plug these into ChatGPT, ask it to reword it several ways (mimicking the likely path a plagiarizer might take to actually generate content) and do some diff/comparison as part of the evaluation.
Even so, I agree with most of the comments that it would be exceeding difficult to truly identify this stuff since you can ask ChatGPT to reword things in very specific ways or just edit it yourself (aside from blatant false positives)
108 comments
[ 2.6 ms ] story [ 147 ms ] threadThere may also be escalating social and perhaps legal penalties too.
And that is without even considering bots.
Respect. Even though I disagree with a need for such tools - it doesn’t matter if content was written by a human or by a machine. What matters is whether it’s easy to read and worthwhile.
This is also a problem with human-generated content.
If you dig well, there are people who do really good content out there
“Need” may be arguable, but why disagree?
It matters a lot instead. On the top of my mind I can think about few reasons:
- if you've to train a NLP model you'd rather do with data that is not autogenerated as AI generated content is rarely used to train new models (like generating a bounce of dogs pic with dall-e and use as input for image detection may not create a precise model)
- if you pay content creators to generate content and they use ChatGPT that's definitely a breach on contract and also a problem
- many search engines (e.g. Google) already heavily penalize auto generated content
- avoid cheating at exams / officials test/certs
However some advantages can disappear when you put constraints on the output such as quality and correctness.
So whilst the end result might be less statistically significant in terms of was it human or AI generated it can overall be also less useful to the end user.
Only if you suppose that the ideal output is superhuman. In the case of OpenAI et al, that's arguably the case, but those aren't the players that are going to get into an arms race with detection anyway. They want it to be relatively easy to detect AI generated content, because they're not in the plagiarism business, and anti-plagiarism measures will get the public and media off their backs. And nobody who is interested in targeting plagiarism has nearly the funding to build their own LLM on a level that matters.
So if there's an arms race in the near term, I expect it will be with postprocessors instead. These will be much smaller models (i.e. runs in your browser, or at least on a small backend machine) that take the output of ChatGPT and tweak it to fool detectors. They won't care about maximizing quality or accuracy, but will just care about preserving meaning while erasing statistical signs of AI generation.
I don't know if the business case for that will be there. It's there for selling papers, and almost certainly some people will try their hand at these models just for the challenge and/or to prove a point.
Most humans can’t write say an essay to save their life.
And those who do write very well tend to have their own signature.
Whilst it’s not 100% accurate we’ve managed to fairly successfully attribute a lot of unknown works to specific authors based on their known works.
So if you create a generator that produces output equals to say top 1% of human authors I’m not entirely sure that you can get one that doesn’t have its own signature.
Because whilst as you said most humans produce output that is statistically indistinguishable from most other humans the output that tends to survive selection bias and become known works is quite distinguishable by definition.
So you don’t even need to get to superhuman capability you just need to get to a high enough output quality that it would limit the statistical search space from billions to millions or even thousands.
The main failure mode I see here is students dramatically improving and throwing the system off. If someone gets a tutor or goes to writing workshops, you don’t want to accuse them of plagiarism just because they got better. But there may be ways you could deal with that, like having the student submit new samples.
I’m not even sure if ML is absolutely necessary for this or not.
False positives in a plagiarism tool are pretty bad IMO. It should definitely skew towards "we can't be certain" rather than "definitely AI".
Sorry.
Your text is likely to be written entirely by AI
False positives and all.
Not sure how a student could prove that the negative, apart from videotaping themselves writing the essay, or only being allowed type it on an airgapped machine owned by the university.
Doesn't feel very feasible to implement.
Why? If there is 75% confidence a students report was generated using ChatGPT, then that’s enough to sit down with the student and discuss the content in person and see if they actually know it. A tool such as this could help the teacher having to avoid doing this with every student, and also reinforce to students that if they don’t actually know the material there’s still a chance they get caught.
> the consequences for honest students would be too severe to actually act
Only if acting is immediately accusing them of plagiarism rather than working with the student to ensure it’s really their work.
https://people.com/human-interest/twins-defamation-lawsuit-m...
Maybe the education system will finally learn that asking students to merely recite information that can be found by anyone, anywhere, in less than 10 seconds isn't helpful in judging their understanding of the subject, except in very limited scenarios.
The students who will do this already do this though, just with different source material I suppose.
> Maybe the education system will finally learn that asking students to merely recite information that can be found by anyone, anywhere
I think we’ll see an increased amount of face to face assignments like exams or essays written in class only, where students can’t use these tools. It solves the problem entirely and is arguably better - but increases education cost.
I think the other thing we'll see is a nanny state approach by some educational institutions. Falling in a trap of being sold on software to "block" or "monitor" students using these tools. It would be easy to implement on a campus network to a certain extent (correlating student network login to URL accessed and potentially MitM) but the reality is smart students will know better and will use phone hotspots and VPN. The other dark side to consider is that the owners of ChatGPT could provide logs of user accounts and queries to higher education as a service.
At the end of the day my guess is all approaches are going to be tested at some level. But the cat is out of the bag and this is going to generate some very interesting countermeasure solutions/approaches along the way.
Ofc, in reality universities aren't (or not only, and in most cases not primarily) about learning, but about credentialism. Employers and various social systems outsource to universities the role of verifying that people actually know something, or are the "right" sort of person, or other signals.
I don't know how one goes about fixing that, or if it's even possible, but I'd like to see more acknowledgement of it. Fixing "cheating" feels like the equivalent of looking to clever programming to fix product bugs.
It's like expecting a guarantee from the church that all their exorcisms render the host demon-free.
A degree is an artifact of a belief system. That there are credentialed idiots among us means the belief system itself is flawed.
Larger projects have a role to play in education, but ultimately if you can't pass the proctored exams, you don't pass the course.
ChatGPT's response: https://imgur.com/a/u7iBBaX
GPTZero's response: https://imgur.com/a/89icz2X
To be clear, my story was ChatGPT-assisted. But I wonder why ChatGPT couldn't detect it correctly like GPTZero?
Here's the link to my non-paywall story for references:
https://medium.com/humor-bytes/i-your-nba-highlights-broadca...
On the topic of assistance, where do detectors draw the line between AI-augmented, aided, and generated writing?
Like with anything else this is something that would require society to come to a consensus.
And there are various issues with AI assisted or generated content depending on the context. For example educational institutions may see this as cheating whilst corporate entities might fear legal challenges in relation to copyright and IP.
Can an algorithm write an article about the 2008 financial crisis? Of course! it has read God knows how many Wikipedia articles, books and online discussions about it. But everything the AI knows is because some humans put in the work and documented everything.
If we don't know something, be it a historical fact or a scientific model, we have the ability to go out in the real world and record the data. Can ChatGPT fly to Yemen and interview refugees? Can ChatGPT tear apart the circuitry of a home appliance, reverse engineer it and write a blog post about it? Can ChatGPT go to a lab and make chemical experiments to create new compounds?
No, it can only regurgitate what it knows and even what it doesn't know.
ChatGPT will absolutely steal the job of all the "journalists" that just regurgitate what others have already written. Original research and field reporting will be left to the real journalists for God knows how many decades. If your job consists of actually making new things or collecting real-world data then I think it will be safe for the decades to come.
Again, it’s definitely correlated, even strongly correlated, but that’s not good enough for plagiarism detection. You can’t go accusing students of academic dishonesty based on a tool that gets it wrong multiple times on a couple dozen samples.
https://news.ycombinator.com/item?id=34557189
The start and end were apparently written by AI.
What does this mean? Is this proof we're in a simulation and I'm actually a mouth piece for an AI that's reached the singularity?
Would be interesting how much they actually catch.
And secondly, this is a never-ending race. Even if it were to be able to detect ChatGPT content with 100% accuracy today, it would just be used to assist in training another model to defeat it.
Even so, I agree with most of the comments that it would be exceeding difficult to truly identify this stuff since you can ask ChatGPT to reword things in very specific ways or just edit it yourself (aside from blatant false positives)