One of the scariest and most depressing aspects when it comes to ChatGPT (so far) is that people already try to use it in such a way.
Same for people that are already talking about using it for medical diagnoses and such, without ever having seen a single proper independent and peer reviewed study when it comes to chatgpts accuracies in specific fields (and that is ignoring all the other potential short term and long term problems when it comes to even considering something like that).
On the bright side, dumb people have always been around. It's not like it's getting worse, in fact I would bet it's getting better. The meta-phenomenon is not something new here.
I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation.
And because GPTs and chatbots are built as very convincing bullshit generators, some percent of the time (majority of the time) they will say things that check out - and that provides enough fuel to start building false assumptions that everything else it says is true.
> I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation.
Which is always extremely confusing to me because I know people who do this also ;P.
> I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation.
It's interesting that OpenAI slow-rolled the release of some of their other projects due to a worry that they'd cause harm in the wrong hands, but seems to have just let ChatGPT rip. I don't think "makes it easy to pollute StackOverflow" rises to the level of a serious enough harm to justify keeping the tech locked away, but it's interesting that we're at least starting to get a glimpse of actual harms.
The article already states why this is such an issue: The results could be correct or incorrect (the model has no idea either way), but are delivered in the style of a well-researched answer. They can also be automated, so there's tons of these deceptive answers.
Right, it sounds like the scale is a particular problem, you'd expect the feedback mechanisms to gradually weed out nonsense answers, but if 90% of all answers were nonsense, humans couldn't keep up with.
I have trialled live coding with it and while amazing it suddenly generates half outputs and gets stuck. Right no, no, its swamping their volunteers, not suddenly making their service better.
I don't see it this way; Without curated information we wouldn't have a resource of valuable Q and A's for AI to base itself off of. I'm ignorant on what particular text the model was trained on but I have to assume it had a lot of details from online resources. It knows about coding languages and common problems so it has trained it's information form something. Without stack overflow or some kind of relationship between openAI ChatGPT suggested answers and confirmation that it was valid the information would becomes pointless for future frameworks since everyone would be stuck using ChatGPT for answers and it would have no place to find the answers. Maybe I'm missing something a context of some kind but I don't see it providing valid answers without the wealth knowledge like a place that stackoverflow can provide.
There's a big concern in the long run: if ChatGPT seriously reduces the number of people who visit StackExchange domains, there will be no dataset no for GPT 4 / ChatGPT 2. E.g. what if a brand new programming language gains popularity, or new libraries with very different patterns of use?
It's actually a separate concern from the spam / content bloat described in the linked post, but they complement each other in creating amplified harm for StackExchange-like platforms: some fraction of users may stop visiting the site (because ChatGPT answered without links) and some other fraction of users will submit spam to try and earn points.
Also, based on all the public info about InstructGPT (the closest ChatGPT "family member"), all of StackExchange is definitely in the training via OpenAI's "filtered Common Crawl", if it isn't also included as a special over-weighted training set (English Wikipedia, for instance, was over-weighted in GPT 3 training).
I've asked it to write me some Python code for the Redlock pattern [1] and whilst it generated text that explained it correctly the code wouldn't run - it assumed the class it needed was called `RedLock` when instead it is simply `Lock`.
I've also asked it how you can make a Flask application not write any cookies on a specific route. First it told me I can use the `Flask-NoCookies` package, gave me the `pip install flask-nocookies` command and showed me how to use its decorator. Great! Except there's no such package called `Flask-NoCookies`. It's just made it up.
I've just repeated that question now and it's told me I "can use the `@app.route()` decorator and specify the `no_cookie` option", again, this doesn't exist and it's just made it up.
There are also examples people have posted to Twitter where it's returned references to books, papers and URLs that it's entirely fabricated, too.
As opposed to the multitude of horrible or incorrect answers written by humans? I think anybody who uses this thing for a few minutes understands that it's not infallible.
I posted two answers to two different threads just to play with it. One got accepted, another didnt. I validated both answers before posting them. Answers were posted well before the ban on AI generated answers was even considered.
Nevertheless, Stackoverflow suspended my account for a week. Of course, it's only for a week, I'll survive and lesson learned, but at the time answer was posted, it did not violate any rules.
I've been messing around with it for a few days trying random interview style questions, and its almost always subtly wrong. Its still very cool but clearly not there yet.
I doubt this ban will succeed. Automation makes the production cost fall close to zero, while the detection cost remains relatibely high.
More ardent reputation based vetting could occur, but that would also impact genuine new contribitors and cut of the inflow of those while further boosting an already bolstered activist gatekeeper contingent on the platform scaring of well intentioned fresh posters.
33 comments
[ 2.7 ms ] story [ 85.8 ms ] threadSame for people that are already talking about using it for medical diagnoses and such, without ever having seen a single proper independent and peer reviewed study when it comes to chatgpts accuracies in specific fields (and that is ignoring all the other potential short term and long term problems when it comes to even considering something like that).
And because GPTs and chatbots are built as very convincing bullshit generators, some percent of the time (majority of the time) they will say things that check out - and that provides enough fuel to start building false assumptions that everything else it says is true.
Which is always extremely confusing to me because I know people who do this also ;P.
<cough> politicians <cough/>
Considering that, stack overflow is definitely one of the source materials for Gpt training, and probably a very significant one for like Codex.
If i ask on SO it’s because i want a human answer. What is the point of getting a Gpt one?
If I wanted the Got opinion, I can __just ask gpt__ (and btw I already do it always, now).
So the decision is very reasonable.. If anything, it’s sad that i cannot see any way in which it can actually be enforced
This "paradox of reuse" is a really big deal, IMO (blog post on the topic: https://nmvg.mataroa.blog/blog/the-paradox-of-reuse-language...)
It's actually a separate concern from the spam / content bloat described in the linked post, but they complement each other in creating amplified harm for StackExchange-like platforms: some fraction of users may stop visiting the site (because ChatGPT answered without links) and some other fraction of users will submit spam to try and earn points.
I've asked it to write me some Python code for the Redlock pattern [1] and whilst it generated text that explained it correctly the code wouldn't run - it assumed the class it needed was called `RedLock` when instead it is simply `Lock`.
I've also asked it how you can make a Flask application not write any cookies on a specific route. First it told me I can use the `Flask-NoCookies` package, gave me the `pip install flask-nocookies` command and showed me how to use its decorator. Great! Except there's no such package called `Flask-NoCookies`. It's just made it up.
I've just repeated that question now and it's told me I "can use the `@app.route()` decorator and specify the `no_cookie` option", again, this doesn't exist and it's just made it up.
There are also examples people have posted to Twitter where it's returned references to books, papers and URLs that it's entirely fabricated, too.
1 - https://redis.com/redis-best-practices/communication-pattern...
Nevertheless, Stackoverflow suspended my account for a week. Of course, it's only for a week, I'll survive and lesson learned, but at the time answer was posted, it did not violate any rules.
More ardent reputation based vetting could occur, but that would also impact genuine new contribitors and cut of the inflow of those while further boosting an already bolstered activist gatekeeper contingent on the platform scaring of well intentioned fresh posters.