Its insane how they think they can have their AGI and eat it too! The closest thing we have to AGI is like a human child even though that's Synthetic (mankind made) General Intelligence and they can't really be truly relied upon and not learn or formulate sentences you don't like
"The authors asked GPT-4 ADA to create a data set concerning people with an eye condition called keratoconus"
That had me confused for a moment, since there's no GPT-4 model called Ada (the current embeddings model is called that, and there was a GPT-3 LLM model with that name too).
Then I realized they were using ADA as an acronym for Advanced Data Analysis.
For anyone jumping in who hasn't read the article, this is not about hallucinations and some researchers being surprised that GPT-4 doesn't always respond with absolute truth.
Instead it's about how easily it can be used to generate plausible looking datasets that would confirm a hypothesis. It's a warning note to journals about how fake data can more easily be created.
Exactly. It's no different from the fake data sets created by hand in scientific misconduct cases for years, just I guess easier. I guess that's not a good thing, but given even making a fake data set by hand is far easier than generating real data, I'm not sure if this will suddenly make more people fake data.
The volume of bogus research is already growing non-linearly. It suggests that there is a market for fake datasets, which will lead to better AI training to fix this problem.
Having spent significant time implementing machine learning papers before the LLM age, I can promise you over 90% of papers you'll find are full of shit. The claims they make are true in only the most contrived of circumstances and don't hold up under any kind of scrutiny. How exactly they came to these lies (data lies, result lies, omitting lies) is really immaterial. The concept everyone is apparently struggling with is that producing a paper that is entirely lies is not doing the scientific world a disservice: It is not unusual and already happens at scale. Making it even easier might motivate someone to actually figure out a way to ensure papers are reproducible and not full of shit. In essence this is a good thing.
Easier cheating tools allow more people to cheat. There is no particular reason it would lead to less fake data, and with the ability to get fake data in a few sentences rather than a few hours of work is apt to lead more people astray.
This is not particularly surprising, nor is it really bad - generating data that appears valid is really powerful. Although it was a bit funny since I happen to have keratoconus, and I don't hear much about it often.
>“It will make it very easy for any researcher or group of researchers to create fake measurements on non-existent patients, fake answers to questionnaires or to generate a large data set on animal experiments.”
Perhaps I'm naive, but I think the people that want to fake data were already doing it without tools like chatgpt. Especially since a ton of biological data is normally distributed, so it's exceedingly easy to generate plausible fake results for such data without a system as advanced as chatgpt
>You have already gotten the basics of faking data incorrect.
Let's get past the unnecessary antagonism. Can you please explain the above?
Or actually, forget it. You history suggests you don't respond and something like 99/100 of your comments are dead. Maybe this isn't the place for you?
No one believed those who said "make it idiot proof and they'll just make a better idiot", but we should have heeded this warning. They did it! They finally did it! They made a better idiot!
A null result, or the absence of evidence, is not evidence of absence. If you fake a null result, you’re not asserting anything other than you could not measure and collect supporting data using your experiment to prove or disprove a hypothesis. It is difficult for someone doing replication to accuse you of ill-intent, as opposed to faked data that proves your point when anyone else can replicate your experiment and get totally different or even contradictory results.
You still have to give out the statistics that show the null result. E.g. something with a high p-value. You are in fact "confirming the null hypothesis". They aren't any more difficult to replicate than results supporting "the alternative hypothesis".
(The whole binary hypothesis system and culture is a mess though, but that's besides the point.)
However, I think that no one will do the scut work necessary to find that a null result was faked, and even if they do since you the researcher got very little status out of it then it’s believable that you made a mistake, and didn’t falsify data.
GPT definitely wins out if you want more novelty/variety in your fake data and are willing to accept extraordinarily higher cost, less rigor, and less reliability. I'm sure there's some occasion when those criteria win out, but Faker's pretty decent most of the time.
I'm not sure this is much better than the state of the art. Training a model on data and then having it generate new, fake data, is not only easy, it's a standard tool for model boosting.
I wouldn't immediately call creating synthetic data 'poisoning the well' unless it is actually distributed as such. For training models with a minimal amount of quality data, it is a viable method for generating more data to increase the quality of the models. But any legit organization will obviously label synthetic data as such.
Sure, but what do you expect? It's like blaming Boeing for genocide and then trying to get rid of 747s.
There's plenty of research into AI safety. There were some damn coups going on over AI safety. The general public defines AI safety as Skynet and homemade bombs, but it's also things like this - political manipulation, astroturfing, fake data, the risk of another industrial revolution.
It's something we should be slamming the brakes on, but most of the people calling out AI safety are also building their own B52 bombers, so nobody takes them seriously either.
80000 Hours has been telling people to get into AI and nuclear policy for years now. Hopefully we have some competent people in govs who do something.
This is so worrying for me, the amount of digital garbage that can now be generated that is not obviously garbage, that one must read and discern first nonsensical bad text generation and second if factual, is truth becoming a needle in the hay stack? How can this be cleaned?
The issue is that we consider passing peer review to be enough for something to be taken as truth when it should really be reproducibility/replicability. That, and the incentives currently driving academia are absolutely ridiculous. Paper mills are merely a symptom of academia's ills, not the cause of them.
Cases of fabrication have been caught because the fabrication is done so badly, i.e. inplausibly. It's often just imputing some "random" numbers or repeating samples etc. Many would be amazed how technically and mathematically illiterate scientists often are. And probably ones that fabricate even more so.
Maybe it will increase and/or get a bit higher quality with LLM fakery. But as with many "AI bad" themes, the problem isn't that "AI" can fabricate the data. The problem is fucked up institutions and cultures.
Roughly speaking, the whole point of an LLM is to create plausible sounding text, without regard to the truth (which it cannot determine or derive), so it's a tool that is perfectly suited to this sort of malfeasance.
I’ve read this over and over, and believe it - but it’s so easy to forget when it absolutely nails something like debugging or suggesting a CMake edit or finding 5 letter combinations that are reversible with a vowel in the middle, or etc.
It’s just still mind blowing I can get a sarcastic summary of an email in the theme of GlaDOS from Portal and in the same screen get an email proofread.
It’s funny how far “what should the next word be” can go.
It might seem that there are cases where the ability to generate heaps of plausibly sounding text on demand is helpful, and there are cases where it is pretty much the worst possible capability to have.
i don't agree. The way that most LLMs currently generate response might be through this modality. But the purpose of LLMs is to get closer to simulating how human minds and languages operate. A lot of models have been trying to overcome the problem of fabrications. GPT bots like SciSpace's ResearchGPT are trying to do precisely that.
If Arxiv was used in training, some of those scientists were the ones that taught it to do this. Well, a good chunk of whatever they scraped off the Internet, too.
Ironically, I tried to ask ChatGPT to generate a signal + some plausibly background for a potential Supersymmetry particle discovery (with some details about model independent searches) and it started hallucinating like someone whose heart broken and spent the night drinking at the bar.
I chose the wrong field to be able to fake data /S.
A long time ago, I ran into the books The Art of Deception and How to Lie With Statistics. They seemed like a good start on training people to spot deceptions. There were articles here on things like p-hacking. Then, the replication crisis.
While no time now, I’m still interested in making a list of resources (esp free) that tells how to construct good studies, has comprehensive presentation of all categories of mistakes/lies we see in them, examples of each, and practice studies with known errors. Anyone here got good books or URL’s that could go in a resource like that? That could train new reviewers quickly?
If I return to AI or ever work in it, I also planned to teach all of that to AI models to automatically review scientific papers. Might contribute to solving the replication crisis. Anyone who’s doing AI now feel free to jump on that. Get a startup or Ph.D. with a tool that tells us which of the rest are fake.
Somewhere in the mind's eye of science fiction is a world where we have near-unlimited productivity and knowledge and we're all free to pursue our self-actualized lives as we see fit.
But as productivity increases, and as AI improves, in both cases individual greed holds back lifting up the many. And so we end up asking for caution on automating away someone's job, or caution on rapid AI progress.
Does anyone know any good writing on how humanity might fight its way through all these mires of progress to the other side - that science fiction world that may or may not even be possible? How the world might look as these things continue to progress over time? Either fiction or a serious analysis is fine.
Most sci-fi skips straight to "There is no more need for University, we simply ask the AI", missing the "students are using AI to cheat" phase entirely.
The world of Star Trek is one where humanity learns from the devastation of world war three and over a century, creates a society where almost everything is run by computers, money doesn't exist, and people work primarily for their personal satisfaction. But I think true AGI is frowned upon there.
Firm disagree! Example: take a tour of the tower of London and learn about all the nasty medieval tortures we used to inflict on people. Now we don't do that anymore.
Humans learn things collectively via culture and cultural transmission has been an extremely effective tool of knowledge preservation over the generations.
This has been the popular narrative in almost all common written history forever. It’s basically the biggest recurring theme in diaries from the Middle Ages, especially among religious text.
Things are never an upward hockey stick but they also aren’t saw waves skirting a baseline.
Who is "we"? Torturing people to death in equally horrific ways is still routine practice among radical Islamists and Mexican drug cartels, among others.
Yes, but my point us that repeating "humanity learns" can lead us down garden paths into thinking there is some species-level recollection, when history reveals a mixed bag at best.
Completely agree transmission of knowledge is a sticky wicket. Good reminder to stay logistically grounded. I have found many of the ambitious thinkers close to me tend to aspire to a "humanity learns" moment but if they're anywhere near politics they tend to be tempered fairly well by the logistical realities of bringing ideas to pass
158 comments
[ 2.9 ms ] story [ 198 ms ] threadSarcasm aside, once such systems really learn to lie, they will be all too human-like. Perhaps the defining quality of real intelligence is deception.
That had me confused for a moment, since there's no GPT-4 model called Ada (the current embeddings model is called that, and there was a GPT-3 LLM model with that name too).
Then I realized they were using ADA as an acronym for Advanced Data Analysis.
Instead it's about how easily it can be used to generate plausible looking datasets that would confirm a hypothesis. It's a warning note to journals about how fake data can more easily be created.
I think the people that are good at faking data simply don't get caught.
Attacks only ever get better, not worse.
Perhaps I'm naive, but I think the people that want to fake data were already doing it without tools like chatgpt. Especially since a ton of biological data is normally distributed, so it's exceedingly easy to generate plausible fake results for such data without a system as advanced as chatgpt
Let's get past the unnecessary antagonism. Can you please explain the above?
Or actually, forget it. You history suggests you don't respond and something like 99/100 of your comments are dead. Maybe this isn't the place for you?
No one believed those who said "make it idiot proof and they'll just make a better idiot", but we should have heeded this warning. They did it! They finally did it! They made a better idiot!
(The whole binary hypothesis system and culture is a mess though, but that's besides the point.)
However, I think that no one will do the scut work necessary to find that a null result was faked, and even if they do since you the researcher got very little status out of it then it’s believable that you made a mistake, and didn’t falsify data.
GPT definitely wins out if you want more novelty/variety in your fake data and are willing to accept extraordinarily higher cost, less rigor, and less reliability. I'm sure there's some occasion when those criteria win out, but Faker's pretty decent most of the time.
It is harder to keep bad data out than it is to keep it filled with good data.
https://en.m.wikipedia.org/wiki/Brandolini%27s_law#:~:text=B....
There's plenty of research into AI safety. There were some damn coups going on over AI safety. The general public defines AI safety as Skynet and homemade bombs, but it's also things like this - political manipulation, astroturfing, fake data, the risk of another industrial revolution.
It's something we should be slamming the brakes on, but most of the people calling out AI safety are also building their own B52 bombers, so nobody takes them seriously either.
80000 Hours has been telling people to get into AI and nuclear policy for years now. Hopefully we have some competent people in govs who do something.
Maybe it will increase and/or get a bit higher quality with LLM fakery. But as with many "AI bad" themes, the problem isn't that "AI" can fabricate the data. The problem is fucked up institutions and cultures.
I’ve met many “data driven” teams that quickly turn their nose up at bad data
It’s just still mind blowing I can get a sarcastic summary of an email in the theme of GlaDOS from Portal and in the same screen get an email proofread.
It’s funny how far “what should the next word be” can go.
They've been doing some interesting work on factored cognition to avoid these sort of hallucinations [2].
1: https://elicit.com/ 2: https://blog.elicit.com/factored-verification-detecting-and-...
I chose the wrong field to be able to fake data /S.
See:
https://en.wikipedia.org/wiki/Synthetic_data
or in scientific literature:
https://arxiv.org/abs/2208.09191
a few companies are doing exclusively this
While no time now, I’m still interested in making a list of resources (esp free) that tells how to construct good studies, has comprehensive presentation of all categories of mistakes/lies we see in them, examples of each, and practice studies with known errors. Anyone here got good books or URL’s that could go in a resource like that? That could train new reviewers quickly?
If I return to AI or ever work in it, I also planned to teach all of that to AI models to automatically review scientific papers. Might contribute to solving the replication crisis. Anyone who’s doing AI now feel free to jump on that. Get a startup or Ph.D. with a tool that tells us which of the rest are fake.
this stuff is getting old. it doesnt need studies on how an LLM bullshits. nobody needs a study on that, they need an article in a tabloid at best.
But as productivity increases, and as AI improves, in both cases individual greed holds back lifting up the many. And so we end up asking for caution on automating away someone's job, or caution on rapid AI progress.
Does anyone know any good writing on how humanity might fight its way through all these mires of progress to the other side - that science fiction world that may or may not even be possible? How the world might look as these things continue to progress over time? Either fiction or a serious analysis is fine.
Most sci-fi skips straight to "There is no more need for University, we simply ask the AI", missing the "students are using AI to cheat" phase entirely.
Some state may cross between individuals via education, but the individuals still must learn.
History shows that knowledge transmission remains a sticky wicket.
Humans learn things collectively via culture and cultural transmission has been an extremely effective tool of knowledge preservation over the generations.
Things are never an upward hockey stick but they also aren’t saw waves skirting a baseline.
Based on some of the shit my neighbors post online there is but a thin veneer on society that keeps them from doing it now.