The article mentions a student taking his own life. I remember this was huge news a couple years ago at ISCA '19 and something that really shook my decision to pursue academia.
After that event, SIGARCH launched an investigation. After a couple years, here were the results of that investigation.
Worth noting is that the investigation actually initially found __no__ misconduct. Imagine that? A student kills himself, and you conclude it was the victim's fault, and not the environment that drove him.
It was only until this post [1] emerged that they relaunched the investigation.
That is very good work from the ACM. They don't whitewash anything, in fact they even keep the option open to re-assess their position should further details come to light. Impressed.
> It should have been unnecessary that we expose these evidence and challenge the result of the investigation, if the committee can drive a responsible, transparent and thorough investigation
An interesting question : Did ACM require the followup Medium article to update their position? I don't know the details of the case. However, merely updating positions when situations are black and white are some of the easiest scenarios. I wouldn't be impressed if black and white situations are assessed as black and white. This doesn't mean that one shouldn't do so. I'd expect those scenarios to be a bare minimum requirement.
Yes, but you can't really fault them for that because without any evidence to go on it would have been a fishing expedition. So compared to some of these other investigations that I'm familiar with I think they did it by the book.
It’s hardly reasonable to ignore the evidence of a students suicide note making a specific and detailed accusation of academic misconduct. It’s not a fishing expedition when you’re looking for something that you have reason to believe exists.
> They don't whitewash anything, in fact they even keep the option open to re-assess their position should further details come to light.
Did I miss something? All I can gather from the announcement PDF is that “several individuals” have been disciplined to varying degrees, the least severe being just a warning letter. No names named, no other details. Most of the announcement was just reiterating they took the investigation seriously. Kind of hard to determine from the announcement what details have and have not been considered, and whether certain individuals have been punished too lightly, no?
The announcement does mention a confidential report has been submitted for further review. Did anything concrete ever come out?
(I suppose it would at least be relatively obvious after a while which individuals are subject to a 15-year ban.)
It is probably too late to save Computer Science Research. Efforts in that direction are likely wasted. More important is to keep the contagion from spreading to allied fields. Grants probably should stop immediately. People doing serious work will need to move to another area where they might be able to contribute. People evaluating work in these other areas will need to guard against allowing theirs to be overtaken by the same downward spiral.
In perhaps a generation, a similar specialty might be bootstrapped and begin to take on problems had been of interest in the old one. What to call the new specialty will be its smallest problem.
Without any personal context, this stance appears very "baby meets bathwater". In particular, I'm not sure I see anything about this particular situation that renders it purely a problem of CS research. What makes other fields (presumably) immune or less predisposed to these kinds of issues?
Your position also paints CS in very broad strokes; in my experience, the only commonality between some subfields of computer science is that they use computers. Graphics, hardware architecture, programming languages, networks, and so on, are all essentially loosely coupled with their own organizing communities and directions. Some of these subfields are more closely tied to mathematics or electrical engineering than strictly to other parts of computer science. If there is an incurable "contagion" that afflicts all of these, I must admit it hard to believe that this contagion would not prove (if not already be proven) effective beyond the artificial confines of the term "computer science".
Oh it does. I'm interdisciplinary. I've been offered the "opportunity" to be part of two citation rings in Economics, and that's from attending a handful of conferences.
Which if one looks at the state of economics right now, is an object lesson on this kind of stuff never ending well.
The bigger issue here, and what is threatening the integrity of the research, is blind reliance on conference publication count as a proxy for research quality.
Maybe it's time to move on from some of these conferences, and focus on interactions that maximize sharing research findings. I know that is unrealistic, but like every other metric, conference acceptance ceases to have value once the metric itself is what people care about.
I often go back to this talk from Michael Stonebraker about, in his terms, the diarrhea of papers [1]. It's difficult to justify time towards anything that doesn't lead to a near-term publication or another line on your CV.
A really awesome talk - it's nowhere close to my field of research, but it is oh so relevant anyways, thank you for linking it and I'd recommend everyone interested in this larger topic to listen to what Stonebraker had to say on this.
And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.
The top 5 in the last 4 years - which is pretty lunatic when you think about it. A Ph.d - which is meant to extend human knowledge by 1 iota - takes 3 years to write. So what does 5 papers indicate?
The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics?
The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.
REF is 7 years I believe.
2/3 weeks per paper - I'm not sure what stage of your career you are at but I can't see how this would be necessary to evaluate the quality of a paper ( which presumably has already passed peer review and been published in a top tier venue ). Also, a PhD is in effect a trainee researcher. I don't think it's unreasonable to expect a more experienced academic to have a higher output rate.
>which presumably has already passed peer review and been published in a top tier venue
So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money.
As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
No comment on the quality of many IEEE conferences :) I think most people active in a field know what the best venues are.
I certainly don't disagree that there is a lot of randomness to the review process. And frankly picking which papers are likely to be most influential in ten years time is super difficult (e.g. see how often best papers have minimal subsequent impact). Regarding the Ferguson document, I haven't served on a REF review panel, but I think if you can justify it as being influential/high quality through other means that should be acceptable?
OFC you are right - it would be fairly career limiting to try to talk it down I would think! But I am concerned that the REF should work by the reviewers actually looking at the material, rather than looking at where the material was published...
"When a measure becomes a target, it ceases to be a good measure." - Goodhart's Law
It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.
The playing field where one can share research findings is far more uneven, more tilted in favor of the prestigious labs and individuals more than the current field of conferences and the blind review process.
Findings and papers from well known individuals (read: twitter accounts) do get far more attention , more citations. Of course, one can argue that, broadly, well known labs and individuals are wel known because of their tendency to do great work, write better papers. And that’s true. However, the above still holds, in my experience as PhD student in ML. Anecdotally, I have seen instances where a less interesting paper from a renowned lab got more attention and eventually more citations than a better paper accepted at the same venue by a less renowned lab on the same topic.
I completely agree with you and this is a significant issue, because it makes it hard for someone not from one of the established places to get research attention.
I would also argue that with the increased importance of the (mostly) commercial "high-impact" journals this has become worse. I know that some of the professional (non-expert) editors of these journals specifically look at the citation counts of authors before accepting to send them out to review, because their main aim is to get people reading the articles, not necessarily good science.
This is objectively worse. Commercial journals need to be bygone. I feel that the publishing venues in the field of machine learning are a great example of non commercial, (relatively) transparent, open access publications. Look at ICLR, ICML, ACL, NeurIPS, for instance. All of their proceedings are open access. Most papers their are available on arXiv as well (which has its pros and cons both that I'd rather not get into here). Many of them also have an open review process wherein the reviewers post their reviews for all to see in a forum like interface, allowing frequent back and forth with the authors. In the best case, its more akin to assist the authors refine their potentially sound findings, a dynamic process where the paper may get revised a couple times before acceptance.
I believe journals need to adapt the openreview process. There are obvious parallels in the lifecycle of a journal submission (from submission to acceptance) and the openreview process, except that the latter is accelerated (for better or for worse).
Yes, the cloud that the large commercial journals have over the academic process is really worrying. A publication in Nature can mean the difference between tenure and no tenure or getting a grant or no grant.
I have to say that I find the review process of the copernicus journals very interesting. You can see a description here: [1]. Unfortunately I don't work in a related field otherwise I would have published there already.
> We find considerable evidence that, overall, article citations are positively correlated with tweets about the article, and we find little evidence to suggest that author gender affects the transmission of research in this new media
I made a similar comment in another thread here, but the problem is that there will always be non-experts who want to assess the quality of research being done in different fields. These might be deans, department heads, VPs of R+D labs, and so on. Ideally, all research would have clear measurable impact, but a lot of good research doesn't, at least not immediately, so there needs to be some way for non-experts to measure quality. Conference/journal publications and citation counts are highly imperfect solutions, but I'm not sure what the better candidates are.
The department head at least should be something of an expert, and be able to consult with more specialized experts. Then that department head can pass on an evaluation to management.
Most experts (who understand what a department head is) don't want to be department heads. In the UK there used to be a tradition of making someone be head of department for 2 years or so and then letting them become a professor again so that the next junior could do the head of stint! This has faded because of much nastier politics (everyone hates you after you sacked their friends) and a rise in admin due to the industrialization of the bigger departments (its not a job for an amateur any more)
Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense.
When someone is hired (whether for tenure or time-limited), their research as a whole has to be evaluated by external, independent committees who take into account the content of the research and do not base their judgment on indicators only. There is no shortcut around that.
The biggest annoyance nowadays is the decision-makers' insistence on "excellence", though. You cannot have only excellent people everywhere, as per the definition of "excellent", yet this demand is in every fucking guideline for postdocs and tenure-track position. It's absolutely ridiculous.
Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense.
The problem with this assertion is: non-experts are paying for all of this.
Imagine you are an ordinary taxpayer. You are feeling the pinch yourself, you look around you and see infrastructure crumbling, every day the press says healthcare and this and that is underfunded. Now along comes some scientist, he or she wants a few billion for a new particle collider that will make no difference whatsoever to your life, and their only justification for it is "well other scientists say we should get all this money, and they're cleverer than you, shut up".
Can you see why funding something with no accountability might be considered problematic?
There's also the fact that an expert who can't explain something to a non-expert probably doesn't understand it very well themselves...
Only experts can make this judgement in the short term. As time goes by, the reception of the work accumulates and it becomes increasingly possible for non-experts to judge.
It took over 20 years after the Standard Model reached broad acceptance (ie., the experts thought the theory was probably right) for the first supercollider powerful enough to observe the Higgs boson to be financed. This was enough time for policy makers to reach high confidence that the experts had not gone badly wrong and for there to be reasonably well-informed public opinion on the merits of the search.
If non-experts are in charge of individual funding, then you're virtually guaranteed to get nonsensical decisions. It couldn't possibly work, and even if you got it to work, the results would be abysmal.
> Can you see why funding something with no accountability might be considered problematic?
Maybe you're confusing budget decisions with funding and hiring decisions. These are fundamentally different. Universities and research institutions, as well as national funding authorities, get budgets that are decided politically, i.e., by elected representatives. These can have broad categories and guidelines or preferred research areas (e.g. "excellence initiatives"). Budgets are usually allocated well in advance, for instance our national funding authority gets budget security for 4 year periods (if I'm not mistaken). How they spend it is dictated by political guidelines for the respective period and plenty of complicated national and international laws.
In contrast, I was talking about hiring decisions and decisions about individual funding. How can it not be obvious to you that these decisions need to be made by experts on the basis of CVs and scientific project proposals, not by politicians or other laymen?
The better candidate is spending more on the evaluation.
E.g, for my company's twice-yearly evaluation, everybody writes up a short report on the most impactful stuff they've done including evidence, this is evaluated by their manager to give a score, and then there's a series of group meetings between managers to make sure that the scores are calibrated, including looking at all types of metrics that can be dug up and comparing to our written role descriptions for different levels. It takes a lot of time but creates fair scores.
This is extremely labor intensive, but that's the thing: To create anything resembling fair evaluation of a large group of people that do a large set of different things, you need to do things that are labor intensive. Using a simple set of metrics don't cut it.
2 years ago Prof at my old school had a student commit suicide over being pressured to go along with this. You can Google your way to figuring out where and who. Last month Prof finally resigned. No other repercussions. People think academia is some priesthood it's not. It's a business like any other with a law of averages determined number of bad actors.
There's a 'hot' subculture in our society that says 'if you ain't cheating, you ain't trying', that refers to lies as 'hustle', that rewards and embraces deception as just agressiveness and boldness, as a norm for life and business and even a celebration of human nature - as if the worst elements of human nature define us any more than the best, as if we don't have that choice (at least, that's how I am trying to articulate it).
It has predictable results. Where are we going to get reliable research, and anything else, if we can't trust each other. Trust is an incredible business tool - highly efficient when you can take risks, be vulnerable, and don't have worry about the other person. Trust is an incredible tool for personal relationships, for the same reasons, and because if you can't trust them and can't be vulnerable, you have a very limited relationship.
> There's a 'hot' subculture in our society that says 'if you ain't cheating, you ain't trying', that refers to lies as 'hustle', that rewards and embraces deception as just agressiveness and boldness, as a norm for life and business and even a celebration of human nature - as if the worst elements of human nature define us any more than the best, as if we don't have that choice (at least, that's how I am trying to articulate it).
I'm not sure that you intended it this way, but this reads as very oblique (i.e., "wink and nudge"). Which subculture are you referring to, and what particular relationship do you think they have to research in Computer Science?
> I'm not sure that you intended it this way, but this reads as very oblique (i.e., "wink and nudge"). Which subculture are you referring to, and what particular relationship do you think they have to research in Computer Science?
I am referring to no particular subculture. Lots of people around me embrace it, including from all over the political spectrum (if that's what you are thinking).
I think the broader society sets the norms for computer science, as with everything else. For example, when star athletes like Barry Bonds, or entire teams like the Houston Astros, or much of college sports, cheat with few reprocussions (and in the past, that wasn't the case - players were banned and school sports programs were basically shut down, etc.) that affects computer science research.
Yeah, honestly go to any conference in the last decade and you'll see some people who are just... out of place in an academic, like, when they were 19 they listened to a podcast that claimed PhDs made XX% more money, so they decided to do that. These people don't care about research, don't care to understand research, they just want to publish, get their degree, and get paychecks from Google/Facebook/Apple. Luckily I've seen a number of these types of people fail to find any high profile jobs after they graduate, so I guess something is still working.
Reap what you sow. We spend our whole lives being told it's perfectly OK that every corporation and person in power behaving like a sociopath is totally fine and this is the culture you get in return.
Academia WAS designed to be an insulated castle from all that, that's why Newton lived in a shitty apartment while getting money from his mom. The institution of research was supposed to be a rich man's game, people who didn't give a shit about practicality, just one upping each other. Once people realized academics could be leveraged to do cool shit like build A-Bombs, it was over.
> Once people realized academics could be leveraged to do cool shit like build A-Bombs, it was over.
Arguably, it's the opposite. Once people realized academics could do this kind of cool shit, they got showered with money and told to do whatever the they want. That's how we got the incredible scientific and engineering advances of the second half of 20th century.
Then the beancounters started asking questions about what the money actually buys, and research quickly turned into another short-term, self-contained, profit-chasing game, starved for resources and only occasionally producing something actually useful.
Nyquist, Bode, Shannon, etc. didn’t need an A bomb to advance their research which is the bedrock of the digital control systems that microcomputers are.
Yes. They also didn't need to spend 90% of their time chasing grant money and publishing papers.
As it is, if our researchers are spending almost all their time thinking about and doing things other than research, what do we expect?
On a tangent, software industry has a bit of similar problem, with the best developers being forced to enter management roles[0] instead of solving technical problems. That is, a developer progresses from doing shoddy work to doing mediocre work and then, just as they start doing high-quality work, they get told to manage a new cohort of juniors doing shoddy work instead. I wonder if that's why so much software is hot garbage these days.
--
[0] - Whether proper ones on management path, or "fake" ones like principal developer, where you get all the managerial responsibilities with none of the authority.
>The institution of research was supposed to be a rich man’s game, people who didn’t give a shit about practicality, just one upping each other.
Thaaaaannnkkk you. It has technically been reserved for Christian Aristocracy, which is practically a world where there is no desperation for calories. Instead many of the greats had an anxiety of their immortality (see Fourier).
Normalization play a huge role in creating this. There are far too many rules in our society at all levels that make no sense and exist seemingly to benefit the rich and powerful. Highly demanded goods and services are prohibited (drugs, gambling, prostitution). Bureaucratic nonsense at every turn. So much schoolwork that it's practically required to cheat to succeed. Testing with barely any resemblance of real-world conditions. Abstract methods that bore students to death.
Those aren't reasons to act dishonestly and abuse other people. In fact, those are reasons to do the opposite - if you find the world so terrible, do something to make it better.
It's up to you and me. Nobody else is coming to save us.
When people are surrounded by rules that make no sense, many start to question all rules. It's not a justification for acting dishonestly, but it is the cause.
One way we can make the world better is by fixing the rules. Getting rid of ones that are unjust, and making the rest more consistent.
>"One way we can make the world better is by fixing the rules. Getting rid of ones that are unjust, and making the rest more consistent."
I once saw sign on a street: do not do something (do not remember what) and reference to a City Bylaw numbered as 37 thousand and something. That is just for one city. Good luck changing this sheer insanity.
I don't think the quantity of laws is really an issue as long as they are being applied fairly and (and this is the vague part) society views them as generally just.
>"I don't think the quantity of laws is really an issue"
And I think it is. At this quantity the quality will definitely suffer. Besides, I did read some bylaws at some point out of curiosity and without going into details many of them are outright unjust/deficient/etc (in my opinion of course)
> "If a law is unjust, a man is not only right to disobey it, he is obligated to do so."
Incidentally, while I recognize thr popularity of this quote, its fairly ridiculous taken literally for laws which are prohibitory rather than obligatory.
Viewing a prohibition as unjust does not obligate me to violate the prohibition; believing people should be free from government constraint to do something doesn’t require me to do that thing.
“Disregard” or “discount” in place of “disobey” would be more generally valid.
The role of bureaucracy is particularly relevant and interesting. Joseph Tainter argues in The Collapse of Complex Societies that the diminishing returns of increasing layers of bureaucracy are an important reason for the collapse of societies:
“Sociopolitical organizations constantly encounter problems that
require increased investment merely to preserve the status quo.
This investment comes in such forms as increasing size of
bureaucracies, increasing specialization of bureaucracies,
cumulative organizational solutions, increasing costs of
legitimizing activities, and increasing costs of internal control
and external defense. All of these must be borne by levying
greater costs on the support population, often to no increased
advantage. As the number and costliness of organizational
investments increases, the proportion of a society's budget
available for investment in future economic growth must decline.”
I think cheating-culture requires more than just a lot of arbitrary rules. You need to also have friends who can help, role models you can see getting ahead while cheating and so-forth.
It’s just a general breakdown of the rule of law and trust all over our society. When everyone around you is breaking the rules and receiving no repercussions, then following the rules yourself is equivalent to choosing to lose.
We used to have strong institutions that were supposed to help push groups into not choosing the bad corner of the prisoners dilemma, but they all seemed to have degraded. I suppose they could have always been like this and the curtain has just been removed, but I’d argue that the perception that following the rules was the best personal choice is almost as valuable as it being true
Can be trusted with what? Given how complicated life is, I'm actually impressed with how morally people behaves. In that sense, I trust that most people are doing what they can, and they are trying to do no harm to others. But that doesn't mean I trust people to be very competent anyway. Trust what.
Do you disagree with either the specific operationalization of trust in the article or even the abstract concept of operationalization of trust in general?
In republics without absolute rulers, individuals trust each other to maintain the state out of a sense of civic duty, and not to declare themselves king.
This trust is greatly eroded by calls to eliminate direct taxes on land holders. In America the local property tax is the most important thing holding society together. It provides residents with assurances that regardless of how corrupt the public process for distributing legal tender becomes, that the richest cannot simply buy the entire country and turn the continent into a private estate which their descendants will inherit in perpetuity without paying enormous taxes to everyone else.
When James Madison organized the assessment of property taxes at the national level during his presidency it resulted in the 'Era of Good Feelings' and a relative low point of political polarization. In contrast when state governments introduced sales taxes for the first time to reduce property taxes it prolonged the Great Depression, and when NYC gave the largest property tax abatements in the country to Donald Trump it lead to the Trump presidency which increased political polarization.
Lack of trust is a symptom not the cause. I hate it when I hear politicians and the like bemoan a "lack of trust". I don't want to trust you! I want to hold you to account!
Actually trust is both a symptom and a cause. The causality goes both ways.
(1) If you lose trust in someone, you'll be less likely to try to find truth in their statements. Finding truth in someone's statements requires time and attention, and we'll invest that attention in people we trust.
(2) If you find falsehoods in their statements, you will lose trust in them.
Overall in Psychology, trust is a primary indicator of a relationship, marriage, partnership, or organization succeeding or falling apart. It's both a symptom and a cause of all other factors.
Deliberately conflating cause & effect is a classic politician’s move. It doesn’t even have to be a reversal of the two: it’s probably even more frequent when they cite one part of a positive feedback loop as the unambiguous cause and the other as the clear effect.
The classic example of the later are the politicians who rant “if only these people would get married and stay married, they wouldn’t be so poor” and neglect to consider that impoverished communities create the conditions for rampant single motherhood. The desire to raise children does not magically vanish merely because there are exactly zero worthwhile men in your community that aren’t your father’s age.
There is a huge short vs long term bias here. Constantly burning everyone around you requires a long line of new suckers. But, within a community trust can have great long term benefits.
In the old days you had to know your place. WASPs smoked cigars and ran things. Those old guys drinking sherry and wearing tweed helped each other out. The Irish were cops, Italians firemen.
In tech it’s pretty obvious to see various constituencies doing dishonest shit help others out.
Indeed. I wonder if the internet also has something to do with this feeling. In a way, people are looking for justifications to doing things the wrong/illegal way. And in the past the information you got about how the world works is from other people around you. Now you can look up people in the same boat and apply whatever logic they did.
Engagement-optimizing media definitely are a part of it. That means both social media sites and classical news media. Both feed us a heavily warped view of the world - one in which nothing works, and everyone tries to cheat. That's because people being good, following the rules, helping each other and accomplishing things together is not newsworthy, and stands no chance against outrage-inducing stories.
trust emerges in situations with iterated interaction; defecting is most effective in anonymous or discrete situations, since there is less opportunity for punishment.
america has urbanized rapidly in the last half century, at the same time that family formation has broken down and life-long jobs have become a thing of the past. we are atomized and thrust into constant competition. i don't mean to idealize a past that i did not even experience, but there is something to be said for having roots and knowing your neighbors. we arguably have more opportunity at the cost of stable identity -- reputation and trust naturally accrete around the kind of stability we lack.
if you talk to older people, people around my grandparents' age or thereabouts, you will hear that they no longer recognize america, that it is fundamentally different than the culture they grew up in, in terms of values. i find myself thinking about this a lot.
> there is something to be said for having roots and knowing your neighbors
I think knowing your neighbors is overvalued. My evidence is Tokyo and living in transient, largely ethnically homogenous sharehouses — in ethnically homogenous areas — for long periods of time.
Academic communities are often quite small; back when I was publishing and serving on committees it often wasn't hard to figure out who the reviewers were, because a paper proposing a revolutionary new way to do X would be sent to top world experts on X. In those cases, the fact that you could sometimes figure out who was who limited possible bad behavior, because clearly unfair reviews would damage the reviewer's reputation.
Not sure I follow. It sounds like the experts in your example are able to effectively identify fair reviews based on their merit. This would need to be the case for people to pick out unfair reviews (and therefore deter reviewers from writing bad reviews). But it that's the case, anonymity shouldn't incentivize anyone to write bad reviews (since bad reviews would be identifiable and dismissible on their own). If anything, I would think that de-anonymization (implicit in this case, since reviewers are presumably nominally anonymous) would be more likely to empower cult-of-personality effects (or in general, would incentivize looking beyond the merits of a review, to get into an influential person's good books), making bad reviews more influential than they otherwise would be.
Refreshing to see someone on the inside who notices.
From the outside this degradation of values and "social lawlessness" has been apparent for years now, especially since 2016.. Really hope this doesn't spill over.
Someone on the inside of what? I don’t think I have a particularly privelaged view or extra knowledge. Rule of law is like poli sci 101 and I can observe the lack of trust just in the general trend of messaging in mainstream media and discussion boards like this
> It’s just a general breakdown of the rule of law and trust all over our society. When everyone around you is breaking the rules and receiving no repercussions, then following the rules yourself is equivalent to choosing to lose.
More specifically, it's the perception of a breakdown that drives this behavior.
When it comes to cheating, there's a growing perception that "everyone else is doing it" and therefore it's not wrong to play the same games as everyone else.
The current political and social media discourse revolves around ideas that "the system is rigged" combined with a die-hard notion that anyone who disagrees with you is wrong and/or evil. When people are bombarded with these ideas every day on their social media feeds, cheating a little bit to get yourself ahead doesn't feel like cheating. It just feels like leveling the playing field.
> When everyone around you is breaking the rules and receiving no repercussions, then following the rules yourself is equivalent to choosing to lose.
Depends on how you define "lose". And also on what you think the "rules" are.
For example: most drivers in the US routinely exceed the posted speed limit on roads. Is that "violating the rules"? In a legal sense, it is, since if a cop catches you he can give you a ticket and you pay a ifne and points go on your driving record. But nobody considers you a bad person for doing it, and I would argue that doing it, if you don't cause an accident, is not harming anyone. However, obeying the speed limit is also not considered a bad thing; all it really means is you get where you're going a bit slower. The tradeoff is yours to make; you don't "lose" by choosing to obey the posted rule.
Now consider an example at the other end of the spectrum: all of the shenanigans with mortgages and the financial system that caused the crash of 2008. Those who "followed the rules" leading up to the crash--for example, those, like my wife and me, who limited our mortgage and the size of house we bought to what we could comfortably afford--did not "lose". Sure, the value of our home went down, but we had no need to sell it then. It was still a house and we could still live in it just fine. Sure, we didn't have a bigger house with more bells and whistles, but we also didn't have to worry about what might happen if the housing market crashed. In other words, we made a tradeoff not much different from the one made by the person who obeys the speed limit and just gets where they're going a bit slower--but still gets there.
Reminds me of Jugaad [0] or Chabuduo mindset in mainland China.
Basically if all it takes is getting citations, then forming a citation ring is "Chabuduo" and you'll only lose face if you are caught (not good enough).
From the wiki article it sounds like it just refers to extremely improvised engineering, something akin to "bubble gum and baling wire" or "MacGyvering" something. Something like a kludge, but a bit less pejorative.
This kind of collusion is driven far more by perverse incentives than some alleged cultural phenomenon of half-assery people like to attach an incorrect but exotic-sounding foreign phrase to. I can't speak for Jugaad, but 差不多 ("Chabuduo") is not at all appropriate here [1]. Just call it what it is: cheating, collusion and conspiracy.
Unfortunately it is not even a subculture. It is often openly touted in mainstream culture. Steve Jobs is often brought up as a hero of that type of thinking.
You are absolutely right that trust and reliability are very valuable. Societies with high trust tend to be richer and much more productive than societies overriden by cheating and corruption.
Lou Pai is the greatest of all time and every man (or breadwinner) would be a fool to blindly follow any other script in society that is presented to them
Getting a divorce court judge to force you to sell your Enron shares at the top, nuking the regulator’s ability to charge you with insider trading, while you elope with your younger hotter high libido stripper nymph to the mountain you bought?
Doesn't this make the case that we should be building institutions and systems resistant to this type of cheating?
Why should it be possible at all to game Journals in this way? Particularly in Computer Science journals where people think about edge cases for a living...
I don't think we can actually solve this on a case by case basis. Other threads in this discussion have highlighted cultural normalization of greed in one of its various forms and I would tend to agree with that angle.
Those systems that can be built to be resistant to greed would definitely benefit from it - but I think it's more of an issue with society at large.
Heh. We need to set up a research project that redteam/blueteams various journals and conferences. And bug bounties for disclosing zero-days in peer review processes/practices.
(The redeem/blueteam thing is kinda unethical, so maybe the University of Minnesota should do it...)
> Doesn't this make the case that we should be building institutions and systems resistant to this type of cheating?
Absolutely. I think people are intimidated, demoralized (de-moralized) and where once they believed anything was possible, any social problem could be solved (even those old as history, such as women's rights, human rights, etc.), now they've somehow drunk the wrong Kool Aid, some stuff distributed by Jim Jones.
I'm a US expat and escaping this culture is one of the things that's made me happiest - I tend to call it the bullshit culture because my favorite example is... writing a good paper for class is admired - but what's really praised is writing a paper that gets good marks without ever having read the subject matter. Being able to spin lies about a topic you've no understanding of and turn that into a marketable skill is a dark potent for the future of America. I think it's always been somewhat present, but since emerging strongly out of the business world in the eighties it's gained a lot of steam.
We are a society that can benefit from cooperation where everyone gets a fair slice of the pie, but that society is eroded if we praise and not shame those people who betray societal trust and cheat the system.
At least in academia, American universities (and Western universities in general) have had a good track record overall with academic integrity, and it has set them apart. This seems to have degraded recently, perhaps because of the increasing pressure of the “publish or perish” system (or dozens of other potential causes).
We do celebrate people who excel academically seemingly effortlessly. But we don’t celebrate bullshit artists so much in school. In business, and particularly tech, it’s another story.
> At least in academia, American universities (and Western universities in general) have had a good track record overall with academic integrity, and it has set them apart.
The difference is that these collusion rings or bogus studies are discussed and exposed publicly. It's not the case in Asia.
I don't think it's nearly as prevalent in Academia. I agree that that sort of deception is quickly punished in Academia if it's discovered and publicized, but I think it still is viewed positively in private contexts. I think it's more of a cultural issue in general.
You can read a bunch of articles on Bernie Madoff, Martin Shkreli and Billy McFarland that romanticize the cunning with which these folks exploited others. Most pieces on them will present an overall negative tone but often feature some pretty glowing admiration of them. Let's also not forget that tax evasion by Trump was praised repeatedly as him beating the system - that's a pretty common view point, more common (especially when it comes to taxes) than the view that those individuals are failing to pay their fair share from what I've observed at least. Trumps a complicated example due to all the political baggage around him so maybe just look at companies like Apple, Google and Facebook - they regularly offshore large portions of their profits and I really doubt the people working to those ends feel any shame, instead it's likely a "beating the system" motivation.
Are you in academia? I am and the state of the publishing is quite horrific. I simply don't read any papers at least in my field unless I'm specifically interested in some detail during my research, or am handed something specifically by a colleague or sometimes if asked specifically to do a peer-review.
This is a bit of a public secret, but quite widely researchers don't really trust articles anymore, if they ever did. Maybe some plot or dataset may give some insight and maybe some discussion has worthy information to ponder on. But mostly they're just some ads to put in a yet another funding application.
Most articles are just churned out to get some lines to CV or to look good in some metric. Publish or perish has turned into full-on bullshit or perish. The whole peer-review system (which is just around 50 years old anyway) is on the verge of just grinding to a halt due to the stupendous volume of hastily hacked together manuscripts.
I think many are still sort of hoping that this will somehow sort itself out. But the collapse of the quality after the explosion of electronic journals, consolidation of publishing houses and overall structure that doesn't really care at all about what is actually in the papers doesn't give much realistic hope.
It should be noted that there's sort of a "parallel reality" in academia behind the publication show. The ethos for academic integrity is still quite strong, teaching tends to be valued by the community (but not by the system) and face-to-face discussions can be of very high quality. But the signal-to-noise is so low in publishing that it's not really worth following.
We really need to get some new arrangement so that we don't drown in all this bullshit. Word-of-mouth, open data repos, conferences and just blogging and pushing stuff to git repos probably is most that's needed. The publishing structure is becoming just plain unnecessary bureaucracy.
Is scale part of the issue? A lot of academic/publishing culture and norms were established when it was considerably smaller.. number of people, not just number of papers.
SEO is a kind of explicit analogy. Google pagerank was modelled on academic publishing, and it worked until it went live. From that point, links started to decrease as a quality signal.. spam. Publish or perish is a similar sort of dynamic.
Honestly, I think most legible systems for determining merit have these sort of issues. If advancement, accolade, grants or somesuch are determined by a formal system, whatever that system used as a signal or metric becomes corrupted. Hence why Word-to-mouth, open data repos, conferences and just blogging and pushing stuff to git repos does work. It's informal.
Yes, I'm becoming quite convinced that in general its stupid to meter almost anything. Why can't we just starting to give social repercussions to bullshit so we can just trust that people aren't scamming everybody all the time. In general, competition is mostly just waste of everybody's time, and in the end it's usually easiest to win just by cheating.
A sort of reputation system is in place in almost all peer-to-peer societies, it tends to form automatically. I don't think we really need any of this weird mess of a system.
We have Wikipedia, we have open source, we have OSM, we have all sort of things that should be "impossible" given the dismal perception people have of other people. This perception is just plain wrong and really harmful.
The alternative to "metering" is tolerating "waste." One reason tenure declined, for example, was checked-out professors. Perhaps that's a price worth paying. The other half of that coin is brilliant people with full freedom to pursue science unencumbered by bullshit.
It's a hard sell though. The cost of metering is subtle. The do-nothing tenured professor is visible.
Wikipedia, OSS, etc really are the shining beacons. Existence proof for something better. Someone needs to write The Cathedral and the Bazaar, but in non geekish.
This is what annoys me as well and I find it stems from trying to force the profit/salary motive into academia. Gladly it seems that even if the structure is put there, most people in academia don't care about the money per se much. Some care for the status and prestige, but salaries don't get you that in this community.
Perhaps surprisingly to some, many in academia would just like to research and teach with some quite modest salary and don't have to think about money at all. E.g. I would gladly and with no hesitations take a €2000/month tenure and keep on doing what I'm doing just more efficiently for everybody. I've been trying to pitch this idea to the funders here in Finland, but to no avail, they simply don't care if the funding system is useful or not for the academic community or humanity, they're focusing on playing the same old (maybe 10 years or so here) application lottery that's not only waste of time, but corrupts the whole community and even the very content of thinking in academica.
"Money" in academia is really abstract as well, and when discussed its not salary, but funding for projects or students or such. And because the funding structure is so bizarre and convoluted you just see big numbers with currency signs flowing everywhere, but this doesn't seem to have much to do with anything concrete happening around.
If academia becomes a place where you can get rich, the system will be in just years corrupted into some bizarre thing where advertisers advertise to each other for the sake of advertising.
Some of it is intentional "motive hacking." As you say, prestige, research funding and the like are as (or more) operative as salary.
Some of it is unintentional. Before publish or perish, publishing volume probably was a signal for something. I doubt it was ever a signal for high quality research, but low (or no) volume may have been a signal for low quality. Also, formal decision making bodies (like grant makers or tenure committees) tend to gravitate to quantitative, legible metrics.
Whatever the reason initially, publishing volume became a hugely important thing with impacts on many aspects of research.
At the same time, in CS especially, the number of researchers has also ballooned. That's a whole other strain on a system of, at core, knowledge dissemination.
I assume this varies field by field. In my field (computer security and cryptography) we don’t have such an antagonistic view of research papers by our colleagues.
I'm not familiar with those fields, but yes this varies a lot by the field. I'd assume cryptography at least is more math-based (at least theoretical stuff) and the math scene is quite different from empirical sciences or engineering.
The antagonistic view is not just towards my colleagues, I'm not particularly proud of my own papers either. I find it more a nuisance to "pay the bills" and a lot of my research goes unpublished (at least in journals) due to all the IMHO unnecessary hassle involved. Just a blog or something would be a lot nicer and probably would communicate the work better, and would ease the pretension of objectivity which I find mostly causes wrong impressions and makes writing really a chore.
> This is a bit of a public secret, but quite widely researchers don't really trust articles anymore, if they ever did. Maybe some plot or dataset may give some insight and maybe some discussion has worthy information to ponder on. But mostly they're just some ads to put in a yet another funding application.
> It should be noted that there's sort of a "parallel reality" in academia behind the publication show.
So what should be the guidelines for someone who is not a researcher, but an engineer, and hopes to stay informed by reading relevant papers from a specific field. (You know the folks who should apply some of that in practice)
Depends on the field and purpose. First of all, academic papers tend to be quite hard to approach if you aren't in the field, even if the quality of the paper is good. Articles almost necessarily cater to a very specific audience and lots of background is assumed almost by necessity. Also papers are not usually read linearly, researchers learn to get the gist of paper in just a few glances if its close to their own fields, and sort of hop around to see if there's something "unexpected".
Also individual papers tend to focus on one very specific problem at a time. This is typically related to some actual larger "debate" and can be difficult to see if one's not familiar with the larger issue. Also especially conclusions tend to have quite heavy implied assumptions that are just generally accepted in the field.
I "stay informed" mostly by face-to-face discussions and emails and such. I don't read much papers myself, but many of my colleagues do and I just hear from them, or ask them if there is new stuff around related to something I'm pondering.
To get an overall view of "state-of-the-art" I'd recommend starting with masters' or doctoral theses. These typically require more elaborate presentation of the background and its typically put out in more readable terms with less assumptions of the readers background knowledge.
In some fields review articles are a good starting point as well, and they tend to briefly sum up the required background, but my understanding is that some fields don't do those much.
If you read "random" articles, I'd do a quick smell-test before digging in. See if code is available, ignore papers with clear hype in the abstract off-hand. You can also "navigate" the field by following citations, although this can be technically annoying as the publishing format is still tailored towards print, even though very few journals are actually printed anymore. If you hit a paywall, try sci-hub or just move on to a next one unless you're looking for something really specific.
If you have something more specific in mind, just email or call or go talk some researcher that looks to be doing something related to what you are looking for. Researchers tend to be quite eager to answer to the public of their stuff, and its seen as sort of a public service duty as well. Depends on the researcher quite a bit though. Maybe a good starting point would be somebody a bit "lower on the ladder". Maybe a postdoc or a PhD student (this depends on the country as well). Professors tend to be busier and actually may not be that up-to-date with their field (especially on technically detailed level) as they spend most of their time in administration and the funding ratrace.
Depending on the country you can just attend lectures too. At least in Finland university lectures are public by law (with some restrictions on e.g. practical lab stuff etc). You can see if the lecturer doesn't seem too busy after the lecture and just go and ask.
You can also just try go to conferences. They usually have a fee in theory, but I don't think you'll be turned away if you just browse around for posters or so, especially if its a smaller one. The fees are just sort of a scam (long and sad story) and researchers organizing the thing usually don't care about the fees at all.
Yes, I worked in academia up until recently, and keep up with the literature.
My comment was on the historical status of academia in the US as a whole (think last 120 years), not just the current state globally.
You’re lamenting the quality of academic publishing in particular. The US now publishes less than 17% of science and engineering papers, but its papers are often the most highly cited. So yes, there has been a huge increase in the number of papers, and number of low-quality papers, but this isn’t necessarily being driven by the US, as the original comment would have implied.
You claim researchers don’t really trust articles anymore, and I agree that it takes a lot more work to filter out the noise now, and I’m less optimistic that authors are presenting an honest, objective appraisal of their results. But significant research is still happening, and academic publishing is still the primary way that information is disseminated. People seem to rely more on name recognition (author, school, journal) now. It probably varies field but field. I’m in a field where results are often proof-based and that tends to be harder to fake.
I don't consider countries much. There are definitely some differences per country, but I think the variation even within a same university is so large that its hard to infer much based on country of origin. But I acknowledge that I ignored that part of your comment in my reply and maybe in that left a wrong impression. But I meant no implication that this was somehow US-driven.
Internationalism is so ingrained in the academic culture (at least on fields I'm familiar with) that it doesn't even really register what country somebody's from or is working in. There are definitely some differences especially in the more "overt" parts of the culture (hats and robes and different titles etc), but these are of very little significance for anything but some ceremonies.
My working experience is from Finland, Sweden and UK, but in academia people come and go between countries very frequently so colleagues tend to be from all over.
There are at least some stereotypes that some countries are more prone to the e.g. citation rings, but I don't find that very relevant, as I think the whole system is quite broken and the publishing (at least in English language) forums are typically not country specific at all. Probably something like this happens in more or less any country.
> I'm a US expat and escaping this culture is one of the things that's made me happiest...
Really? Cause I'm an American who has lived, studied, and worked overseas. Let's just say it's not American (or generally Western) coworkers and classmates who are NOTORIOUS for cheating.
And I think many of us here who've attended "diverse" universities or work for companies with "multicultural" staff have a pretty damn good idea which cultures and nationalities are more likely to be cheating.
> I tend to call it the bullshit culture because my favorite example is... writing a good paper for class is admired - but what's really praised is writing a paper that gets good marks without ever having read the subject matter.
I don't think this is an American thing. In fact, I've never seen this praised anywhere outside of maybe a few people back in High School.
I worked for a company that expanded rapidly with distributed offices all over the world. One of the growing pains we had was that the managers from certain countries, America included, were very trusting by default. This opened the door to a lot of manipulation from employees in certain other countries (which I'm deliberately not going to name) where getting away with a lie was more or less considered acceptable as long as you weren't caught.
In my time in academia, 20 years ago, cheating specifically was much more common with students (and researchers) from what I would guess was your certain other countries. An example of general corruption, I think.
Funny that you say that, because in my country there is also somewhat of a culture of praising rogues and cheaters, and I had a professor who would say (when talking about cheating by copying in exams, I think) that he spent some years in America, and there no one would ever do such things, because there was a honor system and no one would ever think about do it such a thing.
Not that I ever believed him much about that (I do think he believed what he said...).
YCombinator seems to encourage this. Here's what they say about their dinner events:
"Talks are strictly off the record to encourage candor, because the inside story of most startups is more colorful than the one presented later to the public.
Because YC has been around so long and we have personal relationships with most of the speakers, they trust that what they say won’t get out and tell us a lot of medium-secret stuff."
Anecdotally, I've heard stories about Zuckerberg confessing/bragging about all sorts of nasty things at these dinners.
Really, this stuff should just be shamed. Sadly, too often calling out bad behavior just gets you called a "hater"...
Lol, they're not even trying to hide it. Do you see what people like Paul Graham tweet? They're pretty open about these nasty things and then they get all surprised when people call them out on it.
I've actually just encountered this first fucking hand and it disgusted me. I'm an aspiring founder working on something and met a coworker who had been through YC W17. I wanted to pick his brain on his experience, especially since I've been applying to YC myself.
This coworker told me he and his "startup":
- routinely lied to potential customers on the size of their client list
- misled clients on the depth and completeness of their product
- blatantly broke CA laws to cut cost corners
All of this was done to secure contracts in order to secure more funding. "Always be selling", he said.
He literally fucking said to me that he learned to "be dubious, not deceitful" which is probably one of the most deceitful things I've ever heard.
Made me sick to my stomach and pretty much validated (1) why I never moved to SF in the first place, instead moved to NYC and (2) how much of a fraud YC has become. Absolute fucking madness.
Don't forget the big one, "no one knows what they're doing anyway ", a veteran structural engineer has some gaps in his understanding, so that's no different from me, so I should have doubts about launching a skyscraper startup.
All the myths and stories of pretty much every culture reinforce these beliefs though. Stories are never about passive people putting in 12 hours a day, day after day building upon honest work. Its about slick instant gratification, getting ahead, winning through cunning, etc. The only way to really do that in the real world is through luck or cheating.
What myths and stories are you referring to? Hesiod’s Works and Days is opposed to exactly what you suggest. And the Myth of Eden in Judaism - the received tradition of its origin - is a commentary and the need to toil in life.
Both “Athens and Jerusalem” are principled according to honest toil. And look at the results!
George Washington, Ghandi, MLK, Churchill, Lincoln, Jobs, Torvalds, every religious figure, sports figures, artists, etc. ... they all are about "slick instant gratification, getting ahead, winning through cunning"?
Anytime I see a job description that mentions the world "hustle", and trust me there are quite a few in tech, I immediately discard it... but this description of the culture portrays it more accurately that I could have put into words. I think I shall bring it up next time I see a job description like this and let them know that is the impression they are giving.
Even though I share your cynicism, 'A Hustle' is very different from 'Having Hustle' i.e. the noun means something from the adjective.
Having 'hustle' is actually important at a startup, it's part of the essential aspect of it. I'd argue a 'hacker' has a kind of hustle.
I'm pretty suspicious of these things as well, but I've also come to believe in my many years that a bit of koolaid is fine as long as it has self awareness.
I wish Elon would stop posting all the stupid things he does, but I think that's what you get with 'all the other stuff'.
If Elon were fully polite, conscientious, a 'good listener' I'm not sure Tesla would exist. So we pay for the existence of Tesla by accepting that he's going to do dumb tweets about Crytocoins.
> If Elon were fully polite, conscientious, a 'good listener' I'm not sure Tesla would exist.
I think that's a BS excuse for bad behavior. How would being polite and concientious harm Tesla? It's simply that he has power and has low standards for his behavior.
I know plenty of successful people who do what Musk fails to do.
"I know plenty of successful people who do what Musk fails to do. "
Except you don't know any successful people who've remotely done what he's done either.
Talking about 'Dogecoin' is a little unseemly, but it's nowhere in the realm of 'toxic' or 'bad acting'
Without massive public support and sympathy, Telsa wouldn't exist, it's a movement as much as anything, and so you need kind of a showman.
His appearances on SNL etc. are part of that public drama that keeps Tesla stock going with enough support to keep the legitimacy of the dream alive.
I'm seriously doubtful that a quiet, unassuming person would have been able to do most of that.
Expressive, bombastic characters will by virtue of the volume of their actions, sometimes creep up to the line. It's normal. There's nothing wrong with Elon, he's just a little cheezy and spouts too hard with some things.
In case others wonder or worry, as I do, that few care about the issues in my parent comment, I'll take the unusual step of reporting that it has more upvotes than much of the front page - far more than I've ever seen.
(I'm trying to respect the HN tradition/guideline of not talking about votes. I don't really care about Internet points; I think the implied interest in the issue is worthwhile and applicable in this case.)
Oooh.. this sounds like a great computer science problem.
"How to get an objective rating in the presence of adversaries"
It is probably extensible to generic reviews as well... so things like the Amazon scam. But in contrast to Amazon, conference participants are motivated to review.
I honestly don't see why all participants can't be considered as part of the peer review pool and everybody votes. I'd guess you run a risk of being scooped but maybe a conference should consist of all papers with the top N being considered worthy of publication. Maybe the remaining could be considered pre-publication... I mean everything is on ArviX anyways.
So instead of bids you have randomization. Kahneman's latest book talks about this and it's been making the rounds on NPR, NyTimes etc...
I believe they were thinking of it as a consensus problem, where parties need to agree on an objective evaluation in the presence of adversaries eg. authors, people with very similar/identical publications, plagiarists.
In many such events all participants are required to be part of the peer review pool.
However, they review a limited amount of papers (e.g. 3) - "everybody votes" presumes that everybody has an opinion on the rating of every paper. That does not scale - getting a reasonable opinion about a random paper, i.e. reviewing it, takes significant effort; an event may have 1000 or 10000 papers, having every participant review 3 papers is already a significant amount of work, and getting much more "votes" than that for every paper is impractical.
It's unfeasable and even undesirable for everyone to even skim all the submitted papers in their subfield - one big purpose of peer review is to filter out papers so that everyone else can focus on reading only a smaller selection of best papers instead of sifting through everything submitted. The deluge of papers (even "diarrhea of papers" as called in a lecture linked in another comment) is a real problem, I'm a full-time researcher and I still barely have time to read only a fraction of what's getting written.
In theory you could probably do something like have three runoff rounds, such that low-scoring papers are eliminated before people do their second review.
This issue reeks with the rank smell of base politics and in/out group dynamics, and humans have been fighting, in the abstract, these issues since the time Egyptians were building the pyramids.
How can there possibly be an "objective rating" when career advancement, peer respect, and big money all are in the mix depending upon results?
It's not just personal interest here, it's that we can't tell what is good research at the outset, it might take years to be able to appreciate it. It's like climbing a mountain when you can't see the path ahead and have no map. It might lead to the top, or might lead to a lesser peak, or you have to pass a chasm.
In other words objectives are deceiving and rating is based on objectives.
Example: mRNA inventor being sidelined at her university when her method wasn't famous
Example2: Schmidhuber inventing stuff and being forgotten because data and compute were just too small back then
It's all about building a diverse collection of stepping stones. Any new discovery might seem useless and we can't tell which are going to matter years later, but we need the diversity to hedge against the unknown.
Indeed. There is a lot of talk in my field equating large variance in reviews with bad reviewing, but sometimes it's just because we are humans.
Take for example a paper that presents a very innovative method, but with subpar results; and another one that presents an incremental improvement on some existing method, but with results that advance the state of the art.
Which is better?
Even if you ask knowledgeable, careful and honest reviewers, you will get contradictory responses, because it's highly subjective whether you rate originality as more important than results or vice versa (and other factors, like whether you think the first method can eventually be improved to be useful or not, which is often just an educated guess). I see this happening all the time, and I don't think it's something that can be "fixed", it's just how humans work.
Collusion is one of two major problems with modern research in CS. The other one, perhaps even bigger, is its lack of substance and relevance. Most research is meant to fill out resumes with long lists of papers on impressive-sounding venues and bump institutional numbers in order to get more money and promotions. Never mind how naive, irrelevant, inapplicable or unrepresentative that research is.
It's, of course, a very hard problem to solve. It takes a lot of effort to evaluate the real impact of research.
Agree with first paragraph, less with second. I’m pretty experienced in CS. You do some research for six months and we’ll have a line meeting, I’ll know if you’re actually working on things that matter rather than boosting your CV.
It's sad to hear that. Other research areas like medicine, pharmacy or history probalby have the same problem but nobody is looking for it yet. My guess is, the more money is to be made or raised, the higher the chances for nefarious practices.
I've discovered that the software industry is made up of some of the most dishonest, insecure, power-hungry people on the face of this godless earth.
Only a tiny percentage of developers seem to actually enjoy coding - Most of them have no interest in it and only see it as a mechanism to acquire money, power and influence.
Disinformation is rampant because contrarians are punished and conformists are rewarded. The rot is deep in the guts of the industry. Those who have the most power and the loudest voices hoard all the attention for themselves and are unwilling to give exposure to any alternative views - Their deep insecurity drives them to surround themselves only with yes-people and to block out all critics; avoiding disagreement at all costs... Downvote, suppress, censor...
Powerful people in this industry need to put aside their insecurity by embracing disagreement, allow themselves to change their minds, and give a voice to contrarian views and ideas; even when it risks hurting their current interests.
Powerful people should seek the truth and try to promote the narratives which make the most sense; not the narratives which happen to benefit them the most. Everyone is free to move their money to match the changing narratives, so why do powerful people invest so much effort in keeping the focus on narratives which only maintain the status quo? To protect their friends? To protect the system? That is immoral - Capitalism was not designed for this kind of arbitrary altruism. For every person you 'help', you hurt 100 others.
As much as people love to bash Elon Musk right now, he should be applauded for constantly trying to adapt to the narratives which make the most sense as opposed to rotting in his own filth and succumbing to tribalism like everyone else.
The primary target audience for most of my comments is AI for training purposes (unsupervised learning). I've given up on humans.
Most humans don't have enough background knowledge or sufficiently diverse life experience to make sense of this.
I'd have to write a whole book to explain my reasoning behind this statement. I got a lot of my knowledge from HN articles and comments so it should tie in nicely.
How many industries have you worked in? This is fairly common in many walks of life. As I get older I'm getting better at identifying the lunatics in charge. Often they are nice people, but also cause massive amounts of chaos, because they have no clue what is going on. No wonder they feel the need to exert micro-control.
Anyway, thanks for the rant. Always interesting to look at the bottom of the barrel for the HN rejects :-)
One of solutions is to let individuals choose who to trust. I can pick to not trust the certain person, or certain publication or conference, and have my personalized scientist ranking recounted.
And of course, I could choose to delegate the trust, and to “follow” someone, which would mean to incorporate their rankings, especially in areas where I don’t orient that much.
I think the delegation of trust is already what is kind of happening in terms of academics trusting journals.
I do agree that there's probably some cleverer solution on a personal level, but I think the journal system exists as a kind of guard against untrusted actors, and yet fails.
Isn’t this already how it works? The current state represents everyone doing what you describe... enough people chose to delegate trust to the publications that everyone is fighting to get into, and here we are.
Or, just have real repercussions for unethical behavior such as this. If they've identified these people, revoke reviewer status for them. If it was egregious and they have good evidence, publicly announce the revocation. If getting published has a positive benefit on a career, public information about past unethical behavior with regard to publishing and research should hopefully have a negative one.
> One of solutions is to let individuals choose who to trust. I can pick to not trust the certain person, or certain publication or conference, and have my personalized scientist ranking recounted.
Are you asserting a 'trust' whitelist, a 'distrust' grey or blacklist, or some combination?
Are assertions meant to be linked to or backed by evidence?
Are these assertions publicly visible?
Is there a limit to the number of assertions that can be made?
I think you can see how such a system might devolve into formalized collusion.
*> And of course, I could choose to delegate the trust, and to “follow” someone
In the absence of explicit delegation or following, is trust/distrust intended to be transitive at all (ala PageRank or Advogato WoT)?
I long for the long lost time when research was less of an industry. If you read 19th and early 20th research, it comes of from an alien world. You had to be curious about your subject and smart back then.
I was amazed to read about W.E.B DuBois and his scientific work documenting the effect of systemic racism back in the 1900s in Philadelphia and that it leads to disparate health outcomes. There was a show on PBS that referenced it with respect to covid disparities among minorities showing that Dr. DuBois was way ahead of his time. In fact, we, as a society, still haven't learned anything more than 100 years later!
You also had to have substantial personal resources to do research, at least in the 19th century. Going back to that system would probably reduce the number of people doing science for the wrong reasons, but it would also strongly reduce the number of people doing science at all.
I mean, it's well within possibility for you to spend 20 years as SWE earning six figures, and then retire early to be an independent researcher.
As long as you are happy to explore a field that doesn't require millions of dollars of equipment and aren't competing for fame, there's nothing stopping you.
Because of the strong tendency to scapegoat the specific people named, drive them out of academia, and then celebrate victory while things continue in exactly the same way. (Ok, not exactly -- it improves for a while, people get sneakier, and then it continues in exactly the same way.)
Chipping off the tip of an iceberg isn't a good long term strategy.
I'm not in academia, but in the grand tradition of "why don't you just..." solutions crossed with "technical solutions to people problems":
Would it help at all if rather than participants reviewing 3 papers, each reviewed 2 papers and validated the review of 3 more papers?
This is computer science here, with things like the set NP whose defining characteristic is that it's easier to check a solution than generate it.
I'm imagining having some standard that reviews are held to in order to make them validatable. When validating a review, you are just confirming that the issues brought up are reasonable. Same for the compliments.
Sure, it's not perfect because the validators wouldn't dive in as deep or have as much context as the reviewers, but sitting here in my obsidian tower of industry, it seems like it would at least make collusion attacks more difficult. Hopefully without increasing the already heavy load on reviewers.
(It very much seems like an incomplete solution -- we only have to look at politics and regulatory capture to see how far wrong things can go, in ways immune to straightforward interventions. Really, you need to tear down as many of the obstacles to a culture of trust as you can. Taping over the holes in a leaking bucket doesn't work for long.)
That would only work if the review decisions could be expected to be reasonably consistent from person to person. But, from the article:
> In a well-publicized case in 2014, organizers of the Neural Information Processing Systems Conference formed two independent program committees and had 10% of submissions reviewed by both. The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself.
With this much demonstrated discrepancy between two sets of reviewers, it’s hard to believe that adding a validation step would produce a consistent improvement. How can people be expected to find improperly accepted papers when they have less than 50% agreement on the acceptance of good-faith submissions?
Honestly I think this seeming randomness in acceptance is at the heart of why people might think cheating is acceptable. If the process is not reliable, why bother submitting to it?
>The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself.
I referenced Kahneham's latest book, Noise, above but this is exactly the problem he focuses on. There are solutions.
Wouldn’t a ‘solution’ only work if there actually is some underlying objective quality to a paper? It might be the case that different reviewers disagree because there is no right answer, because there is no such thing as objective quality.
Well wisdom of the crowds suggest that there is a truth... and a conference is charged with representing the best so I think that randomization and meta moderation + other strategies are better in the long run.
The described result is not in conflict with decisions being reasonably consistent.
Suppose that two reviewers independently rank papers 80% on quality and 20% on chance factors. With good odds, the two reviewers will agree with each other on the relative rankings of any given pair of papers. But their lists of the top 10% of papers will largely not be in agreement with each other.
This is in aggregate. If you look at great papers, they are almost surely accepted. Bad papers are almost surely rejected. But those in the middle are a coin toss.
I think this is a really interesting idea. It does seem more effective for validating that reviewers are not torching the work of others — is there a nice way to validate that reviewers are not giving their friends a pass, short of reviewing the manuscript and the review?
The problem with Academic collusion rings is that eventually the ring can hold such influence that all "major" research comes from the ring. As ring members benefit there is no reason for them to switch to an alternate system. If one wants to progress in their chosen field then there is no benefit to publishing in an unread source.
One critical vulnerability in the current reviewing pipeline is that the reviewer assignment algorithm places too much weight on the bids. Imagine if you bid on only your friend's paper. The assignment system, if they assign you to any paper at all, is highly likely to assign you to your friend's paper. If you register duplicate accounts or if there are enough colluders, the chance of being assigned to that paper is extremely high.
Fortunately, this is also easy to detect because your bid should reflect your expertise, and in this case it doesn't. What we showed in our paper is that you can reliably remove these abnormal bids. It's not a perfect solution, but it helps.
Yea, I think this is likely a pretty big issue with highly specialized field - there can be an issue with new participants breaking into the field (due to an entrenched old boys club) along with a difficulty getting enough sample data specific to that corner of academia.
I wonder if we could randomly assign reviewers but allow the reviewers to self-report a level of familiarity on the subject matter in general (ideally in advance) and on the paper topic in particular.
People can also cheat on this self-reported familiarity right? On specialized areas I don't see a solution, the colluders might as well be the only experts in the field so you have to enlist them no matter what. But from the description this doesn't seem like what's happening.
"The colluders hide conflicts of interest, then bid to review these papers, sometimes from duplicate accounts, in an attempt to be assigned to these papers as reviewers."
I think that by involving randomness before any active self-selection you wouldn't necessarily involve specialty-wide clusters but you'd make it a lot harder to actively seek out the documents you want to review.
It might also help to attack the ability to create duplicate accounts. Given how relatively few professors exist in the world I'd assume you could put a lot more effort into duplicate account detection than they are right now.
Coming from the natural sciences, I was really surprised that reviewers get to pick what they review! Despite the deeper problems discussed above, removing this glaring bug in the system would be quite easy. If editors have to choose the reviewer assignment may be less efficient, but at least editors have to think more about who would be a good reviewer, and mostly they also know the players in their field and who may have conflicts of interest.
Coming from a natural science field we have a very different review process (but we also "only" review up to 100 papers per committee). Essentially we have topical sub-committees with ~10-15 members. The authors choose which sub-committee they submit to and there is typically a rearranging process by the program chairs and subcommittee chairs to check if there are some very obvious wrong category submissions. I should note that it's typically a disadvantage to submit to the wrong subcommittee, because if members don't really understand the paper they are much more likely to reject.
Every committee member reads all the papers (and indicates conflicts if necessary). We then have a committee meeting where all papers are discussed and accept/reject is being voted on. In these meetings it does happen that one reviewer picks up a subtle point (or finds e.g. a previous publication), that others have missed and this can lead to the reject of even highly scored papers. Having this many eyes and a discussion about he papers definitely helps IMO. The big difference here is that we don't get 10,000 submissions (more like 1,000).
I was actually very surprised that it is possible to register duplicate accounts at those CSE conferences. We get send a single invite to our work address and need to lock into the system using that email. And we are being nominated to get onto the committee.
It is very interesting to me that people can look at this, the replication crisis, and Sokal Squared, and not see that there is a fundamental flaw in the current model of academia as a for-profit publish-or-perish warzone, and instead declare some disciplines are somehow more bloody.
In CS basically everything is on ArXiv.. so the ideas and information are out there. The conference publication gives exposure and some credential-ism but still the idea is already out there to evaluate.
why to put any artificial limit on the number of papers accepted. If it is quality research, then it should be accepted. A simple majority system should be good enough.
It has long seemed like there was soft collusion in academics anyway. There is plenty of complaining and suspicion about how bad and rigged reviews can sometimes seem to be. There are plenty of rumors of PIs who influence what gets published through back-channel alleys. But even in the absence of outright nefarious activity, there's a reinforcement cycle where bigger labs influence what papers get in, receive more grant money, and get better about influencing what papers get in. Even gently guiding what topics are acceptable, in the long term shuts out newcomers. Just about any graduate student by the time they graduate has reviewed papers in a "double blind" system where they could reliably identify the authors, there are always tell-tale markers and styles. It's really hard to find true anonymity.
I've been thinking on an off about review system that might improve on things. I'm imagining perhaps: reviewers and authors both get to see who they are, and conflicts of interest can be called out by other people after review and before publication; reviewers are chosen at random, not allowed to bid; reviewers are sent a series of pairs of papers, and asked to choose which one they'd rather see, scores and ultimately publication can be decided by rank choice vote rather than reviewer assignment; comments on paper improvement would be completely optional. Would this be better or worse than what we have? Would it deter explicit collusion?
You can’t be as candid in your criticisms if the identity of the reviewers is made available to the authors. This is the main reason they’re made anonymous. If you think there can be a conflict of interest, the main mechanism in most fields is to point who shouldn’t be a reviewer for your paper. If you forget someone and they are asked, it’s also their responsibility to recuse themselves. Of course it’s based on trust. Trust is still the foundation of academia.
I’d agree with all those thoughts with respect to the system we have today. Yes, the goal of de-anonymizing reviews would be to eliminate trust in reviewers as a requirement for achieving impartial and fair reviewing (given the many ways we’ve already seen trust failure), and hopefully the side effect would be to increase overall trust in the system. Making reviewers part of the public record would also allow exposure of bad behavior and explicit collusion, without having to go to great lengths to prove it like they did in the article. I’m not entirely sure that the level of candidness that anonymity invites is necessary or helpful for a healthy and robust review system. But, note that my suggestion above doesn’t depend on critical feedback at all. One of the things I’ve found problematic in today’s review system is that reviewers assign absolute scores to a paper, but every reviewer has their own notion of what is good and bad. It’s common for an entire group of reviewers in one sub-topic to give average lower scores than a group in another sub-topic, meaning that different sections of a given journal or conference have different acceptance rates. My suggestion for rank choice voting is partly to ensure that all reviewers are working in the same units, and are all weighted equally. (But I assume there is a possibility of unintended consequences and/or opportunities to game the system with what I suggested, so I’m curious if people see ways that might happen.)
If reviewers are de-anonymized, the opposite of what you think will happen. Nobody other than the most socially stunted would be willing to write a harsh review and make an enemy for life. People are petty. Most fields in academia are incredibly small, with everyone knowing each other. Reviews would become way more positive and useless. Collusion for positive review is pretty rare and wouldn’t really change with real names anyway because it’s pretty subjective to claim a positive judgement is obviously the result of collusion. It’s also already the job of the editors to collate the reviews and make his own judgment based on them. If one review is overly positive and adds no useful information, it can and will be discarded. The editor can even call for another reviewer if he/she is still undecided. I’ve reviewed papers and recommended acceptance, yet the paper was still rejected; the opposite has happened too. As a reviewer you are only advising the editor.
> Reviews would become way more positive and useless.
In my corner of CS, there are plenty reviews of debatable use that are very negative.
In fact, I have a pet hypothesis that the average score of the lower scoring but accepted papers is negative (scale from -3 to 3). And I wouldn't be surprised if the median paper's score was negative.
Positive uselessness doesn't seem like that much worse.
Are you sure about that? I’m not sure you are hearing and understanding my suggestion. A review would always consist of one paper being ranked above another one. The tone of the review comments would have no bearing on the usefulness of the review. The reason we even have critiques in reviews is to justify the score that the reviewer assigns. Reviewers are specifically asked to produce reasons. What I’m suggesting is different because reviewers would rank two papers against each other and not be expected to justify their decision, nor would they be asked to provide a magnitude for their opinion. Comments and critiques would be allowed, but there’s no need for harsh reviews.
I’ve seen a lot of overly and unnecessarily harsh reviews. Anonymizing enables over-stating criticism, it happens routinely. I don’t think I agree that harsh critique is necessary for a healthy review system. It is already the case that good reviews are not extremely harsh, they focus on the facts and are willing to stand by their statements. I don’t personally know that many researchers who have trouble being direct in person and face to face, or of offering constructive criticism.
> As a reviewer you are only advising the editor
This completely depends on the journal or conference. Quite a few of them, especially the larger ones, do not override reviews casually nor often. And what I’m suggesting is a system where this idea can change, where editors can more easily trust the review results, and won’t need to override the decision.
> People are petty.
This might well be true. And so I’m not entirely understanding your argument. It seems to be simultaneously suggesting we have a problem, and defending the status quo as the way it needs to be. What would you suggest as a way to improve the review system so that pettiness has less influence than it does today?
Ranking only makes sense in a venue where the number of papers or presentations is limited. There many online journals of good reputation that will publish anything over a certain (obviously not perfectly well-defined) bar. The number of papers tends to increase over the years as the numbers of researchers also increase, and this is considered acceptable. Other journals are so selective that they won’t publish anything if nothing is deemed worthy.
You’re also misunderstanding the point of reviews. They also serve as comments to the authors to modify their manuscript and make it acceptable for publication.
I fully support the concerns that the author brings up. It's a big problem and we need to figure out how to address it. At the same time, I can't help but notice that this is coming from the ACM itself. The credibility of ACM has really tanked in my eyes in the last 2 - 3 years. Communications of the ACM magazine is full of articles that belong to the "social engineering magazine" and not to the Computer Science publication. Last time I was renewing my membership, I had to sign some kind of a pledge "not to harass people". Are you kidding me? What are we, 12? Because of that, while it's a very important issue, I am very distracted thinking about ACM itself. I guess this is a lesson that reputation and credibility is important. Once you lose it, everything you say, even if it's truly good, gets colored in a certain way.
One way forward could be to lower the bar for publications.
Once it's no longer about being in the esteemed and scarce "10%", they won't bother because they don't need to. Imagine a process where the only criteria are technical soundness and novelty, and as long as minimal standards are met, it's a "go". Call it the "ArXiv + quality check" model.
Neither formal acceptance to publish nor citation numbers truly mark scientific excellence; perhaps, winning a "test of time award" does, or appearing in a text book 10 years later.
411 comments
[ 1.6 ms ] story [ 296 ms ] threadhttps://omscs.gatech.edu/cs-7641-machine-learning
After that event, SIGARCH launched an investigation. After a couple years, here were the results of that investigation.
https://www.sigarch.org/other-announcements/isca-19-joint-in...
Worth noting is that the investigation actually initially found __no__ misconduct. Imagine that? A student kills himself, and you conclude it was the victim's fault, and not the environment that drove him.
It was only until this post [1] emerged that they relaunched the investigation.
[1] https://huixiangvoice.medium.com/evidence-put-doubts-on-the-...
> It should have been unnecessary that we expose these evidence and challenge the result of the investigation, if the committee can drive a responsible, transparent and thorough investigation
An interesting question : Did ACM require the followup Medium article to update their position? I don't know the details of the case. However, merely updating positions when situations are black and white are some of the easiest scenarios. I wouldn't be impressed if black and white situations are assessed as black and white. This doesn't mean that one shouldn't do so. I'd expect those scenarios to be a bare minimum requirement.
Did I miss something? All I can gather from the announcement PDF is that “several individuals” have been disciplined to varying degrees, the least severe being just a warning letter. No names named, no other details. Most of the announcement was just reiterating they took the investigation seriously. Kind of hard to determine from the announcement what details have and have not been considered, and whether certain individuals have been punished too lightly, no?
The announcement does mention a confidential report has been submitted for further review. Did anything concrete ever come out?
(I suppose it would at least be relatively obvious after a while which individuals are subject to a 15-year ban.)
In perhaps a generation, a similar specialty might be bootstrapped and begin to take on problems had been of interest in the old one. What to call the new specialty will be its smallest problem.
Why do you say that? Do you have any experience in the field?
Your position also paints CS in very broad strokes; in my experience, the only commonality between some subfields of computer science is that they use computers. Graphics, hardware architecture, programming languages, networks, and so on, are all essentially loosely coupled with their own organizing communities and directions. Some of these subfields are more closely tied to mathematics or electrical engineering than strictly to other parts of computer science. If there is an incurable "contagion" that afflicts all of these, I must admit it hard to believe that this contagion would not prove (if not already be proven) effective beyond the artificial confines of the term "computer science".
Which if one looks at the state of economics right now, is an object lesson on this kind of stuff never ending well.
Maybe it's time to move on from some of these conferences, and focus on interactions that maximize sharing research findings. I know that is unrealistic, but like every other metric, conference acceptance ceases to have value once the metric itself is what people care about.
[1] https://youtu.be/DJFKl_5JTnA?t=853
And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.
The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics?
The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.
So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money.
As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.
Findings and papers from well known individuals (read: twitter accounts) do get far more attention , more citations. Of course, one can argue that, broadly, well known labs and individuals are wel known because of their tendency to do great work, write better papers. And that’s true. However, the above still holds, in my experience as PhD student in ML. Anecdotally, I have seen instances where a less interesting paper from a renowned lab got more attention and eventually more citations than a better paper accepted at the same venue by a less renowned lab on the same topic.
I would also argue that with the increased importance of the (mostly) commercial "high-impact" journals this has become worse. I know that some of the professional (non-expert) editors of these journals specifically look at the citation counts of authors before accepting to send them out to review, because their main aim is to get people reading the articles, not necessarily good science.
I believe journals need to adapt the openreview process. There are obvious parallels in the lifecycle of a journal submission (from submission to acceptance) and the openreview process, except that the latter is accelerated (for better or for worse).
I have to say that I find the review process of the copernicus journals very interesting. You can see a description here: [1]. Unfortunately I don't work in a related field otherwise I would have published there already.
[1] https://www.atmospheric-chemistry-and-physics.net/peer_revie...
> We find considerable evidence that, overall, article citations are positively correlated with tweets about the article, and we find little evidence to suggest that author gender affects the transmission of research in this new media
https://journals.plos.org/plosone/article?id=10.1371/journal...
(I've only skimmed the paper, a few months ago).
When someone is hired (whether for tenure or time-limited), their research as a whole has to be evaluated by external, independent committees who take into account the content of the research and do not base their judgment on indicators only. There is no shortcut around that.
The biggest annoyance nowadays is the decision-makers' insistence on "excellence", though. You cannot have only excellent people everywhere, as per the definition of "excellent", yet this demand is in every fucking guideline for postdocs and tenure-track position. It's absolutely ridiculous.
The problem with this assertion is: non-experts are paying for all of this.
Imagine you are an ordinary taxpayer. You are feeling the pinch yourself, you look around you and see infrastructure crumbling, every day the press says healthcare and this and that is underfunded. Now along comes some scientist, he or she wants a few billion for a new particle collider that will make no difference whatsoever to your life, and their only justification for it is "well other scientists say we should get all this money, and they're cleverer than you, shut up".
Can you see why funding something with no accountability might be considered problematic?
There's also the fact that an expert who can't explain something to a non-expert probably doesn't understand it very well themselves...
It took over 20 years after the Standard Model reached broad acceptance (ie., the experts thought the theory was probably right) for the first supercollider powerful enough to observe the Higgs boson to be financed. This was enough time for policy makers to reach high confidence that the experts had not gone badly wrong and for there to be reasonably well-informed public opinion on the merits of the search.
> Can you see why funding something with no accountability might be considered problematic?
Maybe you're confusing budget decisions with funding and hiring decisions. These are fundamentally different. Universities and research institutions, as well as national funding authorities, get budgets that are decided politically, i.e., by elected representatives. These can have broad categories and guidelines or preferred research areas (e.g. "excellence initiatives"). Budgets are usually allocated well in advance, for instance our national funding authority gets budget security for 4 year periods (if I'm not mistaken). How they spend it is dictated by political guidelines for the respective period and plenty of complicated national and international laws.
In contrast, I was talking about hiring decisions and decisions about individual funding. How can it not be obvious to you that these decisions need to be made by experts on the basis of CVs and scientific project proposals, not by politicians or other laymen?
E.g, for my company's twice-yearly evaluation, everybody writes up a short report on the most impactful stuff they've done including evidence, this is evaluated by their manager to give a score, and then there's a series of group meetings between managers to make sure that the scores are calibrated, including looking at all types of metrics that can be dug up and comparing to our written role descriptions for different levels. It takes a lot of time but creates fair scores.
This is extremely labor intensive, but that's the thing: To create anything resembling fair evaluation of a large group of people that do a large set of different things, you need to do things that are labor intensive. Using a simple set of metrics don't cut it.
It has predictable results. Where are we going to get reliable research, and anything else, if we can't trust each other. Trust is an incredible business tool - highly efficient when you can take risks, be vulnerable, and don't have worry about the other person. Trust is an incredible tool for personal relationships, for the same reasons, and because if you can't trust them and can't be vulnerable, you have a very limited relationship.
I'm not sure that you intended it this way, but this reads as very oblique (i.e., "wink and nudge"). Which subculture are you referring to, and what particular relationship do you think they have to research in Computer Science?
I am referring to no particular subculture. Lots of people around me embrace it, including from all over the political spectrum (if that's what you are thinking).
I think the broader society sets the norms for computer science, as with everything else. For example, when star athletes like Barry Bonds, or entire teams like the Houston Astros, or much of college sports, cheat with few reprocussions (and in the past, that wasn't the case - players were banned and school sports programs were basically shut down, etc.) that affects computer science research.
Arguably, it's the opposite. Once people realized academics could do this kind of cool shit, they got showered with money and told to do whatever the they want. That's how we got the incredible scientific and engineering advances of the second half of 20th century.
Then the beancounters started asking questions about what the money actually buys, and research quickly turned into another short-term, self-contained, profit-chasing game, starved for resources and only occasionally producing something actually useful.
As it is, if our researchers are spending almost all their time thinking about and doing things other than research, what do we expect?
On a tangent, software industry has a bit of similar problem, with the best developers being forced to enter management roles[0] instead of solving technical problems. That is, a developer progresses from doing shoddy work to doing mediocre work and then, just as they start doing high-quality work, they get told to manage a new cohort of juniors doing shoddy work instead. I wonder if that's why so much software is hot garbage these days.
--
[0] - Whether proper ones on management path, or "fake" ones like principal developer, where you get all the managerial responsibilities with none of the authority.
Thaaaaannnkkk you. It has technically been reserved for Christian Aristocracy, which is practically a world where there is no desperation for calories. Instead many of the greats had an anxiety of their immortality (see Fourier).
It's up to you and me. Nobody else is coming to save us.
One way we can make the world better is by fixing the rules. Getting rid of ones that are unjust, and making the rest more consistent.
I once saw sign on a street: do not do something (do not remember what) and reference to a City Bylaw numbered as 37 thousand and something. That is just for one city. Good luck changing this sheer insanity.
And I think it is. At this quantity the quality will definitely suffer. Besides, I did read some bylaws at some point out of curiosity and without going into details many of them are outright unjust/deficient/etc (in my opinion of course)
The question of course is, peoples interpretations of which laws are just and unjust are subject to bias and individual incentives.
Incidentally, while I recognize thr popularity of this quote, its fairly ridiculous taken literally for laws which are prohibitory rather than obligatory.
Viewing a prohibition as unjust does not obligate me to violate the prohibition; believing people should be free from government constraint to do something doesn’t require me to do that thing.
“Disregard” or “discount” in place of “disobey” would be more generally valid.
So outlawing theft, rape, fraud, kidnapping, false imprisonment, etc. - that's all unjust?
https://en.wikipedia.org/wiki/Joseph_Tainter
Quoting from the book:
“Sociopolitical organizations constantly encounter problems that require increased investment merely to preserve the status quo. This investment comes in such forms as increasing size of bureaucracies, increasing specialization of bureaucracies, cumulative organizational solutions, increasing costs of legitimizing activities, and increasing costs of internal control and external defense. All of these must be borne by levying greater costs on the support population, often to no increased advantage. As the number and costliness of organizational investments increases, the proportion of a society's budget available for investment in future economic growth must decline.”
We used to have strong institutions that were supposed to help push groups into not choosing the bad corner of the prisoners dilemma, but they all seemed to have degraded. I suppose they could have always been like this and the curtain has just been removed, but I’d argue that the perception that following the rules was the best personal choice is almost as valuable as it being true
This animated gif makes it clear: https://invisible.college/reputation/declining-trust.gif
This trust is greatly eroded by calls to eliminate direct taxes on land holders. In America the local property tax is the most important thing holding society together. It provides residents with assurances that regardless of how corrupt the public process for distributing legal tender becomes, that the richest cannot simply buy the entire country and turn the continent into a private estate which their descendants will inherit in perpetuity without paying enormous taxes to everyone else.
When James Madison organized the assessment of property taxes at the national level during his presidency it resulted in the 'Era of Good Feelings' and a relative low point of political polarization. In contrast when state governments introduced sales taxes for the first time to reduce property taxes it prolonged the Great Depression, and when NYC gave the largest property tax abatements in the country to Donald Trump it lead to the Trump presidency which increased political polarization.
(1) If you lose trust in someone, you'll be less likely to try to find truth in their statements. Finding truth in someone's statements requires time and attention, and we'll invest that attention in people we trust.
(2) If you find falsehoods in their statements, you will lose trust in them.
Overall in Psychology, trust is a primary indicator of a relationship, marriage, partnership, or organization succeeding or falling apart. It's both a symptom and a cause of all other factors.
The classic example of the later are the politicians who rant “if only these people would get married and stay married, they wouldn’t be so poor” and neglect to consider that impoverished communities create the conditions for rampant single motherhood. The desire to raise children does not magically vanish merely because there are exactly zero worthwhile men in your community that aren’t your father’s age.
In the old days you had to know your place. WASPs smoked cigars and ran things. Those old guys drinking sherry and wearing tweed helped each other out. The Irish were cops, Italians firemen.
In tech it’s pretty obvious to see various constituencies doing dishonest shit help others out.
america has urbanized rapidly in the last half century, at the same time that family formation has broken down and life-long jobs have become a thing of the past. we are atomized and thrust into constant competition. i don't mean to idealize a past that i did not even experience, but there is something to be said for having roots and knowing your neighbors. we arguably have more opportunity at the cost of stable identity -- reputation and trust naturally accrete around the kind of stability we lack.
if you talk to older people, people around my grandparents' age or thereabouts, you will hear that they no longer recognize america, that it is fundamentally different than the culture they grew up in, in terms of values. i find myself thinking about this a lot.
I think knowing your neighbors is overvalued. My evidence is Tokyo and living in transient, largely ethnically homogenous sharehouses — in ethnically homogenous areas — for long periods of time.
From the outside this degradation of values and "social lawlessness" has been apparent for years now, especially since 2016.. Really hope this doesn't spill over.
Oh, and good lucking mending this.
More specifically, it's the perception of a breakdown that drives this behavior.
When it comes to cheating, there's a growing perception that "everyone else is doing it" and therefore it's not wrong to play the same games as everyone else.
The current political and social media discourse revolves around ideas that "the system is rigged" combined with a die-hard notion that anyone who disagrees with you is wrong and/or evil. When people are bombarded with these ideas every day on their social media feeds, cheating a little bit to get yourself ahead doesn't feel like cheating. It just feels like leveling the playing field.
Depends on how you define "lose". And also on what you think the "rules" are.
For example: most drivers in the US routinely exceed the posted speed limit on roads. Is that "violating the rules"? In a legal sense, it is, since if a cop catches you he can give you a ticket and you pay a ifne and points go on your driving record. But nobody considers you a bad person for doing it, and I would argue that doing it, if you don't cause an accident, is not harming anyone. However, obeying the speed limit is also not considered a bad thing; all it really means is you get where you're going a bit slower. The tradeoff is yours to make; you don't "lose" by choosing to obey the posted rule.
Now consider an example at the other end of the spectrum: all of the shenanigans with mortgages and the financial system that caused the crash of 2008. Those who "followed the rules" leading up to the crash--for example, those, like my wife and me, who limited our mortgage and the size of house we bought to what we could comfortably afford--did not "lose". Sure, the value of our home went down, but we had no need to sell it then. It was still a house and we could still live in it just fine. Sure, we didn't have a bigger house with more bells and whistles, but we also didn't have to worry about what might happen if the housing market crashed. In other words, we made a tradeoff not much different from the one made by the person who obeys the speed limit and just gets where they're going a bit slower--but still gets there.
Basically if all it takes is getting citations, then forming a citation ring is "Chabuduo" and you'll only lose face if you are caught (not good enough).
[0] https://en.wikipedia.org/wiki/Jugaad
[1] https://news.ycombinator.com/item?id=27052249. This reminds me of the Japanese buzzword bingo of earlier decades.
You are absolutely right that trust and reliability are very valuable. Societies with high trust tend to be richer and much more productive than societies overriden by cheating and corruption.
Like the anecdote that he leased a new car every month so he could avoid registering it with the state.
Getting a divorce court judge to force you to sell your Enron shares at the top, nuking the regulator’s ability to charge you with insider trading, while you elope with your younger hotter high libido stripper nymph to the mountain you bought?
These are our role models
Why should it be possible at all to game Journals in this way? Particularly in Computer Science journals where people think about edge cases for a living...
Those systems that can be built to be resistant to greed would definitely benefit from it - but I think it's more of an issue with society at large.
(The redeem/blueteam thing is kinda unethical, so maybe the University of Minnesota should do it...)
Absolutely. I think people are intimidated, demoralized (de-moralized) and where once they believed anything was possible, any social problem could be solved (even those old as history, such as women's rights, human rights, etc.), now they've somehow drunk the wrong Kool Aid, some stuff distributed by Jim Jones.
Time to get to work.
I'm a US expat and escaping this culture is one of the things that's made me happiest - I tend to call it the bullshit culture because my favorite example is... writing a good paper for class is admired - but what's really praised is writing a paper that gets good marks without ever having read the subject matter. Being able to spin lies about a topic you've no understanding of and turn that into a marketable skill is a dark potent for the future of America. I think it's always been somewhat present, but since emerging strongly out of the business world in the eighties it's gained a lot of steam.
We are a society that can benefit from cooperation where everyone gets a fair slice of the pie, but that society is eroded if we praise and not shame those people who betray societal trust and cheat the system.
At least in academia, American universities (and Western universities in general) have had a good track record overall with academic integrity, and it has set them apart. This seems to have degraded recently, perhaps because of the increasing pressure of the “publish or perish” system (or dozens of other potential causes).
We do celebrate people who excel academically seemingly effortlessly. But we don’t celebrate bullshit artists so much in school. In business, and particularly tech, it’s another story.
The difference is that these collusion rings or bogus studies are discussed and exposed publicly. It's not the case in Asia.
You can read a bunch of articles on Bernie Madoff, Martin Shkreli and Billy McFarland that romanticize the cunning with which these folks exploited others. Most pieces on them will present an overall negative tone but often feature some pretty glowing admiration of them. Let's also not forget that tax evasion by Trump was praised repeatedly as him beating the system - that's a pretty common view point, more common (especially when it comes to taxes) than the view that those individuals are failing to pay their fair share from what I've observed at least. Trumps a complicated example due to all the political baggage around him so maybe just look at companies like Apple, Google and Facebook - they regularly offshore large portions of their profits and I really doubt the people working to those ends feel any shame, instead it's likely a "beating the system" motivation.
This is a bit of a public secret, but quite widely researchers don't really trust articles anymore, if they ever did. Maybe some plot or dataset may give some insight and maybe some discussion has worthy information to ponder on. But mostly they're just some ads to put in a yet another funding application.
Most articles are just churned out to get some lines to CV or to look good in some metric. Publish or perish has turned into full-on bullshit or perish. The whole peer-review system (which is just around 50 years old anyway) is on the verge of just grinding to a halt due to the stupendous volume of hastily hacked together manuscripts.
I think many are still sort of hoping that this will somehow sort itself out. But the collapse of the quality after the explosion of electronic journals, consolidation of publishing houses and overall structure that doesn't really care at all about what is actually in the papers doesn't give much realistic hope.
It should be noted that there's sort of a "parallel reality" in academia behind the publication show. The ethos for academic integrity is still quite strong, teaching tends to be valued by the community (but not by the system) and face-to-face discussions can be of very high quality. But the signal-to-noise is so low in publishing that it's not really worth following.
We really need to get some new arrangement so that we don't drown in all this bullshit. Word-of-mouth, open data repos, conferences and just blogging and pushing stuff to git repos probably is most that's needed. The publishing structure is becoming just plain unnecessary bureaucracy.
SEO is a kind of explicit analogy. Google pagerank was modelled on academic publishing, and it worked until it went live. From that point, links started to decrease as a quality signal.. spam. Publish or perish is a similar sort of dynamic.
Honestly, I think most legible systems for determining merit have these sort of issues. If advancement, accolade, grants or somesuch are determined by a formal system, whatever that system used as a signal or metric becomes corrupted. Hence why Word-to-mouth, open data repos, conferences and just blogging and pushing stuff to git repos does work. It's informal.
A sort of reputation system is in place in almost all peer-to-peer societies, it tends to form automatically. I don't think we really need any of this weird mess of a system.
We have Wikipedia, we have open source, we have OSM, we have all sort of things that should be "impossible" given the dismal perception people have of other people. This perception is just plain wrong and really harmful.
It's a hard sell though. The cost of metering is subtle. The do-nothing tenured professor is visible.
Wikipedia, OSS, etc really are the shining beacons. Existence proof for something better. Someone needs to write The Cathedral and the Bazaar, but in non geekish.
Perhaps surprisingly to some, many in academia would just like to research and teach with some quite modest salary and don't have to think about money at all. E.g. I would gladly and with no hesitations take a €2000/month tenure and keep on doing what I'm doing just more efficiently for everybody. I've been trying to pitch this idea to the funders here in Finland, but to no avail, they simply don't care if the funding system is useful or not for the academic community or humanity, they're focusing on playing the same old (maybe 10 years or so here) application lottery that's not only waste of time, but corrupts the whole community and even the very content of thinking in academica.
"Money" in academia is really abstract as well, and when discussed its not salary, but funding for projects or students or such. And because the funding structure is so bizarre and convoluted you just see big numbers with currency signs flowing everywhere, but this doesn't seem to have much to do with anything concrete happening around.
If academia becomes a place where you can get rich, the system will be in just years corrupted into some bizarre thing where advertisers advertise to each other for the sake of advertising.
Luckily cats can't be herded.
Some of it is intentional "motive hacking." As you say, prestige, research funding and the like are as (or more) operative as salary.
Some of it is unintentional. Before publish or perish, publishing volume probably was a signal for something. I doubt it was ever a signal for high quality research, but low (or no) volume may have been a signal for low quality. Also, formal decision making bodies (like grant makers or tenure committees) tend to gravitate to quantitative, legible metrics.
Whatever the reason initially, publishing volume became a hugely important thing with impacts on many aspects of research.
At the same time, in CS especially, the number of researchers has also ballooned. That's a whole other strain on a system of, at core, knowledge dissemination.
The antagonistic view is not just towards my colleagues, I'm not particularly proud of my own papers either. I find it more a nuisance to "pay the bills" and a lot of my research goes unpublished (at least in journals) due to all the IMHO unnecessary hassle involved. Just a blog or something would be a lot nicer and probably would communicate the work better, and would ease the pretension of objectivity which I find mostly causes wrong impressions and makes writing really a chore.
> It should be noted that there's sort of a "parallel reality" in academia behind the publication show.
So what should be the guidelines for someone who is not a researcher, but an engineer, and hopes to stay informed by reading relevant papers from a specific field. (You know the folks who should apply some of that in practice)
Also individual papers tend to focus on one very specific problem at a time. This is typically related to some actual larger "debate" and can be difficult to see if one's not familiar with the larger issue. Also especially conclusions tend to have quite heavy implied assumptions that are just generally accepted in the field.
I "stay informed" mostly by face-to-face discussions and emails and such. I don't read much papers myself, but many of my colleagues do and I just hear from them, or ask them if there is new stuff around related to something I'm pondering.
To get an overall view of "state-of-the-art" I'd recommend starting with masters' or doctoral theses. These typically require more elaborate presentation of the background and its typically put out in more readable terms with less assumptions of the readers background knowledge.
In some fields review articles are a good starting point as well, and they tend to briefly sum up the required background, but my understanding is that some fields don't do those much.
If you read "random" articles, I'd do a quick smell-test before digging in. See if code is available, ignore papers with clear hype in the abstract off-hand. You can also "navigate" the field by following citations, although this can be technically annoying as the publishing format is still tailored towards print, even though very few journals are actually printed anymore. If you hit a paywall, try sci-hub or just move on to a next one unless you're looking for something really specific.
If you have something more specific in mind, just email or call or go talk some researcher that looks to be doing something related to what you are looking for. Researchers tend to be quite eager to answer to the public of their stuff, and its seen as sort of a public service duty as well. Depends on the researcher quite a bit though. Maybe a good starting point would be somebody a bit "lower on the ladder". Maybe a postdoc or a PhD student (this depends on the country as well). Professors tend to be busier and actually may not be that up-to-date with their field (especially on technically detailed level) as they spend most of their time in administration and the funding ratrace.
Depending on the country you can just attend lectures too. At least in Finland university lectures are public by law (with some restrictions on e.g. practical lab stuff etc). You can see if the lecturer doesn't seem too busy after the lecture and just go and ask.
You can also just try go to conferences. They usually have a fee in theory, but I don't think you'll be turned away if you just browse around for posters or so, especially if its a smaller one. The fees are just sort of a scam (long and sad story) and researchers organizing the thing usually don't care about the fees at all.
For some fields there are some good youtube channels that provide summaries that can get you started. E.g. Two Minute Papers is good for machine learning/machine vision/"AI"/etc related stuff: https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg For computer graphics SIGGRAPH "video papers" are really nice and even entertaining: https://kesen.realtimerendering.com/si...
My comment was on the historical status of academia in the US as a whole (think last 120 years), not just the current state globally.
You’re lamenting the quality of academic publishing in particular. The US now publishes less than 17% of science and engineering papers, but its papers are often the most highly cited. So yes, there has been a huge increase in the number of papers, and number of low-quality papers, but this isn’t necessarily being driven by the US, as the original comment would have implied.
You claim researchers don’t really trust articles anymore, and I agree that it takes a lot more work to filter out the noise now, and I’m less optimistic that authors are presenting an honest, objective appraisal of their results. But significant research is still happening, and academic publishing is still the primary way that information is disseminated. People seem to rely more on name recognition (author, school, journal) now. It probably varies field but field. I’m in a field where results are often proof-based and that tends to be harder to fake.
Internationalism is so ingrained in the academic culture (at least on fields I'm familiar with) that it doesn't even really register what country somebody's from or is working in. There are definitely some differences especially in the more "overt" parts of the culture (hats and robes and different titles etc), but these are of very little significance for anything but some ceremonies.
My working experience is from Finland, Sweden and UK, but in academia people come and go between countries very frequently so colleagues tend to be from all over.
There are at least some stereotypes that some countries are more prone to the e.g. citation rings, but I don't find that very relevant, as I think the whole system is quite broken and the publishing (at least in English language) forums are typically not country specific at all. Probably something like this happens in more or less any country.
This was not my experience of school in the US. Just as one example, two words: Cliff's Notes.
Really? Cause I'm an American who has lived, studied, and worked overseas. Let's just say it's not American (or generally Western) coworkers and classmates who are NOTORIOUS for cheating.
And I think many of us here who've attended "diverse" universities or work for companies with "multicultural" staff have a pretty damn good idea which cultures and nationalities are more likely to be cheating.
I don't think this is an American thing. In fact, I've never seen this praised anywhere outside of maybe a few people back in High School.
I worked for a company that expanded rapidly with distributed offices all over the world. One of the growing pains we had was that the managers from certain countries, America included, were very trusting by default. This opened the door to a lot of manipulation from employees in certain other countries (which I'm deliberately not going to name) where getting away with a lie was more or less considered acceptable as long as you weren't caught.
At once, such a great, and terrible, sentence.
At once such a great, and terrible, sentence.
Not that I ever believed him much about that (I do think he believed what he said...).
Anecdotally, I've heard stories about Zuckerberg confessing/bragging about all sorts of nasty things at these dinners.
Really, this stuff should just be shamed. Sadly, too often calling out bad behavior just gets you called a "hater"...
This coworker told me he and his "startup":
- routinely lied to potential customers on the size of their client list
- misled clients on the depth and completeness of their product
- blatantly broke CA laws to cut cost corners
All of this was done to secure contracts in order to secure more funding. "Always be selling", he said.
He literally fucking said to me that he learned to "be dubious, not deceitful" which is probably one of the most deceitful things I've ever heard.
Made me sick to my stomach and pretty much validated (1) why I never moved to SF in the first place, instead moved to NYC and (2) how much of a fraud YC has become. Absolute fucking madness.
This almost necessitates some cheating to survive and the only people left are those who survived this system and hence the culture slowly rots.
Both “Athens and Jerusalem” are principled according to honest toil. And look at the results!
Having 'hustle' is actually important at a startup, it's part of the essential aspect of it. I'd argue a 'hacker' has a kind of hustle.
I'm pretty suspicious of these things as well, but I've also come to believe in my many years that a bit of koolaid is fine as long as it has self awareness.
I wish Elon would stop posting all the stupid things he does, but I think that's what you get with 'all the other stuff'.
If Elon were fully polite, conscientious, a 'good listener' I'm not sure Tesla would exist. So we pay for the existence of Tesla by accepting that he's going to do dumb tweets about Crytocoins.
I think that's a BS excuse for bad behavior. How would being polite and concientious harm Tesla? It's simply that he has power and has low standards for his behavior.
I know plenty of successful people who do what Musk fails to do.
Except you don't know any successful people who've remotely done what he's done either.
Talking about 'Dogecoin' is a little unseemly, but it's nowhere in the realm of 'toxic' or 'bad acting'
Without massive public support and sympathy, Telsa wouldn't exist, it's a movement as much as anything, and so you need kind of a showman.
His appearances on SNL etc. are part of that public drama that keeps Tesla stock going with enough support to keep the legitimacy of the dream alive.
I'm seriously doubtful that a quiet, unassuming person would have been able to do most of that.
Expressive, bombastic characters will by virtue of the volume of their actions, sometimes creep up to the line. It's normal. There's nothing wrong with Elon, he's just a little cheezy and spouts too hard with some things.
(I'm trying to respect the HN tradition/guideline of not talking about votes. I don't really care about Internet points; I think the implied interest in the issue is worthwhile and applicable in this case.)
"How to get an objective rating in the presence of adversaries"
It is probably extensible to generic reviews as well... so things like the Amazon scam. But in contrast to Amazon, conference participants are motivated to review.
I honestly don't see why all participants can't be considered as part of the peer review pool and everybody votes. I'd guess you run a risk of being scooped but maybe a conference should consist of all papers with the top N being considered worthy of publication. Maybe the remaining could be considered pre-publication... I mean everything is on ArviX anyways.
So instead of bids you have randomization. Kahneman's latest book talks about this and it's been making the rounds on NPR, NyTimes etc...
https://www.amazon.com/Noise-Human-Judgment-Daniel-Kahneman/...
https://en.m.wikipedia.org/wiki/Collusion
However, they review a limited amount of papers (e.g. 3) - "everybody votes" presumes that everybody has an opinion on the rating of every paper. That does not scale - getting a reasonable opinion about a random paper, i.e. reviewing it, takes significant effort; an event may have 1000 or 10000 papers, having every participant review 3 papers is already a significant amount of work, and getting much more "votes" than that for every paper is impractical.
It's unfeasable and even undesirable for everyone to even skim all the submitted papers in their subfield - one big purpose of peer review is to filter out papers so that everyone else can focus on reading only a smaller selection of best papers instead of sifting through everything submitted. The deluge of papers (even "diarrhea of papers" as called in a lecture linked in another comment) is a real problem, I'm a full-time researcher and I still barely have time to read only a fraction of what's getting written.
This issue reeks with the rank smell of base politics and in/out group dynamics, and humans have been fighting, in the abstract, these issues since the time Egyptians were building the pyramids.
How can there possibly be an "objective rating" when career advancement, peer respect, and big money all are in the mix depending upon results?
In other words objectives are deceiving and rating is based on objectives.
Example: mRNA inventor being sidelined at her university when her method wasn't famous
Example2: Schmidhuber inventing stuff and being forgotten because data and compute were just too small back then
It's all about building a diverse collection of stepping stones. Any new discovery might seem useless and we can't tell which are going to matter years later, but we need the diversity to hedge against the unknown.
Take for example a paper that presents a very innovative method, but with subpar results; and another one that presents an incremental improvement on some existing method, but with results that advance the state of the art. Which is better?
Even if you ask knowledgeable, careful and honest reviewers, you will get contradictory responses, because it's highly subjective whether you rate originality as more important than results or vice versa (and other factors, like whether you think the first method can eventually be improved to be useful or not, which is often just an educated guess). I see this happening all the time, and I don't think it's something that can be "fixed", it's just how humans work.
It's, of course, a very hard problem to solve. It takes a lot of effort to evaluate the real impact of research.
Only a tiny percentage of developers seem to actually enjoy coding - Most of them have no interest in it and only see it as a mechanism to acquire money, power and influence.
Disinformation is rampant because contrarians are punished and conformists are rewarded. The rot is deep in the guts of the industry. Those who have the most power and the loudest voices hoard all the attention for themselves and are unwilling to give exposure to any alternative views - Their deep insecurity drives them to surround themselves only with yes-people and to block out all critics; avoiding disagreement at all costs... Downvote, suppress, censor...
Powerful people in this industry need to put aside their insecurity by embracing disagreement, allow themselves to change their minds, and give a voice to contrarian views and ideas; even when it risks hurting their current interests.
Powerful people should seek the truth and try to promote the narratives which make the most sense; not the narratives which happen to benefit them the most. Everyone is free to move their money to match the changing narratives, so why do powerful people invest so much effort in keeping the focus on narratives which only maintain the status quo? To protect their friends? To protect the system? That is immoral - Capitalism was not designed for this kind of arbitrary altruism. For every person you 'help', you hurt 100 others.
As much as people love to bash Elon Musk right now, he should be applauded for constantly trying to adapt to the narratives which make the most sense as opposed to rotting in his own filth and succumbing to tribalism like everyone else.
I realize you might be a human with feelings, but your screed above has a curious structure that looks... off somehow.
Anyway, if you are genuinely angry and wrote the above, I acknowledge your emotions, and don't have anything else to add.
Definitely written by either a human or a really bad AI.
Most humans don't have enough background knowledge or sufficiently diverse life experience to make sense of this.
I'd have to write a whole book to explain my reasoning behind this statement. I got a lot of my knowledge from HN articles and comments so it should tie in nicely.
How many industries have you worked in? This is fairly common in many walks of life. As I get older I'm getting better at identifying the lunatics in charge. Often they are nice people, but also cause massive amounts of chaos, because they have no clue what is going on. No wonder they feel the need to exert micro-control.
Anyway, thanks for the rant. Always interesting to look at the bottom of the barrel for the HN rejects :-)
And of course, I could choose to delegate the trust, and to “follow” someone, which would mean to incorporate their rankings, especially in areas where I don’t orient that much.
Do you think this would work?
I do agree that there's probably some cleverer solution on a personal level, but I think the journal system exists as a kind of guard against untrusted actors, and yet fails.
Are you asserting a 'trust' whitelist, a 'distrust' grey or blacklist, or some combination?
Are assertions meant to be linked to or backed by evidence?
Are these assertions publicly visible?
Is there a limit to the number of assertions that can be made?
I think you can see how such a system might devolve into formalized collusion.
*> And of course, I could choose to delegate the trust, and to “follow” someone
In the absence of explicit delegation or following, is trust/distrust intended to be transitive at all (ala PageRank or Advogato WoT)?
You almost forget now days that "getting a PhD" or "getting a tenured position" has nothing at all to do with the process of scientific research.
As long as you are happy to explore a field that doesn't require millions of dollars of equipment and aren't competing for fame, there's nothing stopping you.
And being landed gentry probably went a long way as well.
Chipping off the tip of an iceberg isn't a good long term strategy.
Would it help at all if rather than participants reviewing 3 papers, each reviewed 2 papers and validated the review of 3 more papers?
This is computer science here, with things like the set NP whose defining characteristic is that it's easier to check a solution than generate it.
I'm imagining having some standard that reviews are held to in order to make them validatable. When validating a review, you are just confirming that the issues brought up are reasonable. Same for the compliments.
Sure, it's not perfect because the validators wouldn't dive in as deep or have as much context as the reviewers, but sitting here in my obsidian tower of industry, it seems like it would at least make collusion attacks more difficult. Hopefully without increasing the already heavy load on reviewers.
(It very much seems like an incomplete solution -- we only have to look at politics and regulatory capture to see how far wrong things can go, in ways immune to straightforward interventions. Really, you need to tear down as many of the obstacles to a culture of trust as you can. Taping over the holes in a leaking bucket doesn't work for long.)
> In a well-publicized case in 2014, organizers of the Neural Information Processing Systems Conference formed two independent program committees and had 10% of submissions reviewed by both. The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself.
With this much demonstrated discrepancy between two sets of reviewers, it’s hard to believe that adding a validation step would produce a consistent improvement. How can people be expected to find improperly accepted papers when they have less than 50% agreement on the acceptance of good-faith submissions?
Honestly I think this seeming randomness in acceptance is at the heart of why people might think cheating is acceptable. If the process is not reliable, why bother submitting to it?
I referenced Kahneham's latest book, Noise, above but this is exactly the problem he focuses on. There are solutions.
Suppose that two reviewers independently rank papers 80% on quality and 20% on chance factors. With good odds, the two reviewers will agree with each other on the relative rankings of any given pair of papers. But their lists of the top 10% of papers will largely not be in agreement with each other.
One critical vulnerability in the current reviewing pipeline is that the reviewer assignment algorithm places too much weight on the bids. Imagine if you bid on only your friend's paper. The assignment system, if they assign you to any paper at all, is highly likely to assign you to your friend's paper. If you register duplicate accounts or if there are enough colluders, the chance of being assigned to that paper is extremely high.
Fortunately, this is also easy to detect because your bid should reflect your expertise, and in this case it doesn't. What we showed in our paper is that you can reliably remove these abnormal bids. It's not a perfect solution, but it helps.
I wonder if we could randomly assign reviewers but allow the reviewers to self-report a level of familiarity on the subject matter in general (ideally in advance) and on the paper topic in particular.
"The colluders hide conflicts of interest, then bid to review these papers, sometimes from duplicate accounts, in an attempt to be assigned to these papers as reviewers."
It might also help to attack the ability to create duplicate accounts. Given how relatively few professors exist in the world I'd assume you could put a lot more effort into duplicate account detection than they are right now.
I was actually very surprised that it is possible to register duplicate accounts at those CSE conferences. We get send a single invite to our work address and need to lock into the system using that email. And we are being nominated to get onto the committee.
I've been thinking on an off about review system that might improve on things. I'm imagining perhaps: reviewers and authors both get to see who they are, and conflicts of interest can be called out by other people after review and before publication; reviewers are chosen at random, not allowed to bid; reviewers are sent a series of pairs of papers, and asked to choose which one they'd rather see, scores and ultimately publication can be decided by rank choice vote rather than reviewer assignment; comments on paper improvement would be completely optional. Would this be better or worse than what we have? Would it deter explicit collusion?
In my corner of CS, there are plenty reviews of debatable use that are very negative. In fact, I have a pet hypothesis that the average score of the lower scoring but accepted papers is negative (scale from -3 to 3). And I wouldn't be surprised if the median paper's score was negative.
Positive uselessness doesn't seem like that much worse.
I’ve seen a lot of overly and unnecessarily harsh reviews. Anonymizing enables over-stating criticism, it happens routinely. I don’t think I agree that harsh critique is necessary for a healthy review system. It is already the case that good reviews are not extremely harsh, they focus on the facts and are willing to stand by their statements. I don’t personally know that many researchers who have trouble being direct in person and face to face, or of offering constructive criticism.
> As a reviewer you are only advising the editor
This completely depends on the journal or conference. Quite a few of them, especially the larger ones, do not override reviews casually nor often. And what I’m suggesting is a system where this idea can change, where editors can more easily trust the review results, and won’t need to override the decision.
> People are petty.
This might well be true. And so I’m not entirely understanding your argument. It seems to be simultaneously suggesting we have a problem, and defending the status quo as the way it needs to be. What would you suggest as a way to improve the review system so that pettiness has less influence than it does today?
You’re also misunderstanding the point of reviews. They also serve as comments to the authors to modify their manuscript and make it acceptable for publication.
Once it's no longer about being in the esteemed and scarce "10%", they won't bother because they don't need to. Imagine a process where the only criteria are technical soundness and novelty, and as long as minimal standards are met, it's a "go". Call it the "ArXiv + quality check" model.
Neither formal acceptance to publish nor citation numbers truly mark scientific excellence; perhaps, winning a "test of time award" does, or appearing in a text book 10 years later.