If this is to appeal to a community, there has to be some kind of incentive for that community to add their comments (content) to this site. What is it? Reputation?
In general, it could be appealing to have an open, established process for gathering public feedback on articles. It seems a scientific community would rather have their own version set up, where they could also curate content (rather than browse by category).
If the content was free and distributed this would be possible. You could start by entering data here and then setup your own instance with your own rules if it proves to be useful.
While reputation might be one allure, getting questions about a paper answered (even if not by the author) is helpful. Pointing out fallacies or holes-in-logic in a paper might be a poor investment of time given the number of papers to comment on.
A grander vision of gathering feedback as part of a normal process seems desirable, but I don't know that the incentives for stakeholders are present to enable that.
That may be great for what it is, but what I really want more than anything is to click on a link and be dropped into a buzzing Reddit for Science, full of scholarly commentary on arXiv posts.
SciRate seems to be trying to be that. It's not buzzing yet, but the UX has its heart in the right place. Unfortunately, I see from other comments that the site is pretty old and never quite managed to take off.
I wonder if my dream is ever going to happen... sigh
I'm not an academic, so don't spend many hours a week reading and thinking about papers.
And even back when I was a grad student, I wouldn't have been willing to bear all the weight of making all these discussions happen by myself. There have to be other people active on the platform to make it worth using.
There's a big chicken-and-egg problem, I suppose. But some subreddits, like r/MachineLearning, have gotten us off the ground at least.
I always wanted something like that too. I just yearned for rapid constructive feedback from individuals with subject matter expertise, interest, and time, rather than waiting 6 months for a response from people without one or more of those things, then having to relay comms through the editor middle-person. What you mention could make a nice arXiv-Reddit joint venture... or SciRate or Fermats Library fills the void _ prayer hands_
I remember scirate is pretty old - the domain was registered in 2007. Given the low number of comments now, and with those are concentrated into a narrow part of arXiv, it's unlikely to take off generally.
scirate.com was one of four options I found for comments on arxiv papers [0]; the other three being researchgate.net, pubpeer.com, and arxiv-sanity.com/discussions
Of the four, scirate.com seems the best option (unless you're focused on machine learning -- then arxiv-sanity is the winner).
I agree that comments on arxiv papers are "unlikely to take off" in either a commercial money-making sense or a "grow as large and popular as HackerNews" sense.
There are about article 500 submissions per day to arxiv [0]. I don't know what the number of monthly active users is for scirate.com, but the density of comments might be expected to be low.
I kinda of worry that if this site succeeded, it would slowly do to papers what news aggregators and the internet already did to news. Grind and incentivize them down into clickbait with no meat, with as few things as possible that people won't immediately understand.
For example, looking over my own papers for examples, a one page note [1] got more scites than a paper I contributed to that introduced several techniques cumulatively dropping the projected cost of simulating a chemistry thing by more than a factor of a million [2]. That's a cherry picked example, the pattern isn't that consistent, but having seen what happened to news sites it still worries me.
...not that the incentives for how to structure and publish papers are that great to begin with.
On the contrary, I've sat on Ivy League tenure panels where humanities members intoned solemnly about H-indices, oblivious to the differences in value of the Lira and the Pound. I also consider much of baseball statistics to be junk science, but at least they have "wins over replacement" that could help with the endless "does author order matter in this subfield?" debates.
I welcome a platform that could allow more sophisticated mind-mapping. I've argued to no avail for years that MathSciNet should be open to machine learning, try to make itself a premier playground for machine learning, but it's an AMS cash cow and leadership prefers a 19th century model. Out to lunch recently with a former AMS president, he suggested I should do this for the ArXiv. That's not the point; one wants to encourage significant existing projects to land on our data.
Math is pretty much bifurcated between tightly-bound field tribes, only reading papers within their field, and roving players who recognize ideas in any form and chase them across fields. Of course, tribal mathematicians would dispute this. Better mind-mapping would reveal the connections between what's out there, rather than imposing a tribal mindset on its organization.
It would be good for mathematics to change this culture. It takes more courage than one might imagine to be a roving player. At hiring time, most departments let the tribes take turns. Just like growing up on the streets, pick a gang for survival, if you want to get hired.
Funny to see this here. I created scirate years ago when I was in academia working in quantum computing. Back when Digg was hot. When I left that world and joined Google, I shut it down as I needed to spend all my time learning to code (true today as well I suppose). It was then rewritten by others. Twice! And relaunched using the old domain name. Many thanks to the current stewards.
Amusingly it was even more useful to me after I left academia. Scirate’s community is almost entirely theoretical quantum computing, and so I was able to keep up with the field by looking every three months or so at the top papers.
As noted in other comments, the end point of vanity voting isn't great. It's not great at discovery, but I have found papers there I would never have read. But because it is a small community it actually does a good job in quantifying what the community feels is important. I suspect if scirate was ever more widely used it would be less useful.
Another interesting phenomenon is that no one ever precisely defined what voting meant. Some people think of it as endorsing. Others as bookmarking. Not sure what impact this has, but sometimes if you score your own paper people will growl at you.
17 comments
[ 5.0 ms ] story [ 52.5 ms ] threadIn general, it could be appealing to have an open, established process for gathering public feedback on articles. It seems a scientific community would rather have their own version set up, where they could also curate content (rather than browse by category).
A grander vision of gathering feedback as part of a normal process seems desirable, but I don't know that the incentives for stakeholders are present to enable that.
SciRate seems to be trying to be that. It's not buzzing yet, but the UX has its heart in the right place. Unfortunately, I see from other comments that the site is pretty old and never quite managed to take off.
I wonder if my dream is ever going to happen... sigh
What would make _you_ contribute to make SciRate more buzzing?
And even back when I was a grad student, I wouldn't have been willing to bear all the weight of making all these discussions happen by myself. There have to be other people active on the platform to make it worth using.
There's a big chicken-and-egg problem, I suppose. But some subreddits, like r/MachineLearning, have gotten us off the ground at least.
Of the four, scirate.com seems the best option (unless you're focused on machine learning -- then arxiv-sanity is the winner).
I agree that comments on arxiv papers are "unlikely to take off" in either a commercial money-making sense or a "grow as large and popular as HackerNews" sense.
[0] https://graphthinking.blogspot.com/2021/12/comments-on-arxiv...
[0] https://arxiv.org/stats/monthly_submissions
For example, looking over my own papers for examples, a one page note [1] got more scites than a paper I contributed to that introduced several techniques cumulatively dropping the projected cost of simulating a chemistry thing by more than a factor of a million [2]. That's a cherry picked example, the pattern isn't that consistent, but having seen what happened to news sites it still worries me.
...not that the incentives for how to structure and publish papers are that great to begin with.
[1]: https://scirate.com/arxiv/2106.11513 [2]: https://scirate.com/arxiv/1805.03662
I welcome a platform that could allow more sophisticated mind-mapping. I've argued to no avail for years that MathSciNet should be open to machine learning, try to make itself a premier playground for machine learning, but it's an AMS cash cow and leadership prefers a 19th century model. Out to lunch recently with a former AMS president, he suggested I should do this for the ArXiv. That's not the point; one wants to encourage significant existing projects to land on our data.
Math is pretty much bifurcated between tightly-bound field tribes, only reading papers within their field, and roving players who recognize ideas in any form and chase them across fields. Of course, tribal mathematicians would dispute this. Better mind-mapping would reveal the connections between what's out there, rather than imposing a tribal mindset on its organization.
It would be good for mathematics to change this culture. It takes more courage than one might imagine to be a roving player. At hiring time, most departments let the tribes take turns. Just like growing up on the streets, pick a gang for survival, if you want to get hired.
[0] http://www.shortscience.org
Amusingly it was even more useful to me after I left academia. Scirate’s community is almost entirely theoretical quantum computing, and so I was able to keep up with the field by looking every three months or so at the top papers.
As noted in other comments, the end point of vanity voting isn't great. It's not great at discovery, but I have found papers there I would never have read. But because it is a small community it actually does a good job in quantifying what the community feels is important. I suspect if scirate was ever more widely used it would be less useful.
Another interesting phenomenon is that no one ever precisely defined what voting meant. Some people think of it as endorsing. Others as bookmarking. Not sure what impact this has, but sometimes if you score your own paper people will growl at you.