Ask HN: What's your "hair on fire" problem?

To answer your specific question, DupDetector was an attempt at two things.

Firstly, to separate concerns - to separate my "curating" efforts of cross-referencing and duplicate finding from my occasional contributions of original material and links. I wanted to see the effects of that activities separately. In particular, I wanted people look at my "submissions" page from my profile to see the submissions, not the cross-referencing.

Secondly, to make the process more comprehensive and more automatic. I wrote a few small scripts and had them generate comments. I never let it run automatically, and always submitted them "by hand". That meant that they weren't as obviously robotic as I would eventually have intended them to be. I have also written a couple of comments and submitted them under that ID because that was part of the separation of concerns mentioned above.

I've learned a lot from the experiment, and over time I will distill that and feed it back to the community. You can see some of the discussion here:

http://news.ycombinator.com/item?id=2013666

http://news.ycombinator.com/item?id=2021596

But what running DupDetector has shown me is that the problem is endemic. There is no way that I can see to reduce the rate of re-submissions. People have got very annoyed when the problem has been pointed out. Indeed, many people don't see it as a problem. The very fact that people see an enormous number of posts by DupDetector tells us two things: There are a lot of duplications, and people don't want to know.

Some discussion here: http://news.ycombinator.com/item?id=2013666

So that's fair enough. As I say, I'm terminating the experiment. I've learned that the current level of noise caused by constant re-submissions of the same stories, and often the exact same items, won't stop. So instead I'm concentrating on finding items that fit with PG's stated preference:

    Stories on HN don't have to be about hacking, because
    good hackers aren't only interested in hacking, but they
    do have to be deeply interesting.

On any day, look at the "newest" page and ask yourself how many are really "deeply interesting", how many are amusing diversions, and how many are diverting, but useless. Ask how much you learn from each one. The answer I suspect will be "not much".

Look at the "news" page and ask the same question. The problem is that if enculturated "old hands" don't read the "newest" page, then the "news" page will deteriorate, because the people voting on new items won't be voting the right things for the right reasons.

We see that now. Lots of submissions get votes because they are, in PG's words, "intensely but shallowly interesting."

So I'm going away for a while to see if I can do that. I'll still read stuff, but every time I do I will consciously remind myself that I can get content-free entertainment pretty much anywhere, and that I'm looking specifically for stuff that's "deeply interesting." If I figure out a way to find it automatically I'll certainly come back and share it, although I don't expect I'll succeed. It's a hard problem, and I'm certainly no smarter than others who have worked on it before me.

Currently the best option seems to be to have a small community of like-minded people. That's what HN used to be, and I think that despite PG's excellent efforts, it's getting too big to hold together.

But I'll work on it, and I'll certainly share any useful findings. If there are any.