12 comments

[ 0.29 ms ] story [ 35.9 ms ] thread
While an amazing article, this has already been posted here. I guess it might be good enough for a repeat?
Wow. I'm quite embarrassed.
Well, it is a good article. HN doesn't really have a mechanism for reviving good stuff beyond reposting it like this. (...by design, I assume, or some people would do nothing but reviving popular stuff.)

Maybe there should be a "best of" HN page, something like the best of Craigslist?

A 'best of' would be really cool, and good for people new to the site.
I'm quite embarrassed.

Nah. Arguably the HN algorithm could do slightly smarter matching on URLs. It's interesting to look at the URLs in this case:

http://www.longnow.org/views/essays/articles/ArtFeynman.php

http://www.longnow.org/views/essays/articles/ArtFeynman.php?...

http://www.longnow.org/views/essays/articles/ArtFeynman.php#...

http://www.kurzweilai.net/articles/art0504.html?printable=1

(Yours is the third. Note how the second invented a query string to deliberately skirt the rules!)

Three of these URLs could be considered equivalent - drop the query string in one and the anchor in the other. The last one, of course, is just totally different.

On the other hand, if you do this, you'd probably get some false matches, because some web pages use query strings to represent different content. Plus you couldn't do things like post neat Google searches directly.

So probably we should resign ourselves to an endless series of duplications, whereupon the geeks pounce, digging up previous versions and posting them all in ever-lengthening lists. Not so bad, really. :)

Edit: how about preemptive duplicate prediction? There's an obvious gap here:

http://www.kurzweilai.net/articles/art0504.html

... and one day, someone's gonna post it. (Actually, it gets redirected to still another.) Or maybe they already did and we could make an even geekier list. But how to check that without actually submitting it?

Edit 2: Do I win for minutiae of the day?

Regarding false matches, we could say "we think this has already been posted here. Do you think so too? If so, do you really want to post it again?"
That's a good idea. Somehow I don't think it's pg's style, though.
You could also try to do content-based duplicate detection: rather than looking at the URL, try to determine whether a new article is substantively similar to a previously-posted one. Getting that to work with a high degree of accuracy would take some thought, however. Perhaps there's some work on this in the information retrieval literature?
I actually worked on this problem for Scribd. The solution isn't too heavyweight, especially if you're matching large blocks of unaltered text.
Hmm, 185, 31834, 191212, 311511... Maybe we can predict when it will be posted again?
I predict 778778.