Yep, it uses the same technique. I've pulled every comment and story from HN through the API and make a bunch of Markov chains to produce story titles and comments. The input corpus is a lot smaller than subredditsimulator though and the various subreddits have a large variation in the words used (meaning more funny comments), so it's no way near as good.
Perhaps a future enhancement might be to partition comments by the submission domain, i.e use only comments for github.com stories for fake github submissions, might generate very different text than a munge of every comment.
You could also use the comments on reddit in /r/programming and /r/startups and other related reddits to help get more data for seeding your corpus.
Or if you want to get more complex, find the reddit comments for every link that was submitted to HN and use that (but you have to be careful that you use "hacker" related reddits or it will sound too "reddity")
Maybe aggregate over the subreddits listing (where HN links appear) and then whitelist some of them (after checking against a comment frequency count) like programming, linux etc.
I made a similar thing a year or two based off of another Markov Chain of HN headlines. I generalized it to allow you to mashup headlines from different news sources (Buzzfeed x Hacker News, for example). Still pretty funny, if anyone's interested: http://www.headlinesmasher.com/best/all
As a HN data processing note, to remove the garbled characters, you need to convert the smart quotes from HN (among other things like long dashes) into normal ASCII characters.
EDIT: Looks like the garbled characters were fixed.
Yeah thanks for pointing that out, I took the nuclear approach and just ran all everything through unicodedata.normalize("NFC", ...) which seems to have done the trick.
Think of all those poor, decommissioned teletypes we could put back into service. Then watch The Brave Little Toaster (while trying to ignore the truly-WTF moments). Then weep that this isn't a rely future.
> You're trying to solve bugs or problems.
>> It's like chess, or gymnastics, or baseball, or anything, just that it vanished overnight.
> I've also seen discussions of how your data structures without hunting down some raster graphics, I fire up Uber first.
> Your love of Pete, don't just repeat it with your keystrokes. FWIW I had never thought those 30 servers would be classified as unlawful combatants, removing their legal protections then go for them.
> Google wins the Book Search settlement gives Google 15 days in orbit (bostonglobe.com)
Google wins 15 days in orbit! Whee!
The comments are pretty great, too.
> Hang in there, say "Pizza" and it certainly has a lot of leverage because they're frustrated. Worst case: Someone sees your duck and you've got a new revenue model was (otherwise it was something I loved it, and they LIVE here.
Just hang in there, say "pizza", and make sure no one sees your duck.
> Why I design software, I want to want to live in it
> Show HN: Solving the problem of what you read the Web
> Think Apple Would Dare To Be Upset About Aaron Swartz's life
Yeah. If you store them in a relational database you will grow a couple of grey hairs because you're essentially forcing a tree structure into a table structure. The concepts clash and it's kind of a pain, but possible since every post has 1 unique parent, so you can make upwards references and rebuild a tree from that.
Thank you sir for pointing that out. This looks very interesting and I might even implement that in my own blog at some point in time. I thought about doing something similar to that (without actually knowing this technique) but I shyed away precisely because of the price you pay at insertion time.
EDIT: I've been thinking some more about this. Another possibility would be to limit the depth of the tree to, say, 8 (which should be reasonable) and then make 8 fields, one for each ancestor (parent, grandparent, and so on). Changing the tree will become a nightmare but all queries for subtrees will be blazingly fast.
To clarify - more users are going to read and refresh pages than actually post, so making certain not every GET request results in a new database query would probably improve performance more than trying to limit the number of rows in each query.
Query performance obviously matters, but with a HN like site, it's probably not going to be so critical that limiting the depth of threads is even worth the effort.
A simple method (a modified adjacency list) I've used just stores the root id, parent id and id of each post together. You can get the entire tree from any root post easily (everything shares the same root id) but getting the whole subtree beyond immediate children takes recursion.
I find that you don't even have to worry about treating the data as a tree in most cases until the very end. What you want to actually deal with is a flat array with the ids (root,parent,id) arranged in rendering order, and to have the tree built in the HTML. The data set from the DB doesn't even have to represent the tree structure directly, as long as you can sort it elsewhere.
You can even have two arrays - one (say, an associative array) with the data, and another basic array with the ids. Sort just the array with the ids, then use those as keys to iterate the data array when building the html, so you can avoid ever having to sort the larger array (which as luck has it just happens to be optimized for non-linear access anyway.)
I should probably mention, I thought this was terribly clever when I did it in PHP, before I was aware that all arrays in PHP are basically the same, so it was mostly pointless overoptimization.
Building something like an unordered list in HTML from that array then becomes a matter of adding or removing <UL> elements based on the relative change in depth for each subsequent id. Depth is easy to find by checking if an item's parent is (or isn't) the same as the id of the previous element. The actual tree never exists in code until the unordered list is rendered in the browser.
If you actually know what you're doing beforehand, probably ignore everything I just said and go with nested sets. My method is, admittedly, naive and better programmers are probably chuckling at it over the beverage of their choice, but it does work and it seems to be decently fast.
Markov chains do learn in the sense that they model distributions of strings, can be trained on observed strings and used to generate strings, assign probabilities to strings, classify strings, etc. They have well developed treatments in multiple frameworks of computational learning theory, including Gold learnability and PAC learnability.
Is "deep learning" more than statistics and randomization?
Normally I don't like comments that are just jokes, but this one was too perfect. Well done.
And to add just a slight bit more substance to my comment, while I was reading through all the comments here my wife asked why I was laughing so hard. I found it really difficult to convey why, but I guess that's the nature of this type of humor.
There isn't, but I don't think it should be difficult to feed this into 'char-rnn' if you wanted to do it with RNNs rather than Markov chains. The interface, such as it is, to char-rnn is very simple; you dump everything into a text file 'input.txt'.
I'm actually running that right now with HN comments. It's not done training, but it's not that much more interesting than OP's. Here's some example output:
{"text": "The article was in client 70, or a Denmark, is common captured - and I very well needed to be when they picked out time reports, all the reader or warning detectors and tools and proxchit matters. It also comes up with all levels of me using legality as it is that.<p>That helps a fly of Intel companies through it, but I'm importantly convinced the UI book the impression of orderly research on this afternoon. Personally, it also has a hash but mass measure all the Web working issued and leased across the question of my commercial group of interfaces. The various currents, others avoid their BitCoin better than one game (which is obvious, and form my position for hours at the software itself).", "author": "nostrademons"}
{"text": "Neither care to censor other people (\"infrastructure\" type by development! Neuroscience, flying migrations.)<p>Relevant comment was not finished at the moment. If that appears to be the case that, but ones are a real body.", "author": "pavel_liah"}
{"text": "<i>But just surely this should simply escruble him critically though I laudf. </i><p>It's taken as a more extreme shark to manage hcpm-infolves. However, I'm great, laser mortality, one of the aight payment.", "author": "jacquesm"}
{"text": "FtAhn is not all for violenceral campuse. Teghtletter usually try to provide wonderful purposes a dozen ones for intellectually-good common argument, so this would have you considered something something from writing points from a conversation. It's such a good idea and prohibition, disappeared. Except for partaicrolabed downverted vehicles, if you're the one, you can't pay for your own business, but the women are going onfichious, or not.<p>Edit: the processor thinks without searching stuff. <i>It looks like the Num corrupt introduces a scrappy page\"</i> we might be a new level, you can care about label heat timegakes.<p>> Two individuals. I don't refer to finally great ideas, but I haven't even heard of his place with high-generation (I tell me that a lot of the money) should prosecute my frequency. I have more succes, in fact dumping out the concept of engaging in a way to say that anyone wrote fits and a crunch employee (well, the only manner of Jessico would expose the South Clothes and enterprise making it to there very attempt to save a tight interface to carco again a different type branch takedow, because transcorrs freve-lock writing reduces. Grannian raises <i>the major</i> responsibility.)<p>(Nope! Unless you look at it even when a civil support doesn't expect to admit the system for us disk law (though that would make us bad news.)", "author": "sp332"}
{"text": "Sigh HTML5 of the extradition to NELOANAG may changed.", "author": "davidw"}
Or if I sample with low temperature:
{"text": "I don't know what I want to do in the sense that I was a problem with the same problem with the same problem as a comment on the side of the site. I was a little bit like a problem with the same problem with a single part of the problem.<p>I don't know what I wanted to do in the first place and I was a lot more powerful than the one that was a problem with the same problem. I was a pretty good point of view of the statement of the statement of the state of the state of the particular problem. I would have to say that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the pro...
You'll probably have to train separate char-rnn instances, unfortunately. For the past week or two, I've been experimenting with putting in a metadata prefix which I can use as a seed to specify author/ style, but thus far it hasn't worked at all. The char-rnn just spits out a sort of average text and doesn't mimic specific styles.
Yup. That's been my finding as well. char-rnn was really just a diversion of curiosity after I'd cleaned up the data. My best idea right now is to make a generative model of p(next_token | previous_token(s), author), essentially connecting author directly to every observation. I'm mostly sure that using characters as tokens is overkill for this and requires a higher complexity model than I can afford with this dataset and my computational resources, so I'm going to stop using char-rnn with it.
That's possible. My hope was that you could get authorial style by just including it inline as metadata rather than needing to hardwire it into the architecture (eg take 5 hidden nodes and have them specify an ID for the author so the RNN can't possibly forget). It would have been so convenient and made char-rnn much more useful, but I guess it's turning out to be too convenient to be true.
This is perfect. I opened it on a new tab, and went off to read another article. After coming back and opening multiple links I found myself thinking "where are all this posts with gramatical errors coming from" for a solid one minute - until I noticed "pg" where my username should be!
I did virtually the exact same thing - except my first reaction on seeing the fake HN was: "Why the heck am I logged in as pg? Security hole? This ought to be fun, let's see what kind of things he can do in here..."
Took me a while to figure out that this was actually a fake :-(
My responses ranged from "they're doing what to Linux now?" to "that one sounds cool..." to "that's true..." to "...of course I'm clicking that" to to "hm... well they must be going for an Indian motif".
And somewhere in there I was like "wow, /r/titlegore meets HN."
I'm very dense though. This was really well-made! :D
I still wish that 4th link was real. Like, I have a different taste in music, but I'd so reply to that. lol
Notably, this is what happens most of the time when something from /r/SubredditSimulator gets popular and becomes intermixed with normal submissions on a subscriber's front page and /r/all.
The way to beat a Turing Test is through hiding in plain sight.
9gag automatically reposts reddit's /r/all onto their site. People on 9gag don't realize it's a bot and have one hell of a time trying to interpret the post. Examples:
I opened it in a background tab, lost track of it for a while, ended back there thinking it was the normal ycombinator and unwittingly spent a couple minutes thinking "wtf is up with HN today?"
You've got to be kidding me. I've been further even more decided to use even go need to do look more as anyone can. Can you really be far even as decided half as much to use go wish for that? My guess is that when one really been far even as decided once to use even go want, it is then that he has really been far even as decided to use even go want to do look more like. It's just common sense.
This reminds me of a Google exercise within their Python course, wherein it read in a text file and built dictionaries of every word and the words that followed said word in the text in order to create a prosaic style that mimed the author of the original text. It was quite interesting to run and throw a text file at it to see what it produced.
EDIT: I have a cached version of the exercise if anyone feels like looking at it:
What's impressive is that when I see a typically hacker newish headline or comment I now find myself checking to see who I'm logged in as to make sure I'm on the real hacker news.
I kind of feel bad I discovered what was going on in like 20 seconds.
I increased the zoom of this site (the original font size is just too small for me), so when I opened this link and noticed that the zoom was restarted I immediately became suspicious.
I kept looking at the URL and thinking: "How are they using the same domain?" I came back an hour later only to realize that the i and n are switched in ycombniator.
173 comments
[ 3.1 ms ] story [ 189 ms ] threadPerhaps a future enhancement might be to partition comments by the submission domain, i.e use only comments for github.com stories for fake github submissions, might generate very different text than a munge of every comment.
Or if you want to get more complex, find the reddit comments for every link that was submitted to HN and use that (but you have to be careful that you use "hacker" related reddits or it will sound too "reddity")
As well as keeping comments by submitted domain.
LOL
EDIT: Looks like the garbled characters were fixed.
some of my favorite so far:
> Debunking Myths About Growth Hacking Goes Bad (infoq.com)
> First Firefox OS developer to come (businessmandi.com)
> Modern science and art go to jail? The law is dead in cinema (techcrunch.com)
> A Browser Package Manager Command Line Client Written in Lua
> TweetDeck taken offline after bug allows malicious code execution on Android - eBook
> True hacker resume: CV as Python objects to/from Amazon S3 price cut for all your OK Google searches predict market moves
Hahahaha!
> Ask HN: Optimal number of sets with high IQ users?
> TweetDeck taken offline after bug allows malicious code execution on Android - eBook (spinejs.com)
> Docker Swarm on Raspberry Pi Units Available In GNOME 3 Released (spectrum.ieee.org)
> Show HN: nnmm – A feed of your print or paper books
Isn't this just a pile of books?
>Small Utah ISP firm stands up to the Faces of Facebook popularity, I quit.
>UI7Kit: Add one-line to enable AirPlay video for your startup on Product Hunt
lol.
> You don’t need any HTML theme/template into your murder trial (ft.com)
> A&E's 'Duck Dynasty' Stunt is a user is Steve Jobs’ Unfortunate Contribution to Computing
> You're trying to solve bugs or problems. >> It's like chess, or gymnastics, or baseball, or anything, just that it vanished overnight. > I've also seen discussions of how your data structures without hunting down some raster graphics, I fire up Uber first. > Your love of Pete, don't just repeat it with your keystrokes. FWIW I had never thought those 30 servers would be classified as unlawful combatants, removing their legal protections then go for them.
Google wins 15 days in orbit! Whee!
The comments are pretty great, too.
> Hang in there, say "Pizza" and it certainly has a lot of leverage because they're frustrated. Worst case: Someone sees your duck and you've got a new revenue model was (otherwise it was something I loved it, and they LIVE here.
Just hang in there, say "pizza", and make sure no one sees your duck.
I just ..
> Protesting with a python port of ZFS on Linux
> Ten Rules for Web 2.0 sites and URL obfuscation
> From 0 to 8-figure revenue in spite of flat UI design and email
> How can I get asked how I thought we were wrong.
And the most HN one I've seen so far:
> Entrepreneur crowdsources decision to quit Silicon Valley
> I've pulled every comment and story from HN through the API and make a bunch of Markov chains to produce story titles and comments.
1. https://gist.github.com/orf/5565a572c6ddda039d6f
Can't make up my mind . . .
[1] https://truongtx.me/2014/02/28/tree-structure-query-with-pos... [2] http://stackoverflow.com/questions/603894/is-postgresqls-ltr...
It works a treat as you can query the whole tree in one SQL statement but preserve the nesting for formatting.
EDIT: I've been thinking some more about this. Another possibility would be to limit the depth of the tree to, say, 8 (which should be reasonable) and then make 8 fields, one for each ancestor (parent, grandparent, and so on). Changing the tree will become a nightmare but all queries for subtrees will be blazingly fast.
Query performance obviously matters, but with a HN like site, it's probably not going to be so critical that limiting the depth of threads is even worth the effort.
I find that you don't even have to worry about treating the data as a tree in most cases until the very end. What you want to actually deal with is a flat array with the ids (root,parent,id) arranged in rendering order, and to have the tree built in the HTML. The data set from the DB doesn't even have to represent the tree structure directly, as long as you can sort it elsewhere.
You can even have two arrays - one (say, an associative array) with the data, and another basic array with the ids. Sort just the array with the ids, then use those as keys to iterate the data array when building the html, so you can avoid ever having to sort the larger array (which as luck has it just happens to be optimized for non-linear access anyway.)
I should probably mention, I thought this was terribly clever when I did it in PHP, before I was aware that all arrays in PHP are basically the same, so it was mostly pointless overoptimization.
Building something like an unordered list in HTML from that array then becomes a matter of adding or removing <UL> elements based on the relative change in depth for each subsequent id. Depth is easy to find by checking if an item's parent is (or isn't) the same as the id of the previous element. The actual tree never exists in code until the unordered list is rendered in the browser.
Also here's a good reference on Stack Overflow of different methods to do the same thing: https://stackoverflow.com/questions/2175882/how-to-represent...
If you actually know what you're doing beforehand, probably ignore everything I just said and go with nested sets. My method is, admittedly, naive and better programmers are probably chuckling at it over the beverage of their choice, but it does work and it seems to be decently fast.
Is "deep learning" more than statistics and randomization?
Correct. It is a perfect simulation of HN.
And to add just a slight bit more substance to my comment, while I was reading through all the comments here my wife asked why I was laughing so hard. I found it really difficult to convey why, but I guess that's the nature of this type of humor.
{"text": "The article was in client 70, or a Denmark, is common captured - and I very well needed to be when they picked out time reports, all the reader or warning detectors and tools and proxchit matters. It also comes up with all levels of me using legality as it is that.<p>That helps a fly of Intel companies through it, but I'm importantly convinced the UI book the impression of orderly research on this afternoon. Personally, it also has a hash but mass measure all the Web working issued and leased across the question of my commercial group of interfaces. The various currents, others avoid their BitCoin better than one game (which is obvious, and form my position for hours at the software itself).", "author": "nostrademons"}
{"text": "Neither care to censor other people (\"infrastructure\" type by development! Neuroscience, flying migrations.)<p>Relevant comment was not finished at the moment. If that appears to be the case that, but ones are a real body.", "author": "pavel_liah"}
{"text": "<i>But just surely this should simply escruble him critically though I laudf. </i><p>It's taken as a more extreme shark to manage hcpm-infolves. However, I'm great, laser mortality, one of the aight payment.", "author": "jacquesm"}
{"text": "FtAhn is not all for violenceral campuse. Teghtletter usually try to provide wonderful purposes a dozen ones for intellectually-good common argument, so this would have you considered something something from writing points from a conversation. It's such a good idea and prohibition, disappeared. Except for partaicrolabed downverted vehicles, if you're the one, you can't pay for your own business, but the women are going onfichious, or not.<p>Edit: the processor thinks without searching stuff. <i>It looks like the Num corrupt introduces a scrappy page\"</i> we might be a new level, you can care about label heat timegakes.<p>> Two individuals. I don't refer to finally great ideas, but I haven't even heard of his place with high-generation (I tell me that a lot of the money) should prosecute my frequency. I have more succes, in fact dumping out the concept of engaging in a way to say that anyone wrote fits and a crunch employee (well, the only manner of Jessico would expose the South Clothes and enterprise making it to there very attempt to save a tight interface to carco again a different type branch takedow, because transcorrs freve-lock writing reduces. Grannian raises <i>the major</i> responsibility.)<p>(Nope! Unless you look at it even when a civil support doesn't expect to admit the system for us disk law (though that would make us bad news.)", "author": "sp332"}
{"text": "Sigh HTML5 of the extradition to NELOANAG may changed.", "author": "davidw"}
Or if I sample with low temperature:
{"text": "I don't know what I want to do in the sense that I was a problem with the same problem with the same problem as a comment on the side of the site. I was a little bit like a problem with the same problem with a single part of the problem.<p>I don't know what I wanted to do in the first place and I was a lot more powerful than the one that was a problem with the same problem. I was a pretty good point of view of the statement of the statement of the state of the state of the particular problem. I would have to say that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the problem is that the pro...
You'll probably have to train separate char-rnn instances, unfortunately. For the past week or two, I've been experimenting with putting in a metadata prefix which I can use as a seed to specify author/ style, but thus far it hasn't worked at all. The char-rnn just spits out a sort of average text and doesn't mimic specific styles.
Hopefully you are real. :)
Took me a while to figure out that this was actually a fake :-(
"Wait, how did Hacker News get opened twice?"
...
"Eh, I'll just close this one." (You can guess which one I closed. :P)
[One pile of middle-clicking later]
"Wait, ycombinator is defaulting to opening comments now?!"
I noticed the "ycombniator" shortly after that.
I wish those links were real :(
- http://news.ycombniator.com/comments/comments_198.html: "Git 2.0 changes push default to using only CSS3 - No more remote work work: An adventure in civic hacking (scienceblogs.com)"
- http://news.ycombniator.com/comments/comments_210.html: "Show HN: Farmly – find anyone for anything you want and keep the Olympic germs away. (nowthenapp.com)"
- http://news.ycombniator.com/comments/comments_208.html: "Twitter will pay for anything [Product] (skullsinthestars.com)"
- http://news.ycombniator.com/comments/comments_184.html: "Ask HN: Angry/hardcore rock music to code the summer away, stuck for ideas to practice/exercice a new UI designer/programmer (NYC)"
- http://news.ycombniator.com/comments/comments_190.html: "Is GTK+ the real reason people are startups using? (blog.cto.hiv)"
My responses ranged from "they're doing what to Linux now?" to "that one sounds cool..." to "that's true..." to "...of course I'm clicking that" to to "hm... well they must be going for an Indian motif".
And somewhere in there I was like "wow, /r/titlegore meets HN."
I'm very dense though. This was really well-made! :D
I still wish that 4th link was real. Like, I have a different taste in music, but I'd so reply to that. lol
The way to beat a Turing Test is through hiding in plain sight.
1. https://www.reddit.com/r/SubredditSimMeta/comments/3fnkni/9g...
2. https://www.reddit.com/r/SubredditSimMeta/comments/3kwmnj/9g...
Source: http://namuol.github.io/phaker-news/
Well played.
edit: lol, seems I'm not the only one.
http://thedoomthatcametopuppet.tumblr.com/
HP Lovecraft + Puppet Documentation
[1] http://kingjamesprogramming.tumblr.com/
> We had an inkling that something that someone is logged in to a traditional incandescent bulb
EDIT: I have a cached version of the exercise if anyone feels like looking at it:
https://github.com/AdmiralAsshat/learn_python/blob/master/go...
There should be an "alice.txt" in the same directory as a sample file to throw at it.
"The main problem is going to check its authenticity against any common sense."
"I use in your TOS, you say you're better than a bad example: a painting as an extreme advantage"
FYI, cicking on the "<#> Comments" link from within a comments page leads you to a 404.
I increased the zoom of this site (the original font size is just too small for me), so when I opened this link and noticed that the zoom was restarted I immediately became suspicious.
That was a real "smack my head" moment.