I put this together to help browsing the 'ask HN: who's hiring' threads. It's nothing fancy, but it helped find a couple leads. Maybe it can help you too!
I put this together to help browsing the 'ask HN: who's hiring' threads. It's nothing fancy, but it helped find a couple leads. Maybe it can help you too!
Thanks. I thought about adding that. The posting text is pretty unstructured so I thought a search would get 80% of the way there. Tricky to extract city & type & pay from what is basically a wall of text. Before this I was doing a bunch of ctrl+f nextnextnext searches on the original content. One thing you can do is click the little 'x' to right of a post to hide any post you're not interested in or already contacted. That will persist until you clear your cookies so you can kinda personalize it a bit.
I'm using a custom scraper to get the results. I request the job pages through a random http proxy every so often, parse them, and update the cached json of all the comments. Getting a reliable, free, machine-friendly list of http proxies is harder than I thought. It was automatic, but the proxy provider I was using went down (doh!) so now I manually pull one from a list and trigger the scrape.
On the server I'm using a tiny express app (node.js) + redis to store hidden posts per use based on a cookie.
On the browser just a little bit of twitter boostrap + jquery + some custom UI helpers to generate the page. The searching and hide/show is done within the browser itself. I try to involve the server as little as possible.
To answer your second question:
It will update as soon as I kick off another scrape. I hope to re-enable the automation on this soon so it will only lag 10-15 minutes behind the actual page. Right now it could be quite a while (up to 12 hours).
Yeah! ctrl-f was tough. Especially as the comments spanned multiple pages and weren't sortable by date.
hacker news black listed the ip address of my server after scraping 1 post's comments about once every fifteen minutes for a day. The best way I could think to get around this was an http proxy.
Very awesome. Got any more projects up your sleeve?
Is it better to have an xml feed and load that instead of a JSON file? I wrote a scraper that dumped university courses at my school to a JSON file. I would load that JSON file and load the elements to a table. Mine was pretty dang slow.
I have FlashBlock, and your site was totally blank until I let flash through. Do you have a flash element that blocks the rest of the page from loading?
Can I ask how you parsed job postings from the rest? I have a few programs that would greatly benefit from the ability to distinguish the jobs from normal posts. It doesn't seem that they are different in html...
I presume he's not parsing and instead simply manually entering the Who's Hiring threads and crawling those. If you look down the list you'll see a few entries that are not job posts.
Correct - I'm just parsing all the comments and only showing the "top level" ones. I use this library I wrote: https://github.com/brianc/node-hacker-news-parser which builds a tree structure of the comments.
30 comments
[ 0.20 ms ] story [ 98.9 ms ] threadGreat job! Much better than ctrl-f-ing my way through postings [just like you were doing].
To answer your first question:
I'm using a custom scraper to get the results. I request the job pages through a random http proxy every so often, parse them, and update the cached json of all the comments. Getting a reliable, free, machine-friendly list of http proxies is harder than I thought. It was automatic, but the proxy provider I was using went down (doh!) so now I manually pull one from a list and trigger the scrape.
On the server I'm using a tiny express app (node.js) + redis to store hidden posts per use based on a cookie.
On the browser just a little bit of twitter boostrap + jquery + some custom UI helpers to generate the page. The searching and hide/show is done within the browser itself. I try to involve the server as little as possible.
To answer your second question:
It will update as soon as I kick off another scrape. I hope to re-enable the automation on this soon so it will only lag 10-15 minutes behind the actual page. Right now it could be quite a while (up to 12 hours).
Yeah! ctrl-f was tough. Especially as the comments spanned multiple pages and weren't sortable by date.
Is it better to have an xml feed and load that instead of a JSON file? I wrote a scraper that dumped university courses at my school to a JSON file. I would load that JSON file and load the elements to a table. Mine was pretty dang slow.
But your site holds up fine.
Here's my site if you're at all interested: http://lo.leet.la/jola/bootstrap/docs/sunny.html#
Relvant JS code: http://lo.leet.la/jola/bootstrap/docs/assets/js/sunny.js
There are absolutely tons in NY.
Edit: And the YC job postings are located at http://news.ycombinator.com/jobs