Author here. I scraped 8 years of data on Who is Hiring? threads. The data is available through the article and on GitHub. Spoiler alert: people want to hire full time software engineers in San Francisco. Happy to answer any questions.
I've always found it really "interesting" how much resistance the industry shows to hiring part time and/or remote, the two changes in hiring which would really ease their supposed labor issue. I refuse to believe the dogma about labor shortages until I see companies actually going out and hiring all the available qualified employees. All the nitpicking over what constitutes "qualified" which isn't about being able to do the work is just an excuse to exert butts-in-seats dominion over workers.
The number of electrical engineers employed in the US has been going down for years. One wonders what will happen when an entire generation is raised to think that “technology”=web pages and games. Who will build new computers?
#standardSQL
WITH whoishiring_posts AS (
SELECT id from `bigquery-public-data.hacker_news.full`
WHERE `by`="whoishiring" AND type="story"
)
SELECT text
FROM `bigquery-public-data.hacker_news.full`
WHERE type="comment"
AND parent IN (SELECT id from whoishiring_posts)
This is great! I've been wondering for a while how to efficiently load a comment tree without spamming the Hacker News API with hundreds of requests, and this seems like it would work nicely.
But the official Hacker News API is really painful to use. Specifically because each comment is its own API call (?! ).
I know the HN staff mentioned they're working on a V2 API, but until then the HN HTML tree is so clean, I think this is an exceptional case where scraping the HTML yields a cleaner result that's also less load on the HN servers.
EDIT: Like the other comment on this thread mentioned, using https://hn.algolia.com/api seems ideal here.
Neat. I'd be interested in seeing how many job posts are reposted each month, and which companies post the most. I have noticed many jobs/companies that are present almost every month.
The rainbow coloring of the graphs is distracting.
I had “plans” a while back to put together a cynically-named “Who’s Not Hiring” script that did just that. Look at which companies post the same or extremely similar jobs month after month. Life got in the way and I abandoned the project early, but it would be interesting to see the results of such an effort.
I'm on a team at my current job with ~10 other engineers, and at my previous job I was on a team of ~12. With growth, retirements, and turnover at both jobs we are always hiring. My rough approximation from both jobs (spanning about 10 years) is that we're able to hire one person every 6 months. I'm not a hot tech geographic region, and the field I work in is somewhat niche, but there are many non-cynical reasons a job might be posted indefinitely.
Indefinitely posting while hiring only once every six months is actually the poster child of the problem. If it really takes six months to find someone to fit the role, you might want to take a look at your hiring process because as you must be rejecting (or ignoring the applications) a lot of great, qualified people.
IMHO if someone did take the time to publish the metrics of companies who perpetually post the same positions, it would be a fair counter balance.
I'd argue six months is about average for the jobs I've seen and had over the years. Not every role needs an off-the-shelf front end dev.
A lot of things really do require significant domain and industry specific knowledge. For those roles, it takes a very long time to turn up even one minimally qualified candidate. It's easy to say "find someone bright who can learn on the job", but that's not practical in the cases I've seen.
Many things need either specialized education or experience in a similar role to be effective within a 2-3 year timeframe. When on-the-job training would take multiple years, it's worth spending six months to a year to find someone who can fill the role. At least in the industries I've worked in, the roles that require extensive and rare training/experience outnumber the roles that don't.
If you take six weeks to hire someone (let alone six MONTHS!) then you’re probably missing out on candidates who have other good, quicker options. I remember doing a phone screen with a major telco whose name starts with A, ends with T and has a T in the middle. It seemed to go well, then silence (as often happens with tech interviews). Well I did a few interviews at other companies, picked one, signed everything, moved my family across the country, and started work there. About 9 weeks after that phone screen, they finally got back to me with: “We thought the initial interview went great! How about coming onsite for the next step?” LOL
Our hiring process doesn't take 6 months -- we're only able to find a candidate, interview them, negotiate, and have them accept an offer every 6 months. Our biggest problem is just finding people to apply. In the last ~8 years I count only about 100 total applicants to our jobs (I'm not counting internship job listings where we got a lot of student applicants). We tried posting jobs on HN Who's Hiring for about 6 months and got zero serious candidates, so we stopped.
Out of curiosity, do you put a competitive salary range on your posts? I suspect that if you put a range that is above the median for your locale you might have more bites. Only other reason I can think is if your technology stack is outdated or you are located in an area where no one wants to live.
What's your definition of an "off the shelf" front end dev? And if you want to pigeon hole just front end developers (I meant industry wide), what is it you think can possibly be so special about the UIs of over half of everything that someone with many years of experience (possibly from a variety of industries) can't come in and quickly be effective? Isn't there value of someone who can handle new challenges over someone siloed into a single area?
I would say your comment come across as elitist and the evidence I have is also my personal experience. I've done it many times and rarely has anyone not been happy with the results thus far. Obviously there are industry specific areas of expertise that can be deep, and I can see where there are times you might have to hold out for the right candidate. But I think the figures you give reflect stem much more from personal bias than reality. Sadly, a lot of HR agrees with you which is why it's it's become needlessly hard to hire/find a job these days.
My apologies, I should have left "front end" out entirely. Front end has nothing to do with it, I was just using it as an example of a role that is frequently somewhat standardized, interchangeable, and often does not require knowledge of an additional domain.
My point was that many other roles do require expertise in multiple domains as well as industry specific experience. These jobs are relatively hard to fill. They're also quite common outside of consumer tech companies, in my experience.
I apologize if this seems elitist, but some things simply cannot be picked up in a few weeks on the job. Those jobs tend to have a lower supply of qualified candidates and take longer to fill.
Same!! Like with most tools, web scraping can be used for nefarious reasons very easily. I'm glad he provided that caveat. Signs of someone truly trying to create good work and not just clickbait.
A little off topic... but I love this because it gives me an idea.
I've been looking at LinkedIn with some immensely negative feelings for some time now. I've always wished someone would just disrupt the hell out of them, because they don't deserve the near-monopolistic position they're in. They just don't care about their users!
And before someone says, "well [pushes up glasses] their real users are recruiters that they sell $20k/year seats to their search tool". Yeah, as a hiring manager I've tried using it. Awful! :)
And jobs boards are nothing new, I guess, but something feels really nice about how much bigger these "Who is Hiring?" threads are getting on HN. Maybe we really do need a jobs-centric "social network" for industry verticals. Beyond terrible UX and general disrespect, LinkedIn also just doesn't do enough with their data to make it useable. Maybe that's because it's just too hard to do for every job and industry out there.
And maybe that next-gen professionals' social network can be the first open and free one?
Or maybe I'm just wishing too hard for something that can't/won't be done.
LinkedIn has become plagued by self-help-like stories and other self promoting, vague and meaningless write ups, in my experience.
It wasn't always like this. I used to have a reasonably cool feed centered towards technologies and news that are at least partly relevant to me and my industry. No more. It's all 'I gave a homeless person lunch and today they are the #1 duck breeder in Brazil' or some other nonsense.
I don't know when this started or if it is localized, but it renders the 'place to find useful information' aspect of social media completely void. There's no amount of reasonable curating effort that will save a news feed like that.
I try not to be too negative about things but without some serious change I'll use a static LinkedIn profile and visit it for a fraction of a second every week - to check for messages - and nothing more.
I agree. But since I am young I have a lot of people who are still trying to upgrade to a "real job" after having graduated in the last 1-2 years and these articles are disproportionately shared by those people.
Usually the articles are about how they sent 200 applications and it was all worth it to get that one acceptance, or are re-share spam about some manager taking a chance on a new hire or some shit. I haven't done any work into curating my feed since I rarely scroll through my feed anyway, but I think LinkedIn could do much better at putting industry-related news articles or blogs in there with less perseverance-porn
The only thing I really use LinkedIn for at this point is searching for 2nd degree connects / alumni at companies I'd like to work at, and looking up the professional backgrounds of people that I will be meeting with. However, being able to do this is super critical to networking and I almost always end up paying for premium whenever I'm job searching.
The median HN reader is missing the point of LI because it's not for developers. So you think "man, this sucks". You're like a carpenter looking at a welding machine saying "who needs this huge thing?".
We developers have so many social networks to be found at - SO, github, angel.co, random reddit or discord groups.
But others don't, and linkedin provides amazing value for them as a place to advertise their skillset.
I used SO for candidate search. It worked well for SF, but I recently moved to LA to start a team down here and it was just ridiculously... bad. LinkedIn was better, and that's saying something. But I think it's the scale effects of the social network at play here. SO just doesn't have the scale LinkedIn does. Neither does angel.co, though I can't say I've tried to do recruiting on github or reddit (is that something people do?).
What I am saying is that after having used this thing for so long, I want to improve upon it. If it were open, like HN's Who's Hiring threads, I could just run a BigQuery job on it and do a sophisticated search for the candidates I want. I can maybe even do that and open it up for anyone to use, or sell it, whatever! It would provide value for people.
But the thing that sucks about LinkedIn, aside from the spam by their growth-at-all-costs team, aside from the 5 second page load time, forget the terrible caliber of thought leadership... the BIG THING that makes it really hard to use is their data, or what they expose of it. And I'm wondering if it's just because figuring out the true years of experience (or something else) of someone is too hard for all jobs, but I know some tricks I can use to gauge that for an engineer.
Because if I know everyone I'd want to hire is on the network, I would pay a lot of money to get full access on querying that data. But what LinkedIn has built for it is just sad. And the biggest barrier to building an amazing recruiter search tool is the fact that the entire network belongs to this giant corporation that isn't even doing a good job generating value from it!
When I last used the recruiter tools, I didn't see a way to filter for years of experience in a specific job type (e.g. software engineering) or a set of roles.
There's no way to filter for people who have 10+ years' experience in software engineering roles. Or someone who has 5+ years' experience in CTO or VP Eng roles.
I have a theory that LinkedIn the product is only a platform for LinkedIn the company to tell us how great it is to work for them.
Add to that the "thought leaders", the quotes and other FB-ish artefacts, without forgetting a constant stream of brown-nosing and if you have 20% of the content that is interesting, you can consider yourself lucky.
In contrast, I hang in an industry specific slack with folks all over the world and it's so much better.
I don't have a solution to offer with regards to an open platform, but I think curated communities are the way to go.
And maybe that next-gen professionals' social network can be the first open and free one?
Volunteers have to have somewhat cushy lives to be able to give their time and expertise for free. Stallman was willing to live by his ideals. He was also intermittently homeless while doing so.
These ideas of "We can rebuild it. We can make it better. We can also do it for free!" tend to be unrealistic fantasies.
Stallman also had the advantage of living within the university system for a long time while he built his reputation, so once he was on his own, he could use his reputation to solicit services. The average person has to do more to keep a roof over their head.
Right? I think it could be done. I’ve been talking to friends about this for a couple weeks now.
I’m actually kind of interested in the people that have done “3000 applications etc etc.” The 100-300 applications are fairy common, but 3k?
Also, LinkedIn? more like LockedIn. There was a time where I enjoyed the service; I liked connecting with people doing interesting things from cool backgrounds.
But, I wonder if it hurts more than it helps... I feel like a lot of jobs are filled before they’re flown, every other job is meant for people who already have a job.
I like reading the job postings though; a lot of them are like non sequitur slam poetry.
This might be my naïveté (or the ipa), but I feel like it’d be cool to have a reversed scenario where I can get info on the team, the project, etc. before even reaching out (i.e. companies apply to individuals, but I think I may have reinvented recruiting? I know this may be unrealistic...)
Idk sometimes it feels like LinkedIn offers a non-service, talent will always be employed and/or sought; if everyone else is just a company’s 1000th pick aren’t they just wasting their time?
Why didn't you just use the Algolia API? I used the Algolia API to build a 24 / 7 live stream of randomly selected highly voted HN vídeos https://www.crowdform.co.uk/hntv saved me a lot of time
The data on "Freelancer? Seeking freelancer?" is interesting and I think it can be misleading.
The disparity between freelancers seeking work and people seeking freelancers on HN is huge - way more people looking for work, than people looking to hire. That makes it feel like there is an oversupply of freelancers, which would drive down the pay rates.
However, I think it's more of a reflection of the HN audience than the broader market. I.E. HN readers don't hire as many contract/freelance employees. This aligns with what we've seen in practice as well. If software is the company's core competency, they will be less likely to hire contract or outsource.
On the other hand, there is a huge need for non-software companies to hire senior devs and they hire a lot of contract/freelance. Demand is high and rates are good ($120-200/hr)...at the right companies. So, if you are a contractor looking for work, I would target companies where software isn't their primary business.
Source: I run a company that helps senior software engineers find contract work. (Shameless plug: www.facetdev.com)
I get the comment is a bit snarky, but if you dig into this guys site it's required that you have worked 3+ years at a "top tier company" with a rotating ticker of FAANG and other giant SV behemoths on it's front page ad, which IMO is fairly pretentious and warrants a bit of a call out.
Author here. I scraped 8 years of data on Who is Hiring? threads. The data is available through the article and on GitHub. Spoiler alert: people want to hire full time software engineers in San Francisco. Happy to answer any questions.
People who are saying that the author should have used an API are missing the point; sometimes the only way to gather data is by resorting to scraping, and this article details how one could go about doing that, by using an interesting target: HN.
To the author: Good job, I'm sure many people will find your article useful! I wish I would've had this article at hand a couple of years ago when I needed to aggregate data from several websites that provided no API.
> sometimes the only way to gather data is by resorting to scraping
Nowadays, web scraping has a nonzero risk of hitting legal issues for sites when "the only way to gather data is by resulting to scraping", especially when an API already exists with a ToS/Guidelines on how the data should be obtained and used. And even moreso if the data itself has monetary value.
The new way to marginalize disposable team members is to replace "interns" with "remote" workers.
The savings are likely shifted over to cutting costs for the on-site office footprint, instead of optimizing for freebie slave labor from the lesser skilled, green college newbies. This also probably carries a quality over quantity trade off, in terms of hours spent working (college kids sweating 70 hour burn-out weeks, because paying dues or whatever).
Tech is mature enough, that there are enough veterans in the talent pool shrewdly compromising the level of their engagements, and there is less novelty to attract wide-eyed, coffee fetching would-be script kiddies anymore.
If you're still feeling froggy with your data, I'd love a follow up with some NLP term frequency used in the posts.
Did intern frequency drop or did the sheer increase in posts shrink their overall percentage? The concern I have there is the idea of becoming too "top heavy" where no one is interested in helping interns grow and only want to recruit experienced people.
You raise a great question. I also wonder if Hacker News as a platform tends towards more senior people, I know that I usually feel like one of the least experienced people in most comment sections.
While the article seem intended as a web scraping tutorial, it's good to remember that there is an official HN api [1] in case you really want to do data science on the site.
I was recommended to this professional hacker, He helped me to expose all my partner’s secret that kept my marriage intact, who my spouse was cheating with and all…, Contact him on: alimohammedprohacker @ g m a i l . com contact whatsapp phone : + 1 6 1 9 6 3 2 5 9 2 6 his services include hacking (hint: mobile phones, Instagram, Facebook, gmail,twitter, whatsapp, kik, bank account, iphones, MeetMe, Snapchat, WeChat, hike etc.), tracking, cloning ,upgrading result,preventing you from been hacked or tracked,Adding any important account to your account without account owner knowing he can also help you to spy on your spouse so that you'll know whether he or she is cheating. he can also teach hacking at a very affordable price.
I know a professional hacker SPARRO who has worked for me once this month. He offers legit services such as clearing of bad records online without being traced back to you, He clone phones, hack Facebook account, instagram, WhatsApp, emails, Twitter, bank accounts, Skype, FIXES CREDIT REPORTs, track calls,clear criminal records. He also help retrieve accounts that have been taking by hackers. His charges are affordable, reliable and 100% safe. Contact him via address below...
Email..SPARROHACKER (AT) GMAILCOM
Text no.. +1 (9 7 8) 4 3 4 0 9 5 3
Whatsapp: +9 7 1 5 8 6 2 8 4 4 9 8
84 comments
[ 3.1 ms ] story [ 161 ms ] threadEven though the Hacker News API (https://github.com/HackerNews/API) is somewhat old, it's a much more kosher way of getting data.
Even better is to use the public data dump in BigQuery (https://console.cloud.google.com/marketplace/details/y-combi...). Quick query to get all top-level comments in posts by whoishiring:
I know the HN staff mentioned they're working on a V2 API, but until then the HN HTML tree is so clean, I think this is an exceptional case where scraping the HTML yields a cleaner result that's also less load on the HN servers.
EDIT: Like the other comment on this thread mentioned, using https://hn.algolia.com/api seems ideal here.
I wrote a thin Python library on top: https://github.com/santiagobasulto/python-hacker-news
Here's an example:
Full working example: https://gist.github.com/santiagobasulto/b3a3bb94a26c1798b965...The rainbow coloring of the graphs is distracting.
IMHO if someone did take the time to publish the metrics of companies who perpetually post the same positions, it would be a fair counter balance.
A lot of things really do require significant domain and industry specific knowledge. For those roles, it takes a very long time to turn up even one minimally qualified candidate. It's easy to say "find someone bright who can learn on the job", but that's not practical in the cases I've seen.
Many things need either specialized education or experience in a similar role to be effective within a 2-3 year timeframe. When on-the-job training would take multiple years, it's worth spending six months to a year to find someone who can fill the role. At least in the industries I've worked in, the roles that require extensive and rare training/experience outnumber the roles that don't.
I would say your comment come across as elitist and the evidence I have is also my personal experience. I've done it many times and rarely has anyone not been happy with the results thus far. Obviously there are industry specific areas of expertise that can be deep, and I can see where there are times you might have to hold out for the right candidate. But I think the figures you give reflect stem much more from personal bias than reality. Sadly, a lot of HR agrees with you which is why it's it's become needlessly hard to hire/find a job these days.
My point was that many other roles do require expertise in multiple domains as well as industry specific experience. These jobs are relatively hard to fill. They're also quite common outside of consumer tech companies, in my experience.
I apologize if this seems elitist, but some things simply cannot be picked up in a few weeks on the job. Those jobs tend to have a lower supply of qualified candidates and take longer to fill.
I've been looking at LinkedIn with some immensely negative feelings for some time now. I've always wished someone would just disrupt the hell out of them, because they don't deserve the near-monopolistic position they're in. They just don't care about their users!
And before someone says, "well [pushes up glasses] their real users are recruiters that they sell $20k/year seats to their search tool". Yeah, as a hiring manager I've tried using it. Awful! :)
And jobs boards are nothing new, I guess, but something feels really nice about how much bigger these "Who is Hiring?" threads are getting on HN. Maybe we really do need a jobs-centric "social network" for industry verticals. Beyond terrible UX and general disrespect, LinkedIn also just doesn't do enough with their data to make it useable. Maybe that's because it's just too hard to do for every job and industry out there.
And maybe that next-gen professionals' social network can be the first open and free one?
Or maybe I'm just wishing too hard for something that can't/won't be done.
From LinkedIn's perspective, that's a feature, not a bug.
It wasn't always like this. I used to have a reasonably cool feed centered towards technologies and news that are at least partly relevant to me and my industry. No more. It's all 'I gave a homeless person lunch and today they are the #1 duck breeder in Brazil' or some other nonsense.
I don't know when this started or if it is localized, but it renders the 'place to find useful information' aspect of social media completely void. There's no amount of reasonable curating effort that will save a news feed like that.
I try not to be too negative about things but without some serious change I'll use a static LinkedIn profile and visit it for a fraction of a second every week - to check for messages - and nothing more.
Well actually the bragging here often doesn't even get a veneer of humbleness.
WITH 12 DOLLARS IN YOUR POCKET
AND A WIFE AND KID BACK HOME......
Usually the articles are about how they sent 200 applications and it was all worth it to get that one acceptance, or are re-share spam about some manager taking a chance on a new hire or some shit. I haven't done any work into curating my feed since I rarely scroll through my feed anyway, but I think LinkedIn could do much better at putting industry-related news articles or blogs in there with less perseverance-porn
A good running collection of the 'best' of LI. Not a great running collection, but a good enough one.
We developers have so many social networks to be found at - SO, github, angel.co, random reddit or discord groups.
But others don't, and linkedin provides amazing value for them as a place to advertise their skillset.
What I am saying is that after having used this thing for so long, I want to improve upon it. If it were open, like HN's Who's Hiring threads, I could just run a BigQuery job on it and do a sophisticated search for the candidates I want. I can maybe even do that and open it up for anyone to use, or sell it, whatever! It would provide value for people.
But the thing that sucks about LinkedIn, aside from the spam by their growth-at-all-costs team, aside from the 5 second page load time, forget the terrible caliber of thought leadership... the BIG THING that makes it really hard to use is their data, or what they expose of it. And I'm wondering if it's just because figuring out the true years of experience (or something else) of someone is too hard for all jobs, but I know some tricks I can use to gauge that for an engineer.
Because if I know everyone I'd want to hire is on the network, I would pay a lot of money to get full access on querying that data. But what LinkedIn has built for it is just sad. And the biggest barrier to building an amazing recruiter search tool is the fact that the entire network belongs to this giant corporation that isn't even doing a good job generating value from it!
So let's open it up!
The problem is, "filtering" for jobs across multiple industries is harder than doing it for one industry.
There's no way to filter for people who have 10+ years' experience in software engineering roles. Or someone who has 5+ years' experience in CTO or VP Eng roles.
Add to that the "thought leaders", the quotes and other FB-ish artefacts, without forgetting a constant stream of brown-nosing and if you have 20% of the content that is interesting, you can consider yourself lucky.
In contrast, I hang in an industry specific slack with folks all over the world and it's so much better. I don't have a solution to offer with regards to an open platform, but I think curated communities are the way to go.
https://www.extremetech.com/wp-content/uploads/2013/07/micro...
Volunteers have to have somewhat cushy lives to be able to give their time and expertise for free. Stallman was willing to live by his ideals. He was also intermittently homeless while doing so.
These ideas of "We can rebuild it. We can make it better. We can also do it for free!" tend to be unrealistic fantasies.
I’m actually kind of interested in the people that have done “3000 applications etc etc.” The 100-300 applications are fairy common, but 3k?
Also, LinkedIn? more like LockedIn. There was a time where I enjoyed the service; I liked connecting with people doing interesting things from cool backgrounds.
But, I wonder if it hurts more than it helps... I feel like a lot of jobs are filled before they’re flown, every other job is meant for people who already have a job.
I like reading the job postings though; a lot of them are like non sequitur slam poetry.
This might be my naïveté (or the ipa), but I feel like it’d be cool to have a reversed scenario where I can get info on the team, the project, etc. before even reaching out (i.e. companies apply to individuals, but I think I may have reinvented recruiting? I know this may be unrealistic...)
Idk sometimes it feels like LinkedIn offers a non-service, talent will always be employed and/or sought; if everyone else is just a company’s 1000th pick aren’t they just wasting their time?
The disparity between freelancers seeking work and people seeking freelancers on HN is huge - way more people looking for work, than people looking to hire. That makes it feel like there is an oversupply of freelancers, which would drive down the pay rates.
However, I think it's more of a reflection of the HN audience than the broader market. I.E. HN readers don't hire as many contract/freelance employees. This aligns with what we've seen in practice as well. If software is the company's core competency, they will be less likely to hire contract or outsource.
On the other hand, there is a huge need for non-software companies to hire senior devs and they hire a lot of contract/freelance. Demand is high and rates are good ($120-200/hr)...at the right companies. So, if you are a contractor looking for work, I would target companies where software isn't their primary business.
Source: I run a company that helps senior software engineers find contract work. (Shameless plug: www.facetdev.com)
FTFY.
To the author: Good job, I'm sure many people will find your article useful! I wish I would've had this article at hand a couple of years ago when I needed to aggregate data from several websites that provided no API.
Nowadays, web scraping has a nonzero risk of hitting legal issues for sites when "the only way to gather data is by resulting to scraping", especially when an API already exists with a ToS/Guidelines on how the data should be obtained and used. And even moreso if the data itself has monetary value.
The new way to marginalize disposable team members is to replace "interns" with "remote" workers.
The savings are likely shifted over to cutting costs for the on-site office footprint, instead of optimizing for freebie slave labor from the lesser skilled, green college newbies. This also probably carries a quality over quantity trade off, in terms of hours spent working (college kids sweating 70 hour burn-out weeks, because paying dues or whatever).
Tech is mature enough, that there are enough veterans in the talent pool shrewdly compromising the level of their engagements, and there is less novelty to attract wide-eyed, coffee fetching would-be script kiddies anymore.
Did intern frequency drop or did the sheer increase in posts shrink their overall percentage? The concern I have there is the idea of becoming too "top heavy" where no one is interested in helping interns grow and only want to recruit experienced people.
[1] https://github.com/HackerNews/API
If so, then you want to check out this free training how to make 6 figure income in 2019
https://pst.cr/CVC1J https://pst.cr/CVC1J