Ask HN: Where can we find the unsexy jobs?
Everything is double-digit YOY growth with impending exit, and kubernetes all-the-things, and DevSecOps, and SRE, and web-scale etc etc.
Where are the places running their setup out of a rack in a rando datacenter grandfathered into an affordable Edgecast plan running a LAMP stack on Debian using borg for backups?
This may read like satire, but I promise I'm seriously asking.
Or am I just way too old?
Related: https://boringtechnology.club/
221 comments
[ 1.6 ms ] story [ 257 ms ] threadI'm going to say, on a random job board, this is actually the vast minority of jobs. Most, even at FAANG, have lots of unsexy maintenance that just needs to get done. Even in fancy machine learning land (where I work) 99% of the work is fairly 'boring' in the traditional sense.
Also, there is a strong bias to talk about 'sexy' work outside the company. For a variety of dev marketing reasons. Nobody does a conference talk about the uninteresting boring work that gets done.
Id go to that
I learned that the job really should’ve been called “Data Janitor”. 90-95% if the work was cleaning data sets and making sure that last update in some ML library we used didn’t mess up something.
EDIT: I was considering doing this myself recently, so maybe you'll like the idea, too: Just walk physically to all the mom&dad shops in your area and ask them if they want a free website, if they allow you to put Google text ads on the side. If you have time, you can build those websites and super cheaply host them themselves, and it'll certainly help those shops get their feet wet in the digital age. Plus you build up some ad/link inventory which you can monetize later. The reason why I considered doing it is because I'm 100% sure that people will tell you a lot of technical stuff that they need solved, if you're there and willing to listen. And those small shops are going to need small cloud-free solutions.
Where is this true? I'd say this ecosystem is more dependent upon Docker Hub et al.
Just to clarify: Docker isn’t a dependency management solution, but for most people it’s an important part of the solution.
I loved helping those small businesses but a lot of them don't have a lot of money to burn so going with a monthly rate they can plan on is powerful, as is managing expectations. A lot wanted Facebook for $500. There are patterns that emerge and one could build an in house stack to handle most everything I'm sure, just increase your monthly as they add services.
Mind explaining what this means? Did they want content generation for posts? Or did they actually just want someone to post what they had on file? Or did they just want advertising help?
My family has small businesses and I've thought about helping others, but the asks seem too all over the place.
> Mind explaining what this means?
As a freelancer, what I’ve seen is requests for building a Facebook-scale social network with all the scalable infrastructure and other bells and whistles, mobile app integration, etc., sometimes for a specific community (lawyers, doctors, or some such), that offer $500 as total budget.
I've lost count of the number of customers who said "this should be easy, I'm sure you can knock it up in an afternoon" whilst I'm thinking there is probably two years work in what you are asking for.
Other times, yeah, they want some crazy machine learning driven whatever with bells and whistles and social and apps on every major platform and blah blah blah and don't get why that will take at a bare minimum dozens of very talented and expensive people, and probably multiple years, to achieve. Which is the "Facebook for $500" thing.
Maybe you should consider Perl? I'm only half joking. I have a few Perl projects running unchanged for 15 years and hopefully for the next 15 years as well.
Plus I still use one service in production which was written in C++ in 2009. It's statically linked and as such the binary survived 3 major OS upgrades unmodified. Around 2015 I put it behind Nginx just to be sure with all the OpenSSL issues being discovered. It was designed to be multi-threaded to fully take advantage of a dual-core Athlon with 2GB RAM. Oh wow is that one blazing fast on today's hardware.
If you built an app targeting Ruby 1.8.7 (~15 yrs old now?) with maybe rack and sinatra as your only dependencies using only the bare feature set that was available then, it would easily run on Ruby 3 with no code modifications. It would be a lot nicer than Perl too I bet.
If you rely on industry standard software libraries however, you have to keep in mind that keeping up with the industry standard is in the bargain.
OTOH, if the culture of a language is that mature, popular libraries tend not to receive breaking changes, even if the principal maintainer reads a blog post and falls in love with New Paradigm X, maintenance can be much cheaper/easier, while still using plenty of 3rd party libraries.
In my experience only owners who had enough momentum from previous businesses with that setup or who just maintained a fairly small pool of revenue (and so didn't change.)
Also the economy went through a lot of M&A because its hard in the middle, failures are catastrophic, and paying some guy who will probably take you to beer lunches to host your stuff makes you look smart to all the other up and coming CEOs.
So unless the company is extremely young (say younger than 10 years) then there will be tech of all ages and all levels of sexy.
I briefly worked for an agency early in my career where most of the clients had WordPress or Magento sites with low to moderate traffic. They would usually only need a few hours a month each, so it wouldn't have made sense for them to hire full time programmers or sysadmins.
There’s a ton of business that simply need basic websites and those still need to be developed by someone.
Most businesses don’t need anything complex. It’s just basic sites and emails.
Keeps me busy, and automating as much as I can, before I ride off into the sunset. Sometimes the shiny ring isn't what you need to chase.
We also deploy with Chef and an in-house application deployment system written in Perl, do monitoring with mon, and understand that a sysadmin is a subject matter expert.
Hiring, too.
This. There are thousands of great "Tier 3" (not FAANG, not Fortune 500, but still large) companies out there in the US that will pay $100k-$200k with great benefits, 9-5 culture, and good job security. The SV startup world is pretty irrelevant at this point with the shift to remote IMO.
And then you pick local smaller companies and then you have you what you look for.
E.g. years ago a company might have run their own LAMP stack to host a website or a small online store or whatever.
But in this age of SaaS it is easier for these smaller shops to just use something like Wix or Shopify to handle it for them, complete with fancy wysiwyg editors etc.
I used to get requests to "add payments" or something to websites I used to help out with so that they could basically do an online shop. At the time I actively rejected those requests as there were huge headaches around things like PCI compliance and the like. Fast forward a decade or more and with GDPR and Cookie laws etc it is a total non-brainer to use a hosted service where they handle all of this, rather than spin up your own ecommerce solution.
It does not make sense to roll your own unless you have a large engineering team, at which point you move beyond some random thing running in some random data center somewhere.
Of course not everything online is ecommerce or CMS-oriented - there will be niche and custom stuff that is not handled well by SaaS offerings.
On LinkedIn.
This is the best answer that you will get.
For historical reasons, this is also heavily tied to the adoption of EDI. So I'd start looking for boring tech stacks in areas that have EDI documents associated with them: https://en.wikipedia.org/wiki/X12_Document_List
Back in the day when we had to hire a guy to go down to the cage every day to repair yet another faulty machine, we would have given our left arm for the ability to magically spin up a new VM of any size with a baked image by just running an API call. It's freakin' magic.
I also love, love, love plain Fargate. Not as stupidly complicated as K8s, but I can still run containers without ever thinking about managing hosts. Just keep my containers up to date, push new ones, and trigger a deploy. Never have to "upgrade my cluster to keep from using EOL versions" or "migrate ingress to a new API version" or some other ridiculous thing. It just works, and all I have to think about is my app/container.
I miss it not one jot.
They lost about 50k in sales that day, still less than the cost-of-ownership for redundancy.
PS. Please keep conversations civil. You probably don’t know me, but I’m the least “scared of failure” person you’ll ever meet.
Same goes for the HashiStack. I really don't want to work with Kubernetes anymore simply to deploy some containers.
They also pay much lower than even enterprise webshit, despite requiring increasingly more specialized knowledge.
I suspect this where the real "expertise" in Building Software Systems lies. Stable, Steady, Fault-Tolerant, least MTBF etc. characteristics are the name of the Game.
Just fair warning: you seem to be under the assumption that “boring” means “straightforward and just makes sense.” It does not. If you’re looking for that… good luck! Update if you find somewhere and I’ll join you.
I would recommend anybody this approach.
Rule of thumb for Germans, if they write a fax phone number on the imprint or the website looks 90s style... don't waste your time, skip them.
I went to visit out there and kinda dig the area.
If you happen to be looking for a DevOps or SRE role, check a large banks job boards, you'll be surprised how many open roles are available.
The point is - it's difficult to make generalizations about orgs that big.
Many use a lot of proprietary software, which can be a type of hell nobody understands till you end up as an expert in a niche area and spend your day dealing with vendors and tech support to fix anything. After 5 years you are an expert on "EnterpriseSoftwareCorpA On-Prem Elastic Scaling Stack and Integration to EnterpriseSoftwareCorpB On-Prem Data Stack" which nobody really sees as valuable, so you are now trapped in your career.
I know a bunch of people who wisely let go of 10+ years of their experience in some aging software so they could start again as an AWS solution architect. I'm sure it's was brutal doing that in their 40s/50s, with family, etc. It was tough, but 2 years later they are in a far better career position than those who are still praying for something to happen with those Enterprise stacks.
A lot of the hell that is enterprise software is how systems talk to each other. I'll give an example. Project is started to bring data from Vendor A Product into Product from Vendor B. Vendor A claims system is "open" but there is no documentation outside of some cryptic .NET examples. Even when a successful connection is made, the data and data model makes no sense and nearly impossible to map to what users see in UI.
Vendor A gets a call. Vendor A repeats "open" and brings an "SME" to explain data model. SME is really a Sales Engineer who concludes Vendor A has another product that will expose the data in usable way. Start evaluating this product, and it turns there is a lot overlap between features in Vendor A new product and Vendor B's product. Plus, you still need to use .NET and a proprietary connection to get data from A -> B. Plus,Vendor A's new product does data transformations in a black box..nobody knows what exactly.
Vendor A and B are pointing fingers and each trying to make a case for why their product needs to do X features. Nobody understands this .NET library, so a consulting company is used to build data pipeline from Vendor A -> B.
Granted, a lot of the above wouldn't be acceptable today and a lot of these types of systems are going away and being replaced by ones that are actually designed to be interoperable with other systems. This type of story is hopefully going away sooner than later.
I use a lot of proprietary systems in my current job at extra-big company, but at least they use stuff like SQL, RESTful APIs, etc. I can understand our Data and how it maps. Those are transferable skills.
I can only hope that FAANG isn't building systems where everything is proprietary and make no sense to anybody outside of the Eng team that built them.
I'm under the impression that gRPC is popular, and in some places GraphQL.
That being said, companies like GE, Siemens, and PTC are trying desperately to capture that space as well with their Saas/PaaS.. I won't say they are crap, but it's just more lock-in under the guide of "open." One of them has already gone the way of Watson.
YMML will vary in manufacturing. If you can jump to one of these companies that is on their journey from legacy systems to more open ones, you can definitely land in a good spot. Just ask the right questions when you interview there.
A lot of my Python expertise on Windows comes from working with that system. I used Python because 1) I already knew it, 2) I could easily run parts on either Windows or Linux and 3) all of the internal APIs I needed access to had Python wrappers available (or I could easily write one in Boost Python at the time). I forget exactly which Python libraries I used, but there was some Win32 COM going on, some ctypes and other Python based GUI automation, as well as a lot of process management (reports tended to hang quite a bit needing killing/restarting) and ETL work.
I spent roughly 7 of my 9 years at that company working in part on maintaining the integrations of that 3rd party system with our own internal systems. Yes, a significant chunk of my time at the company, but what software it was is just a footnote on my CV. Instead all of the interesting integration work I did and the efficiencies gained are elaborated upon (e.g. with 8 hours of development effort, I was able to automate away a previously 8-hour manual task that had to be done monthly).
And more of “places running near archaic .NET/JVM versions that somehow manage to combine bureaucracy with lack of organization”.
Majority of the services were older .NET framework projects with some other stuff scattered around. They had a sizable mainframe team, but we’re trying to migrate away from that platform.
I have a friend who works for a major low-code software company. They're doing quite well financially because of all the excitement around low-code. The product is good if you stay within the boundaries of what it can do. Some managers people think they can replace their enterprise Tableau/Spotfire/PowerBI license with low-code and they get bitten very badly.
Finding engineers for a low-code environment is a challenge. You need to understand software development well enough that you can build something because loops, conditional statements, all of those concepts are there. You also need to find somebody who is willing to possibly lock their career into a single tool and forgo the benefits of knowing a general purpose language like C#, Python, etc.
Some companies have success with finding technically minded business people or IT folks who don't enjoy coding and training them. They can thrive and build some nice apps. Lots of folks can't make the leap and fail. Software Engineers are probably the worst bunch to try an convince because the opportunity cost is too high.
Business / money making / crucial systems? No. Some random HR survey application? Maybe. Sadly Microsoft seems to have convinced a number of folks in our organization that this tool set is appropriate for all our development.
I'll second this. Strongly.
It is not so much boring as more of an amalgam of scar tissue and duct tape that accumulates over time and is a maintenance nightmare. You might run into the rare place that does things "right" with old technology but those are rare.
I do think some new things are "crazy" but I'll take "crazy new" before "crazy old" most days of the week.
but of course it also depends on how clever the people beforehand have been, is it stuff tied to impossible knots that cross over 5 parallel dimensions, where nobody knows what it does or how it's doing it, just that "it somehow works, so we don't touch it, that guy was a wizard", or is it just layers of faith held together with duct tape and rope, where people tried to fix stuff over years, throwing in patches that "probably should help, I think, maybe", which could be condensed into less than a third of the size, when rewritten...
I suspect that there's a fair amount of overlap—not total, but quite a bit—between companies that were doing old stuff The Right Way, with lots of well-considered automation and high-quality backups and well-documented, repeatable, largely scripted configuration, and companies that are on some of the "sexy" tech now. So it might be even harder than it used to be to find a company doing things the "boring" way but who don't have a horrible, barely-functioning mess on their hands.
I'd think some of the nerdier, niche tech places probably run things OK and not super new-school. Something like Rsync.net, maybe. Possibly places that like BSD in general will tend not to be on the new hotness. Difficulty: those sorts of places tend to have pretty low head counts so it may be hard to land a job at one (go figure, much of the tried and true stuff Just Works and doesn't require a ton of babysitting if you halfway know what you're doing)
Are there any examples of this? I’m coming up on 20 years into my career and i’ve genuinely never seen a large company that had their IT function ticking over sweetly. Every single one had aspirations and had some things nailed to a greater or lesser extent but none were “done”.
If anything, things are a LOT better today. The old mis-configured MS Exhange host hanging off the office DSL line is gone these days. The MD / CEO’s password is unlikely to be Password1 nowadays.
Well, yeah. It’s been 10 years, and we need to change the password every 3 months, also, you told me not to use ‘Password’, so the password is now ‘CompanyName40’
I've seen a lot of modern tech from other large companies too (banks, retailers etc.) - the culture and pace of development might be different but their tech stacks are very up to date. Even if some of those companies have an old mainframe still running somewhere, they have tonnes of other software too, most of it much more up to date.
If only it were so... A lot of teams in these companies are heavily into fad-chasing, so your random internal web app that has 1 user per hour will be deployed in fully-scalable manner on the company's k8s cluster (which is shit, because company didn't put nearly enough people into keeping it running).
And I think companies 500-1000 of the Fortune 1000 would be even more interesting to work for.