Ask HN: What can I do with my DevOps Experience?

34 points by chigwe01 ↗ HN
I have been fortunate enough to roll out the microservice (kubernetes) infrastructure for one of the largest websites on the internet.

However, I know that I do not want to do Ops work professionally. I kinda just fell into it right out of college and chose it because I saw the potential.

However, I feel like that knowledge has gone to complete waste and wonder if I could leverage it in some unique ways.

What would you recommend that I do?

36 comments

[ 4.3 ms ] story [ 81.2 ms ] thread
Why do you think it’s gone to waste? We have large teams of devops/SRE at my (large big tech) company. Are you interested in SRE roles?

I’m not really in this area, but something like Google SRE handbook might get you excited about it.

(comment deleted)
Move into an adjacent field? What do you want to do professionally?
I am learning full stack and no code development. Hated the fact that all my work was at the infrastructure layer and not the actual development.

I am also shipping my ideas for the first time ever. Super excited about it.

One thing you could do is work for an infrastructure-related company as a developer. Alternatively, if you work at a small company that doesn't have a dedicated infrastructure engineer, you'll still get plenty of chances to flex your infra muscles while still filling other roles and being close to the product.
(comment deleted)
I am personally using it to leverage vast amounts of cash off people and using that to do stuff I actually give a crap about. The difference between me and you is that I have come to terms with this being purely an earner, which there is nothing wrong with.
two obvious ideas:

- learn server side language(s) & become backend dev

- get an IT sales job

You can work as a developer and have excellent ops skills through work experiance. The dev in devops. This is a skill that employers are looking for. Also since you have worked in operations you will understand better how to write applications that have good operational properties monitoring etc.
Move adjacent to do Security work. That is, all companies need security help, and they know it and will pay for it. You help them.

Often Developers see security as a distraction from building features, so working with companies to ensure they have strong security is a good career.

I worked with a bank building a "secure cloud" on top of a public cloud. That is, our team got a long list of requirements, like "storage buckets encrypted at rest", we'd watch public cloud changes, then alert or fix the issue. The bank got to use the public cloud, but also had a much finer-grained control of a lot of security concerns.

DevOps is a very loose term in today's market.

It can range from developer, to pure support engineer, to a jack of all trades, to yaml engineer, to CI/CD maintainer, to Google defined SRE.

Figure out what you want to do, and what you dont want to do, filter job postings and ask questions early in the process.

I'm a "Lead DevOps Engineer", and what that means at my company is that I lead a team doing all of our infrastructure automation(terraform, python), CI pipeline creation and maintenance, update and maintain all the non-prod environments (k8s, docker compose, swarm), infrastructure architecture on all major new projects (since we'll be building the automation to bring those projects up and maintain them), we also get everything else that seems hard to other teams. We build the tools and automation that we use, as well as the tools other teams use. This is important because we provide a common interface to all our teams, even though we work across AWS, Azure, OCI, and on-prem. The developers dont care which cloud their stack is on, that is abstracted away from them.

What we (DevOps) emphatically do not do is on-call. We have a "SRE" team for that (not SRE in any sense of what you think, they deploy new app versions to prod and manage datadog, and manage on-call, they are incapable of writing even basic python).

> What we (DevOps) emphatically do not do is on-call. We have a "SRE" team for that (not SRE in any sense of what you think, they deploy new app versions to prod and manage datadog, and manage on-call, they are incapable of writing even basic python).

This sounds like one of the most screwed up environments I've heard of in a while!

My previous title was sre. I'm looking for a new job and it is a horrible experience because I need to every time dig into what exactly their sre is.

Which is even more annoying because every sre job is some "everyday is different" bs.

Very, very screwed up. Duct tape and bubble gum. My goal over the next 24 months is to get enough of the projects, automation, and other sustainability issues resolved that I can tackle the SRE team problem.

Starting with firing 90% of them once all processes are automated away and they can no longer justify headcount(3x the headcount of DevOps). It will be a battle, that VP has nothing but SRE under them and the only reason they werent fired years ago is because of their connections with the CEO.

Sounds like they renamed the app support team to dev ops, and the ops team to SR and were done. Pretty standard move in some organisations to "keep up with the times".
DevOps is a cultural movement, not a job tittle, but if it is to be used as a job tittle to represent the values of the cultural movement one would assume that you'd never operate by "throwing things over the wall" to another team with no skills to contribute fixes, automation etc....
Was a movement. It's long dead on the water and became a soup of different loosely defined roles.
> However, I know that I do not want to do Ops work professionally

It'd be helpful to know why you don't want to do Ops work. There are many roles that fall outside of DevOps that have the same downsides (frequent oncall, unpredictable workloads, further from the product, CEO doesn't understand what you do, etc). There are also roles that fall under DevOps that don't have some of those downsides (infrastructure architect-type roles).

For example, there are people in the comments recommending security, which is great for job security, but generally bad if you want to do interesting work. Usually developers do most of the actually interesting security work while security teams check boxes for SOCII audits unless you're working for a security product, which can be pretty competitive.

I got into tech because I wanted to ship software that people used. Then when I got my first job out of colllege, I chose the k8s team because I saw the career potential in kubernetes.

Now, I feel like i have such unique knowledge, application of k8s at such a large scale, and I no longer want to use professionally. Makes me feel like my career till this point, has basically gone to waste.

I have done oncall, CI/CD, Kubernetes, dev tooling, terraform and some other devops stuff.

Perhaps consider applying to one of the companies building ecosystem tooling around Kubernetes or other schedulers? Your at-scale operational knowledge will be invaluable - it's something many people do not have.

Actual in-the-trenches experience of distributed systems at scale is fairly rare. Anything that helped you acquire that while being paid is likely not wasted!

(comment deleted)
An idea: work on something that will help other people to never have to learn what you had to learn about k8s.
> software that people used

I’m assuming by this you mean non-technical people.

It feels like if you can handle the chaos, an early-stage startup is the way to go for you. A single developer can significantly contribute to product discussions and infra platform decisions simultaneously.

Perhaps also look into positions outside of traditional tech? Med-tech, green-tech, etc are all in desperate need of infra experience, but tend to not have siloed roles like tech-tech has.

I absolutely agree not to get into security and try to stick a little closer to programming, with the same caveat that if you are doing some programming for a security company then maybe it would be a little better.
Would you please care to reflect on the following problem area, which you might be uniquely positioned to understand and help address?:

Personally, I find it alarming and problematic (though understandable and not surprising, obviously) that so many industry (services, jobs, etc.) are dependent on the big three public clouds (G, A, and M).

- Those owners (and their tenants) can decide to boot customers on whatever (illogical, political) whim; I think this is not purely fantasy.

- Despite whatever portability may exist due to the containerization and microservice standards, those owners essentially practice the "capturing" of customers (and workers) through pretty complex sets of knowledge for running, deploying, and maintaining the services. (Always be upgrading, whether for features, security, API/tool/ecosystem/business updates, etc.).

Sure, there are many other hosts. But I think there is (maybe) something where one could say:

  1) Okay, it is (stupid and uncomfortable) to depend on G, or A, or M, given all that they do, even if they abstract away a lot of details the industry has come together to solve.

  2) The solution is (stabilize, document, and publish/educate) regarding a set of tools/practices for being truly "infra agnostic"--to build and run the same microservices, but on any given set of hosts that one can own/rent/etc.
I hope I made that clear. Maybe this already exists (if so, please tell me), but I'm pretty sure it's not optimal yet, if so.

For background, and some of the above motivations aside, as someone who worked in a big company that owned its infra and did not use the big 3, and seeing all the jobs requiring experience with (one of) the big 3 public clouds, even though studying up would be doable, it seems like a fool's errand to me and a huge turn-off to learn even one much less all of the big 3 public clouds' ins and outs, because I know they will just change and I know for sure they are not thinking companies that will do the right thing.

Anything you could comment on regarding whether this is or is not a real problem (or related to one) that you might be uniquely able to solve by leveraging your experience would be very interesting to hear.

> to learn even one much less all of the big 3 public clouds' ins and outs, because I know they will just change

You'd be amazed (by the sound of it) at how little changes, at least in AWS.

I'm not convinced that people "forgetting" how to run hardware and network gear is a real problem. Most people never knew to start off with.

If it is a real problem, a market solution will emerge that replicates the API-driven provisioning experience of public cloud on hardware. Several companies are working on this, and we'll see if they are successful or not.

(comment deleted)
You advice is absolutely what I wanted to hear! I dont want to be a devops engineer. However, my unique situation had positioned me to see some things and I am trying to see how I can leverage into other things such as a Startup, consulting (I have never done this before)etc.

My low leverage answer was to make awesome youtube video tutorials.

Thank you so much for your response!

If interested in consulting, perhaps you could shoot me an email. Address is in my profile. I may actually need exactly your expertise right now.
(comment deleted)
Infosec. More money, no on call except for SOC roles. Lots of folks who need secure cloud envs, container envs, cicd, etc.
If startups/small companies are of any interest, there is often a large amount of horizontal maneuverability due to people often wearing multiple hats. For example I joined one startup (~12 employees) as the sole DevOps engineer but it quickly turned into not just DevOps but security (everything from dull SIEM monitoring, to red and blue teaming), database admin/optimization, backend development, customer onboarding, and even a little frontend. Most of those came through because I was the only one with experience handling them or a specific issue (but not always on paper) and I actively offered to help. I could have also avoided them if I wanted to, but when you can actually trust your small group of colleagues to not screw you over, its a good time branching out and learning new things at the same time. By the time I left I was spending most of the time doing security and hiring to build out teams (~100 employees when left), both my preference. Would strongly recommend small startups where you talk to the CEO practically every day and you get the sense employees are pretty happy.
I had a similar experience in two startups.

I can really recommend this path.

(comment deleted)
Companies love people with infra experience and you can easily pivot if you show initiative, I did!