HN Explorer

Ask HN: Was hired to improve company's devops, founder won't listen to my ideas

79 points by milquetoastaf ↗ HN
Three weeks ago I joined a fast growing startup on the West Coast to improve their dev operations and processes. The company is seeing crazy growth but they are still doing things like pushing directly to master and using prod deploys to test things out. I was hired to help improve this and implement things like staging environments, autoscaling of resources, and other DevOpsy stuff.

Since joining I've received considerable pushback on my ideas. Obviously I am the new guy and don't expect to just dictate people around nor is that my intention. However what really gets me is that the founder just straight up does not seem to trust me. Every suggestion I give - even things as industry standard such as the use of a staging environment - is met with suspicion and doubt and the insistence that I hadn't done enough proper "research" on the idea. I find this highly offensive considering I've been working in DevOps for almost seven years and before that was a SysAdmin.

So after two weeks, I got the approval to create a staging branch. I wrote up a huge document on the staging workflow, covering edge cases such as hotfixes and rollbacks. I put time on people's calendars and walked them through it including the founder. Yet he still continues to resist the change and pushes directly to master, ignores the new processes I put in place, and generally just does what he wants. He also keeps harping that I need to "earn his trust." I find this baffling - I mean, I was hired for a reason right? Why would they hire me if they didn't trust me?

I am starting to get frustrated, especially since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app. I'm trying to think of the best way to approach this diplomatically. Do any HNers have some pointers or advice to give? I've worked with difficult clients before but I've never faced such a huge lack of trust.

77 comments

[ 4.6 ms ] story [ 121 ms ] thread
invalidOrTaken
Only thing I can think of: If you have freedom of action (you can choose what to do with your time), then do something small, perfect, and seamless. Present it as a fait accompli. That's your foot in the door. Repeat.
invalidOrTaken
Later note: reading through all the responses, I'm impressed at what a great resource the minds and experience of HN are.
_skybird
Don't make suggestions. Tell them, what they are doing wrong and why.. and then show them what to do. Trust seems like an excuse for being too lazy to adapt to changes.
itronitron
If it were me, I would want to run a dev system off the master branch and then tag versions that get the OK and are deployed to the production system. That would give the founder some sense of immediate gratification while insulating the production system from goofs. disclaimer << I am not a devops person
lacker
It sounds like you have a different idea of what you were hired to do than the founder does. Talk to the founder and figure out what they want you to be doing.

If your manager is a different person than the founder, your manager should be able to help you resolve this.

Often companies at some stage realize they need better devops procedures, but also realize they can't slow down their pace of progress, and there are tricky tradeoffs to be made. I encourage you to not be insulted when people have concerns - they may just be concerned that you do not yet have all the context about the company as they do, rather than critical of your experience.

cweagans
Leave. There are better environments to work in.
chrisbennet
This. You aren’t happy and you can’t make your boss happy. Accept that it was a mistake, “admit” that it was your fault for not understanding what they wanted, and go your separate ways. Don’t try to get a “dig” in on your way out, just leave on a good terms as you can.
scorpioxy
I agree. Especially when the boss starts talking about "trust" and not listening to your proposals. Getting blamed for other people's decisions when you've already warned about the consequences is a flag that you've already lost the battle. Some people just can't be reasoned with.

Granted, saying leave is easy to do but the OP might not be able to due to their financial situation or other circumstances. So maybe leaving is not the right option but I don't believe this relationship can be salvaged so checking out(either mentally or physically) is the only way forward, as I see it.

timb07
Alternatively, get ready to leave.

But before you do, push back against the founder. The next time he pushes directly to master, immediately revert the commit, but cherry-pick it over to a development or staging branch instead. Then notify him what you've done and why (including that you're trying to prevent breaking changes being made on master, like happened previously).

How the founder responds to this will determine whether you leave, or whether you stay a bit longer and perhaps gain the founder's trust. But still expect you'll have to leave. :/

stephen82
It's like reading the same story, but with different protagonist and type of job.

I had a similar situation like yours, but I was in the traditional Sys Admin category.

Basically I had to deal with toxic behavior for a number of years and in reality what happened was to stay where I first started for the whole time.

When they had an issue, I would suggest solutions and useful alternatives, but I would get nothing but "No"s and "don't you dare do such thing!", so they could do their ways. Now, when they would mess up badly with something, they would come back running to me to fix their shit.

I was never trusted nor had they believed in me and my skills to improve our entire infrastructure.

I got tired arguing and fighting for nothing as I was getting nowhere.

Eventually, I automated my tasks to the point of not working, like doing nothing at all for like...4 years or so(?) and right before leaving, I took notes based on my scripts' implementations and then deleted them for good.

Lastly, the headquarters decided to close down our branch.

I don't think I would like to work for such type of environment anymore, but I'm afraid the majority are like this more or less.

jhot
I did the same thing of automating my own tasks. Only lasted about a year though because I got too bored.
newman8r
Wouldn't the company own the scripts you wrote to automate their processes while you worked there? Sounds kind of spiteful not to hand them over, although I can empathize with your frustration (been in similar situations as well).
stephen82
I guess, but I am pretty sure if they knew, they would fire me therefore I could not risk it.

So, the best thing to do was to delete them as a whole and move on with my life.

newman8r
Yeah I gotcha, sometimes that's all you can do. Sounds like they were terrible managers. I hadn't even considered the (obvious) fact that automating your own job can be dangerous.
justin_vanw
What is the negative consequence of breaking the site for an hour? Is it "some people are a little annoyed", "nobody notices", or "customers start calling and are very upset" or even "customers file lawsuits"?

The amount of friction you add to the development velocity has to be proportioned to the potential downsides. A system that shows doctors xray images in the ER cant go down or people might die. A site like twitter can go down and slightly improve the life of its users. It sounds like you are recommending big company super risk averse solutions at a startup, which is just silly. Besides, staging environments never work, for a long list of reasons. You should be recommending continuous integration and a focus on proper unit tests.

dfee
> A site like twitter can go down and slightly improve the life of its users.

Ha!

whack
> Besides, staging environments never work, for a long list of reasons

Can you go into more detail on this? I would have thought that having a staging environment where all changes get battle-tested by the wider team, before being deployed to prod, would be a good thing.

[deleted]
(comment deleted)
Xeago
Unless you can spin up and tear down a staging environment provisioned with data and instrumentation I too don’t believe in it. Without that you’ll be using something shared, where either one thing at a time is subjected to testing or you’re testing multiple things at once which could conflict (including falsely positive) with eachother.

Hope that provides a new perspective. If the team is small, or the product is not susceptible to concurrent development you might not have this phenomenon.

Chyzwar
Staging env should be (prod - 1) version. The whole point is to test thing together with other changes. Stage-environment is created to eliminated "works on my machine" problems.
bjourne
But in that case you have qualified people actually performing battle-testing. If you don't, then a staging area can be more than useless because developers think that making sure their code is bug free is someone else job. Also in most companies testing is deemed inferior to programming, so the least capable is assigned testing duties while the better programmers write the code. Which is wrong because battle-testing software takes just as much skill as developing it.
websitescenes
Having a staging environment is pretty standard these days. We set one up from day one at our startup. It doesn't really take much effort and your app benefits from being decoupled from the domain. This allows you to run app instances concurrently without relying on domain specific details. It also allows you to physically test changes before pushing to production.

I can hardly see how a staging environment is "recommending big company super risk averse solutions at a startup, which is just silly". A staging environment has never slowed us down; if anything it gives us the confidence to push to master sooner.

stestagg
I think most of the positive statements across this thread are generally correct, and most of the negative statements show a lack of awareness/inexperience:

My experiences are:

1. Staging environments are typically imperfect and have issues

2. Staging environments often help with product stability despite (1.)

3. Having staging envs is common, and could be considered standard in many companies, but..

4. Staging environments are a cost-benefit trade-off and should be considered on a product-by-product case.

5. CI and Unit testing are also common approaches to increasing product stability, and should be subjected to the same cost-benefit analysis per-product (yes, even unit tests), It's not an either-or with staging environments

6. staging envs should be prod-1 but often they end up being (prod-1 + some workarounds + some cruft left over from a failed deployment + other noise) over time.

7. Running a fully prod-equivalent staging environment can be prohibitively expensive (exact replica of prod configuration including infrastructure, realistic copies of prod data at volume, etc..) so usually some compromises are made to give a best-effort simulation of prod (fewer resources, etc..).

8. 'Battle testing' in staging is not always done, it's not uncommon for staging to be largely not looked at, except for whatever automated testing/alerting is running in that environment

9. It's ok for staging environments to test multiple changes at once, the typical intent of a staging area is to identify the presence of problems before they hit prod. The debugging & isolation of what caused problems is a secondary issue.

Your experiences may differ from the above, but they still stand

[edit: formatting]

websitescenes
Good points. Your description is largely similar to my own experiences. On number #6, do you mean that the code base deviates from what is in production? IMO if staging has deviated from production then your pipeline is broken. There is not a single piece of staging specific code in our app. Whatever works on staging works on production. This is accomplished with ENV variables instead of hard coded environment specific details. We actually push directly from staging into production.
hluska
> On number #6, do you mean that the code base deviates from what is in production? IMO if staging has deviated from production then your pipeline is broken.

I'm not the person you're replying to, but in my experience, this only happens if you get 100% buy in to the dev ops approach. If you get less than 100% buy in, you'll often end up with greasy environments with the extreme end being a founder type writing code directly in prod.

[dead] cam3ham
lol, you go on and on about heavy process at a startup and then recommend CI w/ full unit test coverage?
halis
Maybe he just doesn't like what you're selling. Find out what direction the company wants to go and help them get closer to that.

I would recommend temporary feature branches, that frequently merge into master and also release branches for the code that you want to push to production. You can always do hotfix branches off of the release branch.

There's always that wet dream of no branches at all, every edge case automatically tested and pushing right to production if the tests pass. But this is really not practical for most places of business.

But if you can promote some meaningful amount of: 1) Unit tests 2) Integration tests 3) Functional tests That automatically run when code is pushed, that's a good first step.

Also, if you can get them on a regular release cadence of monthly or better, that would also be preferable, and you should be able to release code to pretty much any environment with the click of a button and with little to no manual intervention.

Oh and use Vault. :)

Stick with the basics and it will take them a long way.

zwickerbne
My suggestion is similar to the ones specified here.

Part of the DevOps concepts is to try to take a scientific unbiased approach that is to experiment, measure and make more changes. You could always treat implementing these changes as experiments to see what works.

Simplicity, trust, team rapport and support from top down really helps in order to implement change.

If the team, founder and manager aren't really interested in changing you can either work on influencing and correcting their mindsets, fight them on it or move on.

danielvf
As you said, the "earn trust" doesn't really make sense, so I'd guess there is actually some other reason. Is the way you've implemented the staging plan making life more difficult in some way? Does the founder have some other objective in mind?

Once I worked with a company that had a super smart founder who hired a super smart sales manager. The sales manager quickly noticed that the company did not follow the usual industry practices around quoting projects, and on his own initiative started rolling out a new system, including training sales people, and changing the company's software systems. There was increasing friction with the founder over everything, and eventually the sales manager was let go/quit.

Somehow the sales manager never understood that the core issue the founder had with his plan was that even though 95% of the industry operated with a secretive, price-what-the-customer-can-pay sales process, the whole company had been built over decades on transparent pricing for everyone. It was big part of the trust the customers had for the company and quite a competitive advantage.

Which is all to say that to you, devops means "production doesn't go down", but to the founder, it could mean "Anyone can push to production even faster on more servers." :)

Since the founder has basically said he doesn't trust you, it's likely there's something you are doing that he doesn't like. I hear that you are offended by the founders lack of trust, but being offended is a really good way of missing the actual meaning when he is talking. If things continue as they are, both of you are going to be very unhappy.

Given that there's this disconnect, I'd really spend some time with him to work out what it is that he actually wants -- and when he says he wants something, push deeper to find out the why behind it! Sometimes people ask for a "universal plugin system for our app, so any can extend it" when all they really wanted was "we can translate the app to French".

It's aways possible that the founder is a loon in this area, but I'd say that's only a 10% chance. 80% bet that there is a miscommunication.

derekp7
The way I've had to explain this to coworkers when dealing with our former manager -- Take what he says, then figure out what he really means, and from there deliver what he needs. But dress what he actually needs into something that looks like what he meant to say, ignoring what he actually said.
iovrthoughtthis
A manager that cannot communicate what they need clearly is not a good manager.
danieltillett
An employee who can't understand what a manager is communicating is not a good employee.

More seriously communication is a two way street and problems can occur at either end.

iovrthoughtthis
I disagree.

It is the sender who is in control of the format. You have to ensure you’re sending something that the reciever will understand.

Good receivers probe and negotiate a better format but ultimatly it is the sender who is responcible for the communication.

Between peers I think it is more acceptable to require more negoatiating the format but in a manager -> employee dynamic (or any other power imbalance) it is the managers responcibility to be understood.

Reliquishing that control is disempowering and stunts personal growth IMO.

I think my view point requires a specific belief about peoples ability to do things and grow. Without that I can see how judging others for their ability to understand makes sense.

I believe in people though.

ivanmaeder
> Given that there's this disconnect, I'd really spend some time with him to work out what it is that he actually wants…

This. Spend time with the guy, try to understand where he's coming from. When you do talk I'd suggest you:

- Make it clear that your motives are to help the company. I.e., you both have the same interests

- Listen, don't try to win the argument, don't push your solutions. The aim is to understand what he thinks is needed

- You can very well explain how you're feeling. Maybe you feel that you're not really sure what you're doing there given that none of your suggestions are accepted. Tell him

- But don't blame, don't presume ill feeling or ignorance. Ask him for suggestions

There's a chance that this is never going to work out and as somebody else said, you're set up for failure. I think you can figure this out if you do as much as you can on your side to understand him, and he still doesn't do his part to help you do well at your job (whatever that is! It may not be what you think).

Something else…

I'm not sure this is part of the issue but I've seen this a lot: you work on an amazing new way of doing things. You figure it all out, document it, dot the i's and cross the t's. And then when you present it nobody is interested. Why? Regardless of whether it's a great idea, you didn't involve anyone else. You need to bring people along with you: ask about their problems, ask for input, collaborate as much as possible…

lazyant
So you've been given the responsibility but not the authority. You've been set up for failure. Demand clarification of what's expected from you and the backing to carry it out, because right now you are not able to do your job.
gnulinux
Sounds like a very toxic company. You're there for a reason, if you have 7 years of experience you do not deserve this attitude. In the West Coast, you can probably find a new job in 3 minutes anyway. Leave and find a new job.
mysticlabs
If the founder is who you report to then you need to schedule a 1-on-1 with them in private and ask them to clarify why they hired you.

If the person you report to is someone else, you need to have this conversation with them first. Then have a follow up meeting with your boss and the founder and clarify your role and position to the founder with your manager.

Just be upfront, and say hey I’m the new guy, and I’m getting pushback, and want to understand my role better so I can contribute my expertise and create value for the team. Not here to step on anyone’s toes, just trying to gather more information as I haven’t been on boarded properly and I’ve been here nearly 3 weeks.

Then let them tell you what they want.

If the response is not what you agreed to when hired and the role they want you to fill is not what you thought then look for another job. Better figure this out now then waste months or years working in a toxic environment. DevOps is one of the most sought after positions in the world right now, don’t settle for being bullied or working in a toxic environment.

At the same time, you need to establish a clear line of communication first. Collect data, and then analyze what the situation is.

Many startups don’t know why they’re hiring people and often aren’t sure why or what they’re hiring for. For example, I got recruited for business development from a company then in the interview realized they weren’t looking for business development. They were looking for an office manager to babysit the developers so they could not be in the office. When I asked about how I would take calls in an open office next to the dev team, and what kind of flexabity I would have to broker deals and deal structures they didn’t even know what I was talking about. The founders basically just wanted an exec assistant with a fancier title.

It’s entirely possible you thought you were hired for devops but they don’t even know what devops is and think you’re just an IT guy or sys admin. That’s why you need to gather more info and figure out if this is something you even want to be involved in.

[dead] princetontiger
Give it another month, and if behavior doesn't change... well, get out!!!
karmakaze
Get aligned on values as others here have said. We had a similar situation with types of tests and coverage for a product that did not yet find fit. In retrospect everyone could see that it was overkill. After the pivot we now commit to master run fast unit tests in CI, fast acceptance tests in staging and most valuable journey tests for core user actions, like sign up/in do the main action, get notified. If it passes you can push to prod or do some more ad-hoc in staging. Pipeline is fast, have stability, and everyone's happy. Top pri after prod issue is not blocking CI. If not a quick fix, revert til green. We also have a separate dev integration env for testing changes with some uncertainty. Go fast and only break minor things.
edoceo
Presumably you made it through a rigorous process to be hired? And maybe have some background in the space (it looks like you do).

I'm a former coder turned CEO. What I see here is a leader who is afraid to let go. And at the same time blocking proven models. And after that you carry the fail-bag.

One time, 15+ years ago I joined a small company as an IT Director - reports to CEO - had a pretty good time investment through the interview process. The second day I had similar feeling to what you've expressed. I quit. Sharp talent can find a new gig - quick.

Like is too short for you to waste time explaining that 2>1 to folks who are paying you to tell them that - and then ignore you.

eximius
If you're willing to relocate, the company I work for is in need of someone like you...
karmakaze
Also add that 'flake tests' were a sore point until it became clear which ones were actually the system failing and which tests didn't reflect actual usage. By keeping stats on test faulures it was clear which were causing distraction time and fix them or system cause first. After that we got stability much more quickly than always treading seemingly random errors.
djangowithme
"So after two weeks, I got the approval to create a staging branch." - lol
poulsbohemian
>founder has pushed breaking changes directly to master

Is founder also CEO? If founder is ceo and is coding rather than working on sales / marketing / traction, then you need to run briskly away

em-bee
why? if he is a tech founder, and there are co-founders that work on sales and marketing, why should he give up the CEO position? or do you mean to say that from his behavior he is obviously not qualified?

greetings, eMBee.

poulsbohemian
Put simply: Lead Developer != CEO.

Unless it is super early stage (pre-employees, pre-revenue, pre-everything), the CEO should not be hands-on-keyboard. If they are, there are several problems:

1) Delegation. They are demonstrating a management failure to delegate.

2) Even in a technology-oriented business, the job of the CEO is not to cut code. The function of the CEO is to manage and lead all the things related to the growth and success of the business (think: sale, marketing, hiring and recruiting, raising capital, interacting with investors, and a whole lot more). If they want to undertake the technical functions of the business, they should become CTO / VP of Engineering / etc and turn the reigns over to a CEO who can execute on the business.

3) Even if there are other team members engaged in sales and marketing, see point #2 - the CEO is fundamentally responsible for the growth and financial success of the business and it is highly unlikely that technical tasks should be their priority.

cbanek
It sounds like you're in a rough spot. Luckily, it sounds like from what you're talking about that there's a lot you can offer and there's a lot of low hanging fruit that can be picked.

Here's what I suggest:

1. Find the easiest things to do that will save people time, or fix problems that people have been talking about. Especially useful if founder is one of the people helped.

2. Once you've found some easy things to do (<1wk), figure out which ones you can do without any help, resources, or authority.

3. Do it, and show it to him, or talk it up with the team and save them some time.

4. Tell the founder you've saved people time, and him money. Invest in some of your next ideas to save more time. Don't ask for permission. Just say you're doing it.

5. Success builds on success.

One thing I'd definitely point out is sometimes you have to go skunkworks. It doesn't sound like you've been there long and they are harping on you to ship something. You likely have plenty of time to just do some easy things, like automating paperwork or forms.

In terms of your staging workflow, again I'd apply concepts that I don't need permission for. Instead of blocking commits to master or even needing a staging branch, you could write some automation that takes a git hash. There's lots of ways to work around needing permissions or official things. Never underestimate the use of a button to do something, rather than typing it into a terminal.

It also sounds like you may have people problems. Is your founder stressed? It sounds like he's having a rough time with scaling growth. Find out what is stressing him out at work and use your talent to eliminate it.

If all else fails, use FIRE. (and quit.)

Good luck.

natbobc
It doesn't sound like a great situation to be in. It sounds like the founder has strong opinions. I suspect he subscribes to Trunk Based Development[1] and Continuous Delivery[2]. He also likely wants to follow the practises that "leading organisations" follow in "Accelerate: State of DevOps Report"[3]. You might find it beneficial to research those subjects prior to discussions with him.

  1 - https://trunkbaseddevelopment.com/
  2 - https://continuousdelivery.com/
  3 - https://cloudplatformonline.com/2018-state-of-devops.html
teh_klev
Just some friendly advice. When you're referencing stuff elsewhere on the web do it like this:

[1] https://trunkbaseddevelopment.com/

[2] https://continuousdelivery.com/

[3] https://cloudplatformonline.com/2018-state-of-devops.html

This means the links are clickable. Using the code indentation thing you did there causes all sorts of grief for mobile device users.

natbobc
Thanks! Was faffing with it... the "help" isn't very helpful on how it formats. Sadly can't edit now. :/
imron
Can't believe this comment is so low.

I would guess this is exactly what is happening, an it sounds like OP is trying to push git flow or similar. I would push back against that as well. Trunk based development beats git flow every time.

anon31415926535
What could help you stay sane is to recast how you view your job.

Here is one way to think about it: your job is to help establish a culture of improved reliability. You are accountable for improving the process but you don't have the authority to decide the process.

A large part of being successful in this role is sales.

There is a pattern that works fairly well at starting a new engagement. Initially, the main goal is not to fix things; instead you need to focus on building trust. As you've found out, without trust, you won't have success.

The easiest way to build trust is to find a problem that the team acknowledges and is small. Tell them how you will solve the problem and have them agree. Solve the problem. Then repeat with another problem until you have some trust.

A good indication that you have enough credibility is when they stay asking for your input. Now, you need to find a problem that they are blind to or have given up on solving. Convince them it's a problem worth solving. Metrics are your friend here. Then get buy-in for your proposed solution. Keep visibility add the work is solved. Sacrifice overall speed for immediate partial wins.

At this point, hopefully you've reached a credibility tipping point and established a good working relationship.

I also had advice for dealing with change resistance, but that is not your problem. Your problem is building your personal credibility by delivering something the team cares about and wants solved.

Try not feel insulted. It's not possible. They're only human. :)

epynonymous
i think a little conflict is fine, working through conflict can often times build trust, use this as a learning experience because you're always going to have conflict, even if this conflict feels like a core belief mismatch. i find the founder is a bit too super direct and lacks tact, i'm a senior director, and have never told anyone to earn my trust, it's a given, i gain nothing by explicitly stating this, but that's besides the point, different strokes, different folks.

i also see a bit of a power struggle here, because your role, perhaps at other companies, has been more control and process oriented, this is a startup with few processes and control seems to be completely with the founder.

here's what i suggest, change is inevitably hard to deal with, it takes time and it's very natural for people to resist, so the first thing for you is to decide whether you're ok with this taking a long time, if not, there are a ton of other opportunities out there, don't waste time. if you're willing to stick it out and see value in the business, in the role, then figure out how to do things in incremental steps aka non-earth shattering, it shouldn't be, adhere to my god-like, every company does it this way devops pattern or else (even if it is god-like). baby steps where it's easier for the team/founder to accept, less friction/inertia.

another thing is through statistics, founders change caused this which impacted that, we had 3h downtime because founder's change hosed the system, this caused roughly $20,000 hit to revenue, or we lost x users as a results of this. if he's ok with this after repeatedly pointing this out, maybe it's because his goal is different, he's looking for speed of execution as opposed to short term revenue gain, in which case you need to try to align and figure out ways to break things even faster, but with simple ways of rolling back. maybe he/she wants a/b type testing with quick rollbacks in case stuff hits the fan. obviously aligning with the founder is going to be critical.

i also suggest a book titled "what got you here won't get your there". you should treat every new role with an open outlook, there are always new challenges to sharpen your skills and experience, if everything's similar to what you did before, you're essentially not learning much.