Worked for AWS for just under a year and a half; this mirrors my experience exactly.
Poorly designed, failure-prone, brittle internal tooling was a time and energy sink to the point where even the most trivial deployment change was a nail-biter. Automated tooling and tests that were ostensibly created to make life easier were the number one pain point and constantly failed in obscure ways that required cutting tickets to teams responsible for the automation.
More often than not these tickets would be ignored and issues would persist for weeks beyond what's reasonable.
I'd actually force myself to write even trivial code on weekends to ensure my ability to develop didn't atrophy as a result of lack of use.
That, paired with the awful corporate culture and constant fear of PIP/job loss forced me to finally leave.
Generally a very unpleasant experience all around, sans the compensation and name on the resume.
>I'd actually force myself to write even trivial code on weekends to ensure my ability to develop didn't atrophy as a result of lack of use.
I did not mention this in my blog post, but I have actually done similar. That's incredible. Thanks for sharing this comment. Our tenures were almost the exact same length too.
When I worked for a BigTechCo, I was doing exercism every single morning for ~30 minutes for the same reason. It really is incredible. I know a few of my former colleagues were doing the same. Today I'm on a team with 4 other engineers, so I keep busy coding.
I thought that the deployment pipelines are part of their core business considering AWS has such products that are meant to be used by customers. I assume that the internal tools are different than what AWS customers use for deployments?
A lot of internal teams don't use the AWS software for various reasons. So like, CodePipeline might be great, but there's an internal analogue (I'm not sure the detail I can go into here) that is awful that a lot of teams use. There's been an internal movement to try to get all teams onto AWS services, but it's incredibly slow moving and there's no timeline that I know of.
The entire CodeBuild suite kind of sucks when you stack it up to options available on the market including GitHub Actions, GitLab Runners, Azure DevOps, literally almost anything.
If the internal analogue is worse than the CodeStar suite, then yikes.
Codestar is simpler than and has too many feature gaps compared to the internal offerings, which makes building and deploying Amazon scale multi-tier, multi-environment services difficult.
Oh man, we disagree! I miss Pipelines (the internal version of CodePipeline). Having worked at startups and now at Google, that tool is honestly best in class.
FWIW, I don't miss Quilt, or VersionSets, or Apollo, or Hydra, or TOD, or NAWS Pipelines bridge. But Pipelines itself, brining all of those tools together is amazing!
I was once pitched a startup by some Amazon vets whose elevator pitch was "Apollo for everyone". I literally laughed out loud, which was followed by some very awkward assurances from them that they were not, in fact, trying to be funny.
> Poorly designed, failure-prone, brittle internal tooling was a time and energy sink to the point where even the most trivial deployment change was a nail-biter.
Considering rather good quality APIs they expose to the external world, this is shocking to read.
Everyone says this, but in terms of discoverability of the actual apis I have found the opposite is true. Azure tends to have docs with small mistakes, but they are extremely thorough. With AWS it seems to be a tossup: You can be lucky and have complete and accurate information, or you can be fucked.
You should have just switched teams. Amazon has amazing teams working on deep technology, they also have huge systems with technical debt and stress. People dont realize how different it can be inside the company.
I've been on 4 different teams at Amazon (now migrating to a 5th). It depends entirely on the team and as well as your current level. Some do loops just like with external candidates. Others barely require more than a conversation with someone on the team -- though this one is a huge red flag. Teams looking for warm bodies are seldom good teams.
Most of the time its somewhere in between. You'll speak with the manager, speak with the team, you each review each other's artifacts, and if everyone's happy, you swap over. The balance of power is pretty nice. I've dropped out of the process many, many times after starting talking with management. Or even just learning more about what the team itself does (I couldn't live with myself if I worked on pre-roll ads, for example. So I noped out of that one).
Unless you’re in Focus or Pivot, you’re a free agent. When Amazon is performance managing you in any sense it isn’t up to your existing manager if you can move, it would be an L10 (VP) exception on the receiving side to allow you to move, even if the hiring manager thinks you’re great.
that can't be right (but i have no idea). In the new group, nobody has any desire to at least do a reference check with the previous group? That would be the first thing as a hiring manager i would try to check for, say "this person is a d-bag that the entire group hated, and we were about to pip him/her anyway"? That would be kind of an equivalent of an approval, even if there is no formal approval.
I mean, yes, having your manager say "I will destroy your career if you think about leaving" would be a kind of approval needed, but short of that level of viciousness it sounds like no?
Not familiar with Amazon culture so I don't know if that'd be considered acceptable behavior. I hope not!
> have you considered that the manager may rightly have bad things to say? or that is just never warranted.
The manager should put those bad things _in writing_ during performance reviews instead of trying to prevent their employees from leaving by dunking on them after they are informed about the fact.
Christ, I wish PIPs were that easy where I work. Some days it feels like the public sector trying to get rid of employees that coast for months at a time before we can work our way through the process to can them.
Find out what they drink for "Team alcohol evenings."
Whiskey team? Ok, probably somewhat stressful but doable. Wine team? Plush and cushy, with a line of people who want to be on that team out the door. Vodka team? Oh hell no. Etc.
There are so many data sources that are all queryable via API. There are a couple basic greasemonkey scripts already floating around but as an SDE it really isn't that hard to write something yourself... It does take a bit of experience to get a good heuristic on weighting the different data points but you can do way better than blindly guessing.
I will say as someone who HAS intelligently looked through this data for most teams at Amazon, there is an absolutely MASSIVE difference between the top 10% of teams and the median, and an even larger difference between than and the bottom 10%, which by all metrics appear to truly be hell on earth.
Amazon runs an annual survey with typically quite high participation rates, it is very detailed, and the results are often illuminating (and like many things at Amazon, widely accessible to all people).
They all look shining from the outside, problems start when you join the new team and realize the pile of st you will be dealing with.
Btw this happens in most companies, tons of tech debt. And those who created that st are off onto new projects recreating the exact same mess all over again.
It is a cycle that never ends.
Only chance is to join early and be in for the long run.
I've made a great career of being a code janitor and doing what others don't want, while having a good attitude. In the last few years, I just rewrite everything I possibly can in Rust and I've yet to regret it.
Most people don't know how to do this, because it's actually hard, and I don't think it's being taught much in programmer education.
I certainly had to learn "on the job", and probably spent 10 years before I got good at it.
I learned the most by fixing bugs. After a while, I started seeing the patterns of why this bug had occurred, and started writing code do avoid the traps.
Before I left Amazon I tried transferring to multiple teams after speaking with hiring managers. They all (IIRC 4 separate ones) wanted either a full on-site loop or at minimum two coding interviews).
I was an L5 with about 5-6 years of experience, 1.5 at that point at Amazon.
Hiring managers have a lot of latitude at Amazon, so there isn't a consistent standard. I don't want to share too much PII but I was in a similar situation. I was able to transfer teams after an informal chat with a few engineers on the team (discussing past projects, the new team's product vision, etc.). There was no whiteboarding involved.
The hiring managers are themselves managers of the team e.g. for a team of devs, a software development manager. In parent comment's case, the manager likely asked for opinions from the team after the chats before giving the final approval.
IME, the best managers are the ones who trust and enable their teams to make the right decisions. It would be a red flag if the manager was just giving out offers without allowing the team to provide their input.
Poorly designed, failure-prone, brittle internal tooling was a time and energy sink
Sometimes I get frustrated at my own job for some reason or another, but I appreciate learning that the grass isn't really greener anywhere else & even places where tech is the core business are having a lot of the same problems.
Completely agree. Interesting how the internal dev satisfaction polling always shows that the vast majority of devs just absolutely love the internal tooling and processes. Might have something to do with the polls not actually being anonymous. When hr/polling teams are asked about anonymity they generally skirt or ignore the question, but I've learned that enough metadata is collected to identify any responder in pretty much any internal poll.
There is very little transparency as to how the polling data is used. I'm sure many employees don't feel comfortable giving honest answers on many of the questions. Also the surveys go far beyond tooling question - they ask about future career plans, happiness with the company, happiness with management, whether employees are interviewing for other jobs, etc. It's not hard to see how certain answers to those questions could color the responder in a very bad light from managements perspective.
This is the general feeling of what I see from AWS support as well as a consumer of the platform.
I'm battling something for weeks now which is a completely broken ass piece of shit inside AWS. I spent the first 3 weeks trying to get attention for this via support tickets and someone to take it seriously, resulting in escalating to account management. I've been on calls with the programme managers, been apologised to constantly and absolutely nothing has been done that is productive. Not only that they sheepishly suggested other customers were in the same shit.
My conclusion at the end is that critical bits of AWS are held together with only a couple of people who actually know how it works and they don't even have the ability to do fix anything...
Not that the consuming company I work for isn't a complete mess either but I expected better from AWS.
I know the types of companies that are Microsoft shops come with their own problems, but my quality of life via tooling at two Microsoft shops has been great.
I think they mean places which use Visual Studio, (usually) C#, Windows Server, Azure, etc. Lots of components of this stack tend to work a bit better if you use all the others. Common in finance, because of Excel.
You get most of your RSU comp after two years; vesting is back-loaded. Many (most?) people wash out before they collect, which obviously works out well for Amazon.
Someone on another Amazon related thread mentioned that 50% leave after 2 years and 80% leave after 4.
If you do the math on their TC to see how it actually works out from month to month, it makes sense. Unless you get more RSU's, you take a huge hit in pay, unless you weren't cashing in on them to make ends meet.
I worked a contract there and the team barely understood how some of those configuration panes functioned. It was always a case of staring at someone else's config and then at yours until you found enlightenment. And the target kept changing, so there was a very bad week where my computer worked for 80 minutes total. I coded like hell during that time, pair programmed with the other contractors or did math on a whiteboard for the rest.
Another guy on the team intimated once in a while that he didn't like the team dynamics, but he played his cards pretty close to the chest. After he gave notice, on what we all thought was supposed to be his last day, we showed up in the morning just to find his computer and all of his equipment in a pile on a desk. We never did figure out if we or he were off by one or it was his middle finger to the team.
All the more mysterious because later that day we did discover his middle finger: The lead dev, who seemed to be one of the more reasonable people on the team, and the resident know-it-all both discovered that someone had fiddled with their configurations in ways that broke the system but produced obscure errors and hard to notice differences (like punctuation).
I took a couple lessons away from that project, and reinforced some that I'd already had. Perhaps the most important of which is: Bragging rights for being first mover are very, very short-lived. Once everyone else has copied it, the most accurate adjective for your version is 'old', or 'primitive', not 'first', and absolutely under no circumstances 'best'. Developers don't like bullshit. Don't pretend like your liabilities are assets.
Man, that's crazy. You can go to big boy jail for that, haha.
When I quit Amazon, they ran to my desk and unplugged my dev machine, since they thought I was going to leave a software "time bomb". I thought that was the stupidest concern in the world, but maybe it does actually happen.
It was maybe his second gig. A lot of us are still little monsters at that point. Not to say that’s okay, but just to say I wasn’t that surprised. What I was surprised by was that I didn’t think he had that kind of animosity. Pretty good at hiding his feelings.
I never understood that “shut them out” reaction for people giving notice. Getting laid off, maybe. But I pick the time and place where I hand over my 2 weeks. I have all day or even a few days to do it, but I have known for a month or three that this was coming, I’ve known for days when it was likely to come, and I’ve known for hours or even days that it was happening today.
You don’t think if I was going to do something shitty that I would have done it first? You guys are really underestimating my intelligence, or at least my ability to plan. But then that’s probably part of why I’m leaving.
Also they say that liars think everyone else is lying, thieves think everyone else is stealing. So I wonder what this says about the paranoid person? Are they a danger to the company?
I worked at Amazon from 1998 to 2003. Those were exactly my experiences. Clearly they don't care about the internal development experience and probably never will.
My Amazon experience was similar. Especially the lack of coding due to the emphasis on devops/meeting culture. I have since felt that once a product reaches maturity, additional headcount (butts in seats) are operators first and secondly a swarm for something like a broadly scoped breaking change, runtime upgrade or platform migration. There is plenty of fat to trim even in aws despite it being among the more secure jobs to have at the company.
I’ve seen work slow down across a few teams and it just leaves me in a place of boredom and complacency with a major technical itch. Despite this itch looking to be scratched, I cannot seem to find the time/energy to pursue coding regularly outside of work in the limited free time I have. Maybe if I were single, no kids, dogs…
No voluntary layoff plan you could take advantage of to get some severance pay? Might have been worth holding off a week to see if they rolled one of those out. Water under the bridge now, though.
Yup - I heard about this from a friend of mine. The offer seemed pretty good: lump sum 4 - 9 months base salary depending on tenure, extra for COBRA for 12 weeks, benefits through end of the year, etc.
Amazon's internal tooling really is the ninth circle of hell. It seems unsalvageable too; ostensibly the teams are supposed to be customer obsessed and the developers are the customer, but there's no accountability for anything to work well at all.
How was the work/life balance though? Did you find you were able to have downtime and rest or were you constantly worrying about unfinished work and looming deadlines?
I don't think I've ever heard a testimony of Amazon having good work life balance. The best you hear is that some people get lucky and things are "ok" on that front.
My work life balance at Amazon as an SDE is great but I don't have 24/7 oncall, make of that what you will.
That being said, I agree that the internal tooling at Amazon is not that great. It's actually a place where newer employees tend to clash with older Amazonian because it compares unfavourably with GitHub actions or any modern CI/CD while being significantly better than anything that existed in the early 2010s.
I've heard a lot of testimonies of SDEs maintaining pipelines that are making them miserable.
This is really baffling to me. In fact, since leaving Amazon in June, I've been really frustrated by how much extra work setting up a CI/CD pipeline is in (say) Drone and Argo/Flux than with Amazon's internal tooling. You can set up a standard Amazon pipeline with a single command and answering a few prompts about naming, whereas with the open-source systems you have to hand-craft everything yourself, hack in a way to update the infra repo every time the source repo changes (I blogged more about that here: https://blog.scubbo.org/posts/ci-cd-cd-oh-my/), and it seems like Drone literally doesn't have metrics that indicate broken builds. To say nothing of how well-integrated Pipelines is with alerting and ticketing - yet _another_ integration you would have to build for yourself in OSS-land.
Literally the only advantage I'm aware of for OSS systems is that they allow you to use whatever language, dependencies, build system, etc. that you want - which, yes, fair, if that's something that you need, then that's a deal-breaker, but for the 99.99...% of internal cases that Pipelines works for, it (seems to me to) work flawlessly and smoothly.
What are some unfavourable comparisons that I'm missing?
Perhaps it's just a green engineer thing. I know GitHub actions, CircleCI, Drone, etc... I don't know pipelines and sometimes it feels like dark magic to me.
As far as the other tooling goes, I often find that they are very Java-oriented (which is fine considering the AWS background) and everything in Python feels janky.
Ah! OK, now "dark magic" I can totally understand. Pipelines _is_ a little opaque, which is a totally fair criticism. In my experience it does _most_ of the things that you would want if you know a few fairly common (and easily searchable) incantations ("How do I add a new stage to this pipeline?", "How do I put tests after this stage?", etc.) - but I can definitely sympathise with the feeling of "this tool is doing a bunch of stuff for me and I don't understand how it's doing it". Thanks!
One point that I've found _really_ strange after transitioning from Pipelines to other CI/CD (I'm using Drone and Argo, but I think this applies for Flux as well) is that infrastructure-definition repos define the _image-version_ that they deploy to a particular stage, rather than just defining the source repo (and automatically deploying the latest image-version which has passed all preceding tests). That seems...odd. Unless you hack together your own automation[0], that means you need two actions to get new App Code out to an environment:
* Push the App Code commit (and have a new image-version built)
* Make a change to your Infra Code, updating the desired image version
Am I missing something?
---
I'm surprised to hear that you consider AWS to be Java-oriented. When I think AWS, I think TypeScript - to the extent that the primary reason I started learning TypeScript was to be able to write CDK more fluently (and, then, to write Lambdas without a whole bunch of Java type boilerplate - though Python also works there).
[0] Which I've done, but the fact that I had to hand-build it suggests I'm doing something that the tool author's discourage
I've actually really been of two minds about this since leaving AWS, though it's been a few years. On one hand, I remember spending an absurd amount of time debugging esoteric internal tooling errors that were un-googleable. The cases where you were lucky enough to find someone else who had the same issue on the internal stack-overflow or wiki were the good ones.
On the other hand, I've actually found myself missing features from amazon internal tools at the jobs I've had since. One example: I remember a pipelines feature where you could look at any change and quickly tell exactly how far that change had made it in your pipeline. At my current job, determining exactly when change X deployed to region Y has been a multi-step exercise in manually comparing commit hashes.
Right!? The observability on OSS CI/CD pipelines seems to be pretty lacking. Maybe I'll look for an opportunity there for my next position... :)
In fairness, I guess this is the tradeoff you get for being forced into "one and only one way to do it". The benefit is that all the tools are likely to play nice with one another "all the way through the flow", because they only have one representation format to be compatible with. The downside is, well, there's only one way to do it; and if that way happens to suck for you (as, for instance, for an ex-coworker who needed to build multiple different versions of the same code against different base OS images for...reasons), the tools are going to hinder more than they help.
Hi, recently ex-Amazonian here. It very much varies team-to-team. I was both careful and lucky enough to always be on teams where work-life balance was actively prioritized, with managers literally telling people to take more PTO, ensuring that stressful oncall experiences were a) compensated with unofficial time-off and b) taken as a prompt to invest in paying down tech debt to avoid repeat experiences. My experiences seem to have been better than the industry average.
But then I saw how some teams (predominantly, but not exclusively, based in Seattle) worked, and....yeah. Overall, not great.
same reasons here that contributed towards deciding leaving Amazon despite it being a dream job entering the industry (i did have a significant "pull factor" to another job that accelerated the timing). when you realize you wouldn't be happy even with the best case, it's time to go. However, i've heard many reports of talented, smart people doing fulfilling, challenging work at Amazon and its probably the luck of the draw on what team you get staffed on. So the case could be made that you couldve simply tried for a transfer.
i suspect that, if you work on a "slow and kludgy product", the responsible thing to do would have been to write a memo laying out why it sucks and what needs to be done. I've seen a rare few folks actually get heard and get the mandate to do what needed to be done. but I know it can feel overwhelming for most and ultimately its the responsibility of the GM/PM to know whether this is true or just the whining of an underling with no context.
also kudos for having the guts/integrity to do this before thanksgiving. make it count!
> 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc…
This gives the lie to Jeff Bezos' whole "Day 1" philosophy. If you're at a Day 1 company you don't spend 40% of your time fighting with the crappy internal tools.
Not sure what the Amzn situation is but something I've noticed at few previous gigs that there're some folks who claim they spent significant time wrestling with tooling but when you actually go in and make them formulate the issue it's either something really obvious or a 10-min google/github search like 9/10 times. These also weren't coming from people who I considered lazy or unintelligent (well, for the most part) so I can't really explain it logically. I suspect it may have something to do with motivation or cultural issues ("not my job" sort of folks).
I've seen people complain about it who were wrong, but I've also seen people fail to complain about it when they really should have been.
I just worked at a fang company, and here's an example - to search CI logs to see if a certain log happened frequently you would : WRITE A SCRIPT to download 100MB CI logs and search them for you.
Your example does not sound as outrageous as you seem to think - I can imagine a number of scenarios where this would be totally acceptable solution assuming we’re talking about software engineers here. Consider the fact that most startups don’t even have dedicated teams running CI unlike at fangs and it’s up to devs to set everything up. Yet I don’t really hear about folks complaining about internal tooling at early startups
The fact that you don't think that's ridiculous is a great example of terrible tooling be accepted as "normal."
Any startup can get this by piping their jenkins (etc) logs into ELK/Splunk/Sumo. And indeed this is exactly what I built at the last startup I worked for.
But again I was talking about a FAANG company not having a way to search logs (again, each individual log file is 100mb and there are 100k of them generated a day).
If it's not clear why that's entirely unacceptable, imagine your team is running the CI for over 10,000 engineers and you're landing changes to this CI system daily. Engineers are seeing all kinds of logs and bugs daily and you need to ascertain if these errors are new, unique to some subset of jobs, lead to failed jobs, etc.
If you're smaller you'd hopefully use something like Datadog, where you can easily ingest logs and have alert triggers from log lines. It's probably harder at bigger firms, where internal politics get in the way of having nice things.
When I worked at google you could =) j/k moma was terrible but most issues were easily searchable via a combination of email list/bug tracker/codesearch
Or you reach out to the owning teams/oncalls and half the time they have no idea how their own code/product works for anything nontrivial because most of it was written several generations of attrition ago.
Haha, I found this quite funny :) I'm sure there's an element of truth to it considering how complicated the code base must be for the various services
This mirrors my experience with my prestigious job I had at SiemensVDO in 2005/2006. Same stuff of wasting time on anything else than coding. He said 10% of his time was coding. I suspect mine was like 20% and I felt miserable after a year. The honeymoon of getting to write embedded code for real cars, cars that would be driven by millions of people only lasted like 3 to 4 months. After that the job had nothing else to teach me and over a little year later I quit too, with a total of one month shy of 1.5 years there.
Amazon has a lot of bad internal tools, but this person's experience doesn't match mine (being here for 8 years) at all
> 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc…
The tools for code submission, pull requests, pipelines, metrics, and logging are fantastic. Google is better. Most companies aren't.
I have never spent 40% of my time battling internal tools....
> 20% of my time in meetings
Developers complain when they're not invited to meetings, and they complain when they're invited. On my team we brutally introspect the value of every meeting, and if it looks like it's not delivering value, we find a new process.
> 20% of my time writing unit tests to hit the 100% coverage requirement of the codebase I worked on.
This makes no sense. This isn't a company mandate, every team is free to determine what code coverage percentage makes sense for them. Give this feedback to your tech lead, nearest Sr. SDE or PE -> 100% test coverage should never be "required"
> 10% of my time tracking down bugs in other team’s codebases for either internal tools or frameworks and trying to get them to acknowledge the problem by filing tickets.
>I have never spent 40% of my time battling internal tools....
In my experience, not at Amazon, long tenure employees get used to the quirky tools but the impact on new employees can be massive. Same with bad code bases, bad documentation and so on.
I really think this is true. Like compare the CR process to Github's PR process. To me, it's horrendous. But to others who are embedded, maybe they love it.
Yeah, comparatively, I find Github's process to be bizarre. Why do I have to fork a repo just to propose a change to it? Why does my "Pull Request" (really it's a Push Request, but it's called a Pull Request because of the 2 repo requirement) have to be associated with a remote repo in the first place?
(I wouldn't say I love the CR process - making multi-package CRs is definitely flawed, and I had a Sage question open with the Builder Team for nearly a year where they admitted as much - but I certainly prefer it to Github's)
But I'm pretty new to the outside world and really keen to understand alternative perspectives. What's good about Github's process to you?
You can do PRs from branches in the same repo on GitHub just fine if you want to. Having them in a separate repo makes more sense because 1) it doesn't require the dev submitting a PR to have anything above and beyond simple read access to the target repo, and 2) there's no concern that someone might depend on WIP branches created in those private forks, so they can be deleted or rebased at will.
The reason why it's called a "pull request" is because you're requesting the owner to pull your changes in; I don't see what this has to do with the number of repos in the picture.
> You can do PRs from branches in the same repo on GitHub just fine if you want to. Having them in a separate repo makes more sense because 1) it doesn't require the dev submitting a PR to have anything above and beyond simple read access to the target repo
Right, that's my point, though - why is there no distinction between "ability to create a real, actual, complete branch on a repo" and "ability to create a 'fake' branch that only exists for the purposes of diffs for a change request"?
For contrast, the flow I was used to inside Amazon was:
1. Clone the repo locally
2. Make my changes locally
3. Run a command that creates a 'fake branch' on the main upstream repo, which is used as the reference for the change. This works even if I don't have push-permissions on the repo (in which case, I can still push my change once it gets approval by clicking a button in the UI, whereupon a service account will push "on my behalf")
Whereas for Github, the process (if you don't have push-permissions) appears to be:
1. Fork the repo to my own Github account
2. Clone that locally (equivalent of 1. above)
3. Make my changes locally (equivalent of 2. above)
4. Push my changes to my own forked repo
5. Run a command (either CLI or UI) that creates a Pull Request from my repo to the target repo
Sure, it's only two extra steps - but I don't see why that friction has to exist in the first place. It gets _much_ worse if your change is open for a while (which will happen to coincide with the case when you don't have push permissions - that is, when you're contributing to code that you don't own), because then you need to resolve rebase locally and push to your fork rather than just being able to update in the UI (Github UI doesn't support rebase-pulls, only merges).
I see your point. FWIW I personally like it because it lets me move between my desktop and my laptop almost seamlessly - just push any changes on one end, and pull them on the other. It also means that, if any (or even all) devices suddenly die on me, whatever I was working on is still safely in my private repo.
But, yes, this does mean pushing things routinely a lot, not just when it's time to make a PR.
Fair enough! I only have a single development machine, so that advantage was invisible for me - but that makes sense!
You could theoretically get the benefits of both approaches, though: have your own personal repo to which you push "in-progress" commits for durability and portability, but maintain Amazon's tooling which generates PRs with a diff between a _local_ commit and the target (by, behind the scenes, generating the ephemeral fork from which to Request a Pull), and permitting updates to that PR from local (not necessarily "pushed to an online repo") commits. That's _still_ advantageous over GitHub's model, because:
* If you don't want to have a personal repo, you don't have to
* Even if you do, the process of updating a PR is simpler and more flexible when executed purely with local Git commands rather than by manipulating a remote repo
Fair enough! I thought I remembered it being a (likewise invisible) branch on the upstream repo, not a separate repo entirely - but I've been out for a while and might be misremembering. Regardless, as you say, it's immaterial - so long as the tooling makes that invisible, the UX is the same.
I don't think Github's PR process is that great. E.g. it misses the ability to diff 2 revisions of a CR if there is only one commit - which would happen if the next revision is a fixup and doesn't create a new commit. I usually used this to review changes incrementally. I find it rather sad that github doesn't have it and one needs to push multiple commits for the sake of making something incrementally reviewable.
Maybe so, but don't you think I talk to new employees? It's half of my job to support my whole team and deliver through others.
I battled those tools when I started. I watched them get better.
I've seen what new hires struggled with 5 years ago and what they struggle with 1 year ago.
Night and day.
The tools have gotten a lot better.
Here's the other ugly truth: That "40% of struggling with internal tools" may be saving the engineer 300% of time of having to implement the same from scratch themselves. Software engineering isn't all algorithms and data structures. A lot of it is just boilerplate code hooking up A to B. And better leave that boilerplate code to the internal tool that you have to figure out how to configure than implement it yourself.
> Maybe so, but don't you think I talk to new employees?
Everyone below a certain level talks to new employees. Do I believe you take their concerns seriously and actively try to help? Based on my own experience with PEs as well as your comment history I think you absolutely do not.
In general my experience with PEs at Amazon led me to conclude that the vast majority of them are:
- entitled
- lazy
- egotistical
- less technically useful than the average l5 engineer
> The tools for code submission, pull requests, pipelines, metrics, and logging are fantastic.
THANK YOU. I feel like I'm taking crazy pills whenever I see people bash Builder Tools - Pipelines in particular. Compared with what seems to be available in the Open Source world, it's fucking stupendous, and has silky-smooth integration.
> On my team we brutally introspect the value of every meeting, and if it looks like it's not delivering value, we find a new process
Oh yeah I love the multiple hours we have spend every week 'introspecting' processes, just to throw out one of the dozen we'd already defined and add another one. And this 'introspection' typically boils down to the loudest, most ambitious mouthbreathers forcing their BS down everyones throats so that they can jot down their amazing process contributions in their promo doc. Brutal is the right word.
But it's the difference between an individual carpenter making a rocking chair for himself and his family, or maybe making a couple to sell to his friends, and being a structural engineer.
Bikeshedding is not a necessary antipattern to the process, but large software projects absolutely need group collaboration, and a discussion of processes, tools, and best practices.
'Buidling things with code' is not just coding. It can include design, architecture, collaboration, planning etc. There are many high quality, highly complex, open source projects that don't rely on many of the 'processes', meetings, toxic competiveness, thought-policing, and beuracracy found in AWS.
Based on your comment history (which is consists of about 0% technically interesting topics, and about 100% pro-amazon 'tales from a super-senior principal engineer guyyyss') tells me you don't like engineering, you like politics and beuracracy. Which is okay!
Is this just me or aside from 40% spent on tooling (which I don't know what the deal is in this case but tbh I've seen some folks really spent their time inefficiently trying to make everything perfect) the breakdown looks completely normal for an established project at medium to big corp?
While it's not just you, and there are lots of folks who think that's just fine: if someone hires a dev and then make them spend only 10% doing the job they applied for, whoever hired them lied about the job, and the product they're hiring for has fundamental problems that you can either commit to solve, or burn bodies over in the hopes that you get transferred out to a different position and it's not your problem anymore. One of those may be good for business, but it's not the one that makes for a good company.
So 20% spent on tests and another 20% on meetings where presumably you decide what to build is not part of the job they applied for? Nonsense! If we’re doing hot takes here there’s no way everyone else on his team is spending 40% of their time on tooling without management doing anything about it
What a lovely thought, I do hope that's true, but everything I've seen about this story and people replying to it points to "no, actually, everyone else apparently spends 40% on fighting their internal tooling, too, and not a single manager seems to give a fuck". So there's that.
It's really difficult getting hundreds or thousands of people working together. Tooling, communication, etc become really critical. Otherwise, it's just chaos. As a result, you spend a lot less time "actually working" and a lot more time "collaborating" and "project managing". Spending less than 50% of your time on "actually working" is very normal in large companies
Was thinking that same. The 40% spent on tooling to deploy code and of course on call time is never fun, but the other time seems normal. But leaving only 10% of real development is not great and the author can probably do better. It is a good reminder that if you get to do greenfield development or generally enjoy your work, don't be quick to jump ship just for higher pay/prestige.
To be completely honest, what he describes seems to be what I've run into at nearly every stop I've been at as a consultant. I honestly never thought of it as hell, but I've also never been in a much better, tech tooling-friendly environment like at one of the FAAMNGs.
I suspect the people who end up leaving those environments for a more normal F500 company are going to be in for a shock in terms of work ergonomics and pay. This guy was lucky that he was able to squirrel away so much tech-worker money to be able to quit without a backup plan.
Why did you waste 10 years of your life there if it was that bad to you? Legit curious here. I can’t imagine spending more than a couple months in an awful situation.
Does anyone know what role or title the the author might have had there? They mentioned DevOps but then also in another paragraph they mentioned front-end development.
Yes I am well-aware of how SDEs work at Amazon. However there is a role at AWS called SDE - System Development Engineer which is very much DevOps which is very different than SRE and the reason I was asking.
My experience has been similar at other large companies (MSFT, Google). My advice is to look for founder-led teams (not companies). If there are plenty of founders still on the team, they will likely care a lot about the product and code and hold everyone to high standards which make the work easier in the long run. On the other hand when I've been on teams where all the original founders left, it's been a constant struggle to make anyone care about anything. At that point there is no ownership and most people are there for a paycheck. Lots of sloppy code gets shipped, everyone approves anything that looks like code, and eventually adding a single line anywhere feels like playing Russian roulette. Extreme short term thinking takes hold and no one cares that in 2-3 years the code will be unmaintainable because they don't plan to be on the team by then.
Yup, and the other flipside is devs being extremely nitpicky and regurgitating irrelevant Amazon principles BS in code reviews because they're trying to get promoted.
That one is my favorite. At Amazon, "bias for action" is a knee-jerk managerial phrase that typically means, "I'm going to spew ambiguous/ignorant bullshit into the room and you do something meaningful with it."
It was much better 5+ years ago. It's impossible to maintain high code standards when a company grows as fast as Google has. Eventually the law of large numbers applies. It's probably still better due to starting from a stronger base, but they are all converging.
There’s really a percent growth rate above which you just can’t build watchers fast enough to keep making sure the new people integrate. The more time the watchers spend trying to watch the less time they have to demonstrate the culture by their actions, and less time to reduce code rot personally.
So you either hope that others will be inspired to do better by your example, or you become a gatekeeper who loses respect because nobody likes to be lectured by someone who isn’t walking the walk.
I spent 10 years at Google (left in April) and I do think it's much better than how Amazon is described in this thread. Not perfect, but really still quite good.
Why better than Meta? My impression was that Meta have outstanding engineering (regardless of what else is happening with their products and company direction).
Meta has a more a hacker culture "move fast and break things". On the other hand, Google is well-known for its engineering culture (they wrote numerous articles and books about it).
>adding a single line anywhere feels like playing Russian roulette
I worked at a company where all 4 founders were still there (15 years in) and that was very much our reality. I think the founders still cared, but that didn't matter - beyond a certain size (>100 employees), the majority of the people in the org didn't care the same way.
Jamie Zawinski thought there were two kinds of people. Those who want to make a company successful, and those who want to work at a successful company. He was salty enough about what happened at Netscape Corp that he quit the industry and opened a bar.
I think you might have cause and effect mixed up a bit there. I’ve absolutely worked with people who left because they were unappreciated, and they were unappreciated because they wrote unmaintainable code and were defensive about it. Being a quitter and writing shitty code are very compatible personality traits.
To an extent then, if the original authors are still around, they were either appreciated or feared, but you probably won’t know which just from the interview. If they split it may be good riddance.
I do have a couple of mystery cases where I believe that someone who previously thought very highly of their own work had a change of heart, and realizing what they’ve done and how hard it would be to fix it, have decided it would be easier to start over someplace else. That may be true but I would recommend that it’s a character building exercise if you at least try to clean up your mess before leaving. There will be other messes, from other blind spots.
I completely understand where the author is coming from. I did literally the same thing. I am fortunate to be in a similar circumstance where I have the savings to weather the storm. The only difference is I was working for a highly dysfunctional manager and team. Towards the last days before I resigned, I was nearly on the brink of panic attacks from every meeting I had with my manager. I could see that my skills as a developer in doing actual, meaningful work had severely atrophied in a short period of time. To anyone that had to make difficult decisions like this, do what is best for you and your loved ones because a job shouldn't define you.
If you're thinking of joining big tech, try to join a team that develops an internal tool.
You have as much impact and are exposed to as much scale (maybe not in terms of TPS but in terms of data, etc.), but you usually have much lighter development processes and a much more frequent release cycle. Your ops will also be a lot lighter.
I'm an early career software engineer and am curious - why do people say working "at scale" like it's a good thing? Why is that desirable? What are you doing differently from someone who doesn't work "at scale"?
it just means they deal with some obvious and non-obvious problems having to do with being A Big Deal. (there's only a little bragging involved.) the most usual example is scaling up. say you have a website which you run on a server, and it's humming along serving users. everything's great. but then you get popular, and your one lil server can no longer keep up. So you swap that in for a bigger server and everything's great again. but then you get even more popular. and suddenly there don't exist bigger computers. so then you split up the programs on that one computer so that you can use two big ones and serve more people than you could before with one. and then three servers. and so on.
Someone who's dealt with websites at scale before is going to be able to skip the first 20 of that process, so if you're a startup dreaming to get big, you find someone that's worked "at scale" eg Amazon or Google to design and build your systems and avoid some pain points like the website going down because the system can't keep up.
If you look at tools in general as something that's intended to help other people solve some problem - and derive some enjoyment from that aspect of your job as a tool writer - tooling that operates "at scale" usually also helps more people.
Making software work for single use is relatively simple, making software that runs at 100% uptime, with large numbers of "users"/"tasks"/etc is an entirely different level of development and design.
The actual functionality is a very small part of software development.
Making it work within its constraints (memory/CPU/network/storage in the small scale, ability to handle peaks while maintaining uptime at a larger scale), as well as dealing with the "non-functionals" like AuthN/AuthZ, obervability, performance, etc is actually the "hard part".
Then add in business "non-functionals" like "minimize CapEx/OpEx", "meet external SLAs with measurable and reportable KPIs", "meet corporate and regulatory standards and requirements" etc.
All of that is much larger than the actual function being developed.
> This is not counting the weeks where I was “on call” and forced to drop all of this to work on a backlog of DevOps related issues.
I want to say sorry at the beginning if I misread what the author (you?) intended when you wrote this. And I'm coming at this from someone who is not a developer and who, compared to the lofty Software Engineer salaries in this industry, feels underpaid and underappreciated for doing the operations/administration work.
With that said: I truly wish those who are leaving their roles where this kind of requirement exists--feels "forced"--would speak more loudly about the need to have actual people doing system administration work. This is especially true, I feel, in large companies who operate "the Cloud" and who ought to know better; someone needs to be doing that work and it can't always, or usually, be the people who are writing the features and implementing the updates to the core product you are selling.
But what seems like has happened is everyone in the tech industry has forgotten it, or named it "Site Reliability Engineer" with a job of 55% failure analysis, 35% coding, and 10% "are the infrastructure and products actually online and functional." And then this role gets looked down on and paid less because the people in the role--people like me--are not seen as "delivering" "value".
Which culminates in the proper software developers seeing it as a thing they are forced to do, resulting in disdain, and furthering the cycle.
The SRE role comes from Google, where they are neither looked down on nor underpaid. Perhaps other companies have corrupted that title to mean something else.
How, pray tell, is someone utterly unfamiliar with a code base supposed to be able to deal with unforeseen issues in production for that service?
(Worse, the incentives for improvements to production quality, and thus on-call quality, are utterly mis-aligned.)
Ironically, I say this as someone presently on-call for a bunch of stuff for which I am utterly unfamiliar with the code base of, and have no time to become familiar with. It's going predictably badly.
So just page that person directly (the dev, for the purposes of this argument).
(And because I feel like this is bound to draw a strawman, steelman this: while there are definitely pages that might not get routed to BE eng in particular, assume we're routing pages to the person responsible for that system. I.e., in the face of infrastructural problems, those pages get routed to something like "infra eng", although IME there's very little that can be done with those in the middle of the night…)
This is the pat answer Amazon gives to defend this absurd practice, but it breaks down really easily.
>If your code breaks something, you should fix that code. Who else should?
What if it wasn't my code, but code written by someone 3 years ago who quit because most people only work at the company for 2 years? And it's in a part of the codebase I've never touched. That's a much more likely scenario.
The problem is that he has a big pile of half-working spaghetti code that he never has time to touch except when it malfunctions in the middle of the night
The problem behind that is that Amazon is a completely dysfunctional corporate hellscape. Like TFA said, you just don't have time or resources to actually fix things
You should join Amazon and do that, and you can come back here and apologize in a couple of years when you get pipped for wasting too much time on legacy code
Ideally incident handling should "just" be rolling back the broken change. Fixing the problem should be done in the morning with no time pressure, not in the middle of the night half asleep with customers on the other side of the world yelling at you. Of course it's not always that simple, but most of the time that's what on call should be about
It would be nice if things only broke during "business" hours and didn't have real world impact. Nevermind impact millions of people around the world. But if you look at the customers of say code that is running cloud infrastructure it is running airlines reservations/checkins, government workloads, banks, hospitals, critical infrastructure, netflix, gaming services. That's a lot of things that can't typically wait for morning.
You must have not tried god mode, the internal tool that flips the table on all of that SDE-0 bullshit. If you work at AMZN, search broadcast for "god mode hackathon" and start using it. If you are external, watch the hype videos here: https://photos.app.goo.gl/8enVGZLYFvpcgj2V9
He's a digital nomad who took off to Mexico (and Belize) and seems a bit lost in where he is going (he says it himself). I've been there myself (except I moved to Vietnam), so I understand it.
This is a lot deeper than just writing tests or 40% internal tooling issues. I also suspect that being remote, he feels like his hands are tied in a company that isn't used to working remotely. Or at least, if I was struggling with tooling, I'd work to find a way to make it better.
I also never complain about writing tests. I can't tell you how many times tests have saved my bacon or resulted in writing cleaner code.
it is when the focus is on having 100% coverage and not on test quality. When 100% becomes the metric it tends to get gamed pretty heavily with tests that have such a huge amount of mocks as to make the tests useless, or doing something to make it hit a branch, but ignoring actually verifying it cause that branch is just a log statement.
everybody on my team. Are you seriously implying I was put on some kind of special "extra testing required" probation system where the unit test coverage requirement was upped from 95% to 100%?
It sounds kind of absurd really. I've never once been asked to sign a contract that required me to do 100% code coverage. As others have noted in the thread, it is like some mythical number any way. It sounds to me like this wasn't going to end well since the expectations were wonky from the start.
It is absolutely possible to set the coverage threshold in Jest to 100% for all categories and they are set to that in the codebase I worked on. I would spend hours trying to that last 0.02% covered sometimes.
I still don't understand why that would cause testing to be hard, but since you won't explain it... I guess it leaves me thinking the problem might not be entirely them.
It's quite possible to get line coverage to 100%. It doesn't really mean anything because it might be covering those lines getting executed in one very particular order - and all bets are off if that order is different.
I work closely with the CR-time coverage/enforcement tool. The tool expects 70% new line coverage. The number was chosen arbitrarily, but individual teams/managers can choose to avoid the rule or set a different threshold.
It's not ridiculous to expect some, *configurable*, amount of test coverage for newly generated code, is it?
No, reasonable levels of testing are part of the job, it's when you get hard and fast rules from leadership like "don't lower coverage, reward increasing coverage" that it starts breaking down, we had that rule and someone gamed it using the methods I described up to 100% coverage.
Former Amazonian here. The learning is unmatched. I've been in different FAANGs but the way that Amazon pushes one to learn and challenge the status quo is unthinkable. I never saw "no, you can't do that, that system/business/area is sacred". Everything is up for grabs all the time. While there are pathological side effects to this (AWS promotion-per-launch, constantly battle for scope), the amount of knowledge one can gain in different areas from world-class people is pretty incredible. One of my systems at Amazon had more traffic in one day than my system at another FAANG had in 5 years. Working on a system that needs to support 5, 6, 7 MILLION TPs to perform complex operations was a great lesson for me.
Yet the internal tooling has been cited by several here as being one of the major time wasters and reasons they don't want to work there. It seems sacred to me...
I would say that 1) the experience varies by team 2) there's a lot of truth to that. But it's not a matter of being sacred, but the (sad) fact that Amazon being a ruthlessly data-driven business, investment on the tools and on reducing technical debt (real or perceived) is always prioritized down in OP1s.
I've been at AWS for almost 6 years now. It depends on what you mean by "internal tooling". But considering all the tooling I've used over the years at least 75% of the internal tooling that was in use when I joined is no longer in use as it has been replaced by newer, better versions of what came before.
Things definitely move at a slower pace than at new startups, but systems within AWS still change at a pretty fast pace for such a large organization. A fair number of employees don't stay past two years because of how compensation works, so they don't stay long enough to observe tooling changes, and are left with the assumption that tools are sacred. The reality is that it takes about two years between major rewritten versions of most tools. Then there is a leap forward, stasis for a while while feedback is gathered and the limits of the existing tooling are found, then in roughly two years there is another leap forward, etc. This churn requires work to keep up with, so some teams also fall behind if their product team prioritizes features over keeping up, so they may also be stuck on old versions of tooling for even longer.
So in summary I'd say yes in the short term some of the tools appear sacred. In the long term pretty much every internal tool or framework is discarded and replaced by newer versions on a fairly regular cadence.
Correct. If you're a PM on AWS and you launch a service, promotion chances will increase substantially. This is the main reason why the product portfolio is so bloated with services that are minor variations of other, existing services. Someone sees a minor use case there, PRFAQs it, voilà!
I hated the internal tooling so much that I switched to a manager role almost immediately and only went back to IC when I was sufficiently situated to not have to code anything that wasn't a flashy proof-of-concept.
373 comments
[ 0.18 ms ] story [ 275 ms ] threadPoorly designed, failure-prone, brittle internal tooling was a time and energy sink to the point where even the most trivial deployment change was a nail-biter. Automated tooling and tests that were ostensibly created to make life easier were the number one pain point and constantly failed in obscure ways that required cutting tickets to teams responsible for the automation.
More often than not these tickets would be ignored and issues would persist for weeks beyond what's reasonable.
I'd actually force myself to write even trivial code on weekends to ensure my ability to develop didn't atrophy as a result of lack of use.
That, paired with the awful corporate culture and constant fear of PIP/job loss forced me to finally leave.
Generally a very unpleasant experience all around, sans the compensation and name on the resume.
I did not mention this in my blog post, but I have actually done similar. That's incredible. Thanks for sharing this comment. Our tenures were almost the exact same length too.
a) How to make it stick
b) How to maximize its efficacy/efficiency
Would be awesome.
https://exercism.org/
I initially thought it was a misspelling of exorcism, and OP was making a joke.
The entire CodeBuild suite kind of sucks when you stack it up to options available on the market including GitHub Actions, GitLab Runners, Azure DevOps, literally almost anything.
If the internal analogue is worse than the CodeStar suite, then yikes.
FWIW, I don't miss Quilt, or VersionSets, or Apollo, or Hydra, or TOD, or NAWS Pipelines bridge. But Pipelines itself, brining all of those tools together is amazing!
Considering rather good quality APIs they expose to the external world, this is shocking to read.
Public facing products have competition. Internal tools have a captive audience that must use them, no matter what.
Most of the time its somewhere in between. You'll speak with the manager, speak with the team, you each review each other's artifacts, and if everyone's happy, you swap over. The balance of power is pretty nice. I've dropped out of the process many, many times after starting talking with management. Or even just learning more about what the team itself does (I couldn't live with myself if I worked on pre-roll ads, for example. So I noped out of that one).
Not familiar with Amazon culture so I don't know if that'd be considered acceptable behavior. I hope not!
The manager should put those bad things _in writing_ during performance reviews instead of trying to prevent their employees from leaving by dunking on them after they are informed about the fact.
And from what I understand at Amazon, it's kind of the same thing, but PIPs have been aggressively weaponized.
Whiskey team? Ok, probably somewhat stressful but doable. Wine team? Plush and cushy, with a line of people who want to be on that team out the door. Vodka team? Oh hell no. Etc.
... I'm kidding. Sort of. But not entirely.
There are so many data sources that are all queryable via API. There are a couple basic greasemonkey scripts already floating around but as an SDE it really isn't that hard to write something yourself... It does take a bit of experience to get a good heuristic on weighting the different data points but you can do way better than blindly guessing.
I will say as someone who HAS intelligently looked through this data for most teams at Amazon, there is an absolutely MASSIVE difference between the top 10% of teams and the median, and an even larger difference between than and the bottom 10%, which by all metrics appear to truly be hell on earth.
Tech survey results??
OP was making it sound like you could get the results for a specific team you'd want to transfer to.
Btw this happens in most companies, tons of tech debt. And those who created that st are off onto new projects recreating the exact same mess all over again.
It is a cycle that never ends.
Only chance is to join early and be in for the long run.
It's s lot of fun with the right attitude.
And hopefully no one else regrets it...
Thus is the circle of (engineering) life.
Just kidding. Please write maintainable code.
Not in the requirements, sorry.
Most people don't know how to do this, because it's actually hard, and I don't think it's being taught much in programmer education.
I certainly had to learn "on the job", and probably spent 10 years before I got good at it.
I learned the most by fixing bugs. After a while, I started seeing the patterns of why this bug had occurred, and started writing code do avoid the traps.
I prefer the term troubleshooter specialising in legacy code, but "code janitor" definitely has felt more appropriate at times.
I was an L5 with about 5-6 years of experience, 1.5 at that point at Amazon.
IME, the best managers are the ones who trust and enable their teams to make the right decisions. It would be a red flag if the manager was just giving out offers without allowing the team to provide their input.
Sometimes I get frustrated at my own job for some reason or another, but I appreciate learning that the grass isn't really greener anywhere else & even places where tech is the core business are having a lot of the same problems.
I figured this was pretty common. I've only had one job where I didn't feel the need to do this.
Well ... maybe?
N1 doesn’t really help, but it’s a start.
I'm battling something for weeks now which is a completely broken ass piece of shit inside AWS. I spent the first 3 weeks trying to get attention for this via support tickets and someone to take it seriously, resulting in escalating to account management. I've been on calls with the programme managers, been apologised to constantly and absolutely nothing has been done that is productive. Not only that they sheepishly suggested other customers were in the same shit.
My conclusion at the end is that critical bits of AWS are held together with only a couple of people who actually know how it works and they don't even have the ability to do fix anything...
Not that the consuming company I work for isn't a complete mess either but I expected better from AWS.
At least you have that.
My skills have atrophied and my pay is meh.
If you do the math on their TC to see how it actually works out from month to month, it makes sense. Unless you get more RSU's, you take a huge hit in pay, unless you weren't cashing in on them to make ends meet.
With the flat share price, there's no 4 year cliff anymore...
Another guy on the team intimated once in a while that he didn't like the team dynamics, but he played his cards pretty close to the chest. After he gave notice, on what we all thought was supposed to be his last day, we showed up in the morning just to find his computer and all of his equipment in a pile on a desk. We never did figure out if we or he were off by one or it was his middle finger to the team.
All the more mysterious because later that day we did discover his middle finger: The lead dev, who seemed to be one of the more reasonable people on the team, and the resident know-it-all both discovered that someone had fiddled with their configurations in ways that broke the system but produced obscure errors and hard to notice differences (like punctuation).
I took a couple lessons away from that project, and reinforced some that I'd already had. Perhaps the most important of which is: Bragging rights for being first mover are very, very short-lived. Once everyone else has copied it, the most accurate adjective for your version is 'old', or 'primitive', not 'first', and absolutely under no circumstances 'best'. Developers don't like bullshit. Don't pretend like your liabilities are assets.
When I quit Amazon, they ran to my desk and unplugged my dev machine, since they thought I was going to leave a software "time bomb". I thought that was the stupidest concern in the world, but maybe it does actually happen.
I never understood that “shut them out” reaction for people giving notice. Getting laid off, maybe. But I pick the time and place where I hand over my 2 weeks. I have all day or even a few days to do it, but I have known for a month or three that this was coming, I’ve known for days when it was likely to come, and I’ve known for hours or even days that it was happening today.
You don’t think if I was going to do something shitty that I would have done it first? You guys are really underestimating my intelligence, or at least my ability to plan. But then that’s probably part of why I’m leaving.
Also they say that liars think everyone else is lying, thieves think everyone else is stealing. So I wonder what this says about the paranoid person? Are they a danger to the company?
I am glad I am doing a master's in parallel because outside that I haven't written code in a while ...
I’ve seen work slow down across a few teams and it just leaves me in a place of boredom and complacency with a major technical itch. Despite this itch looking to be scratched, I cannot seem to find the time/energy to pursue coding regularly outside of work in the limited free time I have. Maybe if I were single, no kids, dogs…
That being said, I agree that the internal tooling at Amazon is not that great. It's actually a place where newer employees tend to clash with older Amazonian because it compares unfavourably with GitHub actions or any modern CI/CD while being significantly better than anything that existed in the early 2010s.
I've heard a lot of testimonies of SDEs maintaining pipelines that are making them miserable.
Literally the only advantage I'm aware of for OSS systems is that they allow you to use whatever language, dependencies, build system, etc. that you want - which, yes, fair, if that's something that you need, then that's a deal-breaker, but for the 99.99...% of internal cases that Pipelines works for, it (seems to me to) work flawlessly and smoothly.
What are some unfavourable comparisons that I'm missing?
As far as the other tooling goes, I often find that they are very Java-oriented (which is fine considering the AWS background) and everything in Python feels janky.
One point that I've found _really_ strange after transitioning from Pipelines to other CI/CD (I'm using Drone and Argo, but I think this applies for Flux as well) is that infrastructure-definition repos define the _image-version_ that they deploy to a particular stage, rather than just defining the source repo (and automatically deploying the latest image-version which has passed all preceding tests). That seems...odd. Unless you hack together your own automation[0], that means you need two actions to get new App Code out to an environment:
* Push the App Code commit (and have a new image-version built)
* Make a change to your Infra Code, updating the desired image version
Am I missing something?
---
I'm surprised to hear that you consider AWS to be Java-oriented. When I think AWS, I think TypeScript - to the extent that the primary reason I started learning TypeScript was to be able to write CDK more fluently (and, then, to write Lambdas without a whole bunch of Java type boilerplate - though Python also works there).
[0] Which I've done, but the fact that I had to hand-build it suggests I'm doing something that the tool author's discourage
On the other hand, I've actually found myself missing features from amazon internal tools at the jobs I've had since. One example: I remember a pipelines feature where you could look at any change and quickly tell exactly how far that change had made it in your pipeline. At my current job, determining exactly when change X deployed to region Y has been a multi-step exercise in manually comparing commit hashes.
In fairness, I guess this is the tradeoff you get for being forced into "one and only one way to do it". The benefit is that all the tools are likely to play nice with one another "all the way through the flow", because they only have one representation format to be compatible with. The downside is, well, there's only one way to do it; and if that way happens to suck for you (as, for instance, for an ex-coworker who needed to build multiple different versions of the same code against different base OS images for...reasons), the tools are going to hinder more than they help.
But then I saw how some teams (predominantly, but not exclusively, based in Seattle) worked, and....yeah. Overall, not great.
i suspect that, if you work on a "slow and kludgy product", the responsible thing to do would have been to write a memo laying out why it sucks and what needs to be done. I've seen a rare few folks actually get heard and get the mandate to do what needed to be done. but I know it can feel overwhelming for most and ultimately its the responsibility of the GM/PM to know whether this is true or just the whining of an underling with no context.
also kudos for having the guts/integrity to do this before thanksgiving. make it count!
This gives the lie to Jeff Bezos' whole "Day 1" philosophy. If you're at a Day 1 company you don't spend 40% of your time fighting with the crappy internal tools.
I just worked at a fang company, and here's an example - to search CI logs to see if a certain log happened frequently you would : WRITE A SCRIPT to download 100MB CI logs and search them for you.
Any startup can get this by piping their jenkins (etc) logs into ELK/Splunk/Sumo. And indeed this is exactly what I built at the last startup I worked for.
But again I was talking about a FAANG company not having a way to search logs (again, each individual log file is 100mb and there are 100k of them generated a day).
If it's not clear why that's entirely unacceptable, imagine your team is running the CI for over 10,000 engineers and you're landing changes to this CI system daily. Engineers are seeing all kinds of logs and bugs daily and you need to ascertain if these errors are new, unique to some subset of jobs, lead to failed jobs, etc.
You can also interpret it like - “always start from scratch. You don’t have anything”
> 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc…
The tools for code submission, pull requests, pipelines, metrics, and logging are fantastic. Google is better. Most companies aren't.
I have never spent 40% of my time battling internal tools....
> 20% of my time in meetings
Developers complain when they're not invited to meetings, and they complain when they're invited. On my team we brutally introspect the value of every meeting, and if it looks like it's not delivering value, we find a new process.
> 20% of my time writing unit tests to hit the 100% coverage requirement of the codebase I worked on.
This makes no sense. This isn't a company mandate, every team is free to determine what code coverage percentage makes sense for them. Give this feedback to your tech lead, nearest Sr. SDE or PE -> 100% test coverage should never be "required"
> 10% of my time tracking down bugs in other team’s codebases for either internal tools or frameworks and trying to get them to acknowledge the problem by filing tickets.
So, software engineering?
In my experience, not at Amazon, long tenure employees get used to the quirky tools but the impact on new employees can be massive. Same with bad code bases, bad documentation and so on.
(I wouldn't say I love the CR process - making multi-package CRs is definitely flawed, and I had a Sage question open with the Builder Team for nearly a year where they admitted as much - but I certainly prefer it to Github's)
But I'm pretty new to the outside world and really keen to understand alternative perspectives. What's good about Github's process to you?
The reason why it's called a "pull request" is because you're requesting the owner to pull your changes in; I don't see what this has to do with the number of repos in the picture.
Right, that's my point, though - why is there no distinction between "ability to create a real, actual, complete branch on a repo" and "ability to create a 'fake' branch that only exists for the purposes of diffs for a change request"?
For contrast, the flow I was used to inside Amazon was: 1. Clone the repo locally 2. Make my changes locally 3. Run a command that creates a 'fake branch' on the main upstream repo, which is used as the reference for the change. This works even if I don't have push-permissions on the repo (in which case, I can still push my change once it gets approval by clicking a button in the UI, whereupon a service account will push "on my behalf")
Whereas for Github, the process (if you don't have push-permissions) appears to be: 1. Fork the repo to my own Github account 2. Clone that locally (equivalent of 1. above) 3. Make my changes locally (equivalent of 2. above) 4. Push my changes to my own forked repo 5. Run a command (either CLI or UI) that creates a Pull Request from my repo to the target repo
Sure, it's only two extra steps - but I don't see why that friction has to exist in the first place. It gets _much_ worse if your change is open for a while (which will happen to coincide with the case when you don't have push permissions - that is, when you're contributing to code that you don't own), because then you need to resolve rebase locally and push to your fork rather than just being able to update in the UI (Github UI doesn't support rebase-pulls, only merges).
But, yes, this does mean pushing things routinely a lot, not just when it's time to make a PR.
You could theoretically get the benefits of both approaches, though: have your own personal repo to which you push "in-progress" commits for durability and portability, but maintain Amazon's tooling which generates PRs with a diff between a _local_ commit and the target (by, behind the scenes, generating the ephemeral fork from which to Request a Pull), and permitting updates to that PR from local (not necessarily "pushed to an online repo") commits. That's _still_ advantageous over GitHub's model, because:
* If you don't want to have a personal repo, you don't have to
* Even if you do, the process of updating a PR is simpler and more flexible when executed purely with local Git commands rather than by manipulating a remote repo
I appreciate the perspective, thank you!
I battled those tools when I started. I watched them get better.
I've seen what new hires struggled with 5 years ago and what they struggle with 1 year ago.
Night and day.
The tools have gotten a lot better.
Here's the other ugly truth: That "40% of struggling with internal tools" may be saving the engineer 300% of time of having to implement the same from scratch themselves. Software engineering isn't all algorithms and data structures. A lot of it is just boilerplate code hooking up A to B. And better leave that boilerplate code to the internal tool that you have to figure out how to configure than implement it yourself.
Everyone below a certain level talks to new employees. Do I believe you take their concerns seriously and actively try to help? Based on my own experience with PEs as well as your comment history I think you absolutely do not.
In general my experience with PEs at Amazon led me to conclude that the vast majority of them are:
- entitled
- lazy
- egotistical
- less technically useful than the average l5 engineer
THANK YOU. I feel like I'm taking crazy pills whenever I see people bash Builder Tools - Pipelines in particular. Compared with what seems to be available in the Open Source world, it's fucking stupendous, and has silky-smooth integration.
Oh yeah I love the multiple hours we have spend every week 'introspecting' processes, just to throw out one of the dozen we'd already defined and add another one. And this 'introspection' typically boils down to the loudest, most ambitious mouthbreathers forcing their BS down everyones throats so that they can jot down their amazing process contributions in their promo doc. Brutal is the right word.
Quit, and go make sourdough bread, shit.
But it's the difference between an individual carpenter making a rocking chair for himself and his family, or maybe making a couple to sell to his friends, and being a structural engineer.
Bikeshedding is not a necessary antipattern to the process, but large software projects absolutely need group collaboration, and a discussion of processes, tools, and best practices.
Based on your comment history (which is consists of about 0% technically interesting topics, and about 100% pro-amazon 'tales from a super-senior principal engineer guyyyss') tells me you don't like engineering, you like politics and beuracracy. Which is okay!
I suspect the people who end up leaving those environments for a more normal F500 company are going to be in for a shock in terms of work ergonomics and pay. This guy was lucky that he was able to squirrel away so much tech-worker money to be able to quit without a backup plan.
And good on you for doing it and great you don't have obligations to provide for others. What a freeing decision you made for your self!
No, no, this sounds pretty much like every team I was on for the nearly-decade I was there.
There is no central SRE teams, SysAdmins, etc. There is no throwing it over the fence.
There is nobody better to fix a production issue than the person who wrote the code which broke. Plus, you are more motivated to write better code.
I would expect Google to have better engineering discipline than Amazon or Meta.
So you either hope that others will be inspired to do better by your example, or you become a gatekeeper who loses respect because nobody likes to be lectured by someone who isn’t walking the walk.
olladecarne's story lines up very well with my own experience working on a post-startup-acquisition org at Google.
>adding a single line anywhere feels like playing Russian roulette
I worked at a company where all 4 founders were still there (15 years in) and that was very much our reality. I think the founders still cared, but that didn't matter - beyond a certain size (>100 employees), the majority of the people in the org didn't care the same way.
To an extent then, if the original authors are still around, they were either appreciated or feared, but you probably won’t know which just from the interview. If they split it may be good riddance.
I do have a couple of mystery cases where I believe that someone who previously thought very highly of their own work had a change of heart, and realizing what they’ve done and how hard it would be to fix it, have decided it would be easier to start over someplace else. That may be true but I would recommend that it’s a character building exercise if you at least try to clean up your mess before leaving. There will be other messes, from other blind spots.
You have as much impact and are exposed to as much scale (maybe not in terms of TPS but in terms of data, etc.), but you usually have much lighter development processes and a much more frequent release cycle. Your ops will also be a lot lighter.
I'm an early career software engineer and am curious - why do people say working "at scale" like it's a good thing? Why is that desirable? What are you doing differently from someone who doesn't work "at scale"?
Someone who's dealt with websites at scale before is going to be able to skip the first 20 of that process, so if you're a startup dreaming to get big, you find someone that's worked "at scale" eg Amazon or Google to design and build your systems and avoid some pain points like the website going down because the system can't keep up.
The actual functionality is a very small part of software development.
Making it work within its constraints (memory/CPU/network/storage in the small scale, ability to handle peaks while maintaining uptime at a larger scale), as well as dealing with the "non-functionals" like AuthN/AuthZ, obervability, performance, etc is actually the "hard part".
Then add in business "non-functionals" like "minimize CapEx/OpEx", "meet external SLAs with measurable and reportable KPIs", "meet corporate and regulatory standards and requirements" etc.
All of that is much larger than the actual function being developed.
I want to say sorry at the beginning if I misread what the author (you?) intended when you wrote this. And I'm coming at this from someone who is not a developer and who, compared to the lofty Software Engineer salaries in this industry, feels underpaid and underappreciated for doing the operations/administration work.
With that said: I truly wish those who are leaving their roles where this kind of requirement exists--feels "forced"--would speak more loudly about the need to have actual people doing system administration work. This is especially true, I feel, in large companies who operate "the Cloud" and who ought to know better; someone needs to be doing that work and it can't always, or usually, be the people who are writing the features and implementing the updates to the core product you are selling.
But what seems like has happened is everyone in the tech industry has forgotten it, or named it "Site Reliability Engineer" with a job of 55% failure analysis, 35% coding, and 10% "are the infrastructure and products actually online and functional." And then this role gets looked down on and paid less because the people in the role--people like me--are not seen as "delivering" "value".
Which culminates in the proper software developers seeing it as a thing they are forced to do, resulting in disdain, and furthering the cycle.
(Worse, the incentives for improvements to production quality, and thus on-call quality, are utterly mis-aligned.)
Ironically, I say this as someone presently on-call for a bunch of stuff for which I am utterly unfamiliar with the code base of, and have no time to become familiar with. It's going predictably badly.
(And because I feel like this is bound to draw a strawman, steelman this: while there are definitely pages that might not get routed to BE eng in particular, assume we're routing pages to the person responsible for that system. I.e., in the face of infrastructural problems, those pages get routed to something like "infra eng", although IME there's very little that can be done with those in the middle of the night…)
If your system/product/service is down because you have a dependency on something that broke -- well it's up to that team to fix their code.
>If your code breaks something, you should fix that code. Who else should?
What if it wasn't my code, but code written by someone 3 years ago who quit because most people only work at the company for 2 years? And it's in a part of the codebase I've never touched. That's a much more likely scenario.
The problem behind that is that Amazon is a completely dysfunctional corporate hellscape. Like TFA said, you just don't have time or resources to actually fix things
Totally normal. Not crazy at all. Take ownership.
He's a digital nomad who took off to Mexico (and Belize) and seems a bit lost in where he is going (he says it himself). I've been there myself (except I moved to Vietnam), so I understand it.
This is a lot deeper than just writing tests or 40% internal tooling issues. I also suspect that being remote, he feels like his hands are tied in a company that isn't used to working remotely. Or at least, if I was struggling with tooling, I'd work to find a way to make it better.
I also never complain about writing tests. I can't tell you how many times tests have saved my bacon or resulted in writing cleaner code.
Are you trolling?
It is absolutely possible to set the coverage threshold in Jest to 100% for all categories and they are set to that in the codebase I worked on. I would spend hours trying to that last 0.02% covered sometimes.
It does make me wonder why it would take you hours to do that though... was the code that complicated to test for some reason?
It's not ridiculous to expect some, *configurable*, amount of test coverage for newly generated code, is it?
Things definitely move at a slower pace than at new startups, but systems within AWS still change at a pretty fast pace for such a large organization. A fair number of employees don't stay past two years because of how compensation works, so they don't stay long enough to observe tooling changes, and are left with the assumption that tools are sacred. The reality is that it takes about two years between major rewritten versions of most tools. Then there is a leap forward, stasis for a while while feedback is gathered and the limits of the existing tooling are found, then in roughly two years there is another leap forward, etc. This churn requires work to keep up with, so some teams also fall behind if their product team prioritizes features over keeping up, so they may also be stuck on old versions of tooling for even longer.
So in summary I'd say yes in the short term some of the tools appear sacred. In the long term pretty much every internal tool or framework is discarded and replaced by newer versions on a fairly regular cadence.
Can you elaborate? You mean a new service gets someone promoted?