> Twitter does not have separate development, test, staging, and production environments.
It's legitimately difficult to imagine any software architecture running smoothly like this, much less a highly successful product with hundreds of engineers working on it...
It's necessarily criminal, but it seems everyone's suspicions about Twitter's codebase is being proven right in media res.
I'm suspicious of this one. It may be misinterpreting what's happening, but we need more details. Specifically, there's obviously some dev environment - people weren't literally editing files directly in production. Then, at some scale you may actually reject the idea of staging and instead concentrate on canary deployments, monitoring, and fast rollbacks.
I'm not saying the situation wasn't bad, but the quote doesn't tell us much without explaining how did it work instead. There's a lot of huge projects with only production environment. Probably more than people are comfortable thinking of.
With a service of this size creating a copy of prod is a non-trivial problem, and there are classes of bugs that will only manifest against prod data. A pristine, miniature pretend-prod just isn't that useful.
What I've seen work in the past is that you develop your changes locally, run a local instance of whatever service you're changing, but local instance is backed by stateful layers in prod.
Feature flags are ingrained in the culture, and you'll be expected to do things like gate your code to some small set of users at first. Employees are a good option, or test users created in prod if you're especially wary of your changes.
I only know of this leading to one SEV, where the feature flags broke in such a way that everything was suddenly enabled for everyone.
I hear Google is a fairly successful company so I think this is a limitation of your imagination. In your defense, I have argued this with very intelligent people who share the same limitation.
At some point, creating a second copy of prod is infeasible or becomes so expensive that one must revisit and discard assumptions about what is actually possible. Duplicating a small to medium set of micro services with a bunch of SaaS integrations isn’t typically a big deal. Much can be done by leveraging a cloud provider’s flexible capacity. At some point that stops working, e.g. when you are the cloud provider. AWS isn’t outsourcing reliability to GCP or vice versa. Other comments have already mentioned outages driven by differences between prod and staging/testing/etc — those issues only get worse at scale.
Flagging is the best option in this situation. I flagged your original post (and upvoted this one). That eventually makes it disappear along with the replies if there is not much traffic.
> Disengaged
CEO:
CEO
Jack Dorsey
had
recruited
Mudge
personally.
They
got
along
well,
and
Mudge
has
never
suspected Dorsey
of
harboring
bad
intent.
But
Dorsey, the
high-profile
CEO
of
one
of
the
most
prominent
companies
on
earth,
was
experiencing
a
drastic loss
of
focus
in
2021.
Dorsey
attended
meetings sporadically,
and
when
he
did,
he
was
extremely
disengaged.”
In
some
meetings
—even
after
he
was
briefed
on
complex
corporate
issues
—
Dorsey
did
not
speak
a
word.
Mudge
heard from
his
colleagues
that
Dorsey
would
remain
silent
for
days
or
weeks.
Worried
about
Dorsey's
health,
the senior
team
mostly
tried
to
cover
up
for
him,”
but
even
mid-
and
lower-level
staff
could
tell
that
the ship
was
rudderless
Worried about Dorsey's health, the senior team mostly tried to cover up for him
This seems like a failure of the corporate model in general. 'Everyone knows' the CEO/dictator/monarch is unable to function, but senior leaders cover it up (poorly), the board looks the other way, shareholders are deceived or at least kept ignorance, and the rabble (that's you) are confused at best and dismissed as crazy if they intuit the state of affairs correctly.
Why do we institutionalize such systems when the patterns of failure are so glaringly obvious and have been endlessly repeated?
14 comments
[ 3.8 ms ] story [ 48.0 ms ] threadEdit: Some people really failing to see the point here, which is that it's meaningless to suggest someone should be prosecuted without specifying why.
It's legitimately difficult to imagine any software architecture running smoothly like this, much less a highly successful product with hundreds of engineers working on it...
It's necessarily criminal, but it seems everyone's suspicions about Twitter's codebase is being proven right in media res.
I'm not saying the situation wasn't bad, but the quote doesn't tell us much without explaining how did it work instead. There's a lot of huge projects with only production environment. Probably more than people are comfortable thinking of.
What I've seen work in the past is that you develop your changes locally, run a local instance of whatever service you're changing, but local instance is backed by stateful layers in prod.
Feature flags are ingrained in the culture, and you'll be expected to do things like gate your code to some small set of users at first. Employees are a good option, or test users created in prod if you're especially wary of your changes.
I only know of this leading to one SEV, where the feature flags broke in such a way that everything was suddenly enabled for everyone.
At some point, creating a second copy of prod is infeasible or becomes so expensive that one must revisit and discard assumptions about what is actually possible. Duplicating a small to medium set of micro services with a bunch of SaaS integrations isn’t typically a big deal. Much can be done by leveraging a cloud provider’s flexible capacity. At some point that stops working, e.g. when you are the cloud provider. AWS isn’t outsourcing reliability to GCP or vice versa. Other comments have already mentioned outages driven by differences between prod and staging/testing/etc — those issues only get worse at scale.
You will of course end up with some bugs that can't (for mysterious reasons) be exactly replicated in each environment. That is not the same thing.
This seems like a failure of the corporate model in general. 'Everyone knows' the CEO/dictator/monarch is unable to function, but senior leaders cover it up (poorly), the board looks the other way, shareholders are deceived or at least kept ignorance, and the rabble (that's you) are confused at best and dismissed as crazy if they intuit the state of affairs correctly.
Why do we institutionalize such systems when the patterns of failure are so glaringly obvious and have been endlessly repeated?