"Since the database is used by everyone, it becomes cared for by no one. Startups don’t have the luxury of a DBA, and everything owned by no one is owned by infrastructure eventually"
I think adding a DBA or hiring one to help you layout your database should not be considered a
'luxury'...
Yeah I mean, hiring one person to own that for 5-10 teams is pretty cheap... Cheaper than each team constantly solving the same problems and relearning the same gotchas/operational stuff that doesn't add much value when writing your application code.
The kitchen sink database used by everybody is such a common problem, yet it is repeated over and over again. If you grow it becomes significant tech debt and a performance bottleneck.
Fortunately, with managed DBs like RDS it is really easy to run individual DB clusters per major app.
The downside is then you have many, many DBs to fight with, to monitor, to tune, etc.
This is rarely a problem when things are small, but as they grow, the bad schema decisions made by empowering DBA-less teams to run their own infra become glaringly obvious.
Not a downside to me. Each team maintains their own DB and pays for their own choices.
In the kitchen sink model all teams are tied together for performance and scalability, and some bad apple applications can ruin the party for everyone.
Seen this countless times doing due diligence on startups. The universal kitchen sink DB is almost always one of the major tech debt items.
I’m a DBRE, which means it’s somehow always my fault until proven otherwise. And even then, it’s usually on me to work around the insane schema dreamt up by the devs.
Multi-tenant DBs can work fine as long as every app has its own users, everyone goes through a connection pooler / load balancer, and every user has rate limits. You want to write shitty queries that time out? Not my problem. Your GraphQL BFF bullshit is trying to make 10,000 QPS? Nope, sorry, try again later.
EDIT: I say “not my problem,” but as mentioned, it inevitably becomes my problem. Because “just unblock them so the site is functional” is far more attractive to the C-Suite than “slow down velocity to ensure the dev teams are doing things right.”
Done that as well; it still becomes my problem because teams without RDBMS knowledge eventually break it, and… then I get paged.
Full Stack is a lie, and the sooner companies accept that and allow people to specialize again, and to pay for the extra headcount, the better off everyone will be.
I disagree I guess. Multiple companies I’ve worked at have broken up their shared db into many dbs that individual teams own the operations of, and it works just fine. At significant scale in traffic and # of eng. No central dbas needed - smaller databases require much less skills to manage. The teams that own them learn enough.
I don't think Jira and Confluence different markdown setup is due to them not sharing their databases. It is just poor product management from Attlasian.
But then the DB Team – if you have one – is responsible for 50 databases, each full of their own unique problems.
This will undoubtedly go over poorly, but honestly I think every data decision should be gated through the DB Team (again, if you have them). Your proposed schema isn’t normalized? Straight to jail. You don’t want to learn SQL? Also straight to jail. You want to use a UUIDv4 as a primary key? Believe it or not, jail.
The most performant and referentially sound app in the world, because of jail.
Serial integers always work better than any uuid as PKs, but the thing with uuid4 is that it disrupts any kind of index or physical ordering you decide to put on your data.
Uuids are really for external communication, not in-system organization.
Serial index forces a synchronisation point on every entity that can create records. If this is only ever a single database that’s fine but plenty of apps can’t scale this way.
Anything non-k-sortable in a B[+,-]tree will cause a ton of page splits. This is a more noticeable performance impact in RDBMS with a clustered index (MySQL's InnoDB, MS SQL Server) [0], but it also impacts Postgres [1] in multiple [2] ways.
No single team should not be responsible for all databases. If such team exists they will either become bottleneck for every other team (by auditing carefully each schema change) or become bloated and not utilized 90% of time, or (most common) they will become nearly useless or even harmful - they will not be really responsible and they will act as dumb proxy - they will introduce latency to the schema updates, but they will not bother to check them very well (why would they? they are not responsible for the whole product, just for the database), some DB refactoring/migrations will be totally abandoned because DB team will make them too painful.
DB team could act as an auditor and expert support, but they should never be fully responsible for DB layer.
> If such team exists they will either become bottleneck for every other team (by auditing carefully each schema change)
That’s the point. Would you send a backend code review to a frontend team? Why do DBs not deserve domain expertise, especially when the entire company depends on them?
> they are not responsible for the whole product, just for the database
I assure you, that’s a lot to be responsible for at scale.
> DB team could act as an auditor and expert support, but they should never be fully responsible for DB layer.
Again, the issue here is when the DB gets borked enough that a SME is required to fix it, they effectively do become responsible, because no CTO is going to accept, “sorry, we’ll be down for a couple of days because our team doesn’t really know how this thing works.”
And if your answer is, “AWS Premium Support,” they’ll just tell you to upsize the instance. Every time. That is not a long-term strategy.
It's because I hate databases and programming separately. I would rather slow code then have to dig into some database procdure. Its just another level of separation thats too mentally hard to manage. Its like... my queries go into a VM and now I have to worry about how the VM is performing.
I wish and maybe there is a programming language with first class database support. I mean really first class not just let me run queries but almost like embedded into the language in a primal way where I can both deal with my database programming fancyness and my general development together.
Sincerely someone who inherited a project from a DBA.
> I mean really first class not just let me run queries but almost like embedded into the language
Not quite embedded into the OS, but Django is a damn good ORM. I say that as a DBRE, and someone obsessed with performance (inherent issues with interpreted languages aside).
The closest thing to what you're describing is Prisma in Node. It generates a Typescript file from your schema so you get code completion on your data. And it exists somewhere between a query builder and a traditional ORM.
I have worked in many languages with many ORMs and this has been my personal favorite.
The language you’re talking about is APEX. I believe it comes from Oracle and is the backend language for Salesforce development. You’ll like the first class database support but that’s about it.
Lots of interesting comments on this one. Anyone have any good resources for learning how not to fuck up schema/db design for those of us who will probably never have a DBA on the team?
Good question. We don't have a DBA either. I've learned SQL as needed and while I'm not terrible, it's still daunting when making the schema for a new module that might require 10-20 tables or more.
One thing that has worked well for us is to alway include the top-most parent key in all child tables down yhe hierarchy. This way we can load all the data for say an order without joins/exists.
Oh and never use natural keys. Each time I thought finally I had a good use-case, it has bitten me in some way.
Apart from that we just try to think about the required data access and the queries needed. Main thing is that all queries should go against indexes in our case, so we make sure the schema supports that easily. Requires some educated guesses at times but mostly it's predictable IME.
Anyway would love to see a proper resource. We've made some mistakes but I'm sure there's more to learn.
LINQPad is awesome and EF Core is just this good so I can see why some would just choose not to deal with SQL.
With that said, this still sounds like a strange situation - most colleagues, acquaintances and people I consulted know they way around SQL and dropping down to 'dbset.FromSql($"SELECT {...' is very commonplace out of the need to use sprocs, views or have tighter control over the query.
Perhaps I undersold myself a little. By the time I got my first job I was fairly well versed in SQL querying, and these days I feel comfortable writing what I'd consider complex queries. That is with various window functions, nested queries, recursion (though I try to avoid that) etc, and I have a good handle on what the query optimizer likes and doesn't like.
But schema design is something else. I still take my time doing that.
Especially since our application is written with backwards compatibility in mind, so changing schema after it's deployed is something we try very hard to avoid.
But yeah, when hiring we require they are comfortable writing "normal" SQL queries (multiple joins, aggregation etc).
The moment you have two databases is the moment you need to deal with data consistency problems.
If you can't do something like determine if you can delete data, as the article mentions, you won't be able to produce an answer to how to deal with those problems.
Management problem masquerading as a tech problem.
Being shared between applications is literally what databases were invented to do. That’s why you learn a special dsl to query and update them instead of just doing it in the same language as your application.
The problem is that data is a shared resource. The database is where multiple groups in an organization come together to get something they all need. So it needs to be managed. It could be a dictator DBA or a set of rules designed in meetings and administered by ops, or whatever.
But imagine it was money. Different divisions produce and consume money just like data. Would anyone imagine suggesting either every team has their own bank account or total unfettered access to the corporate treasury? Of course not. You would make a system. Everyone would at least mildly hate it. That’s how databases should generally be managed once the company is any real size.
Why would you make it a shared resource if you don’t have to?
Decades of experience have shown us the massive costs of doing so - the crippled velocity and soul crushing agony of dba change control teams, the overhead salary of database priests, the arcane performance nightmares, the nuclear blast radius, the fundamental organizational counter-incentives of a shared resource .
Why on earth would we choose to pay those terrible prices in this day and age, when infrastructure is code, managed databases are everywhere and every team can have their own thing. You didn’t have a choice previously, now you do.
You wouldn’t but in any decent sized organization you will have to. If it is an organization that needs to exist there will be some common set of critical data.
In my experience, isolated (repeated) data storage paradigm is even more common at large organizations. They share data via services, ETLs, event buses, etc.
To me this is semantics. So it’s a data warehouse rather than a database. Ok. Or we share data from a common source via “services” - ok but that’s another word for a database and a client (using http to do the talking doesn’t really change anything).
I’m not saying literally every source of data has to be shared and centrally managed. I’m also not saying “rdbms accessed via traditional client and queried via sql” when I say database. I’m just saying a shared database of some shape is inevitable.
Ok, but the OP and the article are talking specifically about a directly shared rdbms scenario, not some nebulous concept of shared data.
Also, operationally it’s not “semantics” at all. You don’t get into (many) operational problems with analysts sharing a datawarehouse. You absolutely do with online apps sharing a rdbms, they aren’t the same thing.
No, neither the article nor the post I originally replied to specify anything more specific than “database”. Database != RDBMS.
A data warehouse is a type of database and is does need to be managed. Your assertion that it is easier to manage is orthogonal to my assertion that there will always be a central database to manage in an organization of decent size.
...I worked at a large software organization where larger teams had their own bank account, and there was a lot of internal billing, etc, mixed with plenty of funny-money to go along with it. That's not a contradiction, though, it perfectly illustrated your point for me.
I would love to see this type of thing from multiple sources. This reflects a lot of my own experience.
I think the format of this is great. I suppose it would take a motivated individual to go around and ask people to essentially fill out a form like this to get that.
One suggestion if we're gonna standardize around this format. Avoid the double negatives. In some cases author says "avoided XYZ" and then the judgment was "no regrets". Too many layers for me to parse there. Instead, I suggest each section being the product that was used. If you regret that product, in the details is where you mention the product you should have used. Or you have another section for product ABC and you provide the context by saying "we adopted ABC after we abandoned XYZ".
I don't recommend trying to categorize into general areas like logging, postmortems, etc. Just do a top-level section for each product.
> The markup cost of using RDS (or any managed database) is worth it.
Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a full-time in-house DBA.
Once you get past the point where the markup can pay for one or more full-time employees, I think you should consider doing that instead of blindly paying more and more to scale RDS up. You're REALLY paying for it with RDS. At least re-evaluate the choices you made as a fledgling startup once you reach the scale where you're paying AWS "full time engineer" amounts of money.
That's a huge instance with an enterprise license on top. Most large SaaS companies can run off of $5k / m or cheaper RDS deployments which isn't enough to pay someone. The amount of people running half a million a year RDS bills might not be that large. For most people RDS is worth it as soon as you have backup requirements and would have to implement them yourself.
Definitely--I recommend this after you've reached the point where you're writing huge checks to AWS. Maybe this is just assumed but I've never seen anyone else add that nuance to the "just use RDS" advice. It's always just "RDS is worth it" full stop, as in this article.
To some extend that is probably true, because when you’ve built a business that needs a 500k/year database fully on RDS it’s already priced into your profits, and switching to a self-hosted database will seem unacceptably risky for something that works just fine.
> Most large SaaS companies can run off of $5k / m or cheaper RDS
Hard disagree. An r6i.12xl Multi-AZ with 7500 IOPS / 500 GiB io1 books at $10K/month on its own. Add a read replica, even Single-AZ at a smaller size, and you’re half that again. And this is without the infra required to run a load balancer / connection pooler.
I don’t know what your definition of “large” is, but the described would be adequate at best at the ~100K QPS level.
RDS is expensive as hell, because they know most people don’t want to take the time to read docs and understand how to implement a solid backup strategy. That, and they’ve somehow convinced everyone that you don’t have to tune RDS.
If you're not using GP3 storage that provides 12K minimum IOPS without requiring provisioned IOPS for >400GB storage, as well as 4 volume striping, then you're overpaying.
If you don't have a reserved instance, then you're giving up potentially a 50% discount on on-demand pricing.
An r6i.12xl is a huge instance.
There are other equivalents in the range of instances available (and you can change them as required, with downtime).
For MySQL and Postgres, RDS stripes across four volumes once you hit 400 GiB. Doesn't matter the type.
The latency variation on gp3 is abysmal [0], and the average [1] isn't great either. It's probably fine if you have low demands, or if your working set fits into memory and you can risk the performance hit when you get an uncached query.
12K IOPS sounds nice until you add latency into it. If you have 2 msec latency, then (ignoring various other overheads, and kernel or EBS command merging) the maximum a single thread can accomplish in one second is (1000 msec / 1 sec / 2 msec) = 500 I/O. Depending on your needs that may be fine, of course.
> If you don't have a reserved instance, then you're giving up potentially a 50% discount on on-demand pricing.
True, of course. Large customers also don't pay retail.
> An r6i.12xl is a huge instance.
I mean, it goes well past that to .32xl, so I wouldn't say it's huge. I work with DBs with 1 TiB of RAM, and I'm positive there are people here who think those are toys. The original comment I replied to said, "large SaaS," and a .12xl, as I said, would be roughly adequate for ~100K QPS, assuming no absurdly bad queries.
>Most large SaaS companies can run off of $5k / m or cheaper RDS deployments which isn't enough to pay someone.
After initial setup, managing equivalent of $5k/m RDS is not full time job. If you add to this, that wages differ a lot around the world, $5k can take you very, very far in terms of paying someone.
You don't get the higher end machines on AWS unless you're a big guy. We have Epyc 9684X on-prem. Cannot match that at the price on AWS. That's just about making the choices. Most companies are not DB-primary.
I think most people who’ve never experienced native NVMe for a DB are also unaware of just how blindingly fast it is. Even io2 Block Express isn’t the same.
To be fair, most networked filesystems are nowhere near as good as EBS. That’s one AWS service that takes real work to replicate on-prem.
OTOH, as noted, EBS does not perform as well as native NVMe and is hilariously expensive if you try. And quite a few use cases are just fine on plain old NVMe.
Thats because EBS is a network block device and not a network filesystem - that would be EFS. And with network block devices you get the same perf and better compared to EBS.
Funny enough, the easiest way to experience this is probably to do some performance experimentation on the machine you code on. If it's a laptop made in the last few years, the performance you can get out of it knowing that it's sipping on a 45W power brick with probably not great cooling will make you very skeptical of when people talk about "scale".
RDS pricing is deranged at the scales I've seen too.
$60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare.
Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this? If its so fundamental to your product and needs such uptime & redundancy, what are the odds its also reasonably small?
> Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this?
Most software startups these days? The blog post is about work done at a startup after all. By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. If you’re a small startup, saving a couple hundred bucks a month by self managing your database is rarely a good choice. There’re more valuable things to work on.
>By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options.
By the time your db is big enough to cost an unreasonable amount on RDS, you've likely got so much momentum that getting off is nearly impossible as you bleed cash.
You can buy a used server and find colocation space and still be pennies on the dollar for even the smallest database. If you're doing more than prototyping, you're probably wasting money.
That’s just another way of saying the opportunity cost isn’t worth paying to do the migration.
Optionality and flexibility are extremely valuable, and that is why cloud compute continues to be popular, especially for rapidly/burstily growing businesses like startups.
I don't mean to pick on your specific comments, but I find these analysis almost always lack a crucial perspective: level of knowledge. This is the single biggest factor, and it's the hardest one to be honest about. No one wants to say "RDS is a good choice . . . because I don't know how nor have I ever self managed a database."
If you want a different opportunity cost, get people with different experience. If RDS is objectively expensive, objectively slow, but subjectively easy, change the subject.
> No one wants to say "RDS is a good choice . . . because I don't know how nor have I ever self managed a database."
I don't think that's accurate. I've self-managed databases, and I still think that RDS is compelling for small engineering teams.
There's a lot to get right when managing a database, and it's easy to screw something up. Perhaps none of the individual parts are super-complicated, but the cost of failure is high. Outsourcing that cost to AWS is pretty compelling.
At a certain team size, you'll end up with a section of the team that's dedicated to these sorts of careful processes. But the first place these issues come up is with the database, and if you can put off that bit of organizational scaling until later, then that's a great path to choose.
I disagree here. This falls apart when you zoom out one step. I'm perfectly capable of managing a database. I'm also capable of maintaining load balancers, redis, container orchestrators, Jenkins, perforce, grafana, Loki, Oncall, individually. But each of those has the high chance of being a distraction from what our software actually does.
Its about tradeoffs, and some tradeoffs are often more applicable than others - getting a ping at 7am on a Sunday because your ec2 instance filled it's drive up with logs and your log rotation script failed because it didn't have a long enough retey is a problem I'm happy to outsource when I should be focusing on the actual app.
Lack of expertise in some particular technology is simply another opportunity cost. I can learn how to operate a production DB at scale (I have racked servers and run other production workloads) but as cofounder/CTO in a startup is that the best use of my time?
If the cost of a hosted DB is going to sink the company, then of course, I will figure it out and run it myself. But it’s not, for most startups. And therefore that knowledge isn’t providing much leverage.
Starting an AI company with deep expertise in training models - that is an example of knowledge providing huge leverage. DB tech is not in this bucket for most businesses.
People do not really understand the value of the former. Even dealing with financial options (buy/sell and underlying) which are a pure form of it, people either do not understand the value, or do so in a very abstract way they do not intuit.
Good point. And, since you brought up financials, you also see this when people use a majority of their savings to lump sum pay off a mortgage. They take an overweighted view of saving on interest and, IMO, underweight the flexibility of liquidity.
In the small SaaS startup case, I’d say the production database is typically the most critical single piece of infra, so self hosting is just not a compelling proposition unless you have a strong technical reason where having super powerful database hardware is important, or a team with multiple people who have sysadmin or DBA experience. I think both of those cases are unusual.
I’ve been the guy managing a critical self-hosted database in a small team, and it’s such a distraction from focusing on the actual core product.
To me, the cost of RDS covers tons of risks and time sinks: having to document the db server setup so I’m not the only one on the team who actually knows how to operate it, setting up monitoring, foolproof backups so I don’t need to worry that they’re silently failing because a volume is full and I misconfigured the monitoring, PITR for when someone ships a bad migration, one click HA so the database itself is very unlikely to wake me at 3am, blue/green deploys to make major version upgrades totally painless, never having to think about hardware failures or borked dist-upgrades, and so on.
Each of those is ultimately either undifferentiated work to develop in-house RDS features that could have been better spent on product, or a risk of significant data loss, downtime, or firefighting. RDS looks like a pretty good deal, up to a point.
I like fiddling with databases, but I totally agree with this. Unless you really need a big database and are going to save 100k+ per year by going self managed then RDS or similar just saves you so much stress. We've been using it for the best part of 10 years and uptime and latency have consistently been excellent, and functionality is all rock solid. I never have to think about it, which is just what I want from something so core to the business.
I am good at databases (have been a DBA in the past), and 100% agree with this. RDS is easy to standup and get all the things you mentioned, and not have to think about again. If we grow to the point where the overhead is more than a FT DBA, awesome. It means we are successful, and are fortunate to have options.
Unfortunately there are so many people and teams who thinks that simply running their databases on RDS means that they're backed up, highly-available and can be easily load balanced, upgraded, partitioned, migrated and so on which is simply not the case with the basic configuration.
RDS is a great choice, for prototyping and only for production if you know what you're doing when setting it up.
FWIW, this is common in all cloud deployments, people assume that running something "severless" is a magical silver bullet.
Well…just using the defaults when creating an RDS Postgres in the console give you an HA cluster with two read replicas, 7 days of backups restorable to any point in time, automatic minor version upgrades, and very easy major upgrades. So unless you start actively unchecking stuff those are not entirely invalid assumptions.
I agree, but I also classify some of these as "learn them once and you're all set".
Maybe it takes you a month the first time around and a week the 10th time around. First product suffers, the other products not so much. Now it just takes a week of your time and does not require you to pay large AWS fees, which means you are not bleeding money
I like to set up scrappy products that do not rack up large monthly fees. This means I can let them run unprofitable for longer and I don't have to seek an investor early, which would light up a large fire under everyone's butts and start influencing timelines because now they have the money and want a return asap.
I'll launch a week later - no biggie usually. I could have come up with the idea a month later, so I'm still 3 weeks early ;)
It doesn't work for all projects, obviously, but I've seen plenty of SaaS start out with a shopping spree, then pay monthly fees and purchase licenses for stuff that they could have set up for free if they put some (usually not a lot) effort into it. When times get rough, the shorter runway becomes a hard fact of life. Maybe they wouldn't have needed a VC and could have bootstrapped and also survived for longer.
Learning it all is what gave me an appreciation for RDS! I’ve self managed a number of Postgres and MySQL databases, including a 10TB Postgres cluster with all of the HA and backup niceties.
While I generally agree as far as initial setup time goes, I favor RDS because I can forget about it, whereas the hand rolled version demands ongoing maintenance, and incurs a nonzero chance of simple mistakes that, if made, could result in a 100% dataloss unrecoverable scenario.
I’m also mostly talking about typical, funded startups here, as opposed to indie/solo devs. If you’re flying solo launching a tiny proof of concept that may only ever have a few users, by all means run it yourself if you’d like, but if you’ve raised money to grow faster and are paying employees to iterate rapidly searching for PMF…just pay for RDS and make sure as much time as possible is spent on product features that provide actual business value. It starts at like $15/month. The cost of simply not being laser-focused on product is far greater.
> you've likely got so much momentum that getting off is nearly impossible as you bleed cash.
Databases are not particularly difficult to migrate between machines. Of all the cloud services to migrate, they might actually be the easiest, since the databases don't have different API's that need to be rewritten for, and database replication is a well-established thing.
Getting off is quite the opposite of nearly impossible.
Lots of cases. It doesn't even have to be a tiny database. Within <1TB range there's a huge number of online companies that don't need to do more than hundreds of queries per second, but need the reliability and quick failover that RDS gives them. The $600k cost is absurd indeed, but it's not the range of what those companies spend.
Also, Aurora gives you the block level cluster that you can't deploy on your own - it's way easier to work with than the usual replication.
Once you commit to more deeply Amazon flavored parts of AWS like Aurora, aren't you now fairly committed to hoping your scale never exceeds the cost-benefit tradeoff?
Or you're realistic about what you're doing. Will you ever need to scale more than 10x? And on the timescales where you do grow over 10x, would it be better to reconsider/re-architect everything anyway?
I mean, I'm looking after a 4 instance Aurora cluster which is great feature wise, is slightly overprovisioned for special events, and is more likely to shrink than grow 2x in the next decade. If we start experiencing any issues, there's lots of optimisations that can be still gained from better caching and that work will be cheaper than the instance size upgrade.
There’s still a defined cost to swapping your DB code over to a different backend. At the point where it becomes uneconomical, you’re also at a scale you can afford rewriting a module.
That’s why we have things like “hexagonal architecture”, which focus on isolating the storage protocol from the code. There’s an art to designing such that your prototype can scale with only minor rework — but that’s why we have senior engineers.
Sure, but if you're paying anywhere near list price for your on-prem hardware at scale you're also doing it wrong. I've never seen a scenario where Amazon discounts exceed what you would get from a hardware or software vendor at the same scale.
It's more interesting how cloud services are sold like any other consumables or corporate services.
No one runs their own electricity supply (well until recently with renewables/storage), they buy it as a service, up to a pretty high scale before it becomes more economic to invest the capex and opex to run your own.
If my scale exceeds the cost benefit tradeoff, then I will thank God/Allah/Buddah/Spaghetti Monster.
These questions always sound flawed to me. It's like asking won't I regret moving to California and paying high taxes once I start making millions of dollars? Maybe? But that's an amazing problem to have and one that I may be much better equipped to solve.
If you are small, RDS is much cheaper, and many company killing events, such as not testing your backups are solved. If you are big and you can afford a 60K/yr RDS bill than you can make changes to move on-prem. Or you can open up excel and do the math if your margins are meaningfully affected by moving on-prem.
I assume that you do that math on all your new features too, right? The calculation of how much extra money they will bring in?
On some level, AWS/GCP/California relies on you doing this calculation for the things that you can do it on easily (the savings of moving away), while not doing this calculation on things where it's hard to do (new development). That way, you can pretend that your new features are a lot more valuable than the $Xk/year you will save by moving your infra.
>The calculation of how much extra money they will bring in?
Yes, I've done the math. The piece you are missing is, saving money on infra will bring in $0 new dollars. There is a floor to how much money I can save. There is no ceiling to how much money the right feature can bring in. Penny pinching on infra, especially when the amount of money is saved is less than the cost of an engineer is almost always a waste of time while you are growing a company. If you are at the point where you are wasting 1x,2x,3x of an engineers salary of superflous infrastructure - then congratulations you have survived the great filter for 99% of startups.
>That way, you can pretend that your new features are a lot more valuable than the $Xk/year you will save by moving your infra.
Finding product market fit is 1000x harder than moving from RDS to On-prem. If you haven't solved PMF, then no amount of $Xk/year in savings will save you from having to shut down your company.
I am well aware of the math on that. Also, switching to faster infra can be a surprising benefit to your revenue, by the way, if it makes your app feel nicer.
The thing is, most features, particularly later in the life of a company, don't have an easy-to-measure revenue impact, and I suspect that many features are actually worth $0 of revenue. However, they cost money to implement (both in engineering time and infra), making them very much net negative value propositions. This is why Facebook and Google can cut tons of staff and lose nothing off their revenue number.
Also, there's a bit of a gambling mentality here which is that a feature could be worth effectively infinite revenue (ie it could be the thing that gives you PMF), so it's always worth doing over things with known, bounded impact on your bottom line. However, improving your efficiency gives you more cracks at finding good features before you run out of money.
Agree. "What if you're wildly successful and get huge?" Awesome, we'll solve the problem then. The other part is what if AWS was a part of becoming successful? IE, it freed my small team from having to worry all that much about a database and instead focused on features.
So moving to/from Aurora/RDS/own EC2/on-prem should be a matter of networking and changing connection strings in the clients.
Your operational requirements and processes (backup/restore, failover, DR etc) will change, but that's because you're making a deliberate decision weighing up those costs vs benefits.
You can use DNS to mitigate the pain of changing those connection strings, decoupling client change management from backend change process, or if you had foresight, not having to change client connection strings at all.
Nope, nope, nope! When you change DNS entries, they will take effect at some point in the future when the cache expires and when your app decides to reconnect. (Possibly after a restart) At that point, why not be sure and change the config?
I mean, DNS change can work, but when you're doing that one-in-years change, why risk the extra failure modes.
I have a small MySQL database that’s rather important, and RDS was a complete failure.
It would have cost a negligible amount. But the sheer amount of time I wasted before I gave up was honestly quite surprising. Let’s see:
- I wanted one simple extension. I could have compromised on this, but getting it to work on RDS was a nonstarter.
- I wanted RDS to _import the data_. Nope, RDS isn’t “SUPER,” so it rejects a bunch of stuff that mysqldump emits. Hacking around it with sed was not confidence-inspiring.
- The database uses GTIDs and needed to maintain replication to a non-AWS system. RDS nominally supports GTID, but the documented way to enable it at import time strongly suggests that whoever wrote the docs doesn’t actually understand the purpose of GTID, and it wasn’t clear that RDS could do it right. At least Azure’s docs suggested that I could have written code to target some strange APIs to program the thing correctly.
Time wasted: a surprising number of hours. I’d rather give someone a bit of money to manage the thing, but it’s still on a combination of plain cloud servers and bare metal. Oh well.
> Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this?
Very small businesses with phone apps or web apps are often using it. There are cheaper options of course, but when there is no "prem" and there are 1-5 employees then it doesn't make much sense to hire for infra. You outsource all digital work to an agency who sets you up a cloud account so you have ownership, but they do all software dev and infra work.
> If its so fundamental to your product and needs such uptime & redundancy, what are the odds its also reasonably small?
Small businesses again, some of my clients could probably run off a Pentium 4 from 2008, but due to nature of the org and agency engagement it often needs to live in the cloud somewhere.
I am constantly beating the drum to reduce costs and use as little infra as needed though, so in a sense I agree, but the engagement is what it is.
Additionally, everyone wants to believe they will need to hyperscale, so even medium scale businesses over-provision and some agencies are happen to do that for them as they profit off the margin.
A lot of my clients are small businesses in that range or bigger.
AWS and the like are rarely a cost effective option, but it is something a lot of agencies like, largely because they are not paying the bills. The clients do not usually care because they are comfortable with a known brand and the costs are a small proportion of the overall costs.
A real small business will be fine just using a VPS provider or a rented server. This solves the problem of not having on premise hardware. They can then run everything on a single server, which is a lot simpler to set up, and a lot simpler to secure. That means the cost of paying someone to run it is a lot lower too as they are needed only occasionally.
They rarely need very resilient systems as they amount of money lost to downtime is relatively small - so even on AWS they are not going to be running in multiple availability zones etc.
RDS is not so bulletproof as advertised, and the support is first arrogant then (maybe) helpful.
People pay for RDS because they want to believe in a fairy tale that it will keep potential problems away and that it worked well for other customers. But those mythical other customers also paid based on such belief. Plus, no one wants to admit that they pay money in such irrational way.
It's a bubble
> $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare.
One of these is not like the others (DBAs are not capex.)
Have you ever considered that if a company can get the same result for the same price ($100K opex for RDS vs same for human DBA), it actually makes much more sense to go the route that takes the human out of the loop?
The human shows up hungover,
goes crazy, gropes Stacy from HR, etc.
Not only that, you can't just have one DBA. You need a team a them, otherwise that person is going to be on call 24/7, can never take a vacation, etc. Your probably looking at a minimum of 3.
You'll need an engineer with database skills, not a dedicated DBA. I haven't seen a small company with a full time DBA in well over a decade. If you can learn a programming language, you can learn about indexes and basic tuning parameters (buffer pool, cache, etc.)
Discount rates are actually much better too on the bigger instances. Therefore the "sticker price" that people compare on the public site is no where close to a fair comparison.
We technically aren't supposed to talk about pricing publically, but I'm just going to say that we run a few 8XL and 12Xl RDS instances and we pay ~40% off the sticker price.
If you switch to Aurora engine the pricing is absurdly complex (its basically impossible to determine without a simulation calculator) but AWS is even more aggressive with discounting on Aurora, not to mention there are some legit amazing feature benefits by switching.
I'm still in agreeance that you could do it cheaper yourself at a Data Center. But there are some serious tradeoffs made by doing it that way. One is complexity and it certainly requires several new hiring decisions. Those have their own tangible costs, but there are a huge amount of intangible costs as well like pure inconvenience, more people management, more hiring, split expertise, complexity to network systems, reduce elasticity of decisions, longer commitments, etc.. It's harder to put a price on that.
When you account for the discounts at this scale, I think the cost gap between the two solutions is much smaller and these inconveniences and complexities by rolling it yourself are sometimes worth bridging that smaller gap in cost in order to gain those efficiencies.
> but I'm just going to say that we run a few 8XL and 12Xl RDS instances and we pay ~40% off the sticker price.
Genuinely curious, how do you that?
We pay a couple of million dollars per year and the biggest spend is RDS. The bulk of those are 8xl and 12xl as you mention and we have a lot of these. We do have savings plans, but those are nowhere near 40%.
Yeah 40% seems like a pipedream. I was at a Fortune 500 defense firm and we couldn't get any cloud provider to even offer us anything close to that discount if we agreed to move to them for 3-4 years minimum. That org ended up not migrating because it was significantly cheaper to buy land and build datacenters from scratch than to rent in the cloud.
Defense firms do a lot more than just government work. Also, there are definitely discounts in govcloud when Fortune 500 companies that operate 30+ datacenters start talking to govcloud providers about potentially migrating to their services.
Some orgs are looking at moving back to on prem because they're figuring this out. For a while it was vogue to go from capex to opex costs, and C suite people were incentivized to do that via comp structures, hence "digital transformation" ie: migration to public cloud infrastructure. Now, those same orgs are realizing that renting computers actually costs more than owning them, when you're utilizing them to a significant degree.
Same experience here. As a small organization, the quotes we got from cloud providers have always been prohibitively expensive compared to running things locally, even when we accounted for geographical redundancy, generous labor costs, etc. Plus, we get to keep know how and avoid lock-in, which are extremely important things in the long term.
I am sure for some use-cases cloud services might be worth it, especially if you are a large organization and you get huge discounts. But I see lots of business types blindly advocating for clouds, without understanding costs and technical tradeoffs. Fortunately, the trend seems to be plateauing. I see an increasing demand for people with HPC, DB administration, and sysadmin skills.
> Plus, we get to keep know how and avoid lock-in, which are extremely important things in the long term.
So much this. The "keep know how" has been so greatly avoided over the past 10 years, I hope people with these skills start getting paid more as more companies realize the cost difference.
When I started working in the 1980s (as a teenager but getting paid) there was a sort of battle between the (genuinely cool and impressive) closed technology of IBM and the open world of open standards/interop like TCP/IP and Unix, SMTP, PCs, even Novell sort of, etc. There was a species of expert that knew the whole product offering of IBM, all the model numbers and recommended solution packages and so on. And the technology was good - I had an opportunity to program a 3093K(?) CM/VMS monster with APL and rexx and so on. Later on I had a job working with AS/400 and SNADS and token ring and all that, and it was interesting; thing is they couldn't keep up and the more open, less greedy, hobbyists and experts working on Linux and NFS and DNS etc. completely won the field. For decades, open source, open standards, and interoperability dominated and one could pick the best thing for each part of the technology stack, and be pretty sure that the resultant systems would be good. Now however, the Amazon cloud stacks are like IBM in the 1980s - amazingly high quality, but not open; the cloud architects master the arcane set of product offerings and can design a bespoke AWS "solution" to any problems. But where is the openness? Is this a pendulum that goes back and forth (and many IBM folks left IBM in the 1990s and built great open technologies on the internet) or was it a brief dawn of freedom that will be put down by the capital requirements of modern compute and networking stacks?
My money is on openness continuing to grow and more and more pieces of the stack being completely owned by openness (kernels anyone?) but one doesn't know.
Even without owning the infrastructure, running in the cloud without know-how is very dangerous.
I hear tell of a shop that was running on ephemeral instance based compute fleets (EC2 spot instances, iirc), with all their prod data in-memory. Guess what happened to their data when spot instance availability cratered due to an unusual demand spike? No more data, no more shop.
Don't even get me started on the number of privacy breaches because people don't know not to put customer information in public cloud storage buckets.
I was part of a relatively small org that wanted us to move to cloud dev machines. As soon as they saw the size of our existing development docker images that were 99.9% vendor tools in terms of disk space, they ran the numbers and told us that we were staying on-prem. I'm fairly sure just loading the dev images daily or weekly would be more expensive than just buying a server per employee.
That’s made possible because of all the orchestration platforms such as Kubernetes being standardized, and as such you can get pretty close to a cloud experience while having all your infrastructure on-premise.
Yes, virtualization, overprovisioning and containerization have all played a role in allowing for efficient enough utilization of owned assets that the economics of cloud are perhaps no longer as attractive as they once were.
Context: I build internal tools and platforms. Traffic on them varies, but some of them are quite active.
My nasty little secret is for single server databases I have zero fear of over provisioning disk iops and running it on SQLite or making a single RDBMS server in a container. I've never actually run into an issue with this. It surprises me the number of internal tools I see that depend on large RDS installations that have piddly requirements.
The problem with single instance is that while performance-wise it's best (at least on bare metal), there comes a moment when you simply have too much data and one machine can't handle. Your your scenario, it may never come up, but many organizations face this problem sooner or later.
I agree, my point is that clusters are overused. Most applications simply don't need them and it results in a lot of waste. Much of this has to do with engineers being tasked with an assortment of roles these days, so they obviously opt for the solution where a database and upgrades are managed for them. I've just found that managing a single containers upgrades aren't that big of an issue.
Ordering that amount of amount of servers takes about one hour with hetzner. If you truly want a complete rack on your own maybe a few days as they have to do it manually.
Most companies don‘t need to scale up full racks in seconds. Heck, even weeks would be ok for most of them to get new hardware delivered. The cloud planted the lie into everyone‘s head that most companies dont have predictable and stable load.
rental period is a month
you can also use hetzner cloud, which is still roughly 10x less expensive then aws and that does not take into account the vastly cheaper traffic
Most businesses could probably know server needs 6-12 months out. There's a small number of businesses in the world that actually need dynamic scaling.
One other appealing alternative for smaller startups is to run Docker on one burstable vm. This is a simple setup and allows you to go beyond the cpu limits and also scale up the vm.
Might be other alternatives than using Docker so if anyone has tips for something simpler or easier to maintain, appreciate a comment.
I do that. In fact I've been doing it for years, because every time I do the math, AWS is unreasonably expensive and my solo-founder SaaS would much rather keep the extra money.
I think there is an unreasonable fear of "doing the routing and everything". I run vpncloud, my server clusters are managed using ansible, and can be set up from either a list of static IPs or from a terraform-prepared configuration. The same code can be used to set up a cluster on bare-metal hetzner servers or on cloud VMs from DigitalOcean (for example).
I regularly compare this to AWS costs and it's not even close. Don't forget that the performance of those bare-metal machines is way higher than of overbooked VMs.
Could you please explain what you mean by "physical backbone connection", as I can't think of a meaning that fits the context.
If you mean dealing with the physical dedicated servers that can be rented from Hetzner, that's what the person you replied to was talking about being not so difficult.
If you mean everything else at the data centre that makes having a server there worthwhile (networking, power, cooling, etc.) I don't think people were suggesting doing that themselves (unless you're a big enough company to actually be in the data centre business), but were talking about having direct control of physical servers in a data centre managed by someone like Hetzner.
(edit: and oops sorry I just realised I accidentally downvoted your comment instead of up, undone and rectified now)
No worries, easy to not foresee every possible way in which strangers could interpret a comment!
But I think that people (at least jwr, and probably even nyc_data_geek saying "on prem") are talking about cloud (like AWS) vs. renting (or buying) servers that live in a data centre run by a company like Hetzner, which can be considered "on prem" if you're the kind of data centre client who has building access to send your own staff there to manage your servers (while still leaving everything else, possibly even legal ownership and therefore deprecation etc. to the data centre owner).
What you're thinking of - literally taking responsibility for running your own mini data centre - I think is hardly ever considered (at least in my experience), except by companies at the extremes of size. If you're as big as Facebook (not sure where the line is but obviously including some companies not AS big as Meta but still huge) then it makes sense to run your own data centres. If you're a tiny business getting less than thousands of website visits a day and where the website (or whatever is being hosted) isn't so important that a day of downtime every now and then isn't a big deal, then it's not uncommon to host from the company's office itself (just using a spare old PC or second hand cheap 1U server, maybe a cheap UPS, and just connected to the main internet connection that people in the office use, and probably managed by a single employee, or company owner, who happens to be geeky enough to think it's one or both of simple or fun to set up a basic LAMP server, or even a Windows server for its oh-so-lovely GUI).
You usually just do colocation. The data center will give you a rack (or space for one), an upstream gateway to your ISP, and redundant power. You still have to manage a firewall and your internal network equipment, but its not really that bad. I've used PFsense firewalls, configured by them for like $1500, with roaming vpn, high availability, point to point vpn, and as secure as reasonably possible. After that it's the same thing as the cloud except its physical servers.
100% agree. People still think that maintaining infrastructure is very hard and requires lot of people. What they disregard is that using cloud infrastructure also requires people.
When talking about Hetzner pricing, please don’t change the subject to AWS pricing. The two have nothing in common, and intuition derived from one does not transfer to the other.
Why? The only reason I'm using Hetzner and not AWS for several of my own projects (even though I know AWS much better since this is what I use at work) is an enormous price difference in each aspect (compute, storage, traffic).
If all you need are some cloud servers, or a basic load balancer, they are pretty much the same.
If you need a plethora of managed services and don't want to risk getting fired over your choice or specifics of how that service is actually rendered, they are nothing alike and you should go for AWS, or one of the other large alternatives (GCP, Azure etc.).
On the flip side, if you are using AWS or one of those large platforms as a glorified VPS host and you aren't doing this in an enterprise environment, outside of learning scenarios, you are probably doing something wrong and you should look at Hetzner, Contabo, or one of those other providers, though some can still be a bit pricey - DigitalOcean, Vultr, Scaleway etc.
Well, in my case at least, what they have in common is that I can choose to run my business on one or the other. So it's not about intuition, but rather facts in my case: I avoid spending a significant amount of money.
I (of course) do realize that if you design your software around higher-level AWS services, you can't easily switch. I avoided doing that.
I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud.
Our calculations said it would cost 3x as much to run our infra on the cloud.
We pushed back, and were greenlit on creating a hybrid architecture that allowed us to launch machines both on-prem and in the cloud (via a direct link to the cloud datacenter). This gave us the benefit of autoscaling our volatile services, while maintaining our predictable services on the cheap.
After I left, apparently my former team was strong-armed into migrating everything to the cloud.
A few years go by, and guess who reaches out on LinkedIn?
The parent org was curious how we built the hybrid infra, and wanted us to come back to do it again.
My funny story is built on the idea that AWS is Hotel California for your data.
A customer had an interest in merging the data from an older account into a new one, just to simplify matters. Enterprise data. Going back years. Not even leaving the region.
The AWS rep in the meeting kinda pauses, says: "We'll get back to you on the cost to do that."
The sticker shock was enough that the customer simply inherited the old account, rather than making things tidy.
"Data transferred "in" to and "out" from Amazon EC2, Amazon RDS, Amazon Redshift, Amazon DynamoDB Accelerator (DAX), and Amazon ElastiCache instances, Elastic Network Interfaces or VPC Peering connections across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction."
My (peripheral) experience is that it is much cheaper to get data in than to get data out. When you have the amount of data being discussed — "Enterprise data. Going back years." — that can get very costly.
It's the amount of data where it makes more sense to put hard drives on a truck and drive across the country rather than send it over a network, where this becomes an issue (actually, probably a bit before then).
AWS actually has a service for this - Snowmobile, a storage datacenter inside of a shipping container, which is driven to you on a semi truck. https://aws.amazon.com/snowmobile/
> Q: Can I export data from AWS with Snowmobile?
>
> Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary. Visit the Snowball Edge FAQs to learn more.
R2 is great. Our GCS bill (almost all egress) jumped from a few hundred dollars a month to a couple thousand dollars a month last year due to a usage spike. We rush-migrated to R2 and now that part of the bill is $0.
I've heard some people here on HN say that it's slow, but I haven't noticed a difference. We're mainly dealing with multi-megabyte image files, so YMMV if you have a different workload.
awesome. I remember reading about this a while ago, but never tried. Since it has the same API i can imagine its not daunting as a multi-cloud infrastructure.
I guess permissions might be more complex, as in EC2 instance profiles wouldnt grant access, etc.
Just to make sure nobody is confused by this - R2 has the same API as S3, not GCS. We had to build a simple abstraction around GCS/S3 to perform the migration. But if you're migrating from S3, it's pretty much drop-in. We even use the AWS-provided S3 Java library (especially convenient for making signed URLs).
There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck!
I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere.
I personally dont care if the docs are crap because usually the first thing I do is update / actually write the docs to make them usable.
For a lot of techs though crap documentation is a deal breaker.
Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written. Quite frequently though its outsourced development agencies that refuse to write it because its "out of scope" and a "billable extra". Which I think is an egregious stance...doxs Should be part and parcel of the project. Mandatory.
I agree that bad documentation is a serious problem in many cases. So much so that your suggestion to write the documentation after the fact can become quite impossible.
If there is only one thing that juniors should learn about writing documentation (be it comments or design documents), it is this: document why something is there. If resources are limited, you can safely skip comments that describe how something works, because that information is also available in code.
(It might help to describe what is available, especially if code is spread out over multiple repositories, libraries, teams, etc.)
(Also, I suppose the comment I'm responding to could've been slightly more forgiving to GP, but that's another story.)
Unfortunately it’s also possible that e.g the company switched from share point to confluence and lost half the entire knowledge base because it wasn’t labeled the way they thought it was. Or that the docs were all purged because they were part of an abandoned project.
> Quite frequently though its outsourced development agencies that refuse to write it
It's also completely against their interest to write docs as it makes their replacement easier.
That's why you need someone competent on the buying side to insist on the docs.
A lot of companies outsource because they don't have this competency themselves. So it's inevitable that this sort of thing happens and companies get locked in and can't replace their contractors, because they don't have any docs.
> the first thing I do is update / actually write the docs to make them usable.
OK so the docs are in sync for a single point of time when you finish. Plus you get to have the context in your head (bus factor of 1, job security for you, bad for the org.)
How about if we just write clean infra configs/code, stick to well known systems like docker, ansible, k8s, etc.
Then we can make this infra code available to an on prem LLM and ask it questions as needed without it drifting out of sync overtime as your docs surely will.
Wrong documentation is worse than no documentation.
"Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written."
I can always guarantee a stream of consciousness one note that should have most of the important data, and a few docs about the most important parts. It's up to management if they want me to spend time turning that one note into actual robust documentation that is easily read.
Just to be clear, after I (and a few others left), they moved everything entirely to the cloud.
Even with documentation on the hybrid setup, they'd need to get a new on-prem environment up and running (find a colo, buy machines, set up the network, blah blah).
Yes, I do believe autoscaling is actually a good use case for public cloud. If you have bursty load that requires a lot of resources at peak which would sit idle most of the time, probably doesn't make sense to own what you need for those peaks.
It's not an either/or. Many business both own and rent things.
If price is the only factor, your business model (or executives' decision-making) is questionable. Buy only the cheapest shit, spend your time building your own office chair rather than talking to a customer, you aren't making a premium product, and that means you're not differentiated.
Elsewhere today I recommended RDS, but was thinking of small startup cases that may lack infrastructure chops.
But you are totally right it can be expensive. I worked with a startup that had some inefficient queries, normally it would matter, but with RDS it cost $3,000 a month for a tiny user base and not that much data (millions of rows at most).
Data isn't cheap never was. Paying the licensing fees on top make it more expensive. It really depends on the circunstance a managed database usually has exended support from the compaany providing it. You have to weigh a team's expertise to manage a solution on your own and ensure you spent ample time making it resilient. Other half is the cost of upgrading hardware sometimes it is better to just outright pay a cloud provider if you business does not have enough income to outright buy hardware.There is always an upfront cost.
Small databases or test environment databases you can also leverage kubernetes to host an operator for that tiny DB. When it comes to serious data and it needs a beeline recovery strategy RDS.
Really it should be a mix self hosted for things you aren't afraid to break. Hosted for the things you put at high risk.
Even for small workloads it's a difficult choice. I ran a small but vital db, and RDS was costing us like 60 bucks a month per env. That's 240/month/app.
DynamoDB as a replacement, pay per request, was essentially free.
I found Dynamo foreign and rather ugly to code for initially, but am happy with the performance and especially price at the end.
For big companies such as banks this cost comparison is not as straight forward. They have whole data centres just sitting there for disaster recovery. They periodically do switchovers to test DR. All of this expense goes away when they migrate to cloud.
Well corporations pay printers to do their printing because they don't want to be in the business of printing. It's the same with infrastructure, a lot of corporations simply don't want to be in the data centre business.
From what I’ve read, a common model for mmorpg companies is to use on-prem or colocated as their primary and then provision a cloud service for backup or overage.
Seems like a solid cost effective approach for when a company reaches a certain scale.
Lots of companies, like Grinding Gear Games and Square Enix, just rent whole servers for a tiny fraction of the price compared to what the price gouging cloud providers would charge for the same resources. They get the best of both worlds. They can scale up their infrastructure in hours or even minutes and they can move to any other commodity hardware in any other datacenter at the drop of a hat if they get screwed on pricing. Migrating from one server provider (such as IBM) to another (such as Hetzner) can take an experienced team 1-2 weeks at most. Given that pricing updates are usually given 1-3 quarters ahead at a minimum, they have massive leverage over their providers because they an so easily switch. Meanwhile, if AWS decides to jack up their prices, well you're pretty much screwed in the short-term if you designed around their cloud services.
> Data is the most critical part of your infrastructure. You lose your network: that’s downtime. You lose your data: that’s a company ending event. The markup cost of using RDS (or any managed database) is worth it.
You need well-run, regularly tested, air gapped or otherwise immutable backups of your DB (and other critical biz data). Even if RDS was perfect, it still doesn't protect you from the things that backups protect you from.
After you have backups, the idea of paying enormous amounts for RDS in order to keep your company from ending is more far fetched.
I agree that RDS is stupidly expensive and not worth it provided that the company actually hires at least 2x full-time database owners who monitor, configure, scale and back up databases. Most startups will just save the money and let developers "own" their own databases or "be responsible for" uptime and backups.
This is because you are using SQL Server. Microsoft has intentionally made cloud pricing for SQL server prohibitively expensive for non-Azure cloud workloads by requiring per-core licensing that is extremely punitive for the way EC2 and RDS is architected. This has the effect of making RDS vastly more expensive than running the same workload on bare metal or Azure.
Frankly, this is anti-competitive, and the FTC should look into it, however, Microsoft has been anti-competitive and customer hostile for decades, so if you're still using their products, you must have accepted the abuse already.
Also, it is often overlooked that you still need skilled people to run RDS. It's certainly not "2-clicks and forget" and "you don't need to pay anyone running your DB".
I haven't run a Postgres instance with proper backup and restore, but it doesn't seem like rocket science using barman or pgbackrest.
So by and large I agree with the things in this article. It's interesting that the points I disagree with the author on are all SaaS products:
> Moving off JIRA onto linear
I don't get the hype. Linear is fine and all but I constantly find things I either can't or don't know how to do. How do I make different ticket types with different sets of fields? No clue.
> Not using Terraform Cloud No Regrets
I generally recommend Terraform Cloud - it's easy for you to grow your own in house system that works fine for a few years and gradually ends up costing you in the long run if you don't.
> GitHub actions for CI/CD Endorse-ish
Use Gitlab
> Datadog Regret
Strong disagree - it's easily the best monitoring/observability tool on the market by a wide margin.
Cost is the most common complaint and it's almost always from people who don't have it configured correctly (which to be fair Datadog makes it far too easy to misconfigure things and blow up costs).
> Pagerduty Endorse
Pagerduty charges like 10x what Opsgenie does and offers no better functionality.
When I had a contract renewal with Pagerduty I asked the sales rep what features they had that Opsgenie didn't.
He told me they're positioning themselves as the high end brand in the market.
Cool so I'm okay going generic brand for my incident reporting.
Every CFO should use this as a litmus test to understand if their CTO is financially prudent IMO.
Do jira pages still take 30 seconds to load, and have all the interaction speed of cold molasses?
Does it have nice keyboard shortcuts yet?
Do I still need to perform an arcane ritual of setup to get the ticket statuses to be what I want?
Linear has been such a breath of fresh air, with such a solid desktop app (on Mac OS) that I don’t ever want to go back. Stuff happens instantly, the layout and semantics are an excellent “90% good enough” that I would happily relegate jira to only the most enterprise of enterprise projects.
At one of the bigger companies I was at we had an on-prem JIRA in the same office building and it was still so slow that I would often forget why I was loading that page
No, Jira loading is relatively OK and on par with other SPAs. It's got a CTRL+SHIFT+P style actions menu for tickets which helps cut down on point and click pain (especially for linking issues etc). Setting up statues and workflows and how they map to a board is relatively straightforward.
There are lots of things where Jira falls short, but the pain points on an under-resourced self hosted instance of ten years ago are nothing like the ones you'll find on Jira cloud today.
Does Jira still have multiple flavours of markdown for different fields and editors? Last I used it, it used a different flavour for creating and editing a ticket. Also another flavour for bitbucket. None of these were compatible and it would convert between them in the backend but I was left confused every time when I would have to switch formatting styles
I remember that from a while back, and getting annoyed - it doesn't appear to be something that annoys me at the moment so it might have been fixed, but on reflection I tend to just use the default rich text editor now.
It takes markdownish input but converts it to rich text as you type - so asterisk-space starts a bullet point list, etc.
I actually can't remember if it has a dedicated markdown mode anymore; the rich text editing supports the usual shortcuts that mean I tend to stick with it.
Interesting. Atlassian also just launched an integration with OpsGenie. I have the same opinion of JIRA. I've tried many competitors (not Linear so far) and regretted it every time.
Two different measurements (time and Atlassian development processes) that are orthogonal.
Anything Atlassian does is mostly quite late and its integration story is so pathetic that it's unsurprising.
Try to have a bitbucket pipeline that pushes to confluence. Seems like a basic integration to have, after all, Confluence has an API (well, actually it has 3 different ones) so surely Atlassian would make a basic thing like "publish a wiki page" a thing you get out of the box.
I'm not sure they just launched anything. OpsGenie has been an Atlassian product for 5 or more years now. I've been using it for 3-4 myself and its been integrated with Jira the whole time.
In fact, OpsGenie has mostly been on Auto-pilot for a few years now.
Agreed on PagerDuty
It doesn't really do a lot, administrating it is fairly finicky, and most shops barely use half the functionality it has anyway.
To me its whole schedule interface is atrocious for its price, given from an SRE/dev perspective, that's literally its purpose - scheduled escalations.
OpsGenie’s cheapest is $9 per user month but arbitrarily crippled, the plan anybody would want to use is $19 per user month
So instead of a factor of ten it’s ten percent cheaper. And i just kind of expect Atlassian to suck.
Datadog is ridiculously expensive and on several occasions I’ve run into problems where an obvious cause for an incident was hidden by bad behavior of datadog.
Looks super interesting, and that $3/month for hobbyists is just low enough to meet my budget for hobby services, but please, for on-call stuff, you gotta have alerts that make phone calls. Nothing else is going to wake me in the middle of the night. This is the #1 feature I expect from an on-call service - you're on-call because you will be called.
(It’s configurable too; we have vibrate-only or silenced modes. Think old-school beeper.)
In the rare case that it doesn’t wake you, we have configurable escalation strategies to alert someone else on your team after a configurable number of minutes.
One of the things about phone calls is that they will continue to ring and ring until you actively decide to answer or ignore; it's not just a half-second "ping!" sound delivered once. This is also useful in meetings when everything else is closed up or silenced, to jar you out.
I usually do not respond immediately to phone notifications, which I can handle async. Phone calls are by definition sync.
Ah yeah, these are no ordinary phone notifications. :)
Heii On-Call will keep alerting you with these “Critical Alerts” until you’ve manually acknowledged. (Or until it escalates to a teammate and they acknowledge…)
And at least on my phone they sound nothing like normal phone notifications, which I personally always have on vibrate and/or DND anyway.
I just started building out on-call rotation scheduling to fit teams that already have an alerting solution and need simple automated scheduling. I’d love to get some feedback: https://majorpager.com
Grafana OnCall can be self hosted for free or you can pay $20 a month, and still always have the option to migrate to self hosting if you want to save money
> Cost is the most common complaint and it's almost always from people who don't have it configured correctly (which to be fair Datadog makes it far too easy to misconfigure things and blow up costs).
I loved Datadog 10 years ago when I joined a company that already used it where I never once had to think about pricing. It was at the top of my list when evaluating monitoring tools for my company last year, until I got to the costs. The pricing page itself made my head swim. I just couldn’t get behind subscribing to something with pricing that felt designed to be impossible to reason about, even if the software is best in class.
Their pricing setup is evil. Breaking out by SKUs and having 10+ SKUs is fine, trialing services with “spot” prices before committing to reserved capacity is also fine.
But (for some SKUs, at least) they make it really difficult to be confident that the reserved capacity you’re purchasing will cover your spot use cases. Then, they make you contact a sales rep to lower your reserved capacity.
It all feels designed to get you to pay the “spot” rate for as long as possible, and it’s not a good look.
I understand the pressures on their billing and sales teams that lead to these patterns, but they don’t align with their customers in the long term. I hope they clean up their act, because I agree they’re losing some set of customers over it.
Another annoying thing is that the billing dashboards do not map clearly to what's on the pricing pages / in the contract. Good luck figuring out the extras for RUM when you have multiple orgs.
Then they have things that I wanted to try for a long time, but... support doesn't care? Repeated "would you like to use this? / very likely, can we try it out? / (silence)". I love their product, but they are so annoying to deal with at the billing level.
> Another annoying thing is that the billing dashboards do not map clearly to what's on the pricing pages / in the contract. Good luck figuring out the extras for RUM when you have multiple orgs.
I, quite literally, was griping to my Datadog CSM about this exact thing last week. They'll email me and be, "Oh, you know you're logging volume this month put you into on-demand indexing rates, right?" and my answer is always, "No, because your monitoring platform makes it nearly impossible for me to monitor it correctly."
You can't reference your contracted volume rates when building monitors out and the units for the metrics you need to watch don't match the units you contract with them on the SKU.
> You can't reference your contracted volume rates when building monitors out and the units for the metrics you need to watch don't match the units you contract with them on the SKU.
Are you referring to the `datadog.estimated_usage.logs.ingested_events` metric? It includes excluded events by default but you can get to your indexed volume by excluding excluded logs. `sum:datadog.estimated_usage.logs.ingested_events{datadog_index:*,datadog_is_excluded:false}.as_count()`
> Datadog makes it far too easy to misconfigure things and blow up costs
I'll give you a fun example. It's fresh in my mind because i just got reamed out about it this week.
In our last contract with DataDog, they convinced us to try out the CloudSIEM product, we put in a small $600/mo committment to it to try it out. Well, we never really set it up and it sat on autopilot for many months. We fell under our contract rate for it for almost a year.
Then last month we had some crazy stuff happen and we were spamming logs into DataDog for a variety of reasons. I knew I didn't want to pay for these billions of logs to be indexed, so I made an exclusion filter to keep them out of our log indexes so we didn't have a crazy bill for log indexing.
So our rep emailed me last week and said "Hey just a heads up you have $6,500 in on-demand costs for CloudSIEM, I hope that was expected". No, it was NOT expected. Turns out excluding logs from indexing does not exclude them from CloudSIEM. Fun fact, you will not find any documented way to exclude logs from CloudSIEM ingestion. It is technically possible, but only through their API and it isn't documented. Anyway, I didn't do or know this, so now i had $6,500 of on-demand costs plus $400-500 misc on-demand costs that I had to explain to the CTO.
I should mention my annual review/pay raise is also next week (I report to the CTO), so this will now be fresh in their mind for that experience.
That’s just the sort of hypothetical scenario that kept running through my head as I tried to find a way for us to use Datadog. I even particularly wanted to use the CloudSIEM product. Bummer.
DatDog is a freaking beast. NY wife works in workday (a huge employee management system) and they have a very large number of tutorials, videos, "working hours" and other tools to ensure their customers are making the best use of it.
Datadog on the other side... their "DD University" is a shame and we as paying customers are overwhelmed and with no real guidance. DD should assign some time for integration for new customers, even if it is proportional to what you pay annually. (I think I pay around 6000 usd annually.
I mostly agreed with OP's article, but you basically nailed all of the points of disagreement I did have.
Jira: Its overhyped and overpriced. Most HATE jira. I guess I don't care enough. I've never met a ticket system that I loved. Jira is fine. Its overly complex sure. But once you set it up, you don't need to change it very often. I don't love it, I don't hate it. No one ever got fired for choosing Jira, so it gets chosen. Welcome to the tech industry.
Terraform Cloud: The gains for Terraform Cloud are minimal. We just use Gitlab for running Terraform pipelines and have a super nice custom solution that we enjoy. It wasn't that hard to do either. We maintain state files remotely in S3 with versioning for the rare cases when we need to restore a foobar'd statefile. Honestly I like having Terraform pipelines in the same place as the code and pipelines for other things.
GitHub Actions: Yeah switch to GitLab. I used to like Github Actions until I moved to a company with Gitlab and it is best in class, full stop. I could rave about Gitlab for hours. I will evangelize for Gitlab anywhere I go that is using anything else.
DataDog: As mentioned, DataDog is the best monitoring and observability solution out there. The only reason NOT to use it is the cost. It is absurdly expensive. Yes, truly expensive. I really hate how expensive it is. But luckily I work somewhere that lets us have it and its amazing.
Pagerduty: Agree, switch to OpsGenie. Opsgenie is considerably cheaper and does all the pager stuff of Pager duty. All the stuff that PagerDuty tries to tack on top to justify its cost is stuff you don't need. OpsGenie does all the stuff you need. Its fine. Similar to Jira, its not something anyone wants anyway. No ones going to love it, no one loves being on call. So just save money with OpsGenie. If you're going to fight for the "brand name" of something, fight for DataDog instead, not a cooler pager system.
I'm right there with you on Jira. The haters are wrong - it's a decent enough ticket system, no worse than anything else I've used. You can definitely torture Jira into something horrible, but that's not Jira's fault. Bad managers will ruin any ticket system if they have the customization tools to do so.
Yeah, usually Jira hate is really convoluted company process hate. Of course the Jira software isn't perfect, but it's fine. Jira's strength and weakness is it's flexibility.
Why gitlab? GitHub actions are a mess but gitlab online's ci cd is not much better at all, and for self hosted it opens a whole different can of worms. At least with GitHub actions you have a plugin ecosystem that makes the super janky underlying platform a bit more bearable.
I've found GitLab CI's "DAG of jobs" model has made maintenance and, crucially for us, optimisation relatively easy. Then I look into GitHub Actions and... where are the abstraction tools? How do I cache just part of my "workflow"? Plugins be damned. GitLab CI is so good that I'm willing to overlook vendor lock-in and YAML, and use it for our GitHub project even without proper integration. (Frankly the rest of GitLab seems to always be a couple features ahead, but no-one's willing to migrate.)
Mhmm that's actually a good point!! I didn't realize that I couldn't do that with GitHub, I never really used partial caching. I just had a lot (a looot) of issues with our kubernetes runner (which I even made sure to be as close to the vanilla docs example as possible). I guess the grass is always greener on the other side :)
Linear has a lot going for it. It doesn't support custom fields, so if that's a critical feature for you, I can see it falling short. In my experience though, custom fields just end up being a mess anytime a manager changes and decides to do things differently, things get moved around teams, etc.
- It's fast. It's wild that this is a selling point, but it's actually a huge deal. JIRA and so many other tools like it are as slow as molasses. Speed is honestly the biggest feature.
- It looks pretty. If your team is going to spend time there, this will end up affecting productivity.
- It has a decent degree of customization and an API. We've automated tickets moving across columns whenever something gets started, a PR is up for review, when a change is merged, when it's deployed to beta, and when it's deployed to prod. We've even built our own CLI tools for being able to action on Linear without leaving your shell.
- It has a lot of keyboard shortcuts for power users.
- It's well featured. You get teams, triaging, sprints (cycles), backlog, project management, custom views that are shareable, roadmaps, etc...
> Cost is the most common complaint and it's almost always from people who don't have it configured correctly (which to be fair Datadog makes it far too easy to misconfigure things and blow up costs).
Datadog's cheapest pricing is $15/host/month. I believe that is based on the largest sustained peak usage you have.
We run spot instances on AWS for machine learning workflows. A lot of them if we're training and none otherwise. Usually we're using zero. Using DataDog at it's lowest price would basically double the cost of those instances.
That would totally be my preference if business users didn't want access.
Getting them to use Github/Gitlab is an argument I've never won. Typically it goes the other way and I end up needing to maintain a Monday or Airtable instance in addition to my ticketing system.
In terms of Datadog - the per host pricing on infrastructure in a k8/microservices world is perhaps the most egregious of pricing models across all datadog services. Triply true if you use spot instances for short lived workloads.
For folks running k8 at any sort of scale, I generally recommend aggregating metrics BEFORE sending them to datadog, either on a per deployment or per cluster level. Individual host metrics tend to also matter less once you have a large fleet.
You can use opensource tools like veneur (https://github.com/stripe/veneur) to do this. And if you don't want to set this up yourself, third party services like Nimbus (https://nimbus.dev/) can do this for you automatically (note that this is currently a preview feature). Disclaimer also that I'm the founder of Nimbus (we help companies cut datadog costs by over 60%) and have a dog in this fight.
I'll be dead in the ground before I use TFC. 10 cents per resource per month my ass. We have around 100k~ resources at an early-stage startup I'm at, our AWS bill is $50~/mo and TFC wants to charge me $10k/mo for that? We can hire a senior dev to maintain an in-house tool full time for that much.
Interesting read, I agree with adopting an identity platform but this can definitely be contentious if you want to own your data.
I wonder how much one should pay attention to future problems at the start of a startup versus "move fast and break things." Some of this stuff might just put you off finishing.
I'm using Pulumi in production pretty heavily for a bunch of different app types (ECS, EKS, CloudFront, CloudFlare, Vault, Datadog monitors, Lambdas of all types, EC2s with ASGs, etc.), it's reasonably mature enough.
As mentioned in the other comment, the most commonly used providers for terraform are "bridged" to pulumi, so the maturity is nearly identical to Terraform. I don't really use Pulumi's pre-built modules (crossroads), but I don't find I've ever missed them.
I really like both Pulumi and Terraform (which I also used in production for hundreds of modules for a few years), which it seems like isn't always a popular opinion on HN, but I have and you absolutely can run either tool in production just fine.
My slight preference is for Pulumi because I get slightly more willing assistance from devs on our team to reach in and change something in infra-land if they need to while working on app code.
Say what you want. The tool then builds that, or changes whats there to match.
I've tried Pulumi and understanding the bit that runs before it tries to do stuff and the bit that runs after it tries to do stuff and working out where the bugs are is a PITA. It lulls you into a false sense of security that you can refer to your own variables in code, but that doesn't get carried over to when it is actually running the plan on the cloud service (ie actually creating the infrastructure) because you can only refer to the outputs of other infrastructure.
CFN is too far in the other direction, primarily because it's completely invisible and hard to debug.
Terraform has enough programmability (eg for_each, for-expressions etc) that you can write "here is what I want and how the things link together" and terraform will work out how to do it.
The language is... sometimes painful, but it works.
The provider support is unmatched and the modules are of reasonable quality.
> There are no great FaaS options for running GPU workloads, which is why we could never go fully FaaS.
I keep wondering when this is going to show up. We have a lot of service providers, but even more frameworks, and every vendor seems to have their own bespoke API.
Right, I still find it faster to manually provision a specific instance type, install PyTorch on it, and deploy a little flask app for an inference server.
Even if others disagree with your endorsements or regrets, this record shows you're actually aware of the important decisions you made over the past four years and tracked outcomes. Did you record the decisions when you made them and revisit later?
Why wouldn't you use Kubernetes? There are basically 3 classes of deployments:
1) We don't have any software, so we don't have a prod environment.
2) We have 1 team that makes 1 thing, so we just launch it out of systemd.
3) We have between 2 and 1000 teams that make things and want to self-manage when stuff gets rolled out.
Kubernetes is case 3. Like it or not, teams that don't coordinate with each other is how startups scale, just like big companies. You will never find a director of engineering that says "nah, let's just have one giant team and one giant codebase".
On AWS, at least, there are alternatives such as ECS and even plain old EC2 auto scaling groups. Teams can have the autonomy to run their infrastructure however they like (subject to whatever corporate policy and compliance regime requirements they might have to adhere to).
Kubernetes is appealing to many, but it is not 100% frictionless. There are upgrades to manage, control plane limits, leaky abstractions, different APIs from your cloud provider, different RBAC, and other things you might prefer to avoid. It's its own little world on top of whatever world you happen to be running your foundational infrastructure on.
The alternatives aren't frictionless either; many items from that image are not specific to Kubernetes. I personally find AWS API's frustrating to use, so even if I were running a one-person shop (and was bound to AWS for some reason - maybe a warlock has cursed me?) I'd lean towards managing things from EKS to get an interface that fits my brain better. It's just preference, though - EC2 auto-scaling is perfectly viable if that's your jam.
The iceberg is fine, but using ECS doesn't absolve you from needing to care about monitoring, affinity, audit logging, OS upgrades, authentication/IAM, etc. That's generally why organizations choose to have infrastructure teams, or to not have infrastructure at all.
I have seen people rewrite Kubernetes in CloudFormation. You can do it! But it certainly isn't problem-free.
ECS Fargate does manage the security of the node up to and including the container runtime. Patches are often applied behind the scenes, without many folks even knowing, and for those that require interruption, a restart of the task will land it on a patched node.
You’re right that if you use a cloud provider, IAM is something that has to be reckoned with. But the question is, how many implementations of IAM and policy mechanisms do I want to deal with?
K8S has a credible local development and testing story, ECS and ASGs do not. The fact that there's a generic interface for load-balancer like things, and then you can have a different implementation on your laptop, in the datacenter, and in AWS, and everything ports, is huge.
Also, you can bundle your load balancer config and application config together. No written description of the load balancer config + an RPM file to a disinterested different team.
One giant codebase is fine. Monorepo is better than lots of scattered repos linked together with git hashes. And it doesn't really get in the way of each team managing when stuff gets rolled out.
Because it works, the infra folks you hired already know how to use it, the API is slightly less awful than working with AWS directly, and your manifests are kinda sorta portable in case you need to switch hosting providers for some reason.
This is my case. I’m one man show ATM so no DBA. I’m still using Kubernetes. Many things can be automated as simply as helm apply. Plus you get the benefit of not having a hot mess of systemd services, ad hoc tools which you don’t remember how you configured, plethora bash scripts to do common tasks and so on.
I see Kubernetes as one time (mental and time) investment that buys me somehow smoother sailing plus some other benefits.
Of course it is not all rainbows and unicorns. Having a single nginx server for a single /static directory would be my dream instead of MinIO and such.
Because I can go from main.go to a load balanced, autoscaling app with rolling deploys, segeregated environments, logging & monitoring in about 30 minutes, and never need to touch _any_ of that again. Plus, if I leave, the guy who comes after me can look at a helm chart, terraform module + pipeline.yml and figure out how it works. Meanwhile, our janq shell script based task scheduler craps out on something new every month. What started as 15 lines of "docker run X, sleep 30 docker kill x" is now a polyglot monster to handle all sorts of edge cases.
I have spent vanishingly close to 0 hours on maintaining our (managed) kubernetes clusters in work over the past 3 years, and if I didn't show up tomorrow my replacement would be fine.
You still need to get mysql installed and configured though. On AWS, it's 30 lines of terraform for RDS on an internal subnet with a security group only allowing access from your cluster.
For that, you get automated backups, very simple read proxies, managed updates of you ever need them. You can vertically scale down, or uo to the point of "it's cheaper to hire a DBA to fix this".
There was no AWS, and there was no Terraform. I did have a shell script that would install what was needed, compile my own version of Mysql, configure it, and go.
If you can do all that in 30 minutes (or even a few hours), I would love to read an article/post about your setup, or any resources you might recommend.
I've just done it a dozen times at this point. Hello world from gin-gonic [0], terraform file with a DO K8s cluster [1] and load balancer, and CI/CD [2] on deploy. There's even time to make a cuppa when you run terraform.
We use this for our internal services at work, and the last time I touched the infra was in 2022 according to git
You can just deploy other applications to Kubernetes, for example you can deploy this operator https://prometheus-operator.dev/ and you get Prometheus and Grafana running with a bunch of dashboards already created. Then you annotate your pods to tell Prometheus what to scrape, and you got monitoring. It also comes with AlertManager for alerting. Same for logging, you deploy Elasticsearch and Kibana and you're good to go.
Uh? Even some larger startups don't have DBAs anymore. For better or for worse. Hell even the place I currently work in, which is not a startup at all has basically no DBA role to speak of.
Places get pretty big with no dedicated DBA resources these days. Last place I was at was a Fintech SaaS with 50 engineers and half a million paying customers.
Running off a couple of medium ( $3k/month each range ) RDS databases with failover setup. ECS for apps.
Databases looked after themselves. The senior people probably spent 20% of a FTE on stuff like optimizing it when load crept up.
Place before that was a similar size and no DBA either. People just muddled though.
This is a sweeping generalization to make, and I think you underestimate how easy it is to achieve redundancy with modern tools these days.
My company uses redundant services because we like to deploy frequently, and our customers notice if our API breaks while the service is restarted. Running the service redundantly allows us to do rolling deploys while continuing to serve our API. It’s also saved us from downtime when a service encounters a weird code path and crashes.
Helm is the only infrastructure package manager I've ever used where you could reliably get random third party things running without a ton of hassle. It's a huge advantage.
Great post. I do wonder - what are the simplest K8s alternatives?
Many say in the database world, "use Postgres", or "use sqlite." Similarly there are those databases that are robust that no one has heard of, but are very limited like FoundationDB. Or things that are specialized and generally respected like Clickhouse.
It's just that, you should start with a handful of backed-up pet servers. Then manually automate their deployment when you need it. And only then go for a tool that abstracts the automated deployment when you need it.
But I fear the simplest option on the Kubernetes area is Kubernetes.
I shunned k8s for a long time because of the complexity, but the managed options are so much easier to use and deploy than pet servers that I can’t justify it any more. For anything other than truly trivial cases, IMO kubernetes or (or similar, like nomad) is easier than any alternative.
The stack I use is hosted Postgres and VKS from Vultr. It’s been rock solid for me, and the entire infrastructure can be stored in code.
This is good advice, if you haven't experienced the pain of doing it yourself, you won't know what the framework does for you. There are limits to this reasoning of course, we don't reimplement everything on the stack just for the learning experience. But starting with just docker might be a good idea.
It’s mainly running your own control plane that is complex. Managed k8s (EKS, AKS, GKE) is not difficult at all. Don’t listen to all the haters. It’s the same crowd who think they can replace systemd with self hacked init scripts written in bash, because they don’t trust abstractions and need to see everything the computer does step-by-step.
I also stayed away for a long time due to all the fear spread here, after taking the leap, I’m not looking back.
The lightweight “simpler” alternative is docker-compose. I put simpler in quotes because once you factor in all the auxiliary software needed to operate the compose files in a professional way (IaC, Ansible, monitoring, auth, VM provisioning, ...), you will accumulate the same complexity yourself, only difference is you are doing it with tools that may be more familiar to what you are used to. Kubernetes gives you a single point of control plane for all this. Does it come with a learning curve? Yes, but once you get over it there is nothing inherent about it that makes it unnecessary complex. You don’t need autoscaler, replicasets and those more advanced features just because you are on k8s.
If you want to go even simpler, the clouds have offerings to just run a container, serverless, no fuzz around. I have to warn everyone though that using ACI on Azure was the biggest mistake of my career. Conceptually it sounds like a good idea but Azures execution of it is just a joke. Updating a very small container image taking upwards of 20-30 minutes, no logs on startup crashes, randomly stops serving traffic, bad integration with storage.
We adopted TFC at the start of 2023 and it was problematic right from the start; stability issues, unforeseen limitations, and general jankiness. I have no regrets about moving us away from local execution, but Terraform Cloud was a terrible provider.
When they announced their pricing changes, the bill for our team of 5 engineers would have been roughly 20x, and more than hiring an engineer to literally sit there all day just running it manually. No idea what they’re thinking, apart from hoping their move away from open source would lock people in?
We ended up moving to Scalr, and although it hasn’t been a long time, I can’t speak highly enough of them so far. Support was amazing throughout our evaluation and migration, and where we’ve hit limits or blockers, they’ve worked with us to clear them very quickly.
My contrarian view is that EC2 + ASG is so pleasant to use. It’s just conceptually simple: I launch an image into an ASG, and configure my autoscale policies. There are very few things to worry about. On the other hand, using k8s has always been a big deal. We built a whole team to manage k8s. We introduce dozens of concepts of k8s or spend person-years on “platform engineering” to hide k8s concepts. We publish guidelines and sdks and all kinds of validators so people can use k8s “properly”. And we still write 10s of thousands lines of YAML plus 10s of thousands of code to implement an operator. Sometimes I wonder if k8s is too intrusive.
I tend to agree that for most things on AWS, EC2 + ASG is superior. It's very polished. EKS is very bare bones. I would probably go so far as to just run Kubernetes on EC2 if I had to go that route.
But in general k8s provides incredibly solid abstractions for building portable, rigorously available services. Nothing quite compares. It's felt very stable over the past few years.
Sure, EC2 is incredibly stable, but I don't always do business on Amazon.
At first I thought your "in general" statement was contradicting your preference for EC2 + ASG. I guess AWS is such a large part of my world that "in general" includes AWS instead of meaning everything but AWS.
K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And lock-in never really went away because it's all coupled with cloud specific stuff anyway.
Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need ISO/PCI/whatever.
I think they're more getting a k8s requiring a whole mess of 3rd party code to actually be useful when bringing it to prod. For EKS you end up having coredns, fluentbit, secrets store, external dns, aws ebs csi controller, aws k8s cni, etc.
And in the end it's hard to say if you've actually gained anything except now this different code manages your AWS resources like you were doing with CF or terraform.
Everything we run our workloads on is based on millions of LoCs, whether it's in the OS, in K8S, in is built-in or external kinds. If you decide to run K8S in AWS, you'll be better of using Karpenter, external-secrets and all these things as they will make your life easier in various ways.
Argo CD, Argo Rollouts, Vault, External Secrets, Cert Manager, Envoy, Velero, plus countless operators, plus a service mesh if you need it, the list goes on. If you're providing Kubernetes as a platform at any sort of scale you're going to need most of this stuff or some alternatives. This sums up to at least multiple million LOC. Then you have Kubernetes itself, containerd, etcd...
> K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform.
I don't know what you have been doing with Kubernetes, but I run a few web apps out of my own Kubernetes cluster and the full extent of my lines of code are the two dozen or so LoC kustomize scripts I use to run each app.
I run my own cluster too, it is managed by one terraform file which is maintained on GitHub [0]. Along with that I deploy everything on here with 1 shell script and a bunch of yaml manifests for my services. It's perfect for projects that are managed by one person (me). Everything is in git and reproducable. The only thing I am doing unconventional is that I didn't want to use github actions, so I use Kaniko to build my Docker containers inside my cluster.
If you're using a K8S cluster just to deploy a few web apps then it's not really a platform that you could provide to an engineering team within a medium-large company. You could probably run your stuff on ECS.
> If you're using a K8S cluster just to deploy a few web apps (...)
It's really not about what I do and do not do with Kubernetes. It's on you to justify your "millions upon millions lines of code" claim because it is so outlandish and detached from reality that it says more about your work than about Kubernetes.
I repeat: I only need a few dozen lines of kustomize scripts to release whole web apps. Simple code. Easy peasy. What mess are you doing to require "millions upon millions" lines of code?
Please don't deflect the question. You claimed you need millions and millions of LoC to get something running on Kubernetes. I stated the fact that I have multiple apps running in my personal Kubernetes cluster and they only require a couple of dozen lines of Kustomize. You are the one complaining about complexity where apparently no one else sees it. Either you're able to back up your claims, or you can't. I don't think you can, actually, and I think that's why you are deflecting questions. In fact, I'd go as far as to claim you have zero experience with Kubernetes, and you're just parroting cliches.
While I love ECS you're not giving k8s enough credit. Nearly every COTS (common off the self) app has a helm chart, hardly any provide direct ECS support. If I want a simple kafka cluster or zookeeper cluster there's a supported helm chart for that, nothing is provided for ECS, you have to make that yourself.
You're both using hyperboles that don't match the reality of the average-sized company using Kubernetes. It's neither "millions upon millions of lines of code" nor "just a few dozen lines of kustomize scripts".
kubeadm + fabric + helm got me 99% of the way there. My direct report, a junior engineer, wrote the entire helm chart from our docker-compose. It will not entirely replace our remote environment but it is nice to have something in between our SDK and remote deployed infra. Not sure what you meant by security; could you elaborate? I just needed to expose one port to the public internet.
> My contrarian view is that EC2 + ASG is so pleasant to use.
Sometimes I think that managed kubernetes services like EKS are the epitome of "give the customers what they want", even when it makes absolutely no sense at all.
Kubernetes is about stitching together COTS hardware to turn it into a cluster where you can deploy applications. If you do not need to stitch together COTS hardware, you have already far better tools available to get your app running. You don't need to know or care in which node your app is suppose to run and not run, what's your ingress control, if you need to evict nodes, etc. You have container images, you want to run containers out of them, you want them to scale a certain way, etc.
To me, it sounds like your company went through a complex re-architecturing exercise at the same time you moved to Kubernetes, and your problems have more to do with your (probably flawed) migration strategy than the tool.
Lifting and shifting an "EC2 + ASG" set-up to Kubernetes is a straightforward process unless your app is doing something very non-standard. It maps to a Deployment in most cases.
The fact that you even implemented an operator (a very advanced use-case in Kubernetes) strongly suggests to me that you're doing way more than just lifting and shifting your existing set-up. Is it a surprise then that you're seeing so much more complexity?
Why not dump your application server and dependencies into rented data center (or EC2 if you must) and setup a coarse DR? Maybe start with a monolith in PHP or Rails.
None of that word salad sounds like startup to me, but then again everyone loves to refer to themselves as a startup (must be a recruiting tool?), so perhaps muh dude is spot on.
I would like to know what you’re being downvoted for. It’s not bad advice, necessarily… this was the way 20 years ago. I mean isn’t hacker news running kind of like this as a monolith on a single server? People might be surprised how far you can get with a simple setup.
I don't want to be negative, but this post reads like a list of things that I want to avoid in my career. I did a brief stint in cloud stuff at a FAANG and I don't care to go back to it.
Right now I'm engineer No. 1 at a current startup just doing DDD with a Django monolith. I'm still pretty Jr. and I'm wondering if there's a way to scale without needing to get into all of the things the author of this article mentions. Is it possible to get to a $100M valuation without needing all of this extra stuff? I realize it varies from business to business, but if anyone has examples of successes where people just used simple architecture's I'd appreciate it.
I work at a startup and most of the stuff in the article covers things we use and solve real world problems.
If you're looking for successful businesses, indie hackers like levelsio show you how far you can get with very simple architectures. But that's solo dev work - once you have a team and are dealing with larger-scale data, things like infrastructure as code, orchestration, and observability become important. Kubernetes may or may not be essential depending on what you're building; it seems good for AI companies, though.
30-40 people; not much TPS but we're not primarily building a web app; we have event-driven data pipelines and microservices for ML data.
If you're primarily building a web app, a monolith is fine for quite a while, I think. But a lot of the stuff in the post is still relevant even for monoliths - RDS, Redis, ECR, terraform, pagerduty, monitoring/observability.
Stackoverflow famously grew huge for a long time on a single Windows box. I don’t recommend that but yeah KISS rule definitely. Floss version: supabase, open telemetry, etc.
You don't need this many tools, especially really early. It also depends on the particulars of your business. E.g. if you are B2B SaaS, then you need a ton of stuff automatically to get SOC2 and generally appease the security requirements of your customers.
That said, anything that's set-and-forget is great to start with. Anything that requires it's own care and feeding can wait unless it's really critical. I think we have a project each quarter to optimize our datadog costs and renegotiate our contract.
Also if you make microservices, you are going to need a ton of tools.
You can scale to any valuation with any architecture. Whether or not you need sophisticated scaling solutions depends on the characteristics of your product, mostly how pure of a software play it is. Pure software means you will run into scaling challenges quicker, since likely part of your value add is in fact managing the complexity of scaling.
If you are running a marketplace app and collect fees you're going to be able go much further on simpler architectures than if you're trying to generate 10,000 AI images per second.
I'm currently early in my career and "the software guy" in a non-software team and role, but I'm looking to move into a more engineering direction. You've pretty much got my dream next job at the moment — if you don't mind me asking, how did you manage to find your role, especially being "still pretty Jr."?
The things I did to get here are honestly kind of stupid. I started out at a defense contractor after graduating and left in the first six months because all the software devs were jumping ship. Went to a small business defense contractor (yep that's a thing) and learned to build web apps with React and Django. Then the pace of business slowed so after about 18 months I got on the Leetcode grind and got into a FAANG. Realized I hated it, so I quit after about 9 months with no job lined up.
While unemployed I convinced myself I was going to get a job in robotics (I actually got pretty close, I had 3 final level interviews with robotics companies), but the job market went to shit pretty much the exact day I quit my job lol. I spent about 6 months just learning ROS, Inverse Kinematics, math for robotics, gradient descent and optimization, localization, path planning, mapping etc. I taught at a game development summer camp for a month and a half, that was awesome. Working with kids is always a blast. Also learned Rust and built a prototype for a multiplayer browser-based coding game I had been thinking about for a while. It was an excuse to make a full stack application with some fun infrastructure stuff.
The backend is no longer running, but originally users could see their territory on the galaxy grow as their code won battles for them.
For the current role, I really just got lucky. The previous engineer was on his way out for non-job related reasons. He had read a lot of the books I had (Code Complete, Domain Driven Design) and I think we just connected over shared interests and intellectual curiosity.
I think that in the modern day, so many people are really just in this space for the paycheck-- and that's okay! Everyone needs to make a living. But I think that if you have that intellectual curiosity and like making stuff, people will see that and get excited. It ends up being a blessing and a curse.
I have failed interviews because of honesty "I would Google the names of books and read up on that subject" or "I think if I was doing CSS then I would be in the wrong role" (I realize how douchey that sounds but I just was not meant to design things, I have tried). But I have also gone further in interviews than I should have because I was really engrossed in a particular problem like path planning or inverse kinematics and I was able to talk about things in plain terms.
I think it's easier to learn things quickly if they are something you're actually interested in, it becomes effortless. Basically I just try to do that so I can learn optimally, then I try to get lucky.
EDIT: Oh I just thought of more good advice. Find senior devs to learn from. They can be kind of grumpy in their online presence, but they help you avoid so many tar pits. I am in a Discord channel with a handful of senior engineers. The best way to get feedback is to naively say "I'm going to do X", they will immediately let you know why X is a bad idea. A lot of their advice boils down to KISS and use languages with strong typing.
I did this myself for a good 15 years or so, but eventually with a family, money became a bit more of a priority, and it's hard to get a good job if all you've worked at is small shops. Any next role in a larger tech company will likely be a downgrade until you can prove yourself out, which of course you may not be able to because things are so different, and motivation will run low because you're being tasked with all the stuff that caused you to leave big tech in the first place. It can be quite miserable to be grouped with a bunch of kids with 3-5 YOE that have no idea how to build something from scratch, and they're outperforming you because they know the system.
In my case it took a good five years and a couple job hops to rebalance. But eventually you get back to a reasonable tech leadership role and back to making some of the bigger decisions to help make the junior devs' lives less miserable.
No regrets, but the five years it takes to rebalance can be pretty hard.
I think that my work is honestly the most important factor in my happiness. I spend 8 hours a day (probably for the rest of my life) at work so it's going to be the thing that impacts me the most psychologically in my life.
After realizing that, I decided I'd try as hard as I possibly could to never have to work at a job that I didn't like. I already didn't want kids so that part is easy. The other part of the equation is saving lots of money. I'm not an ascetic by any means, but I live well below my means on a SWE salary which means I can save quite a bit of money each year.
I also recognize that not wanting to go corporate severely limits my options down the line. But capitalism is all about making money for other people. If I can make someone a lot of money, they're not going to care about if I have the chops to stand up a Kubernetes cluster or write a Next.js app or whatever (I hope).
I don't think I'm super smart, I'd say I'm pretty average for this line of work. But I reckon that most SWEs are focused on learning new technologies to get to their next job, or are overly concerned with technical problems. I like to think that I am pragmatic enough about only doing things that are going to deliver business value to make up for being average in smarts.
Anyways, there's not really a point to this rant. These are just some thoughts I have had about optimizing my career for my own happiness, and how I hope I can stay a hot commodity even though I hate working in the cloud and my software skills aren't bleeding edge.
Currently working at a $100M valuation tech company that fundamentally is built on a Django monolith with some other fluffy stuff lying around it. You can go far with a Django monolith and some load balancing.
Don't need any of it. Start simple. Some may be useful though. The list makes good points. Keep it around and if you find yourself suffering from the lack of something, look through the list and see if anything there would be good ROI. But don't adopt something just because this list says you should.
One thing though, I'd start with go. It's no more complex than python, more efficient, and most importantly IMO since it compiles down to binary it's easier to build, deploy, share, etc. And there's less divergence in the ecosystem; generally one simple way to do things like building and packaging, etc. I've not had to deal with versions or tooling or environmental stuff nearly as much since switching.
After reading through this entire post, I'm pleasantly surprised that there isn't one item where I don't mirror the same endorse/regret as the author. I'm not sure if this is coincidence or popular opinion.
I agree. I’m afraid I’m one of those 00s developers and can relate. Back then many startups were being launched on super simple stacks.
With all of that complexity/word salad from TFA, where’s the value delivered? Presumably there’s a product somewhere under all that infrastructure, but damn, what’s left to spend on it after all the infrastructure variable costs?
I get it’s a list of preferences, but still once you’ve got your selection that’s still a ton of crap to pay for and deal with.
Do we ever seek simplicity in software engineering products?
I think that far too many companies get sold on the vision of "it just works, you don't need to hire ops people to run the tools you need for your business". And that is true! And while you're starting, it may be that you can't afford to hire an ops guy and can't take the time to do it yourself. But it doesn't take that much scale before you get to the point it would be cheaper to just manage your own tools.
Cloud and SaaS tools are very seductive, but I think they're ultimately a trap. Keep your tools simple and just run them yourselves, it's not that hard.
> Do we ever seek simplicity in software engineering products?
Doubtfully. Simplicity of work breakdown structure - maybe. Legibility for management layers, possibly. Structural integrity of your CYA armor? 100%.
The half-life of a software project is what now, a few years at most these days? Months, in webdev? Why build something that is robust, durable, efficient, make all the correct engineering choices, where you can instead race ahead with a series of "nobody ever got fired for using ${current hot cloud thing}" choices, not worrying at all about rapidly expanding pile of tech and organizational debt? If you push the repayment time far back enough, your project will likely be dead by then anyway (win), or acquired by a greater fool (BIG WIN) - either way, you're not cleaning up anything.
Nobody wants to stay attached to a project these days anyway.
Look, the thing is - most of infra decisions are made by devops/devs that have a vested interest in this.
Either because they only know how to manage AWS instances (it was the hotness and thats what all the blogs and YT videos were about) and are now terrified from losing their jobs if the companies switch stacks. Or because they needed to put the new thing on their CV so they remain employable. Also maybe because they had to get that promotion and bonus for doing hard things and migrating things. Or because they were pressured into by bean counters which were pressured by the geniuses of Wall Street to move capex to opex.
In any case, this isn't by necessity these days. This is because, for a massive amount of engineers, that's the only way they know how to do things and after the gold rush of high pay, there's not many engineers around that are in it to learn or do things better. It's for the paycheck.
It is what it is. The actual reality of engineering the products well doesn't come close to the work being done by the people carrying that fancy superstar engineer title.
You know the old adage "fast, cheap, good: pick two"? With startups, you're forced to pick fast. You're still probably not gonna make it, but if you don't build fast, you definitely won't.
For simplicity, software must be well built. Unfortunately, the software development practice is perpetually underskilled so we release buggy crap which we compensate for in infrastructure.
There's an easy bent towards designing everything for scale. It's optimistic. It's feels good. It's safe, defendable, and sound to argue that this complexity, cost, and deep dependency is warranted when your product is surely on the verge of changing the course of humanity.
The reality is your SaaS platform for ethically sourced, vegan dog food is below inconsequential and the few users that you do have (and may positively affect) absolutely do not not need this tower of abstraction to run.
The funny thing is a lot of smaller startups are seeing just how absurdly expensive these service are, and are just switching back to basic bare metal server hosting.
For 99% of businesses it's a wasteful, massive overkill expense. You dont NEED all the shiny tools they offer, they don't add anything to your business but cost. Unless you're a Netflix or an Apple who needs massive global content distribution and processing services theres a good chance you're throwing money away.
We had FB up to 6 figures in servers and a billion MAUs (conservatively) before even tinkering with containers.
The “control plane” was ZooKeeper. Everything had bindings to it, Thrift/Protobuf goes in a znode fine. List of servers for FooService? znode.
The packaging system was a little more complicated than a tarball, but it was spiritually a tarball.
Static link everything. Dependency hell: gone. Docker: redundant.
The deployment pipeline used hypershell to drop the packages and kick the processes over.
There were hundreds of services and dozens of clusters of them, but every single one was a service because it needed a different SKU (read: instance type), or needed to be in Java or C++, or some engineering reason. If it didn’t have a real reason, it goes in the monolith.
This was dramatically less painful than any of the two dozen server type shops I’ve consulted for using kube and shit. It’s not that I can’t use Kubernetes, I know the k9s shortcuts blindfolded. But it’s no fun. And pros built these deployments and did it well, serious Kubernetes people can do everything right and it’s complicated.
After 4 years of hundreds of elite SWEs and PEs (SRE) building a Borg-alike, we’d hit parity with the bash and ZK stuff. And it ultimately got to be a clear win.
But we had an engineering reason to use containers: we were on bare metal, containers can make a lot of sense on bare metal.
In a hyperscaler that has a zillion SKUs on-demand? Kubernetes/Docker/OCI/runc/blah is the friggin Bezos tax. You’re already virtualized!
Some of the new stuff is hot shit, I’m glad I don’t ssh into prod boxes anymore, let alone run a command on 10k at the same time. I’m glad there are good UIs for fleet management in the browser and TUI/CLI, and stuff like TailScale where mortals can do some network stuff without a guaranteed zero day. I’m glad there are layers on top of lock servers for service discovery now. There’s a lot to keep from the last ten years.
But this yo dawg I heard you like virtual containers in your virtual machines so you can virtualize while you virtualize shit is overdue for its CORBA/XML/microservice/many-many-many repos moment.
You want reproducibility. Statically link. Save Docker for a CI/CD SaaS or something.
You want pros handing the datacenter because pets are for petting: pay the EC2 markup.
You can’t take risks with customer data: RDS is a very sane place to splurge.
Half this stuff is awesome, let’s keep it. The other half is job security and AWS profits.
No, not at all. Maybe baffled by the use of expensive cloud services instead of running on your own bare metal where the cost is in datacenter space and bandwidth. The loss of control coupled with the cost is baffling.
I am 10s developer/systems engineer and my eyes kept getting wider with each new technology on the list. I don't know if its overkill or just the state of things right now.
There is no way one person can thoroughly understand so many complex pieces of technology. I have worked for 10 years more or less at this point, and I would only call myself confident on 5 technical products, maybe 10 if I being generous to myself.
Not really, it's just like counting: awk, grep, sed, uniq, tail, etc.
"CloudOS" is in it's early days right now.
You need to be careful on what tool or library you pick.
It is a shame karpenter is AWS only. I was thinking about how our k8s autoscaler could be better and landed on the same kind of design as karpenter where you work from unschedulable pods backwards. Right now we have an autoscaler which looks at resource utilization of a node pool but that doesn’t take into account things like topology spread constraints and resource fragmentation.
Azure Linux support has been added recently [1], and so is available for self-hosted Karpenter (you can specify "imageFamily: AzureLinux" in AKSNodeClass), will be rolled out for the managed experience (Node Auto Provisioning) soon.
643 comments
[ 3.3 ms ] story [ 364 ms ] threadI think adding a DBA or hiring one to help you layout your database should not be considered a 'luxury'...
Maybe you need help with setup for a few weeks/months, and then some routine billable hours per month for maintenance / change advice.
Fortunately, with managed DBs like RDS it is really easy to run individual DB clusters per major app.
This is rarely a problem when things are small, but as they grow, the bad schema decisions made by empowering DBA-less teams to run their own infra become glaringly obvious.
In the kitchen sink model all teams are tied together for performance and scalability, and some bad apple applications can ruin the party for everyone.
Seen this countless times doing due diligence on startups. The universal kitchen sink DB is almost always one of the major tech debt items.
Multi-tenant DBs can work fine as long as every app has its own users, everyone goes through a connection pooler / load balancer, and every user has rate limits. You want to write shitty queries that time out? Not my problem. Your GraphQL BFF bullshit is trying to make 10,000 QPS? Nope, sorry, try again later.
EDIT: I say “not my problem,” but as mentioned, it inevitably becomes my problem. Because “just unblock them so the site is functional” is far more attractive to the C-Suite than “slow down velocity to ensure the dev teams are doing things right.”
Full Stack is a lie, and the sooner companies accept that and allow people to specialize again, and to pay for the extra headcount, the better off everyone will be.
This is how you end up with the infamous "jira and confluence have two different markdown flavors" issue.
This will undoubtedly go over poorly, but honestly I think every data decision should be gated through the DB Team (again, if you have them). Your proposed schema isn’t normalized? Straight to jail. You don’t want to learn SQL? Also straight to jail. You want to use a UUIDv4 as a primary key? Believe it or not, jail.
The most performant and referentially sound app in the world, because of jail.
Uuids are really for external communication, not in-system organization.
They require periodic synchronization. What isn't a big deal at all and is required by many other database features.
PlanetScale uses int PKs [0], and they seem to have scaled just fine.
[0]: https://github.com/planetscale/discussion/discussions/366
[0]: https://www.percona.com/blog/uuids-are-popular-but-bad-for-p...
[1]: https://www.cybertec-postgresql.com/en/unexpected-downsides-...
[2]: https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-pa...
DB team could act as an auditor and expert support, but they should never be fully responsible for DB layer.
That’s the point. Would you send a backend code review to a frontend team? Why do DBs not deserve domain expertise, especially when the entire company depends on them?
> they are not responsible for the whole product, just for the database
I assure you, that’s a lot to be responsible for at scale.
> DB team could act as an auditor and expert support, but they should never be fully responsible for DB layer.
Again, the issue here is when the DB gets borked enough that a SME is required to fix it, they effectively do become responsible, because no CTO is going to accept, “sorry, we’ll be down for a couple of days because our team doesn’t really know how this thing works.”
And if your answer is, “AWS Premium Support,” they’ll just tell you to upsize the instance. Every time. That is not a long-term strategy.
I wish and maybe there is a programming language with first class database support. I mean really first class not just let me run queries but almost like embedded into the language in a primal way where I can both deal with my database programming fancyness and my general development together.
Sincerely someone who inherited a project from a DBA.
Not quite embedded into the OS, but Django is a damn good ORM. I say that as a DBRE, and someone obsessed with performance (inherent issues with interpreted languages aside).
I have worked in many languages with many ORMs and this has been my personal favorite.
[0]: https://github.com/prisma/prisma/discussions/12715
[0]: https://github.com/prisma/prisma/issues/5184#issuecomment-18...
One thing that has worked well for us is to alway include the top-most parent key in all child tables down yhe hierarchy. This way we can load all the data for say an order without joins/exists.
Oh and never use natural keys. Each time I thought finally I had a good use-case, it has bitten me in some way.
Apart from that we just try to think about the required data access and the queries needed. Main thing is that all queries should go against indexes in our case, so we make sure the schema supports that easily. Requires some educated guesses at times but mostly it's predictable IME.
Anyway would love to see a proper resource. We've made some mistakes but I'm sure there's more to learn.
With that said, this still sounds like a strange situation - most colleagues, acquaintances and people I consulted know they way around SQL and dropping down to 'dbset.FromSql($"SELECT {...' is very commonplace out of the need to use sprocs, views or have tighter control over the query.
But schema design is something else. I still take my time doing that.
Especially since our application is written with backwards compatibility in mind, so changing schema after it's deployed is something we try very hard to avoid.
But yeah, when hiring we require they are comfortable writing "normal" SQL queries (multiple joins, aggregation etc).
The neat thing is, you don't. Nobody ever avoids fucking up db design.
The best you can do is decide what is really important to get right, and not fuck that part up.
P.S. to the original person concerned about this though… for your own sake and your successors, please keep trying.
Just do the exercise of deciding what is really important first, so you can make sure you succeed for that stuff.
If you can't do something like determine if you can delete data, as the article mentions, you won't be able to produce an answer to how to deal with those problems.
Being shared between applications is literally what databases were invented to do. That’s why you learn a special dsl to query and update them instead of just doing it in the same language as your application.
The problem is that data is a shared resource. The database is where multiple groups in an organization come together to get something they all need. So it needs to be managed. It could be a dictator DBA or a set of rules designed in meetings and administered by ops, or whatever.
But imagine it was money. Different divisions produce and consume money just like data. Would anyone imagine suggesting either every team has their own bank account or total unfettered access to the corporate treasury? Of course not. You would make a system. Everyone would at least mildly hate it. That’s how databases should generally be managed once the company is any real size.
Decades of experience have shown us the massive costs of doing so - the crippled velocity and soul crushing agony of dba change control teams, the overhead salary of database priests, the arcane performance nightmares, the nuclear blast radius, the fundamental organizational counter-incentives of a shared resource .
Why on earth would we choose to pay those terrible prices in this day and age, when infrastructure is code, managed databases are everywhere and every team can have their own thing. You didn’t have a choice previously, now you do.
You DO have to share data in other ways, usually datawarehouse or services, but that is not the same thing.
I’m not saying literally every source of data has to be shared and centrally managed. I’m also not saying “rdbms accessed via traditional client and queried via sql” when I say database. I’m just saying a shared database of some shape is inevitable.
Also, operationally it’s not “semantics” at all. You don’t get into (many) operational problems with analysts sharing a datawarehouse. You absolutely do with online apps sharing a rdbms, they aren’t the same thing.
A data warehouse is a type of database and is does need to be managed. Your assertion that it is easier to manage is orthogonal to my assertion that there will always be a central database to manage in an organization of decent size.
I think the format of this is great. I suppose it would take a motivated individual to go around and ask people to essentially fill out a form like this to get that.
One suggestion if we're gonna standardize around this format. Avoid the double negatives. In some cases author says "avoided XYZ" and then the judgment was "no regrets". Too many layers for me to parse there. Instead, I suggest each section being the product that was used. If you regret that product, in the details is where you mention the product you should have used. Or you have another section for product ABC and you provide the context by saying "we adopted ABC after we abandoned XYZ".
I don't recommend trying to categorize into general areas like logging, postmortems, etc. Just do a top-level section for each product.
Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a full-time in-house DBA.
https://calculator.aws/#/estimate?id=48b0bab00fe90c5e6de68d0...
Total 12 months cost: 547,441.85 USD
Once you get past the point where the markup can pay for one or more full-time employees, I think you should consider doing that instead of blindly paying more and more to scale RDS up. You're REALLY paying for it with RDS. At least re-evaluate the choices you made as a fledgling startup once you reach the scale where you're paying AWS "full time engineer" amounts of money.
Assuming you have any. You might not, because of AWS.
Hard disagree. An r6i.12xl Multi-AZ with 7500 IOPS / 500 GiB io1 books at $10K/month on its own. Add a read replica, even Single-AZ at a smaller size, and you’re half that again. And this is without the infra required to run a load balancer / connection pooler.
I don’t know what your definition of “large” is, but the described would be adequate at best at the ~100K QPS level.
RDS is expensive as hell, because they know most people don’t want to take the time to read docs and understand how to implement a solid backup strategy. That, and they’ve somehow convinced everyone that you don’t have to tune RDS.
If you don't have a reserved instance, then you're giving up potentially a 50% discount on on-demand pricing.
An r6i.12xl is a huge instance.
There are other equivalents in the range of instances available (and you can change them as required, with downtime).
For MySQL and Postgres, RDS stripes across four volumes once you hit 400 GiB. Doesn't matter the type.
The latency variation on gp3 is abysmal [0], and the average [1] isn't great either. It's probably fine if you have low demands, or if your working set fits into memory and you can risk the performance hit when you get an uncached query.
12K IOPS sounds nice until you add latency into it. If you have 2 msec latency, then (ignoring various other overheads, and kernel or EBS command merging) the maximum a single thread can accomplish in one second is (1000 msec / 1 sec / 2 msec) = 500 I/O. Depending on your needs that may be fine, of course.
> If you don't have a reserved instance, then you're giving up potentially a 50% discount on on-demand pricing.
True, of course. Large customers also don't pay retail.
> An r6i.12xl is a huge instance.
I mean, it goes well past that to .32xl, so I wouldn't say it's huge. I work with DBs with 1 TiB of RAM, and I'm positive there are people here who think those are toys. The original comment I replied to said, "large SaaS," and a .12xl, as I said, would be roughly adequate for ~100K QPS, assuming no absurdly bad queries.
[0]: https://www.percona.com/blog/performance-of-various-ebs-stor...
[1]: https://silashansen.medium.com/looking-into-the-new-ebs-gp3-...
After initial setup, managing equivalent of $5k/m RDS is not full time job. If you add to this, that wages differ a lot around the world, $5k can take you very, very far in terms of paying someone.
The query times are incredible.
OTOH, as noted, EBS does not perform as well as native NVMe and is hilariously expensive if you try. And quite a few use cases are just fine on plain old NVMe.
Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this? If its so fundamental to your product and needs such uptime & redundancy, what are the odds its also reasonably small?
Most software startups these days? The blog post is about work done at a startup after all. By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. If you’re a small startup, saving a couple hundred bucks a month by self managing your database is rarely a good choice. There’re more valuable things to work on.
By the time your db is big enough to cost an unreasonable amount on RDS, you've likely got so much momentum that getting off is nearly impossible as you bleed cash.
You can buy a used server and find colocation space and still be pennies on the dollar for even the smallest database. If you're doing more than prototyping, you're probably wasting money.
Optionality and flexibility are extremely valuable, and that is why cloud compute continues to be popular, especially for rapidly/burstily growing businesses like startups.
If you want a different opportunity cost, get people with different experience. If RDS is objectively expensive, objectively slow, but subjectively easy, change the subject.
I don't think that's accurate. I've self-managed databases, and I still think that RDS is compelling for small engineering teams.
There's a lot to get right when managing a database, and it's easy to screw something up. Perhaps none of the individual parts are super-complicated, but the cost of failure is high. Outsourcing that cost to AWS is pretty compelling.
At a certain team size, you'll end up with a section of the team that's dedicated to these sorts of careful processes. But the first place these issues come up is with the database, and if you can put off that bit of organizational scaling until later, then that's a great path to choose.
Its about tradeoffs, and some tradeoffs are often more applicable than others - getting a ping at 7am on a Sunday because your ec2 instance filled it's drive up with logs and your log rotation script failed because it didn't have a long enough retey is a problem I'm happy to outsource when I should be focusing on the actual app.
If the cost of a hosted DB is going to sink the company, then of course, I will figure it out and run it myself. But it’s not, for most startups. And therefore that knowledge isn’t providing much leverage.
Starting an AI company with deep expertise in training models - that is an example of knowledge providing huge leverage. DB tech is not in this bucket for most businesses.
I’ve been the guy managing a critical self-hosted database in a small team, and it’s such a distraction from focusing on the actual core product.
To me, the cost of RDS covers tons of risks and time sinks: having to document the db server setup so I’m not the only one on the team who actually knows how to operate it, setting up monitoring, foolproof backups so I don’t need to worry that they’re silently failing because a volume is full and I misconfigured the monitoring, PITR for when someone ships a bad migration, one click HA so the database itself is very unlikely to wake me at 3am, blue/green deploys to make major version upgrades totally painless, never having to think about hardware failures or borked dist-upgrades, and so on.
Each of those is ultimately either undifferentiated work to develop in-house RDS features that could have been better spent on product, or a risk of significant data loss, downtime, or firefighting. RDS looks like a pretty good deal, up to a point.
RDS is a great choice, for prototyping and only for production if you know what you're doing when setting it up.
FWIW, this is common in all cloud deployments, people assume that running something "severless" is a magical silver bullet.
Maybe it takes you a month the first time around and a week the 10th time around. First product suffers, the other products not so much. Now it just takes a week of your time and does not require you to pay large AWS fees, which means you are not bleeding money
I like to set up scrappy products that do not rack up large monthly fees. This means I can let them run unprofitable for longer and I don't have to seek an investor early, which would light up a large fire under everyone's butts and start influencing timelines because now they have the money and want a return asap.
I'll launch a week later - no biggie usually. I could have come up with the idea a month later, so I'm still 3 weeks early ;)
It doesn't work for all projects, obviously, but I've seen plenty of SaaS start out with a shopping spree, then pay monthly fees and purchase licenses for stuff that they could have set up for free if they put some (usually not a lot) effort into it. When times get rough, the shorter runway becomes a hard fact of life. Maybe they wouldn't have needed a VC and could have bootstrapped and also survived for longer.
While I generally agree as far as initial setup time goes, I favor RDS because I can forget about it, whereas the hand rolled version demands ongoing maintenance, and incurs a nonzero chance of simple mistakes that, if made, could result in a 100% dataloss unrecoverable scenario.
I’m also mostly talking about typical, funded startups here, as opposed to indie/solo devs. If you’re flying solo launching a tiny proof of concept that may only ever have a few users, by all means run it yourself if you’d like, but if you’ve raised money to grow faster and are paying employees to iterate rapidly searching for PMF…just pay for RDS and make sure as much time as possible is spent on product features that provide actual business value. It starts at like $15/month. The cost of simply not being laser-focused on product is far greater.
Databases are not particularly difficult to migrate between machines. Of all the cloud services to migrate, they might actually be the easiest, since the databases don't have different API's that need to be rewritten for, and database replication is a well-established thing.
Getting off is quite the opposite of nearly impossible.
Also, Aurora gives you the block level cluster that you can't deploy on your own - it's way easier to work with than the usual replication.
I mean, I'm looking after a 4 instance Aurora cluster which is great feature wise, is slightly overprovisioned for special events, and is more likely to shrink than grow 2x in the next decade. If we start experiencing any issues, there's lots of optimisations that can be still gained from better caching and that work will be cheaper than the instance size upgrade.
There’s still a defined cost to swapping your DB code over to a different backend. At the point where it becomes uneconomical, you’re also at a scale you can afford rewriting a module.
That’s why we have things like “hexagonal architecture”, which focus on isolating the storage protocol from the code. There’s an art to designing such that your prototype can scale with only minor rework — but that’s why we have senior engineers.
No one runs their own electricity supply (well until recently with renewables/storage), they buy it as a service, up to a pretty high scale before it becomes more economic to invest the capex and opex to run your own.
These questions always sound flawed to me. It's like asking won't I regret moving to California and paying high taxes once I start making millions of dollars? Maybe? But that's an amazing problem to have and one that I may be much better equipped to solve.
If you are small, RDS is much cheaper, and many company killing events, such as not testing your backups are solved. If you are big and you can afford a 60K/yr RDS bill than you can make changes to move on-prem. Or you can open up excel and do the math if your margins are meaningfully affected by moving on-prem.
On some level, AWS/GCP/California relies on you doing this calculation for the things that you can do it on easily (the savings of moving away), while not doing this calculation on things where it's hard to do (new development). That way, you can pretend that your new features are a lot more valuable than the $Xk/year you will save by moving your infra.
Yes, I've done the math. The piece you are missing is, saving money on infra will bring in $0 new dollars. There is a floor to how much money I can save. There is no ceiling to how much money the right feature can bring in. Penny pinching on infra, especially when the amount of money is saved is less than the cost of an engineer is almost always a waste of time while you are growing a company. If you are at the point where you are wasting 1x,2x,3x of an engineers salary of superflous infrastructure - then congratulations you have survived the great filter for 99% of startups.
>That way, you can pretend that your new features are a lot more valuable than the $Xk/year you will save by moving your infra.
Finding product market fit is 1000x harder than moving from RDS to On-prem. If you haven't solved PMF, then no amount of $Xk/year in savings will save you from having to shut down your company.
The thing is, most features, particularly later in the life of a company, don't have an easy-to-measure revenue impact, and I suspect that many features are actually worth $0 of revenue. However, they cost money to implement (both in engineering time and infra), making them very much net negative value propositions. This is why Facebook and Google can cut tons of staff and lose nothing off their revenue number.
Also, there's a bit of a gambling mentality here which is that a feature could be worth effectively infinite revenue (ie it could be the thing that gives you PMF), so it's always worth doing over things with known, bounded impact on your bottom line. However, improving your efficiency gives you more cracks at finding good features before you run out of money.
So moving to/from Aurora/RDS/own EC2/on-prem should be a matter of networking and changing connection strings in the clients.
Your operational requirements and processes (backup/restore, failover, DR etc) will change, but that's because you're making a deliberate decision weighing up those costs vs benefits.
You can use DNS to mitigate the pain of changing those connection strings, decoupling client change management from backend change process, or if you had foresight, not having to change client connection strings at all.
I mean, DNS change can work, but when you're doing that one-in-years change, why risk the extra failure modes.
It would have cost a negligible amount. But the sheer amount of time I wasted before I gave up was honestly quite surprising. Let’s see:
- I wanted one simple extension. I could have compromised on this, but getting it to work on RDS was a nonstarter.
- I wanted RDS to _import the data_. Nope, RDS isn’t “SUPER,” so it rejects a bunch of stuff that mysqldump emits. Hacking around it with sed was not confidence-inspiring.
- The database uses GTIDs and needed to maintain replication to a non-AWS system. RDS nominally supports GTID, but the documented way to enable it at import time strongly suggests that whoever wrote the docs doesn’t actually understand the purpose of GTID, and it wasn’t clear that RDS could do it right. At least Azure’s docs suggested that I could have written code to target some strange APIs to program the thing correctly.
Time wasted: a surprising number of hours. I’d rather give someone a bit of money to manage the thing, but it’s still on a combination of plain cloud servers and bare metal. Oh well.
Sounds like you are walking massive edge
Very small businesses with phone apps or web apps are often using it. There are cheaper options of course, but when there is no "prem" and there are 1-5 employees then it doesn't make much sense to hire for infra. You outsource all digital work to an agency who sets you up a cloud account so you have ownership, but they do all software dev and infra work.
> If its so fundamental to your product and needs such uptime & redundancy, what are the odds its also reasonably small?
Small businesses again, some of my clients could probably run off a Pentium 4 from 2008, but due to nature of the org and agency engagement it often needs to live in the cloud somewhere.
I am constantly beating the drum to reduce costs and use as little infra as needed though, so in a sense I agree, but the engagement is what it is.
Additionally, everyone wants to believe they will need to hyperscale, so even medium scale businesses over-provision and some agencies are happen to do that for them as they profit off the margin.
AWS and the like are rarely a cost effective option, but it is something a lot of agencies like, largely because they are not paying the bills. The clients do not usually care because they are comfortable with a known brand and the costs are a small proportion of the overall costs.
A real small business will be fine just using a VPS provider or a rented server. This solves the problem of not having on premise hardware. They can then run everything on a single server, which is a lot simpler to set up, and a lot simpler to secure. That means the cost of paying someone to run it is a lot lower too as they are needed only occasionally.
They rarely need very resilient systems as they amount of money lost to downtime is relatively small - so even on AWS they are not going to be running in multiple availability zones etc.
People pay for RDS because they want to believe in a fairy tale that it will keep potential problems away and that it worked well for other customers. But those mythical other customers also paid based on such belief. Plus, no one wants to admit that they pay money in such irrational way. It's a bubble
Come time for force major upgrade shoved down our throat? Downtime, surprise, surprise
One of these is not like the others (DBAs are not capex.)
Have you ever considered that if a company can get the same result for the same price ($100K opex for RDS vs same for human DBA), it actually makes much more sense to go the route that takes the human out of the loop?
The human shows up hungover, goes crazy, gropes Stacy from HR, etc.
RDS just hums along without all the liabilities.
We technically aren't supposed to talk about pricing publically, but I'm just going to say that we run a few 8XL and 12Xl RDS instances and we pay ~40% off the sticker price.
If you switch to Aurora engine the pricing is absurdly complex (its basically impossible to determine without a simulation calculator) but AWS is even more aggressive with discounting on Aurora, not to mention there are some legit amazing feature benefits by switching.
I'm still in agreeance that you could do it cheaper yourself at a Data Center. But there are some serious tradeoffs made by doing it that way. One is complexity and it certainly requires several new hiring decisions. Those have their own tangible costs, but there are a huge amount of intangible costs as well like pure inconvenience, more people management, more hiring, split expertise, complexity to network systems, reduce elasticity of decisions, longer commitments, etc.. It's harder to put a price on that.
When you account for the discounts at this scale, I think the cost gap between the two solutions is much smaller and these inconveniences and complexities by rolling it yourself are sometimes worth bridging that smaller gap in cost in order to gain those efficiencies.
Genuinely curious, how do you that?
We pay a couple of million dollars per year and the biggest spend is RDS. The bulk of those are 8xl and 12xl as you mention and we have a lot of these. We do have savings plans, but those are nowhere near 40%.
It looks like a reserved instance is 35% off sticker price? Add probably a discount and you'd be around 40% off.
Just like any other asset.
Besides, running things locally can be refreshingly simple if you are just starting something and you don't need tons of extra stuff, which becomes accidental complexity between you, the problem, and a solution. This old post described that point quite well by comparing Unix to Taco Bell: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra.... See HN discussion: https://news.ycombinator.com/item?id=10829512.
I am sure for some use-cases cloud services might be worth it, especially if you are a large organization and you get huge discounts. But I see lots of business types blindly advocating for clouds, without understanding costs and technical tradeoffs. Fortunately, the trend seems to be plateauing. I see an increasing demand for people with HPC, DB administration, and sysadmin skills.
So much this. The "keep know how" has been so greatly avoided over the past 10 years, I hope people with these skills start getting paid more as more companies realize the cost difference.
My money is on openness continuing to grow and more and more pieces of the stack being completely owned by openness (kernels anyone?) but one doesn't know.
I hear tell of a shop that was running on ephemeral instance based compute fleets (EC2 spot instances, iirc), with all their prod data in-memory. Guess what happened to their data when spot instance availability cratered due to an unusual demand spike? No more data, no more shop.
Don't even get me started on the number of privacy breaches because people don't know not to put customer information in public cloud storage buckets.
If I had a small business with very clever people I'd be very afraid of what happens if they're not available for a while.
My nasty little secret is for single server databases I have zero fear of over provisioning disk iops and running it on SQLite or making a single RDBMS server in a container. I've never actually run into an issue with this. It surprises me the number of internal tools I see that depend on large RDS installations that have piddly requirements.
On what disk is the actual data written? How do you do backups, if you do?
For example, how long does it take to rent another rack that you didnt plan for?
And not to mention that the cost of cloud management platforms that you have to deploy to manage these owned assets is not free.
I mean, how come even large consumers of electricity does not buy and own their own infrastructure to generate it?
They sure do? BASF has 3 power plants in Hamburg, Disney operate Reedy Creek Energy with at least 1 power plant and I could list a fair bit more...
>For example, how long does it take to rent another rack that you didnt plan for?
I mean, you can also rent hardware a lot cheaper then on AWS. There certainly are providers where you can rent out a rack for a month within minutes
Most companies don‘t need to scale up full racks in seconds. Heck, even weeks would be ok for most of them to get new hardware delivered. The cloud planted the lie into everyone‘s head that most companies dont have predictable and stable load.
Might be other alternatives than using Docker so if anyone has tips for something simpler or easier to maintain, appreciate a comment.
I would have a hard time doing servers as cheap as hetzner for example including the routing and everything
I think there is an unreasonable fear of "doing the routing and everything". I run vpncloud, my server clusters are managed using ansible, and can be set up from either a list of static IPs or from a terraform-prepared configuration. The same code can be used to set up a cluster on bare-metal hetzner servers or on cloud VMs from DigitalOcean (for example).
I regularly compare this to AWS costs and it's not even close. Don't forget that the performance of those bare-metal machines is way higher than of overbooked VMs.
We are using hetzner cloud.. but we are also scaling up and down a lot right now
If you mean dealing with the physical dedicated servers that can be rented from Hetzner, that's what the person you replied to was talking about being not so difficult.
If you mean everything else at the data centre that makes having a server there worthwhile (networking, power, cooling, etc.) I don't think people were suggesting doing that themselves (unless you're a big enough company to actually be in the data centre business), but were talking about having direct control of physical servers in a data centre managed by someone like Hetzner.
(edit: and oops sorry I just realised I accidentally downvoted your comment instead of up, undone and rectified now)
Aka if I add up power (including backup) + backbone connection rental + server deprication I can not do it for the hetzner price..
That was quite imprecise, sorry about that.
But I think that people (at least jwr, and probably even nyc_data_geek saying "on prem") are talking about cloud (like AWS) vs. renting (or buying) servers that live in a data centre run by a company like Hetzner, which can be considered "on prem" if you're the kind of data centre client who has building access to send your own staff there to manage your servers (while still leaving everything else, possibly even legal ownership and therefore deprecation etc. to the data centre owner).
What you're thinking of - literally taking responsibility for running your own mini data centre - I think is hardly ever considered (at least in my experience), except by companies at the extremes of size. If you're as big as Facebook (not sure where the line is but obviously including some companies not AS big as Meta but still huge) then it makes sense to run your own data centres. If you're a tiny business getting less than thousands of website visits a day and where the website (or whatever is being hosted) isn't so important that a day of downtime every now and then isn't a big deal, then it's not uncommon to host from the company's office itself (just using a spare old PC or second hand cheap 1U server, maybe a cheap UPS, and just connected to the main internet connection that people in the office use, and probably managed by a single employee, or company owner, who happens to be geeky enough to think it's one or both of simple or fun to set up a basic LAMP server, or even a Windows server for its oh-so-lovely GUI).
Why? The only reason I'm using Hetzner and not AWS for several of my own projects (even though I know AWS much better since this is what I use at work) is an enormous price difference in each aspect (compute, storage, traffic).
If all you need are some cloud servers, or a basic load balancer, they are pretty much the same.
If you need a plethora of managed services and don't want to risk getting fired over your choice or specifics of how that service is actually rendered, they are nothing alike and you should go for AWS, or one of the other large alternatives (GCP, Azure etc.).
On the flip side, if you are using AWS or one of those large platforms as a glorified VPS host and you aren't doing this in an enterprise environment, outside of learning scenarios, you are probably doing something wrong and you should look at Hetzner, Contabo, or one of those other providers, though some can still be a bit pricey - DigitalOcean, Vultr, Scaleway etc.
Well, in my case at least, what they have in common is that I can choose to run my business on one or the other. So it's not about intuition, but rather facts in my case: I avoid spending a significant amount of money.
I (of course) do realize that if you design your software around higher-level AWS services, you can't easily switch. I avoided doing that.
I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud.
Our calculations said it would cost 3x as much to run our infra on the cloud.
We pushed back, and were greenlit on creating a hybrid architecture that allowed us to launch machines both on-prem and in the cloud (via a direct link to the cloud datacenter). This gave us the benefit of autoscaling our volatile services, while maintaining our predictable services on the cheap.
After I left, apparently my former team was strong-armed into migrating everything to the cloud.
A few years go by, and guess who reaches out on LinkedIn?
The parent org was curious how we built the hybrid infra, and wanted us to come back to do it again.
I didn't go back.
A customer had an interest in merging the data from an older account into a new one, just to simplify matters. Enterprise data. Going back years. Not even leaving the region.
The AWS rep in the meeting kinda pauses, says: "We'll get back to you on the cost to do that."
The sticker shock was enough that the customer simply inherited the old account, rather than making things tidy.
Have people lost the ability to write export and backup scripts?
You can peer two vpc's and as long as you are transferring within the same (real) AZ, it's free: https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-vp...
Even peered VPC's only pay "normal" prices: https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer
"Data transferred "in" to and "out" from Amazon EC2, Amazon RDS, Amazon Redshift, Amazon DynamoDB Accelerator (DAX), and Amazon ElastiCache instances, Elastic Network Interfaces or VPC Peering connections across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction."
It's the amount of data where it makes more sense to put hard drives on a truck and drive across the country rather than send it over a network, where this becomes an issue (actually, probably a bit before then).
> Q: Can I export data from AWS with Snowmobile? > > Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary. Visit the Snowball Edge FAQs to learn more.
https://aws.amazon.com/snowmobile/faqs/?nc2=h_mo-lang
Why would they make it convenient to leave?
When you have enough data, that cost is quite significant.
I've heard some people here on HN say that it's slow, but I haven't noticed a difference. We're mainly dealing with multi-megabyte image files, so YMMV if you have a different workload.
I guess permissions might be more complex, as in EC2 instance profiles wouldnt grant access, etc.
I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere.
I personally dont care if the docs are crap because usually the first thing I do is update / actually write the docs to make them usable.
For a lot of techs though crap documentation is a deal breaker.
Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written. Quite frequently though its outsourced development agencies that refuse to write it because its "out of scope" and a "billable extra". Which I think is an egregious stance...doxs Should be part and parcel of the project. Mandatory.
If there is only one thing that juniors should learn about writing documentation (be it comments or design documents), it is this: document why something is there. If resources are limited, you can safely skip comments that describe how something works, because that information is also available in code.
(It might help to describe what is available, especially if code is spread out over multiple repositories, libraries, teams, etc.)
(Also, I suppose the comment I'm responding to could've been slightly more forgiving to GP, but that's another story.)
It's also completely against their interest to write docs as it makes their replacement easier.
That's why you need someone competent on the buying side to insist on the docs.
A lot of companies outsource because they don't have this competency themselves. So it's inevitable that this sort of thing happens and companies get locked in and can't replace their contractors, because they don't have any docs.
OK so the docs are in sync for a single point of time when you finish. Plus you get to have the context in your head (bus factor of 1, job security for you, bad for the org.)
How about if we just write clean infra configs/code, stick to well known systems like docker, ansible, k8s, etc.
Then we can make this infra code available to an on prem LLM and ask it questions as needed without it drifting out of sync overtime as your docs surely will.
Wrong documentation is worse than no documentation.
I can always guarantee a stream of consciousness one note that should have most of the important data, and a few docs about the most important parts. It's up to management if they want me to spend time turning that one note into actual robust documentation that is easily read.
Even with documentation on the hybrid setup, they'd need to get a new on-prem environment up and running (find a colo, buy machines, set up the network, blah blah).
If price is the only factor, your business model (or executives' decision-making) is questionable. Buy only the cheapest shit, spend your time building your own office chair rather than talking to a customer, you aren't making a premium product, and that means you're not differentiated.
But you are totally right it can be expensive. I worked with a startup that had some inefficient queries, normally it would matter, but with RDS it cost $3,000 a month for a tiny user base and not that much data (millions of rows at most).
RDS make perfect sense for them
That: and trust is hard earned over a long tail which is harder if you are trying to compete on price.
Small databases or test environment databases you can also leverage kubernetes to host an operator for that tiny DB. When it comes to serious data and it needs a beeline recovery strategy RDS.
Really it should be a mix self hosted for things you aren't afraid to break. Hosted for the things you put at high risk.
DynamoDB as a replacement, pay per request, was essentially free.
I found Dynamo foreign and rather ugly to code for initially, but am happy with the performance and especially price at the end.
Just to pay someone else enough money to provide the same service and make a profit while do it
It's like accounting and finance. Yeah a lot of companies use tax firms, but they all have finance and accounting in-house.
Amazon may have their own shipping fleet, most retailers of smaller scale pay someone else to do it for profit.
They need to replicate everything in multiple availability zones, which is going to be more expensive than replicating data centres.
They still need to test their cloud infrastracuture works.
Seems like a solid cost effective approach for when a company reaches a certain scale.
> Data is the most critical part of your infrastructure. You lose your network: that’s downtime. You lose your data: that’s a company ending event. The markup cost of using RDS (or any managed database) is worth it.
You need well-run, regularly tested, air gapped or otherwise immutable backups of your DB (and other critical biz data). Even if RDS was perfect, it still doesn't protect you from the things that backups protect you from.
After you have backups, the idea of paying enormous amounts for RDS in order to keep your company from ending is more far fetched.
Frankly, this is anti-competitive, and the FTC should look into it, however, Microsoft has been anti-competitive and customer hostile for decades, so if you're still using their products, you must have accepted the abuse already.
I haven't run a Postgres instance with proper backup and restore, but it doesn't seem like rocket science using barman or pgbackrest.
1. Hiring someone full time to work on the database means migrating off RDS
2. Database work is only about spend reduction
> Moving off JIRA onto linear
I don't get the hype. Linear is fine and all but I constantly find things I either can't or don't know how to do. How do I make different ticket types with different sets of fields? No clue.
> Not using Terraform Cloud No Regrets
I generally recommend Terraform Cloud - it's easy for you to grow your own in house system that works fine for a few years and gradually ends up costing you in the long run if you don't.
> GitHub actions for CI/CD Endorse-ish
Use Gitlab
> Datadog Regret
Strong disagree - it's easily the best monitoring/observability tool on the market by a wide margin.
Cost is the most common complaint and it's almost always from people who don't have it configured correctly (which to be fair Datadog makes it far too easy to misconfigure things and blow up costs).
> Pagerduty Endorse
Pagerduty charges like 10x what Opsgenie does and offers no better functionality.
When I had a contract renewal with Pagerduty I asked the sales rep what features they had that Opsgenie didn't.
He told me they're positioning themselves as the high end brand in the market.
Cool so I'm okay going generic brand for my incident reporting.
Every CFO should use this as a litmus test to understand if their CTO is financially prudent IMO.
So if that's the upgrade path you're going down I'd expect it to be fantastic.
Linear has been such a breath of fresh air, with such a solid desktop app (on Mac OS) that I don’t ever want to go back. Stuff happens instantly, the layout and semantics are an excellent “90% good enough” that I would happily relegate jira to only the most enterprise of enterprise projects.
There are lots of things where Jira falls short, but the pain points on an under-resourced self hosted instance of ten years ago are nothing like the ones you'll find on Jira cloud today.
It takes markdownish input but converts it to rich text as you type - so asterisk-space starts a bullet point list, etc.
I actually can't remember if it has a dedicated markdown mode anymore; the rich text editing supports the usual shortcuts that mean I tend to stick with it.
Given Atlassian bought OpsGenie in 2018, this either somewhere between quite late and unsurprising.
Anything Atlassian does is mostly quite late and its integration story is so pathetic that it's unsurprising.
Try to have a bitbucket pipeline that pushes to confluence. Seems like a basic integration to have, after all, Confluence has an API (well, actually it has 3 different ones) so surely Atlassian would make a basic thing like "publish a wiki page" a thing you get out of the box.
Nope.
I suppose it comes back to the comparative priorities (as evaluated by recurrent revenue) of ticking rfq boxes vs solving actual problems.
In fact, OpsGenie has mostly been on Auto-pilot for a few years now.
To me its whole schedule interface is atrocious for its price, given from an SRE/dev perspective, that's literally its purpose - scheduled escalations.
OpsGenie’s cheapest is $9 per user month but arbitrarily crippled, the plan anybody would want to use is $19 per user month
So instead of a factor of ten it’s ten percent cheaper. And i just kind of expect Atlassian to suck.
Datadog is ridiculously expensive and on several occasions I’ve run into problems where an obvious cause for an incident was hidden by bad behavior of datadog.
We use iOS “Critical Alerts” and similar on Android that breaks through any Do-Not-Disturb settings. https://heiioncall.com/blog/better-alerting-for-heii-on-call... Would you be willing to give that a shot? It wakes me every time :)
(It’s configurable too; we have vibrate-only or silenced modes. Think old-school beeper.)
In the rare case that it doesn’t wake you, we have configurable escalation strategies to alert someone else on your team after a configurable number of minutes.
I usually do not respond immediately to phone notifications, which I can handle async. Phone calls are by definition sync.
Heii On-Call will keep alerting you with these “Critical Alerts” until you’ve manually acknowledged. (Or until it escalates to a teammate and they acknowledge…)
And at least on my phone they sound nothing like normal phone notifications, which I personally always have on vibrate and/or DND anyway.
Give it a try and I think you’ll like it.
https://allquiet.app
With SMS, Phone Calls and Critical Alerts / DnD override.
We're 5 USD/user.
We try to build as close to our users as possible. Happy for any new try outs! :)
(I am co founder)
I loved Datadog 10 years ago when I joined a company that already used it where I never once had to think about pricing. It was at the top of my list when evaluating monitoring tools for my company last year, until I got to the costs. The pricing page itself made my head swim. I just couldn’t get behind subscribing to something with pricing that felt designed to be impossible to reason about, even if the software is best in class.
Their pricing setup is evil. Breaking out by SKUs and having 10+ SKUs is fine, trialing services with “spot” prices before committing to reserved capacity is also fine.
But (for some SKUs, at least) they make it really difficult to be confident that the reserved capacity you’re purchasing will cover your spot use cases. Then, they make you contact a sales rep to lower your reserved capacity.
It all feels designed to get you to pay the “spot” rate for as long as possible, and it’s not a good look.
I understand the pressures on their billing and sales teams that lead to these patterns, but they don’t align with their customers in the long term. I hope they clean up their act, because I agree they’re losing some set of customers over it.
Then they have things that I wanted to try for a long time, but... support doesn't care? Repeated "would you like to use this? / very likely, can we try it out? / (silence)". I love their product, but they are so annoying to deal with at the billing level.
I, quite literally, was griping to my Datadog CSM about this exact thing last week. They'll email me and be, "Oh, you know you're logging volume this month put you into on-demand indexing rates, right?" and my answer is always, "No, because your monitoring platform makes it nearly impossible for me to monitor it correctly."
You can't reference your contracted volume rates when building monitors out and the units for the metrics you need to watch don't match the units you contract with them on the SKU.
Maddening.
Are you referring to the `datadog.estimated_usage.logs.ingested_events` metric? It includes excluded events by default but you can get to your indexed volume by excluding excluded logs. `sum:datadog.estimated_usage.logs.ingested_events{datadog_index:*,datadog_is_excluded:false}.as_count()`
I'll give you a fun example. It's fresh in my mind because i just got reamed out about it this week.
In our last contract with DataDog, they convinced us to try out the CloudSIEM product, we put in a small $600/mo committment to it to try it out. Well, we never really set it up and it sat on autopilot for many months. We fell under our contract rate for it for almost a year.
Then last month we had some crazy stuff happen and we were spamming logs into DataDog for a variety of reasons. I knew I didn't want to pay for these billions of logs to be indexed, so I made an exclusion filter to keep them out of our log indexes so we didn't have a crazy bill for log indexing.
So our rep emailed me last week and said "Hey just a heads up you have $6,500 in on-demand costs for CloudSIEM, I hope that was expected". No, it was NOT expected. Turns out excluding logs from indexing does not exclude them from CloudSIEM. Fun fact, you will not find any documented way to exclude logs from CloudSIEM ingestion. It is technically possible, but only through their API and it isn't documented. Anyway, I didn't do or know this, so now i had $6,500 of on-demand costs plus $400-500 misc on-demand costs that I had to explain to the CTO.
I should mention my annual review/pay raise is also next week (I report to the CTO), so this will now be fresh in their mind for that experience.
Datadog on the other side... their "DD University" is a shame and we as paying customers are overwhelmed and with no real guidance. DD should assign some time for integration for new customers, even if it is proportional to what you pay annually. (I think I pay around 6000 usd annually.
Jira: Its overhyped and overpriced. Most HATE jira. I guess I don't care enough. I've never met a ticket system that I loved. Jira is fine. Its overly complex sure. But once you set it up, you don't need to change it very often. I don't love it, I don't hate it. No one ever got fired for choosing Jira, so it gets chosen. Welcome to the tech industry.
Terraform Cloud: The gains for Terraform Cloud are minimal. We just use Gitlab for running Terraform pipelines and have a super nice custom solution that we enjoy. It wasn't that hard to do either. We maintain state files remotely in S3 with versioning for the rare cases when we need to restore a foobar'd statefile. Honestly I like having Terraform pipelines in the same place as the code and pipelines for other things.
GitHub Actions: Yeah switch to GitLab. I used to like Github Actions until I moved to a company with Gitlab and it is best in class, full stop. I could rave about Gitlab for hours. I will evangelize for Gitlab anywhere I go that is using anything else.
DataDog: As mentioned, DataDog is the best monitoring and observability solution out there. The only reason NOT to use it is the cost. It is absurdly expensive. Yes, truly expensive. I really hate how expensive it is. But luckily I work somewhere that lets us have it and its amazing.
Pagerduty: Agree, switch to OpsGenie. Opsgenie is considerably cheaper and does all the pager stuff of Pager duty. All the stuff that PagerDuty tries to tack on top to justify its cost is stuff you don't need. OpsGenie does all the stuff you need. Its fine. Similar to Jira, its not something anyone wants anyway. No ones going to love it, no one loves being on call. So just save money with OpsGenie. If you're going to fight for the "brand name" of something, fight for DataDog instead, not a cooler pager system.
So on the standard tech hype cycle, that sounds about right.
- It's fast. It's wild that this is a selling point, but it's actually a huge deal. JIRA and so many other tools like it are as slow as molasses. Speed is honestly the biggest feature.
- It looks pretty. If your team is going to spend time there, this will end up affecting productivity.
- It has a decent degree of customization and an API. We've automated tickets moving across columns whenever something gets started, a PR is up for review, when a change is merged, when it's deployed to beta, and when it's deployed to prod. We've even built our own CLI tools for being able to action on Linear without leaving your shell.
- It has a lot of keyboard shortcuts for power users.
- It's well featured. You get teams, triaging, sprints (cycles), backlog, project management, custom views that are shareable, roadmaps, etc...
Datadog's cheapest pricing is $15/host/month. I believe that is based on the largest sustained peak usage you have.
We run spot instances on AWS for machine learning workflows. A lot of them if we're training and none otherwise. Usually we're using zero. Using DataDog at it's lowest price would basically double the cost of those instances.
You're staying within an ecosystem you know and it seems to offer almost all of the necessary functionality
Getting them to use Github/Gitlab is an argument I've never won. Typically it goes the other way and I end up needing to maintain a Monday or Airtable instance in addition to my ticketing system.
For folks running k8 at any sort of scale, I generally recommend aggregating metrics BEFORE sending them to datadog, either on a per deployment or per cluster level. Individual host metrics tend to also matter less once you have a large fleet.
You can use opensource tools like veneur (https://github.com/stripe/veneur) to do this. And if you don't want to set this up yourself, third party services like Nimbus (https://nimbus.dev/) can do this for you automatically (note that this is currently a preview feature). Disclaimer also that I'm the founder of Nimbus (we help companies cut datadog costs by over 60%) and have a dog in this fight.
I'll be dead in the ground before I use TFC. 10 cents per resource per month my ass. We have around 100k~ resources at an early-stage startup I'm at, our AWS bill is $50~/mo and TFC wants to charge me $10k/mo for that? We can hire a senior dev to maintain an in-house tool full time for that much.
I wonder how much one should pay attention to future problems at the start of a startup versus "move fast and break things." Some of this stuff might just put you off finishing.
As mentioned in the other comment, the most commonly used providers for terraform are "bridged" to pulumi, so the maturity is nearly identical to Terraform. I don't really use Pulumi's pre-built modules (crossroads), but I don't find I've ever missed them.
I really like both Pulumi and Terraform (which I also used in production for hundreds of modules for a few years), which it seems like isn't always a popular opinion on HN, but I have and you absolutely can run either tool in production just fine.
My slight preference is for Pulumi because I get slightly more willing assistance from devs on our team to reach in and change something in infra-land if they need to while working on app code.
We do still use some Pulumi and some Terraform, and they play really nicely together: https://transcend.io/blog/use-terraform-pulumi-together-migr...
Infrastructure should be declared, not coded.
Say what you want. The tool then builds that, or changes whats there to match.
I've tried Pulumi and understanding the bit that runs before it tries to do stuff and the bit that runs after it tries to do stuff and working out where the bugs are is a PITA. It lulls you into a false sense of security that you can refer to your own variables in code, but that doesn't get carried over to when it is actually running the plan on the cloud service (ie actually creating the infrastructure) because you can only refer to the outputs of other infrastructure.
CFN is too far in the other direction, primarily because it's completely invisible and hard to debug.
Terraform has enough programmability (eg for_each, for-expressions etc) that you can write "here is what I want and how the things link together" and terraform will work out how to do it.
The language is... sometimes painful, but it works.
The provider support is unmatched and the modules are of reasonable quality.
I keep wondering when this is going to show up. We have a lot of service providers, but even more frameworks, and every vendor seems to have their own bespoke API.
That being said, Cloudflare is on the path to offering a great GPU FaaS system for inference.
I believe it’s still in beta, but it’s the most promising option at the moment.
https://fluxcd.io/blog/2022/11/flux-is-a-cncf-graduated-proj...
> Weaveworks will be closing its doors and shutting down commercial operations > Alexis Richardson, 5 Feb 2024
https://www.linkedin.com/posts/richardsonalexis_hi-everyone-...
If the project has legs, it's now under CNCF.
Is the project future at risk? https://github.com/fluxcd/flux2/discussions/4544
1) We don't have any software, so we don't have a prod environment.
2) We have 1 team that makes 1 thing, so we just launch it out of systemd.
3) We have between 2 and 1000 teams that make things and want to self-manage when stuff gets rolled out.
Kubernetes is case 3. Like it or not, teams that don't coordinate with each other is how startups scale, just like big companies. You will never find a director of engineering that says "nah, let's just have one giant team and one giant codebase".
Kubernetes is appealing to many, but it is not 100% frictionless. There are upgrades to manage, control plane limits, leaky abstractions, different APIs from your cloud provider, different RBAC, and other things you might prefer to avoid. It's its own little world on top of whatever world you happen to be running your foundational infrastructure on.
Or, as someone has artistically expressed it: https://blog.palark.com/wp-content/uploads/2022/05/kubernete...
I have seen people rewrite Kubernetes in CloudFormation. You can do it! But it certainly isn't problem-free.
You’re right that if you use a cloud provider, IAM is something that has to be reckoned with. But the question is, how many implementations of IAM and policy mechanisms do I want to deal with?
Also, you can bundle your load balancer config and application config together. No written description of the load balancer config + an RPM file to a disinterested different team.
https://github.com/martinvonz/jj https://github.com/facebook/sapling
I see Kubernetes as one time (mental and time) investment that buys me somehow smoother sailing plus some other benefits.
Of course it is not all rainbows and unicorns. Having a single nginx server for a single /static directory would be my dream instead of MinIO and such.
I have spent vanishingly close to 0 hours on maintaining our (managed) kubernetes clusters in work over the past 3 years, and if I didn't show up tomorrow my replacement would be fine.
Admittedly, I was afraid of ever restarting as I wasn’t sure it would reboot. But still…
For that, you get automated backups, very simple read proxies, managed updates of you ever need them. You can vertically scale down, or uo to the point of "it's cheaper to hire a DBA to fix this".
We use this for our internal services at work, and the last time I touched the infra was in 2022 according to git
[0] https://github.com/gin-gonic/gin
[1] https://gist.github.com/donalmacc/0efbb0b377533232da3f776c60....
[2] https://docs.digitalocean.com/products/kubernetes/how-to/dep...
You also need to pay them which is an event.
- I want to spin up multiple redundant instances of some set of services
- I want to load balance over those services
- I want some form of rolling deploy so that I don’t have downtime when I deploy
- I want some form of declarative infrastructure, not click-ops
Given these requirements, I can’t think of an alternative to managed k8s that isn’t more complex.
Running off a couple of medium ( $3k/month each range ) RDS databases with failover setup. ECS for apps.
Databases looked after themselves. The senior people probably spent 20% of a FTE on stuff like optimizing it when load crept up.
Place before that was a similar size and no DBA either. People just muddled though.
My company uses redundant services because we like to deploy frequently, and our customers notice if our API breaks while the service is restarted. Running the service redundantly allows us to do rolling deploys while continuing to serve our API. It’s also saved us from downtime when a service encounters a weird code path and crashes.
Many say in the database world, "use Postgres", or "use sqlite." Similarly there are those databases that are robust that no one has heard of, but are very limited like FoundationDB. Or things that are specialized and generally respected like Clickhouse.
What are the equivalents of above for Kubernetes?
Kubernetes is probably “use postgres”
It's just that, you should start with a handful of backed-up pet servers. Then manually automate their deployment when you need it. And only then go for a tool that abstracts the automated deployment when you need it.
But I fear the simplest option on the Kubernetes area is Kubernetes.
I shunned k8s for a long time because of the complexity, but the managed options are so much easier to use and deploy than pet servers that I can’t justify it any more. For anything other than truly trivial cases, IMO kubernetes or (or similar, like nomad) is easier than any alternative.
The stack I use is hosted Postgres and VKS from Vultr. It’s been rock solid for me, and the entire infrastructure can be stored in code.
I also stayed away for a long time due to all the fear spread here, after taking the leap, I’m not looking back.
The lightweight “simpler” alternative is docker-compose. I put simpler in quotes because once you factor in all the auxiliary software needed to operate the compose files in a professional way (IaC, Ansible, monitoring, auth, VM provisioning, ...), you will accumulate the same complexity yourself, only difference is you are doing it with tools that may be more familiar to what you are used to. Kubernetes gives you a single point of control plane for all this. Does it come with a learning curve? Yes, but once you get over it there is nothing inherent about it that makes it unnecessary complex. You don’t need autoscaler, replicasets and those more advanced features just because you are on k8s.
If you want to go even simpler, the clouds have offerings to just run a container, serverless, no fuzz around. I have to warn everyone though that using ACI on Azure was the biggest mistake of my career. Conceptually it sounds like a good idea but Azures execution of it is just a joke. Updating a very small container image taking upwards of 20-30 minutes, no logs on startup crashes, randomly stops serving traffic, bad integration with storage.
We adopted TFC at the start of 2023 and it was problematic right from the start; stability issues, unforeseen limitations, and general jankiness. I have no regrets about moving us away from local execution, but Terraform Cloud was a terrible provider.
When they announced their pricing changes, the bill for our team of 5 engineers would have been roughly 20x, and more than hiring an engineer to literally sit there all day just running it manually. No idea what they’re thinking, apart from hoping their move away from open source would lock people in?
We ended up moving to Scalr, and although it hasn’t been a long time, I can’t speak highly enough of them so far. Support was amazing throughout our evaluation and migration, and where we’ve hit limits or blockers, they’ve worked with us to clear them very quickly.
My contrarian view is that EC2 + ASG is so pleasant to use. It’s just conceptually simple: I launch an image into an ASG, and configure my autoscale policies. There are very few things to worry about. On the other hand, using k8s has always been a big deal. We built a whole team to manage k8s. We introduce dozens of concepts of k8s or spend person-years on “platform engineering” to hide k8s concepts. We publish guidelines and sdks and all kinds of validators so people can use k8s “properly”. And we still write 10s of thousands lines of YAML plus 10s of thousands of code to implement an operator. Sometimes I wonder if k8s is too intrusive.
But in general k8s provides incredibly solid abstractions for building portable, rigorously available services. Nothing quite compares. It's felt very stable over the past few years.
Sure, EC2 is incredibly stable, but I don't always do business on Amazon.
Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need ISO/PCI/whatever.
Who exactly needs millions of lines of code?
And in the end it's hard to say if you've actually gained anything except now this different code manages your AWS resources like you were doing with CF or terraform.
How many LOCs in the linux kernel again?
I don't know what you have been doing with Kubernetes, but I run a few web apps out of my own Kubernetes cluster and the full extent of my lines of code are the two dozen or so LoC kustomize scripts I use to run each app.
0 https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne...
It's really not about what I do and do not do with Kubernetes. It's on you to justify your "millions upon millions lines of code" claim because it is so outlandish and detached from reality that it says more about your work than about Kubernetes.
I repeat: I only need a few dozen lines of kustomize scripts to release whole web apps. Simple code. Easy peasy. What mess are you doing to require "millions upon millions" lines of code?
Sometimes I think that managed kubernetes services like EKS are the epitome of "give the customers what they want", even when it makes absolutely no sense at all.
Kubernetes is about stitching together COTS hardware to turn it into a cluster where you can deploy applications. If you do not need to stitch together COTS hardware, you have already far better tools available to get your app running. You don't need to know or care in which node your app is suppose to run and not run, what's your ingress control, if you need to evict nodes, etc. You have container images, you want to run containers out of them, you want them to scale a certain way, etc.
Lifting and shifting an "EC2 + ASG" set-up to Kubernetes is a straightforward process unless your app is doing something very non-standard. It maps to a Deployment in most cases.
The fact that you even implemented an operator (a very advanced use-case in Kubernetes) strongly suggests to me that you're doing way more than just lifting and shifting your existing set-up. Is it a surprise then that you're seeing so much more complexity?
Why not dump your application server and dependencies into rented data center (or EC2 if you must) and setup a coarse DR? Maybe start with a monolith in PHP or Rails.
None of that word salad sounds like startup to me, but then again everyone loves to refer to themselves as a startup (must be a recruiting tool?), so perhaps muh dude is spot on.
Right now I'm engineer No. 1 at a current startup just doing DDD with a Django monolith. I'm still pretty Jr. and I'm wondering if there's a way to scale without needing to get into all of the things the author of this article mentions. Is it possible to get to a $100M valuation without needing all of this extra stuff? I realize it varies from business to business, but if anyone has examples of successes where people just used simple architecture's I'd appreciate it.
If you're looking for successful businesses, indie hackers like levelsio show you how far you can get with very simple architectures. But that's solo dev work - once you have a team and are dealing with larger-scale data, things like infrastructure as code, orchestration, and observability become important. Kubernetes may or may not be essential depending on what you're building; it seems good for AI companies, though.
If you're primarily building a web app, a monolith is fine for quite a while, I think. But a lot of the stuff in the post is still relevant even for monoliths - RDS, Redis, ECR, terraform, pagerduty, monitoring/observability.
That said, anything that's set-and-forget is great to start with. Anything that requires it's own care and feeding can wait unless it's really critical. I think we have a project each quarter to optimize our datadog costs and renegotiate our contract.
Also if you make microservices, you are going to need a ton of tools.
If you are running a marketplace app and collect fees you're going to be able go much further on simpler architectures than if you're trying to generate 10,000 AI images per second.
The things I did to get here are honestly kind of stupid. I started out at a defense contractor after graduating and left in the first six months because all the software devs were jumping ship. Went to a small business defense contractor (yep that's a thing) and learned to build web apps with React and Django. Then the pace of business slowed so after about 18 months I got on the Leetcode grind and got into a FAANG. Realized I hated it, so I quit after about 9 months with no job lined up.
While unemployed I convinced myself I was going to get a job in robotics (I actually got pretty close, I had 3 final level interviews with robotics companies), but the job market went to shit pretty much the exact day I quit my job lol. I spent about 6 months just learning ROS, Inverse Kinematics, math for robotics, gradient descent and optimization, localization, path planning, mapping etc. I taught at a game development summer camp for a month and a half, that was awesome. Working with kids is always a blast. Also learned Rust and built a prototype for a multiplayer browser-based coding game I had been thinking about for a while. It was an excuse to make a full stack application with some fun infrastructure stuff.
https://ai-arena.com/#/multiplayer
The backend is no longer running, but originally users could see their territory on the galaxy grow as their code won battles for them.
For the current role, I really just got lucky. The previous engineer was on his way out for non-job related reasons. He had read a lot of the books I had (Code Complete, Domain Driven Design) and I think we just connected over shared interests and intellectual curiosity.
I think that in the modern day, so many people are really just in this space for the paycheck-- and that's okay! Everyone needs to make a living. But I think that if you have that intellectual curiosity and like making stuff, people will see that and get excited. It ends up being a blessing and a curse.
I have failed interviews because of honesty "I would Google the names of books and read up on that subject" or "I think if I was doing CSS then I would be in the wrong role" (I realize how douchey that sounds but I just was not meant to design things, I have tried). But I have also gone further in interviews than I should have because I was really engrossed in a particular problem like path planning or inverse kinematics and I was able to talk about things in plain terms.
I think it's easier to learn things quickly if they are something you're actually interested in, it becomes effortless. Basically I just try to do that so I can learn optimally, then I try to get lucky.
EDIT: Oh I just thought of more good advice. Find senior devs to learn from. They can be kind of grumpy in their online presence, but they help you avoid so many tar pits. I am in a Discord channel with a handful of senior engineers. The best way to get feedback is to naively say "I'm going to do X", they will immediately let you know why X is a bad idea. A lot of their advice boils down to KISS and use languages with strong typing.
In my case it took a good five years and a couple job hops to rebalance. But eventually you get back to a reasonable tech leadership role and back to making some of the bigger decisions to help make the junior devs' lives less miserable.
No regrets, but the five years it takes to rebalance can be pretty hard.
After realizing that, I decided I'd try as hard as I possibly could to never have to work at a job that I didn't like. I already didn't want kids so that part is easy. The other part of the equation is saving lots of money. I'm not an ascetic by any means, but I live well below my means on a SWE salary which means I can save quite a bit of money each year.
I also recognize that not wanting to go corporate severely limits my options down the line. But capitalism is all about making money for other people. If I can make someone a lot of money, they're not going to care about if I have the chops to stand up a Kubernetes cluster or write a Next.js app or whatever (I hope).
I don't think I'm super smart, I'd say I'm pretty average for this line of work. But I reckon that most SWEs are focused on learning new technologies to get to their next job, or are overly concerned with technical problems. I like to think that I am pragmatic enough about only doing things that are going to deliver business value to make up for being average in smarts.
Anyways, there's not really a point to this rant. These are just some thoughts I have had about optimizing my career for my own happiness, and how I hope I can stay a hot commodity even though I hate working in the cloud and my software skills aren't bleeding edge.
One thing though, I'd start with go. It's no more complex than python, more efficient, and most importantly IMO since it compiles down to binary it's easier to build, deploy, share, etc. And there's less divergence in the ecosystem; generally one simple way to do things like building and packaging, etc. I've not had to deal with versions or tooling or environmental stuff nearly as much since switching.
Things were more far more manual and much less secure, scalable and reliable in the past, but they were also far far simpler.
With all of that complexity/word salad from TFA, where’s the value delivered? Presumably there’s a product somewhere under all that infrastructure, but damn, what’s left to spend on it after all the infrastructure variable costs?
I get it’s a list of preferences, but still once you’ve got your selection that’s still a ton of crap to pay for and deal with.
Do we ever seek simplicity in software engineering products?
Cloud and SaaS tools are very seductive, but I think they're ultimately a trap. Keep your tools simple and just run them yourselves, it's not that hard.
Doubtfully. Simplicity of work breakdown structure - maybe. Legibility for management layers, possibly. Structural integrity of your CYA armor? 100%.
The half-life of a software project is what now, a few years at most these days? Months, in webdev? Why build something that is robust, durable, efficient, make all the correct engineering choices, where you can instead race ahead with a series of "nobody ever got fired for using ${current hot cloud thing}" choices, not worrying at all about rapidly expanding pile of tech and organizational debt? If you push the repayment time far back enough, your project will likely be dead by then anyway (win), or acquired by a greater fool (BIG WIN) - either way, you're not cleaning up anything.
Nobody wants to stay attached to a project these days anyway.
/s
Maybe.
Either because they only know how to manage AWS instances (it was the hotness and thats what all the blogs and YT videos were about) and are now terrified from losing their jobs if the companies switch stacks. Or because they needed to put the new thing on their CV so they remain employable. Also maybe because they had to get that promotion and bonus for doing hard things and migrating things. Or because they were pressured into by bean counters which were pressured by the geniuses of Wall Street to move capex to opex.
In any case, this isn't by necessity these days. This is because, for a massive amount of engineers, that's the only way they know how to do things and after the gold rush of high pay, there's not many engineers around that are in it to learn or do things better. It's for the paycheck.
It is what it is. The actual reality of engineering the products well doesn't come close to the work being done by the people carrying that fancy superstar engineer title.
You know the old adage "fast, cheap, good: pick two"? With startups, you're forced to pick fast. You're still probably not gonna make it, but if you don't build fast, you definitely won't.
There's an easy bent towards designing everything for scale. It's optimistic. It's feels good. It's safe, defendable, and sound to argue that this complexity, cost, and deep dependency is warranted when your product is surely on the verge of changing the course of humanity.
The reality is your SaaS platform for ethically sourced, vegan dog food is below inconsequential and the few users that you do have (and may positively affect) absolutely do not not need this tower of abstraction to run.
For 99% of businesses it's a wasteful, massive overkill expense. You dont NEED all the shiny tools they offer, they don't add anything to your business but cost. Unless you're a Netflix or an Apple who needs massive global content distribution and processing services theres a good chance you're throwing money away.
The “control plane” was ZooKeeper. Everything had bindings to it, Thrift/Protobuf goes in a znode fine. List of servers for FooService? znode.
The packaging system was a little more complicated than a tarball, but it was spiritually a tarball.
Static link everything. Dependency hell: gone. Docker: redundant.
The deployment pipeline used hypershell to drop the packages and kick the processes over.
There were hundreds of services and dozens of clusters of them, but every single one was a service because it needed a different SKU (read: instance type), or needed to be in Java or C++, or some engineering reason. If it didn’t have a real reason, it goes in the monolith.
This was dramatically less painful than any of the two dozen server type shops I’ve consulted for using kube and shit. It’s not that I can’t use Kubernetes, I know the k9s shortcuts blindfolded. But it’s no fun. And pros built these deployments and did it well, serious Kubernetes people can do everything right and it’s complicated.
After 4 years of hundreds of elite SWEs and PEs (SRE) building a Borg-alike, we’d hit parity with the bash and ZK stuff. And it ultimately got to be a clear win.
But we had an engineering reason to use containers: we were on bare metal, containers can make a lot of sense on bare metal.
In a hyperscaler that has a zillion SKUs on-demand? Kubernetes/Docker/OCI/runc/blah is the friggin Bezos tax. You’re already virtualized!
Some of the new stuff is hot shit, I’m glad I don’t ssh into prod boxes anymore, let alone run a command on 10k at the same time. I’m glad there are good UIs for fleet management in the browser and TUI/CLI, and stuff like TailScale where mortals can do some network stuff without a guaranteed zero day. I’m glad there are layers on top of lock servers for service discovery now. There’s a lot to keep from the last ten years.
But this yo dawg I heard you like virtual containers in your virtual machines so you can virtualize while you virtualize shit is overdue for its CORBA/XML/microservice/many-many-many repos moment.
You want reproducibility. Statically link. Save Docker for a CI/CD SaaS or something.
You want pros handing the datacenter because pets are for petting: pay the EC2 markup.
You can’t take risks with customer data: RDS is a very sane place to splurge.
Half this stuff is awesome, let’s keep it. The other half is job security and AWS profits.
that would have been around the time when containers entered the public/developer consciousness, no?
There is no way one person can thoroughly understand so many complex pieces of technology. I have worked for 10 years more or less at this point, and I would only call myself confident on 5 technical products, maybe 10 if I being generous to myself.
[1] https://github.com/Azure/karpenter-provider-azure/pull/72
(source: On the team that is developing the provider)