Curious to know how to which degree they are self hosting, seems they'll buy their own servers but I doubt they'll go all the way to buying land and building their own data centers?
When I was there, they rented space in a DC outside Chicago (near the airport), and later near NY for a backup location. The DC in Chicago was fantastic. We had a whole load of racks (about a row in the DC), and we bought our own servers. Running our own DC would have crossed the line, it's a whole huge thing you need to do and it only makes sense at scale if you want to do it well, Basecamp used a tiny fraction of the DC capacity.
A whole load of Racks for running Basecamp? We are talking about 42U per Rack, and total of 420U of Servers?
The scale seems quite massive. At least to the idea / perception of what I had about Basecamp. Would be nice to see those specification and see how much of an improvement it is 10 years later and if we could fit those into 2-3 Racks.
Basecamp started on one single Rackspace server, before I was there. I started at 20 people, left at ~55.
When I left there were I don't remember how many racks exactly, but more than 10, less than 30 in the primary location. 42U of servers in each. There was a mix of a whole load of (Dell, never pay list price!) blade servers, DB appliances (~12 in total across ~6 apps ISTR), Isilon storage[0][1], F5 kit, juniper routers etc. etc. We had some epically fast storage in some of the servers for the time, way faster than SSDs.
Later we added two more sites. One in I think Virginia, one in NY. The one in Virginia was a replica of what we needed to run Basecamp, the one in NY was a half-rack data replication location (I think I got that the right way round). We had 10G fibre (we rented wavelengths not the whole fibre) between each location. We could lose one DC and remain RW for our block data, 2 DCs and we'd have to drop down to RO. Block data was things like uploads, so DBs, search etc. wouldn't have been affected. We could lose one of the /main/ DCs and still be RW for everything.
With all this kit we were able to run both main DCs hot. With our Geo DNS you could hit either of our DCs and you'd get served pages, you could even write to both locations. One DC was always the "RO" DC, it always replicated the databases. If you tried to write to that DC we proxied your request to the RW DC over our 10G links and proxied any more requests you made for n seconds to the RW DC too, at which point we reverted you back to the RO DC.
Now, NY to Virginia isn't that far, so why bother with the hot/hot config? Because it played into the rest of our plan, which was DC failover. With some pretty epic voodoo (Juniper/F5/OpenResty etc.) we could fail over the datacentres, swapping the RO and RW locations. We could also do this if one of the locations was unavailable. We could do this in 4 seconds /without losing a single in-flight request/ (we tested it).
This ended up a bit longer than I was intending, but it illustrates a few things:
- I think people underestimate Basecamp. It's /huge/ (money and users, not employees). Not so much in the tech world (anymore), but even with this kit, even with the (at the time 6) sysadmins that maintained it, it still made a shit load of cash. I guestimated the net-worth of the two owners as in the hundreds of millions of dollars each, entirely because of basecamp. Cash that as a privately owned company all went to the owners (who gave some of it to us, they treated us fairly well). I think it was Patio11 who said that people under estimated the market for software, it's easy to do, these aren't human-scale numbers.
- The owners are right about keeping it private. You might not make "larger yacht than Larry Ellison" money, but you sure as hell have a lot higher chance of making "pretty damn big yacht and not having to work again" sort of money. If I was rolling the dice I know which I'd gamble on.
- I don't think I ever actually calculated it, but the amount of money our servers were worth while running was /immense/. When you went to the DC and looked at the row of racks, you could kinda see just how dense the $ value there was. The efficiencies that the cloud brings gives money to AWS, not to the clients.
- All of this was still /way/ cheaper than running cloud infra. I'm not against the cloud (I use it), but we made more because we had this infra than if we were on the cloud. We were dense when it came to # customers per $ spent on infra.
- The flexibility we had because we controlled everything was also extraordinary. It came at a cost, we had to do it ourselves, but we had a /lot/ of power to make things work the way we wanted to.
- We had access to everything, from the network (hell, even the light :) ) to the JS. This gave us optimisation options not available to a lot of people. Networ...
That is about the amazon.com architecture as late as 2006 (only slightly higher scale of three entire datacenters scattered around virginia and I think dual 40G links between datacenters, but other than that exactly the same principles).
Lot of people think that DC failover needs to be east coast / west coast but you can achieve most of your DC redundancy goals separated by 100 miles or less and have a lot lower latency, higher bandwidth and lower costs. Might want to think about different geographic flood plains and different power companies / grids.
Could still nerd fight about EMPs from nuclear war and a sufficiently massive hurricane or an earthquake out here on the west coast, but at some point you need to accept some risks.
The reason was that I remember DHH stating they were doing 2K RPS with 30 App Server in 2015 ~ 2016. And they were on one Primary DB. ( At least that was what I jot down in my notes ). I was assuming they could fit multiple "App Server" Node inside a single 1U Blade. But even if it was 1U per App Server, that would be 30U + likely a Powerful 4U DB Monster. Along with probably some Cache instances.
Even if the number above does not include redundancy, that still only makes it 2 Racks of Servers with spare.
30 Racks is a lot. What am I missing here? Apart from Storage.
The place I work (CTO for now, looking for opportunities pretty soon) can sustain nearly 400 RPS through a Rails app on a single Performance-L Heroku dyno without a sweat (though we run two min), but it's the tip of the iceberg. The infra to support those web servers is way more than that. Aurora Postgres x 3 for now, a large ES cluster, 2x redis clusters, memcached etc.
Bear in mind that we had I think 6 apps. Basecamp 1, 2 and 3 (all separate infra), Highrise, Campfire, some other internal stuff. Our ES cluster was pretty damn big, redis and memcached too. Juniper, F5, network switches (rack infra) etc. Storage was pretty big, quite a few 4u servers with spinning rust. The blade servers were I think 6 blades in 2u.
Everything was redundant, everything had 2x or more. I honestly can't remember how many racks we had. "More than 10" is my hand-wavey guess.
I don't think DHH was being disingenuous with his "2k on 30 servers" message, it was likely more about the scalability of Rails rather than the infra required to run the app.
Who told you they need to build a massive data center? They are NOT RENTING SERVERS but buying space for the ones they use.
With the budget they can afford to buy super beefy rack servers that occupy no space. With a single rack they can have a ton of late tech servers.
The bulk of the media is served from appliances they give out to ISPs, so their AWS costs might be reasonable.
It's also a huge company and a massive distributed system, so moving that off from AWS is an insane amount of effort and there might be legitimate benefits from AWS' management and autoscaling layer that aren't as valuable for 37Signals which runs simpler, monolithic systems.
Additionally, Netflix falls into one category of website DHH writes about: unpredictable, significant bursts of traffic. When they publish Tiger King during the pandemic there’s a spike in streaming no one could have predicted. When people finish the show it goes away. The arrival time of the next breakout show is likely uncertain, even if it can be modeled using a statistical distribution.
I worked at Basecamp/37signals up until 2015. We had peaks and troughs, with as you might predict traffic ramping at ET morning, and tailing off PT afternoon/evening. We had some European traffic, but it was substantially less than the US.
It still cost /far/ less to buy/rack/power etc. our servers 100% of the time than it would have done to use the cloud and scale up and down with demand.
There is definitely a situation when bursty requirements will make cloud the more obvious answer, but given the near order of magnitude cost increase for cloud it's not as clear cut as just seeing a wave on a graph :)
2. Content Encoding are not done at a regular interval. And most of their work on ML and other quality requirements does not require fixed interval either. i.e They are schedule to work in Spot when AWS price are at its lowest.
Very interesting post! I think there will be a movement away from the cloud in the coming time. People will realize that the cloud comes with it's problems when regulations come down, governments require companies to spy on them etc.
The only way of being really sure of what is going on is to own it yourself. Unfortunately that has become more and more important.
I think there will be more competition for specific types of cloud needs. AWS, GCP, and Azure offer every service you could ever need in one place. At the other extreme, you can pay a colo facility to rack the servers you own. I think the in-between market has the potential to grow. Cheaper than AWS but without all the drawbacks of true self-hosting.
Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host.
Their RDS and ES are 500k. They still need to pay for EC2, S3 and any other services they might be using, like ELB and Elasticache. So probably higher than that.
He also did not mention if they are going to self-host, colocate, or rent bare metal in a more traditional host (like softlayer). Probably colocate as I think that’s how they used to run?
Very important point. There are plenty of skilled people who live in places that don't require that kind of salary in order to afford a decent place to live. Why not hire a guy in Kansas to remote manage your infrastructure?
Silicon Valley culture isn't nearly so critical or valuable that your organization requires it in order to do the necessary work.
Yeah... I don't agree at all with their conclusions about running your own stuff "not that complicated".
Sure, it is not "that complicated" for basic stuff, simple sites, hobby projects or overall quick'n dirty solutions because you're under a budget, as with everything.
Doing infrastructure properly, with SLAs and a very high level of availability and redundancy (both in systems, and in people with the know-how) is incredibly complicated in my opinion.
I bet what they're going to save on the the Cloud provider is going to be spent (and probably even more) in salaries, if they keep the same levels of service.
Specially true considering they pay pretty good salaries.
Managing AWS services also requires staffing, so most likely these would be the same people in a small organization.
Self hosting in my experience can be very reliable if you over provision redundant enterprise grade hardware, once it’s running you can get 6-10 years of usable service life. The hardware cost itself is negligible even more so when amortized over a long life cycle.
> Hire one person to buy, configure, manage hardware is what?
This point gets raised every single time bare-metal comes up for discussion and at this point I'm not sure if people are legitimately unaware or if there are hidden agendas at play.
There are plenty of providers out there who will handle the logistics for you with an interface similar to AWS EC2 - ask for server and get SSH access in minutes/days/weeks (depending if it's an off-the-shelf or custom build).
From there, your existing DevOps guy (which you must have to navigate AWS' complexity) can take over.
It's only at either extremely large scale or very unusual requirements that managing your own hardware makes sense. Until then, you will never be able to beat the economies of scale of managed bare-metal providers.
> There are plenty of providers out there who will handle the logistics for you with an interface similar to AWS EC2...
I am sorry but how's it different from leaving BigCloud for SmallCloud? That's like saying, I'd no longer use Java/Kotlin because Oracle/Google but I'll however use Clojure because JVM.
They [believe that they] don't need to hire anyone. They already have a staff of ops folks that run their cloud operations. And those folks will run their on-prem operations.
Also, Hardware isn't as hard as people make out, you install it once and it runs; if it fails there's a special out-of-band manager built in so you can remotely debug it, datacenters have remote-hands personel.
If you're constantly installing stuff then you're scaling up a lot, and that's the level where it would make sense to have someone hired dedicated for it.
You can also rent hardware.
the thing is: cloud ops are also quite complex but it's a different kind of complexity. I don't know how many people are required.
The largest teams I've been on have been cloud deployments, and some of the smallest teams were bare-metal; but that's just an anecdote.
> Also, Hardware isn't as hard as people make out, you install it once and it runs; if it fails there's a special out-of-band manager built in so you can remotely debug it, datacenters have remote-hands personel.
If you're "cloud native" you've already architected your stack to be horizontally-scalable and distributed (partly because the cloud is so expensive that running at peak load capability all the time would bankrupt you) so hardware failure should matter even less.
MSB don't pay $250k/yr. For $100k/yr they will find someone in USA who will enjoy racking servers, wiring, configuring networks, installing OS, etc. If they look outside of US, they will find it for cheap.
If you run anything non trivial in a cloud, chances are that one person will still be cheaper than the one or more person cloud operations team that you have now.
"Hire one person to buy, configure, manage hardware is what? $250k/year? "
Outside of expensive Silicon Valley and FAANGs, where do you get this $250k/Year ? I think most of us on HN are normalizing this whole "devs make 250k/year" thing when in reality, it is a tiny percentage of people. Tiny when you look at the entire ecosystem.
> Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS?
A company which has a total AWS bill of $500k/yr is a very small one (the article says 500K is just the RDS/ES bill, so total is surely much higher for them).
Our (tiny startup) AWS bill is ~200K/yr and the amount of compute and storage we do could very easily be handled by one single physical server.
Now, sure, if we went to self-hosted we'd split that into multiple servers for redundancy and growth headroom, but in any case definitely less than a dozen boxes.
You don't hire anyone to install and manage the hardware at that scale. Someone from the ops team goes to the colo for a day or two to set it all up and then you only need to go back when something needs replacement, which at that small scale is going to be very rarely.
That's exactly what we did at a previous startup and the company grew to many, many hundreds of people before there was so much infrastructure that it became worth thinking about hiring a dedicated person. By that time you'd be in the 10s of millions in AWS spend.
Also nowadays hardware as well as software are incredibly reliable. You can basically set up a Debian web + postgres server with automatic updates enabled and not need to even log into it for months and months.
Well said! It's amazing how much power you can get in a $10,000 server. If you have a simple stack and consistent demand, self hosting makes a ton of sense.
WIth 500K yearly you can do a lot!! Including renting 3 office spaces in multiple locations in the US and using them as data centers. Offices cost about $1000 - $3000 yearly, he doesn't need a premium space in a big building but for $500K yearly you can do a lot!
In 2 years you would have spent 1M. That is more than enough to have a dedicated devops that travels to all the locations and setup the super beefy servers you can buy with that budget.
Ok but 500k is the whole budget for location, backup location, energy, devops people, operation people, etc.. I run part of our infra on-premise, and part of on AWS.. in the summer, we had issues with our air-conditioner (europe) and we had to turn-off partially our service, now in Winter because of the raising costs, we are discussing again what to do with the on-premise services, employees getting ill, etc but AWS, in another hand, is just a bill.
Very valid points. Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. OPs people kinda shrug and ask that you essentially turn it off and back on again and that usually resolves the issue at that time. There are so many abstraction layers when all the infrastructure is software run and I bet a lot of those introduce performance degradation.
Another element of this is I would think its more fun for a systems person to be closer to the hardware. Specifying, ordering and unboxing servers, disks, switches and setting things up sounds satisfying to me.
They are much less frequent, and at the very least you have full access to the hardware to debug if needed. Can you get me access to the AWS control plane's logs so I can understand why my RDS instance is stuck in "modifying" for 12+ hours?
I'm going to beat its reliability (and successfully do so) because the complexity of my stack is less than 1% of Amazon's. I have way less moving parts (and complex interactions between them) to worry about than Amazon. My only liability is hardware failure, and it turns out that stuff has gotten pretty reliable nowadays.
Am I able to beat Amazon on their wide range of services? Absolutely not - if you're looking for FAANG-level scale, you're talking to the wrong person. But for the handful of services I do need (HTTP LB, app servers, relational database and cache), I beat them because I only need to deliver very specific services and can schedule the risky operations (maintenance, hardware upgrades, etc) around my business' needs, something AWS can't do because as a global provider any time they pick for maintenance would inevitably clash with someone's business-critical hours. Turns out most companies also only require a small chunk of what AWS offers, so the rest of the AWS control plane's complexity and its global scale is actually a major liability.
> AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy.
And yet that one time a client's RDS instance ran out of disk space, it got stuck in "modifying" for 12+ hours (I assume an AWS operator eventually manually logged in and fixed it) with no way for us to recover, forcing us to restore a backup and reconstruct missing data from logs to bring our service back online in a timely manner.
Would it fail the same way on bare-metal? Yes, but at least I can SSH in and fix it immediately. Not to mention, the only reason it ran out of space is because AWS' pricing forced us to be very conservative with storage, where as the bare-metal machines I rent - for far cheaper than AWS' offerings - come with terabytes of storage as standard so this incident would've never happened to begin with.
> The thing that Soul or other similar tools follow is to automatically turn your database into you backend. You don't need to implement any endpoints, Soul already did it for you.
may have no clue anyway how to be reliable - which is totally fine of course.
> You're not going to beat the reliability by going inhouse.
OP was talking about observability and diagnosability. These will be infinitely better/easier with access to the hardware & OS.
But on reliability, if you keep your stack simple and boring, you can pretty easily also beat them on reliability. Complexity is the top enemy of reliability (and security).
Ah yeah, I remember when a friend's company struggled for weeks with some terrible slowdowns on their Azure-based cloud infrastructure. After hours on hours of support calls and emails and weeks of hair-pulling, everything suddenly went better : after careful interviews with Azure support team, they understood that their VMs were running on hosts in two different racks, and that the link between the racks had just been upgraded from gigabit to 10-gig when the ToR switches were replaced.
Interesting. I helped put together a cost estimates for Amazon Workspaces for our development infrastructure and we quickly found it costing us up to a quarter million compared to moving it to a colocated server. AWS is not looking that affordable anymore.
AWS is never affordable, but offers interesting commodities as a service that these days many companies could not afford to build or maintain themselves.
I worked at a hybrid shop that had main MySQL and MongoDB databases on bare metal rented servers and all the new extra features/services built in AWS. It was a pretty good split. We did have some human cost in handling hardware failures and scaling of the hardware but it was way less than cloud hosting for that much CPU and RAM would have been as well as access to higher performance machines. I'm glad that most of the infrastructure handling scripts in Ansible and Chef were made before I got there and only had to make a few new specialized ones.
Oh right, we did have many app servers located with the DBs as well. They were stateless and numerous so could be anywhere. The datacentre with the bare metal did have dedicated links to the nearby AWS region we were using. At one point I think one of their redundant links got severed on purpose by accident, so things were slow for a while.
The microservices in AWS were feature-services that had their own datastores so only paid a single overall request latency than the many app-server to db queries per user request.
> The cloud excels at two ends of the spectrum ... The first end is when your application is so simple and low traffic that you really do save on complexity by starting with fully managed services .. The second is when your load is highly irregular
I think there's a third, which is when the company is so rich it just doesn't matter. Then saving even a little bit of employee time (which are more likely to be the bottleneck in such a company) is worth basically infinite spend.
I worked at a place where they spent $6k/month on cloud, just for a Java app not doing much plus a PostgreSQL database. I came up with a way to save $2k/month (better indexes), when I did it nobody cared or even noticed I think. I know that $2k/month might not sound like much, but I think the same principle would have been true even if it'd been much more. They spent about $2m/year on an external software development agency to create a React project, full of inefficient scrum meetings and billable hours from project managers and so on, and they didn't really care about that either.
I think if you're working at a company that makes a huge amount of money, it just isn't really rational to care that much about spend. I think of it like this: If a person had $100m in the bank, I reckon they're not going to care whether their beer costs $7.00 or $7.10. There are just more important things to worry about.
Sure between $7.10 and $7.00, who cares, but now we are talking more like $7 or $70. If I had $100m I would prefer to pay $7 for the beer and spend the remaining $63 on something more useful.
Companies (particularly young ones) succeed or fail if they are innovating on their products and delivering value to customers. Reducing cloud spend is not doing that.
It may very well improve the bottom line, but when the company is (correctly) more focused on “how do we increase our customers by X% this year”, pursuing those savings is a distraction.
It's true also for companies where CIOs are going to Gartner events and don't want to appear like the only ones not being "cloud-only" or "cloud-first"
Then you end up with BS like the ones described here
https://www.the-investing-desk.com/cloud-costs-are-in-a-bubb...
where to justify the fact that cloud is much more expensive managers start to throw random numbers vaguely justified just to show that cloud is actually better.
I saw similar things in Corporate world even when choosing a colo/service partner vs doing in-house... managers asking us detailed costs for infra needed (servers, network, licenses etc.) and when you were getting something like 30 in-house vs 100 hosted they just threw some random FTE count to support it to get to 110 vs 100.
Managing machine life cycles, data intensive and stateful systems is very hard. I was hoping to see more concrete examples/data points instead of just saying we can also run the same software on our own machines.
DEATH TO THE CLOUD. For far too long they have profited out of the ignorance of those who do not want to learn how to virtualize, or host their own site.
Have you seen the specs of the server you rent? A 2 GHZ core? Really?
Time to spend real money in a real hardware.
Fiber optic has dropped latency to really low levels. You don't need a super close geolocation server anymore.
With the money they spend on the cloud, they can rent small offices all over the world and have a small rack server with beefy servers in a secure server room they can build.
The premise of the cloud is actually a good idea: if you only use the server for 1cpu-hours per day, you should only pay that, and let others use it for the other 23cpu-hours in the day.
The executions we've had so far are terrible. There's huge markup, piles of complex abstractions, and all the other issues mentioned in this article.
When I was a teen, I used to pay this guy in town like $1.5 a month for a user on a *nix machine where I could run scripts and run my own IRC bouncer and stuff. A bunch of people shared this machine (mostly running IRC bouncers, AFAIK). I've no idea now many, it was stable, and nobody was eating up all the CPU. It worked.
I feel that this nostalgic memory of the past is closer to a utopic "cloud" than anything any modern provider is offering.
Vultr has a good 2 dollar tier for this, my only gripe being that they're like Hetzner and will shut your shit down at random in the first 6 months and ask you for your passport or ID or a pic of you holding it. This has happened to me personally on Vultr and Hetzner.
Why Weathergraph (my hourly forecast app for iOS) was never in the cloud:
Because I’d be dumb to spend hundreds of dollars and engineering complexity for a stable load service, when a 20 EUR VPS (10 EUR until recently - see, scaling!) runs everything I need (forecast proxy, geocoding, website, payment verification) just fine :).
It almost seems like their is an opportunity to build a cloud hosting service, where mom and pop data centers can plug in their infrastructure to a cloud service and let the platform manage the renting and latency based routing.
If the software is developed enough, this enables an even more global distribution. Software services don't want to manage machines and data centers don't want to manage customers.
I didn’t listen to the accompanying podcast so I might be missing some details.
It strikes me that it seems like 37signals are just using AWS as a glorified virtual machine with maybe some db hosting.
To me the big benefit of the cloud is using all the little services they provide, like s3, sns, sqs, CloudSearch ECS, cloudwatch and the list goes on and on.
If we were to host all those service ourselves and keep them up to date the operating cost would be huge.
Furthermore I work for a site that is very public and has a lot of for instance ddos threats hanging over head. AWS handles lots of ddos threats all the time and they a much better suited for handling that than our own ops team.
92 comments
[ 0.29 ms ] story [ 161 ms ] threadA whole load of Racks for running Basecamp? We are talking about 42U per Rack, and total of 420U of Servers?
The scale seems quite massive. At least to the idea / perception of what I had about Basecamp. Would be nice to see those specification and see how much of an improvement it is 10 years later and if we could fit those into 2-3 Racks.
Basecamp started on one single Rackspace server, before I was there. I started at 20 people, left at ~55.
When I left there were I don't remember how many racks exactly, but more than 10, less than 30 in the primary location. 42U of servers in each. There was a mix of a whole load of (Dell, never pay list price!) blade servers, DB appliances (~12 in total across ~6 apps ISTR), Isilon storage[0][1], F5 kit, juniper routers etc. etc. We had some epically fast storage in some of the servers for the time, way faster than SSDs.
Later we added two more sites. One in I think Virginia, one in NY. The one in Virginia was a replica of what we needed to run Basecamp, the one in NY was a half-rack data replication location (I think I got that the right way round). We had 10G fibre (we rented wavelengths not the whole fibre) between each location. We could lose one DC and remain RW for our block data, 2 DCs and we'd have to drop down to RO. Block data was things like uploads, so DBs, search etc. wouldn't have been affected. We could lose one of the /main/ DCs and still be RW for everything.
With all this kit we were able to run both main DCs hot. With our Geo DNS you could hit either of our DCs and you'd get served pages, you could even write to both locations. One DC was always the "RO" DC, it always replicated the databases. If you tried to write to that DC we proxied your request to the RW DC over our 10G links and proxied any more requests you made for n seconds to the RW DC too, at which point we reverted you back to the RO DC.
Now, NY to Virginia isn't that far, so why bother with the hot/hot config? Because it played into the rest of our plan, which was DC failover. With some pretty epic voodoo (Juniper/F5/OpenResty etc.) we could fail over the datacentres, swapping the RO and RW locations. We could also do this if one of the locations was unavailable. We could do this in 4 seconds /without losing a single in-flight request/ (we tested it).
This ended up a bit longer than I was intending, but it illustrates a few things:
- I think people underestimate Basecamp. It's /huge/ (money and users, not employees). Not so much in the tech world (anymore), but even with this kit, even with the (at the time 6) sysadmins that maintained it, it still made a shit load of cash. I guestimated the net-worth of the two owners as in the hundreds of millions of dollars each, entirely because of basecamp. Cash that as a privately owned company all went to the owners (who gave some of it to us, they treated us fairly well). I think it was Patio11 who said that people under estimated the market for software, it's easy to do, these aren't human-scale numbers.
- The owners are right about keeping it private. You might not make "larger yacht than Larry Ellison" money, but you sure as hell have a lot higher chance of making "pretty damn big yacht and not having to work again" sort of money. If I was rolling the dice I know which I'd gamble on.
- I don't think I ever actually calculated it, but the amount of money our servers were worth while running was /immense/. When you went to the DC and looked at the row of racks, you could kinda see just how dense the $ value there was. The efficiencies that the cloud brings gives money to AWS, not to the clients.
- All of this was still /way/ cheaper than running cloud infra. I'm not against the cloud (I use it), but we made more because we had this infra than if we were on the cloud. We were dense when it came to # customers per $ spent on infra.
- The flexibility we had because we controlled everything was also extraordinary. It came at a cost, we had to do it ourselves, but we had a /lot/ of power to make things work the way we wanted to.
- We had access to everything, from the network (hell, even the light :) ) to the JS. This gave us optimisation options not available to a lot of people. Networ...
Lot of people think that DC failover needs to be east coast / west coast but you can achieve most of your DC redundancy goals separated by 100 miles or less and have a lot lower latency, higher bandwidth and lower costs. Might want to think about different geographic flood plains and different power companies / grids.
Could still nerd fight about EMPs from nuclear war and a sufficiently massive hurricane or an earthquake out here on the west coast, but at some point you need to accept some risks.
Even if the number above does not include redundancy, that still only makes it 2 Racks of Servers with spare.
30 Racks is a lot. What am I missing here? Apart from Storage.
Bear in mind that we had I think 6 apps. Basecamp 1, 2 and 3 (all separate infra), Highrise, Campfire, some other internal stuff. Our ES cluster was pretty damn big, redis and memcached too. Juniper, F5, network switches (rack infra) etc. Storage was pretty big, quite a few 4u servers with spinning rust. The blade servers were I think 6 blades in 2u.
Everything was redundant, everything had 2x or more. I honestly can't remember how many racks we had. "More than 10" is my hand-wavey guess.
I don't think DHH was being disingenuous with his "2k on 30 servers" message, it was likely more about the scalability of Rails rather than the infra required to run the app.
It's also a huge company and a massive distributed system, so moving that off from AWS is an insane amount of effort and there might be legitimate benefits from AWS' management and autoscaling layer that aren't as valuable for 37Signals which runs simpler, monolithic systems.
It still cost /far/ less to buy/rack/power etc. our servers 100% of the time than it would have done to use the cloud and scale up and down with demand.
There is definitely a situation when bursty requirements will make cloud the more obvious answer, but given the near order of magnitude cost increase for cloud it's not as clear cut as just seeing a wave on a graph :)
2. Content Encoding are not done at a regular interval. And most of their work on ML and other quality requirements does not require fixed interval either. i.e They are schedule to work in Spot when AWS price are at its lowest.
3. Netflix get huge discount anyway.
The only way of being really sure of what is going on is to own it yourself. Unfortunately that has become more and more important.
Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host.
Weird.
He also did not mention if they are going to self-host, colocate, or rent bare metal in a more traditional host (like softlayer). Probably colocate as I think that’s how they used to run?
Both your points are still valid though.
If it's $250k each, that's already $500k. Unless they can find someone who can do both for... $300k?
I don't know.
He doesn't provide enough details to make the math work.
Silicon Valley culture isn't nearly so critical or valuable that your organization requires it in order to do the necessary work.
Sure, it is not "that complicated" for basic stuff, simple sites, hobby projects or overall quick'n dirty solutions because you're under a budget, as with everything.
Doing infrastructure properly, with SLAs and a very high level of availability and redundancy (both in systems, and in people with the know-how) is incredibly complicated in my opinion.
I bet what they're going to save on the the Cloud provider is going to be spent (and probably even more) in salaries, if they keep the same levels of service.
Specially true considering they pay pretty good salaries.
This point gets raised every single time bare-metal comes up for discussion and at this point I'm not sure if people are legitimately unaware or if there are hidden agendas at play.
There are plenty of providers out there who will handle the logistics for you with an interface similar to AWS EC2 - ask for server and get SSH access in minutes/days/weeks (depending if it's an off-the-shelf or custom build).
From there, your existing DevOps guy (which you must have to navigate AWS' complexity) can take over.
It's only at either extremely large scale or very unusual requirements that managing your own hardware makes sense. Until then, you will never be able to beat the economies of scale of managed bare-metal providers.
I am sorry but how's it different from leaving BigCloud for SmallCloud? That's like saying, I'd no longer use Java/Kotlin because Oracle/Google but I'll however use Clojure because JVM.
Go back and re-read :)
They [believe that they] don't need to hire anyone. They already have a staff of ops folks that run their cloud operations. And those folks will run their on-prem operations.
If you're constantly installing stuff then you're scaling up a lot, and that's the level where it would make sense to have someone hired dedicated for it.
You can also rent hardware.
the thing is: cloud ops are also quite complex but it's a different kind of complexity. I don't know how many people are required.
The largest teams I've been on have been cloud deployments, and some of the smallest teams were bare-metal; but that's just an anecdote.
If you're "cloud native" you've already architected your stack to be horizontally-scalable and distributed (partly because the cloud is so expensive that running at peak load capability all the time would bankrupt you) so hardware failure should matter even less.
I think you already pay this guy to manage your AWS instances. so you get the benefit of the full 500k
Outside of expensive Silicon Valley and FAANGs, where do you get this $250k/Year ? I think most of us on HN are normalizing this whole "devs make 250k/year" thing when in reality, it is a tiny percentage of people. Tiny when you look at the entire ecosystem.
> Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS?
A company which has a total AWS bill of $500k/yr is a very small one (the article says 500K is just the RDS/ES bill, so total is surely much higher for them).
Our (tiny startup) AWS bill is ~200K/yr and the amount of compute and storage we do could very easily be handled by one single physical server.
Now, sure, if we went to self-hosted we'd split that into multiple servers for redundancy and growth headroom, but in any case definitely less than a dozen boxes.
You don't hire anyone to install and manage the hardware at that scale. Someone from the ops team goes to the colo for a day or two to set it all up and then you only need to go back when something needs replacement, which at that small scale is going to be very rarely.
That's exactly what we did at a previous startup and the company grew to many, many hundreds of people before there was so much infrastructure that it became worth thinking about hiring a dedicated person. By that time you'd be in the 10s of millions in AWS spend.
Another element of this is I would think its more fun for a systems person to be closer to the hardware. Specifying, ordering and unboxing servers, disks, switches and setting things up sounds satisfying to me.
As oppose to complexities of self-host using self purchased hardware?
You're not going to beat the reliability by going inhouse.
If you run a competent organization, this can be a boon. If you have serious deficiencies, it may well be the opposite.
Am I able to beat Amazon on their wide range of services? Absolutely not - if you're looking for FAANG-level scale, you're talking to the wrong person. But for the handful of services I do need (HTTP LB, app servers, relational database and cache), I beat them because I only need to deliver very specific services and can schedule the risky operations (maintenance, hardware upgrades, etc) around my business' needs, something AWS can't do because as a global provider any time they pick for maintenance would inevitably clash with someone's business-critical hours. Turns out most companies also only require a small chunk of what AWS offers, so the rest of the AWS control plane's complexity and its global scale is actually a major liability.
> AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy.
And yet that one time a client's RDS instance ran out of disk space, it got stuck in "modifying" for 12+ hours (I assume an AWS operator eventually manually logged in and fixed it) with no way for us to recover, forcing us to restore a backup and reconstruct missing data from logs to bring our service back online in a timely manner.
Would it fail the same way on bare-metal? Yes, but at least I can SSH in and fix it immediately. Not to mention, the only reason it ran out of space is because AWS' pricing forced us to be very conservative with storage, where as the bare-metal machines I rent - for far cheaper than AWS' offerings - come with terabytes of storage as standard so this incident would've never happened to begin with.
> The thing that Soul or other similar tools follow is to automatically turn your database into you backend. You don't need to implement any endpoints, Soul already did it for you.
may have no clue anyway how to be reliable - which is totally fine of course.
OP was talking about observability and diagnosability. These will be infinitely better/easier with access to the hardware & OS.
But on reliability, if you keep your stack simple and boring, you can pretty easily also beat them on reliability. Complexity is the top enemy of reliability (and security).
On AWS, you can have a Postgres instance with multi-region availability. How do you plan to do this with your own hardware?
So it's a matter of choice.
This is a good point. I really dont like the centralization on AWS.
The microservices in AWS were feature-services that had their own datastores so only paid a single overall request latency than the many app-server to db queries per user request.
I think there's a third, which is when the company is so rich it just doesn't matter. Then saving even a little bit of employee time (which are more likely to be the bottleneck in such a company) is worth basically infinite spend.
I worked at a place where they spent $6k/month on cloud, just for a Java app not doing much plus a PostgreSQL database. I came up with a way to save $2k/month (better indexes), when I did it nobody cared or even noticed I think. I know that $2k/month might not sound like much, but I think the same principle would have been true even if it'd been much more. They spent about $2m/year on an external software development agency to create a React project, full of inefficient scrum meetings and billable hours from project managers and so on, and they didn't really care about that either.
Companies (particularly young ones) succeed or fail if they are innovating on their products and delivering value to customers. Reducing cloud spend is not doing that.
It may very well improve the bottom line, but when the company is (correctly) more focused on “how do we increase our customers by X% this year”, pursuing those savings is a distraction.
Then you end up with BS like the ones described here https://www.the-investing-desk.com/cloud-costs-are-in-a-bubb... where to justify the fact that cloud is much more expensive managers start to throw random numbers vaguely justified just to show that cloud is actually better.
I saw similar things in Corporate world even when choosing a colo/service partner vs doing in-house... managers asking us detailed costs for infra needed (servers, network, licenses etc.) and when you were getting something like 30 in-house vs 100 hosted they just threw some random FTE count to support it to get to 110 vs 100.
The executions we've had so far are terrible. There's huge markup, piles of complex abstractions, and all the other issues mentioned in this article.
When I was a teen, I used to pay this guy in town like $1.5 a month for a user on a *nix machine where I could run scripts and run my own IRC bouncer and stuff. A bunch of people shared this machine (mostly running IRC bouncers, AFAIK). I've no idea now many, it was stable, and nobody was eating up all the CPU. It worked.
I feel that this nostalgic memory of the past is closer to a utopic "cloud" than anything any modern provider is offering.
Because I’d be dumb to spend hundreds of dollars and engineering complexity for a stable load service, when a 20 EUR VPS (10 EUR until recently - see, scaling!) runs everything I need (forecast proxy, geocoding, website, payment verification) just fine :).
(Link: https://apps.apple.com/cz/app/weathergraph-weather-widget/id... )
Developers keep adding toys that cost money, and then at some point, someone at the top notices the bill, and it's all hands on deck.
The cost due diligence should happened before anything, not when your ass is on fire and you need to save money yesterday.
"Do we need that beefy RDS instance?" ¯\_(ツ)_/¯
"Why does our ElasticCache data keep growing at such a high rate and nothing is expiring?" ¯\_(ツ)_/¯
"Could X be done much cheaper with Y?" ¯\_(ツ)_/¯
"Why did we spin up DynamoDB for this if we already have Postgres?". ¯\_(ツ)_/¯
If the software is developed enough, this enables an even more global distribution. Software services don't want to manage machines and data centers don't want to manage customers.
It strikes me that it seems like 37signals are just using AWS as a glorified virtual machine with maybe some db hosting. To me the big benefit of the cloud is using all the little services they provide, like s3, sns, sqs, CloudSearch ECS, cloudwatch and the list goes on and on. If we were to host all those service ourselves and keep them up to date the operating cost would be huge. Furthermore I work for a site that is very public and has a lot of for instance ddos threats hanging over head. AWS handles lots of ddos threats all the time and they a much better suited for handling that than our own ops team.