I dont think its fair to compare GCP's egress costs to a colo's. A collocation is simply sending your packets straight to the internet, where-as GCP routes your packets over private fiber to its closet POP to your user. Giving you better latency.
The last time I measured AWS latency to anywhere, it tens of milliseconds higher than the super-cheap-and-crappy virtual server service I compared it to.
This is only true with Google as far as I know. And I'm fairly sure that all cloud providers only have maybe 10 edge routers in the US so the utility is limited. You can't have too many edge routers because they rely on IP Anycast which is dangerous to do if your routes are too similar.
You are right, the Google Network magic is awesome. Yet I agree with the opinion here. They charge more than they should. Yet I think in the long run it will become cheaper (AWS is always pushing for price reduction and the others will follow)
When you buy in mbps, you're actually billed based on 95th percentile usage. So this comparison is way off, depending on traffic patterns, 1mb/s committed can work out to about 120GB in a month on average. If you use reasonable GB per mb/s numbers the cloud providers don't look all that bad.
I have posted a few times about how absurdly expensive all the cloud providers are. If you have a baseline load you should be co-locating bare metal. Any excess capacity you need should spill over into your AWS/GCE/Azure account.
For example: A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month.
OVH has absurdly low prices on both. If you have a baseline load take a look. Their enterprise offerings are pretty rock solid and they have multiple data centers with direct fiber backplane interconnects.
IMHO the big cloud providers only make sense if you have a highly elastic load or if you are making extensive use of their as-a-service higher level offerings. If you're running baseline load bare metal destroys them.
I used to use OVH on occasion, but I was really disappointed with their North American-based tech support. Didn't seem particularly plugged-in and English proficiency was shaky. (At this point I'm in AWS because I don't really care about low-point performance and would rather spike up when needed.)
> You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month.
Then you are just trading one fee for another, but the "sysadmin services" you are ordering most likely has slower response time, less resources, less knowledge, and worse uptime guarantees than someone like aws.
Yeah, at the larger sizes it might make sense, but not for everyone.
At smaller sizes too. If you just have one server, it's unlikely that the slower response time will loose you 3000 dollars per month which is the savings of dedicated service versus aws. Many small businesses run on leased hardware with on site support. Rackspace for instance made their fortune in that sector.
Well you're not comparing to AWS without ops, you're comparing to hiring someone for non-AWS vs managing AWS on your own. Outsourced ops know your systems. They have contracted guarantees. They can manage your own systems, hosted systems or even AWS. Using not-AWS saves you enough to pay for that.
I worked for a MSP for many years, and was well acquainted with our regional competitors. Our uptime and theirs was nowhere near AWS-level.
You're getting a different engineer every time you pick up the phone. They most certainly do not know your systems like an internal team would. I regularly saw cascading/circular issues caused by lack of familiarity and/or poor change management.
Owning and administering your own iron makes sense at a certain scale, but it's a much bigger scale than most companies will reach.
Nope you have a number for the person that provides the service and you can call her/him. Unless you are at several million per month spend at AWS have fun getting a hold of someone when sh#t hits the fan.
That might get you one admin (we office workers have a mistaken belief that our salary is most of our cost. It's somewhere just north of half.).
What do you do when they're on vacation/sick/sleeping?
Thing is, I really want us to host our own stuff. For one thing if you don't host anything you lose all competence and then you really are beholden to your cloud provider.
But it seems to be that we are in this situation until someone figures out how to comodify servers - and server orchestration - to the point where this stuff gets cheaper to manage again.
This pendulum always swings. No sooner will we control our own hardware again than someone will come up with a new way to centralize it.
Totally, the thing that's super cool about the amazon instances you don't have to do anything with users, authentication, authorization, networking, dns, backups, monitoring, logging, firewall rules, security patches and upgrades.
With a bare metal server you have to manage all of that.
Oh my goodness man, have you managed Amazon instances? Setting up VPCs, IGW, IAM, DNS, EBS Snapshots (and deletions until recent lifecycle), custom metric in coudwatch, and Security Groups is a seriously challenging task just to learn the basics, much less do it right, much less automate it.
It's not a bed of roses in the physical world either, but you're simply wrong to say it's easy in the "Cloud" and hard in a DC/Bare Metal.
(it was a joke.) There are a zillion things to think about if you're going to have even one machine public on the internet, regardless of the hosting solution. At a small organization there's always that one programmer that's like half sysadmin.
I find it amusing that people pretend without bare metal, you'll get all of that person's time back.
With correct pricing (3-year commitment), you're actually looking at about $8k/yr/server difference, for 64-core servers. If you're at the point where your workload requires ~12 64-core servers, that difference will net you $100k, which you can spend on the ops staffer.
It really, really depends on on your use case, but there are some scenarios where bare metal just makes more sense. Also there are ones where cloud makes everything much easier.
North America-oriented B2B app that's a JavaScript SPA witch connects to some lightweight services fronting a database? I think that's a great contender for the cloud.
500-node Hadoop cluster running 24/7 with heavy load? You are going to see some huge savings by maintaining a Datacenter. You can use the extra 1.7MM/mo. to set up some robust networking equipment and have few people on-site.
Like every IT trend, "Cloud ops" is the trendy mind-slug that eats everyone's brain and makes them say that everyone who doesn't jump on the bandwagon is going to die broke and unloved in the gutter.
Remember how XML meant massively reduced integration costs and firing all your developers because GUI tools would let business managers connect pretty boxes and then kick back to watch the graphs go up and to the right as business exploded?
Yeah, so AWS is great, if you're building something that works well on AWS. And that's a large class of apps - if you're throwing up a Drupal front end to a CRUD line-of-business backend, need to push notifications and send some email, sure. You're dead in the middle of their target market.
Doing something interesting? Doesn't even have to be as massive as the parent suggests - if you're doing something that blows any of the billable parameters out of the sweet-spot, like bandwidth, storage, latency requirements, things that depend on system-locality, etc., you're much better off building, and using AWS for the pieces which can be broken off that don't hit the pain points.
you can also pickup an enterprise flash device and have a few terabytes of 500k+IOPs for a few grand whereas reaching a similar performance profile via provisioned IOPs will cost you 10x as much every month.
You're comparing apples to oranges -- if you want a couple TB of locally attached storage, you'd get an i3.2xlarge for $450/month with 1.9TB of NVMe SSD (plus 8 cores CPU + 60GB RAM).
Using EBS volumes with provisioned IOPS give you some advantages that you don't get with a bare SSD -- like snapshot support and the ability to easily detach it and move it to another server.
you're right, the performance of the local storage on i3 instances is a lot better than normal EBS but still nothing close to what you would get from just buying an Intel P4500 or similar though.
I'm not sure which region you priced, but an m4.16xlarge in us-east-1 (Virginia) is $3.20/hour, or $2304/month.
But if you are comparing it to a purchased server, you ought to be looking at Reserve pricing, a 1 year reserved instance is $16K/year, 3 years is $32K (or you can pay monthly if you want).
Still not as cheap as owning the server, but not quite as bad.
With an m4.16xl, there's no effective difference between a dedicated or default tenancy instance - only one .16xl can run on a physical hypervisor, which is reflected in the pricing, it's $3.20/hour whether it's dedicated or not:
For smaller instances, there is around a 10% increase for dedicated, for example, for an m4.4xl, the price goes to $0.88 for dedicated tenancy versus $0.80 for default.
There is a $2/hour fee for each region in which you have dedicated instances, but that gets lost in the noise if you're using significant AWS resources. (it's $2/hour regardless of how many instances you're running)
We haven't noticed any significant difference between performance on dedicated and non-dedicated instances, and only use dedicated instances where required for compliance/regulatory reasons.
1. Will Dell servers magically appear in your racks all mounted correctly? Shipping delays is a very serious and real problem.
2. Will Dell servers arrived with 0 problems on its components? Any time I ordered meaningful amount of servers, I usually get about 5% fail rate. VMs aren't perfect but you can destroy and recreate in different region almost real time.
3. Ever had to deal with difficult-to-work-with network admin? The cloud is significantly less pain in the ass.
#3 is why we went to Amazon, and its one of the reasons we were successful. Sad as that was. No weeks/months of delays/hassles/fights/sabotage. Just success as fast as we could do our part and figure out the few small hurdles.
Oh it was, but it was the situation we had to operate in.
And Amazon allowed to to achieve amazing things that never would have been possible and that's without the idea of things like Kinesis or Redshift that we couldn't have achieved easily/at all even without personnel problems. You simply could never get 3 new servers or drastically respec an existing one in under a day let alone minutes. We were a small shop so it's not like we had spare servers ready to go.
Networking is a great place for assholes to build empires and exert control. In 20 years of professional engagement with distributed and data center networks, most of these guys (and they are almost always guys) running network orgs are an impediment and spend more time helping out their vendor of choice than anything else.
I've run into awesome network guys in position of power only a few times, and they are 100x employees. The last major rollout of a service that I did was literally 3 months ahead of schedule solely because of the efforts of this awesome network dude snowman as recently put in charge.
We saw some good people, but they generally left because they didn't want to work in that environment.
Eventually replaced with someone helpful, it was amazing what we started to get done. Got lucky there we found the replacement and he stayed long enough to be there when needed.
It was sabotage in the 'oops this project isn't working' or 'oops this really import thing came up, no time' or 'didn't I do that?' sense. Having some plausible deniability. Work slowdown/stoppage stuff. Not actively breaking things (mostly). And it was all directed at maintaining control/power.
Worst person I've ever worked with. When I tell stories at future jobs I fully expect people to not believe me. Hell, people who weren't there for earlier incidents often didn't believer them.
Oh I believe you. I've heard of it before. Was trying to say that if that's an issue, it should be an HR fix more than an infrastructure fix. But I also imagine often that person also convinced management the are an "11x rockstar programmer"...
They were one of the hardest working people in the company. Partially that's because they worked on nonsense projects they made up, and partially that's because they were always putting out fires caused by the fact they did things poorly in the first place.
That's scares me cause I've seen that bit way too many times. There are huddled over the terminal in the evening as all managers leave "Oh look so and so is working so hard" and I am thinking "they are fixing the bug they put in just last week because they didn't bother testing it enough".
Tons of that. Each server was a literal copy of a previous one then modified to fit its new role. So we never got two servers that were the same. They'd have extra junk on them too.
Just basic docker use (or something similar) would have made a HUGE difference. Parts of the department were sent to training but it didn't happen because it would have made him less indispensable.
At one point someone higher up (but not high enough) asked how many servers we had in Amazon and the answer like 10x our physical set.
"And only one of you manages them?"
"On the side, we don't usually have to touch them."
"Then why does it take 4 people for the other servers?"
(Collectively) "Yeah. Why IS that?"
Change control was a big problem too. "Random" changes, things that didn't change at all (even when we could prove it), stuff like that causes SO MANY issues, basically all avoidable.
It's not the hardware you're paying for, it's the services. How much does it cost to employ a systems administrator? A lab tech? And so on. You see exactly the same pattern in software services. People pay more to buy products that are "turn key" versus having to hire someone to do tons of work building out systems or services or what-have-you.
Also, hiring people is a risk, and it doesn't happen instantly. When you have a gap of months without a sysadmin things are bad. Hiring a bad sysadmin is even worse. Not to mention the impossibility of hiring 1/2 or 3/4 of a sysadmin. There's scaling limits there. If you have the resources to DIY, that can be a key competitive advantage, but not everyone does.
Often, you still need a "systems administrator" with the cloud. Except, now he calls himself a DevOps guy and charges a lot more. ;) AWS isn't going to configure itself.
How does that work out? Looking at the per-hour figures in AWS and GCE, they're similar for a 64-core, quarter terabyte RAM machine ($3.20 AWS, $3.40 GCE). Both vendors have ways to get your bill discounted (with google's being easier). AWS's instance pricing has been fairly comparable to other vendors for a couple of years now; it's their bandwidth that's costly.
What are most people running anyway? Using this kind of behemoth server as a 'typical' example seems out of line.
Google base is slightly lower and it gives you 30% off automatically on instances that run 24/7 for the month.
That's typical of on premise deployments. You get big servers because they are so annoying to deploy you might as well fill everything possible in the box.
On the clouds, you would create a group of VM per service, with appropriate size for the service. It is significantly more efficient.
Isn't that $1710 already including the 30% discount? (It works out to $3.4 x 30ish days x 70%). You get similar discounts on AWS, but you have to commit for a year - given the OP's intent of leasing a Dell server for three years, this shouldn't be dismissed.
> A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx)
Cherry-picking a bit there. You're entering a 3-year commitment with the Dell lease, but not with the EC2 lease. Take a 3-year RI on that EC2 instance, and the price drops considerably. If you also take away the 'dedicated' requirement, the 3-year RI is about twice as expensive, not ten times.
Edit: correction from Johnny555 above: this top-tier server doesn't have a 'dedicated' option, so there's not an 'extra' cost for that. It's just twice the price, if you do a 3-year commitment (as you are doing for the dell)
You're paying for the staff time to set up the lower-level infrastructure and keep it running at _least_ as reliably as if you had your own staff on bare metal, quite likely more so. (including scaling it when you need more etc).
It is interesting that they end up charging you this for _bandwidth_ though. Their staff time doesn't actually probably scale with bandwidth, but their prices do.
I think it's probably kind of a way of pricing based on a proxy for customer size, likely revenue, and ability/willingness to pay. Rather than as a % margin on actual costs.
3) hardware components you'll need for redundancy, I've personally replaced network cards because fans burned out or HDDs because of usual failure rate.
4) when I'm done using a n1-highcpu-32 I stop using it and not pay for it, or when I need 5 of those for 2 hours I provision them and stop when I'm done running a workload.
5) upgrades to hardware or additional hardware, intel released skylake, should I buy those and try to sell my Haswell on eBay?
I'm not sure it becomes a simple price of server + colo cost comparison.
With GCP (I use daily), AWS (I used to use) or Azure, what I'm paying for is someone else to focus on things like colo leases, capacity expansion, redundancy, physical security and provide higher order services - load balancers, BigTable/BigQuery, CloudSQL/RDS, GKE, DataProc.
This allows me to run all of infra as close to capacity as possible while avoiding any wasted cycles
However I do miss going to data centers and hear my machines hum and blink, that is quite the feeling.
I just landed a few racks of bladecenters and storage. High 7 figure deal. This was probably the best implementation project ever... everything ran on time, the vendor delivered everything on time, facilities was 100% ready and pre-cabled, etc. I'm incredibly proud of our team -- our biggest issue was someone ordered the wrong SFP. Butter.
Timeline: 120 days procurement (including product evaluation and PoC), 30 days install activity, 10 days buildout. 10 days software build, 15 days testing.
That's a little over 6 months. A ton of vendor/build engineering/tester time. And we'll be at 50% capacity for another 4 months. Or... we could spin the whole thing up on AWS/GCE/Azure in < 30 days and only consume what we need when we need it.
The LDS Church has a prohibition on hot beverages but cold caffeinated are actually allowed. It is a common misconception that all caffeine is not allowed. Homebrewed beer, and homebrewed wine are technically allowed also. Many also ignore the part in the wow about eating meat sparingly while misinterpreting the rest.
How is this is a surprise to anyone? The big players are all pushing their clouds because its a cash bonanza. It's the SaaS model for hardware, make money forever because your customers never own anything.
I've done the math many times and it's orders of magnitude cheaper to colocate as long as you can afford an IT guy and the upfront cost of hardware.
What happens when a harddrive fails in your colo? Unless you have best practice backups, you will lose customer data and trust.
GCloud abstracts the issue of hardware so you can focus on real business development. And to some, that's worth the cost until they can properly afford all the hidden costs of independent hosting.
These are solved problems though. RAID works great around 99% of the time and for the others you can use off-site backup. It's extremely rare that something needs a true 100% uptime. Salesforce and Reddit still go down for maintenance windows in 2017. For one project I setup log streaming with Postgres to the cloud(since inbound BW is free :) ) and ran a hot backup there just in case.
Really though in my experience the cloud is far less reliable than colocation. I've had AWS VMs "degrade" or randomly die countless times but I can't remember the last time my Colo boxes went down, probably not since I last upgraded the OS.
> it's orders of magnitude cheaper to colocate as long as you can afford an IT guy and the upfront cost of hardware.
It may be. But for many people, it does not matter.
Let's say you work at the place (I do) where "compute" cost is like ~0.5% of the total costs. Almost all of the cost is employee's salaries and benefits.
BTW we get (and from what I hear, everyone does) offers from other cloud vendors to switch, and get 1 year free or whatever. And it gets ignored because it is a big risk for a tiny possible gain.
Fair enough. I just think that infastructure is a bigger differentiator of success than most give it credit for. By using the cloud you're giving up the possible performance and money advantages of doing it yourself.
It's really just a new form of vertical integration. If you're working only at the high levels you're not only paying someone to do that rest for you, those people are making profit off of you. Profit that could be yours.
Keep in mind that with cloud providers you're also paying for the SDN that makes dynamic provisioning of VMs and logical network segmentation possible. Scalable SDN is much harder / more expensive than traditional networking.
True, and it's still probably overpriced... but a lot of the SDN expense is in the stateful routing between networks (e.g., traffic going to/from the internet).
I'm not an expert, but it's possible... from what I understand amazon does SDN on CPUs in Dom0, Microsoft uses FPGAs, and Google has burnt their own ASICs... those are all expensive options. Much more expensive than generic networking gear. Possibly 4-5x, but idunno...
I believe Amazon uses commodity NICs + iptables in Dom0 (based on rumors). I have no idea what a custom ASIC would cost, but I think the Jupiter networking gear / Titan NIC that Google built are pretty advanced components that would cost more than commodity stuff.
In any case, the hardware components are fixed costs. What's not fixed is the cost of applying routing and segmentation rules for each packet. There are scarce resources involved here: the hardware has limited memory and limited cycles. You're paying for the use of those scarce resources (e.g., having a memory resident stateful firewall rule for a TCP session). Charging for bandwidth isn't perfect, but it probably correlates pretty closely with the underlying resources and is a much more intuitive unit-of-value for customers.
According to me... Are you arguing that it's not? It stands to reason that dynamic network architectures (SDNs) would be harder and more expensive to design and build than static architectures... it's the difference between managing shared mutable state and managing shared immutable state. On the hardware side, a simple commodity hub or switch is cheaper than a CPU, an FPGA, or burning your own ASIC to do fast SDN.
Cannot agree more with title. I did a comparative model of AWS bills and colo bills in the context of companies of different sizes (https://blog.paxautoma.com/buy-or-rent-the-cost-model-hostin...). It turned out frequently overlooked costs for bandwidth and provisioned IOPS can be responsible of large chunk of the EC2 bill.
Router? Gateway? Firewall? Network Access Control? VLANs? Ability to manage all this through declarative version controlled code w/ rollback?
The costs of doing those (well) yourself are not cheap.
Getting them from a provider that's certified to do them well while giving you software control also isn't cheap.
You're comparing cost of gas per gallon to to expense of miles driven per gallon. Pretty sure on your IRS or corporate expense report those aren't the same.
Heck, even the cost of creating and maintaining documentation for the services has to be enormous. The first time I worked with the AWS IoT services I was extremely impressed by the breadth, depth, and quality of the documentation, and it all seemed up-to-date: the screenshots showed the same interfaces I was seeing, and even documentation on small features or use-cases seemed to be accurate and current.
When you put it all together, you're not paying for bandwidth, you're paying for the power and convenience of a robust, high-quality, high-availability solution that provides extraordinary benefits when looked at holistically.
Also, employees? These are services that are actively maintained and developed."Alternative 1 – Colocation in a Data Center" doesn't include the 100k/yr you'll have to pay an engineer to manage that hardware.
Most enterprises, you're right. I left people off because you can handle a dozen or more DCs with only a couple engineers if you assume (design for) frequent component failure and only do a site visit/decomm/install once or twice a year.
Worth noting that Digital Ocean doesn't actually bill for bandwidth. They say they do in their Droplet template descriptions, but they really don't. I've pushed many many terabytes to/from my Droplets and never received a bill for it. But you need to cap your individual Droplet bandwidth using something like tc[0] around ~400MB/s or they'll shut off the network interface (DDoS detection).
indeed. I'm surprised how buried this comment is, because it's a really good deal (until they change things). anecdotally, DO sent me an email when one of my droplets was sustaining around 200MB/s, but it wasn't a "knock it off" email, it was an "is this intentional" email. is the ~400MB/s figure from your own experience?
(Bandwidth is ambiguous in this context so I'll use "data transfers" instead)
I personally don't see the outrage. AmaGoogSoft overcharges for data transfers because they know they can get away with it and that lowering it won't attract more customers.
Customers with transfer-heavy applications will always buy their servers from providers with unlimited transfers like OVH[0][1], where you can do hundreds of terabytes a month with no extra charges (1.5 Gbps * 3600 * 24 * 30 = 486 TB). Even if AmaGoogSoft lowered their transfer prices by 100 fold their pricing still can't compete with OVH.
Companies with enough engineering resources can always go with the best of both worlds: transfer-heavy servers on OVH, and "regular" servers on AmaGoogSoft. The expensive data transfers will only hit smaller outfits, but these customers won't switch because it's not worth the hassle to split your hosting across two providers.
> I personally don't see the outrage. AmaGoogSoft overcharges for data transfers because they know they can get away with it and that lowering it won't attract more customers.
I know a handful of scientists, myself included, who would consider cloud computing were it not for the expensive egress costs. The ease of spinning up lots of computing power for scientific modelling is useless if retrieving the vast quantities of raw output data is costly. I will have to investigate OVH though as a potential opportunity--thanks.
How much raw output data are we talking about? What order of magnitude? Unless we're talking PB a month EC2 prices seem fairly reasonable. Also keep in mind intra-region communication between instances is free
The cloud pricing model has, I would say, definitely been set for ecommerce uses, not scientific computing. Ecommerce uses use a lot less data, and if they are using a huge amount of data they are likely to have a corresponding amount of revenue to pay the high prices.
Scientific computing -- you've probably got a ton of data, and your 'revenue' from project-based grants doesn't really scale with the amount of data and other computing resource needs you have, at least not to the extent it does for ecommerce.
Possibly AWS could make money on scientific data at prices you could afford... but they probably make the _most_ money by targeting their pricing model at ecommerce, which has the most money to spend.
Possibly there's a business model for a cloud-provider focusing on scientific computing. I mean, there probably are such providers already I don't know about? Maybe? But it'd be a tough business, the people paying are pretty budgetarily constrained, their budgets don't neccesarily go proportionaly as their resource needs go up, they traditionally are (ironically) averse to innovation in IT, there might not be enough likely customers to give the provider the scale they need to make money, etc.
Yeah I suppose my argument is a bit moot given Amazon probably makes plenty of money in the private sector without having to worry about academia. But if they could get the cost down to be on the level of buying and running your own hardware, they could see an emergent market.
In fact, the federal government in its current business-oriented mindset may even lean towards a contractor handled machine to replace ones like Yellowstone at NCAR or those at NOAA NCEP if the price was right.
It would make a lot of sense if grants covered such costs, even at current AWS prices.
Currently, I suspect a lot of IT-heavy grant-funded projects are basically relying on university infrastructure being provided without full (or in some cases any) project-based cost recoup, which universities are increasingly loathe to do. The indirect/overhead margin probably doesn't always really cover it. (And it's not like a university is going to let you use 'overhead' dollars for AWS, _even if_ it would actually be a cost savings compared to the local IT you are using instead!)
It's not really rational economic decisions being made all around.
Proper professional IT is expensive. People still think automating is supposed to save them money, but turns out, nope. Many industries are still relying on unprofessionally provided IT instead of paying for professional IT.
You're likely not running a model just for yourself. You have collaborators at other institutions that need data to compare or combine with something else.
Your grant may require you distribute the output for use by other researchers. That could either entail being hosted by you or by an agency or other entity. But you still have to get the data to them.
A reviewer for your publications may request the data.
That brings up another point that the review process can be upwards of a year.
Oftentimes you have to go through an exploratory data analysis, where you don't even know what your final analysis will entail.
"Amazon EC2, Microsoft Azure and Google Gloud Platform are all seriously screwing their customers over when it comes to bandwidth charges."
Disagree. There's no false advertising here, they're making you pay for their service and convenience of using a combined [Paas, Iaas, Saas ..etc]. It's unfair to view these services as a singular function, you typically touch MANY features/products in production. The cost includes the convenience of offering everything under one roof, because, face it, doing everything by yourself at the SLAs provided by the giants is no trivial task.
Unless you're a BIG company that likes to distract itself with infrastructure instead of building and sharpening the core offerings, chances are that you will NEVER really build anything as reliable, inter-operable, configurable and manageable at cost.
This is simply untrue for bandwidth charges if you're pushing a lot of data.
A streaming video service, for example, might end up paying $0.10/user/hour. So a movie-a-night customer would cost $6. What multiple of reasonable do you think that is?
The fact here isn't quite right. (I'm an employee at Dropbox)
Primarily, until 2 years ago we did lean on S3 for all block storage, but most of the rest of the infrastructure (metadata storage, etc) ran in our own datacenters.
Your point I think you're getting at sounds like something I'd agree with though -- you can wait a bit the cost efficiency starts to be what is important/impactful to work on before shifting your usage away from some of these providers.
About 2-3x. I'm a big fan of cloud computing in general (and AWS in particular as the clear leader, IMO) and helped lead our transition out of colos and into AWS, but the egress charges is the biggest area of AWS that I feel is egregiously over-priced.
Fortunately, it's relatively easy to locate some of your very high egress services outside of AWS. I discourage that for casual optimization, but when you get to Netflix/20 scale, that makes sense to trifle with.
“The best way to express it is that everything you see on Netflix up until the play button is on AWS, the actual video is delivered through our CDN,” Netflix spokesperson Joris Evers said.[1]
This comes off as a bit of a red herring. No one is denying the value of their other offerings. It's the cost of bandwidth which is clearly disproportionate.
Let them charge for extra for the 'value adds', what is the justification for adding these costs to the bandwidth specifically?
I think people raising the issue are just seeking more transparent pricing.
This is definitely true but I think that possibly as the big cloud players devote more and more of their revenue towards developing all kinds of (vendor lockin) APIs each of which does not concern the average user, rather than generic and reliable infrastructure that benefits all users, the value proposition will get worse and worse.
Considering my experience with google adwords service, I wouldn't be surprised if the teams providing these services were under heavy pressure to over-promise, under-deliver, and sneak in as many hidden charges as they possibly can.
Very Evil is 'ad-words' when they first started it was ok but in the late 90's they started those ad-clicks all over the world, all of a sudden people in Nigeria were clicking your ad's for $1/click.
Funny thing is every time Google added a new service you were opted in, until you turned it off, you didn't know there was a new service unless your monthly bill went up 10X, and went on like this for years.
99% of our biz was europe-usa, we had turned off all ad's outside those area's, but google kept creating new products and featuring our ad's all over Africa, where we had never shipped a product in our entire history.
Finally simply quit using ad-words, gave up with their games.
AWS is evil, Google is Evil.
If you can't touch it, you don't own it; That includes your data.
That includes your server's, that includes your pipe
I disagree strongly with this. Before all these cloud providers websites were not noticeably less reliable than they are now. I was around back when Apache and CGI was all there was, even then uptime was so good that it was rare to hit a website that was down.
There's a lot of koolaid being thrown around by the companies with cloud to sell. Unfortunately these also happen to be the big "market leaders" so it's hard to deny what they're saying and get taken seriously.
It's like six sigma, agile, or stack ranking. The big guys are doing it so it must be the right thing to do... Right? Until everyone realized it's mostly a ploy to make money selling books and conference tickets, or in this case rent out a bunch of excess capacity for huge profit.
I disagree with reliability as well. Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two. With the cloud you have to worry a lot more about fault tolerance whereas with dedicated equipment simple offline backups are often enough to meet any reasonable SLA.
> Before all these cloud providers websites were not noticeably less reliable than they are now.
Before all of these providers existed:
1) The volume of internet traffic that exists today didn't exist then. Mobile devices didn't exist. Mobile devices that aren't always connected to the internet and consume hours of our day didn't exist. Large downloads (1GB) didn't exist. They couldn't exist because the infrastructure that exists now can properly support it...at scale.
2) Websites that had large volumes of traffic had top tier expensive admins to maintain them (surprise surprise, Amazon did and turned it into a service)
> Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two.
3) What's your definition of "most"? Sources for those numbers please?
As we've gotten more device computers have gotten more powerful. Server software has gotten better. HTTP has Keepalive and multiplexing now. Encryption and networking are offloaded to hardware and we have a lot more cores.
I would guess a single server can handle thousands of times as many users as it could handle years ago. HAProxy, Netty, Nginx, and others can handle over a million (simple) HTTP requests per second. That's more requests than Google.com gets.
Most as in I've been watching over 100 AWS VM's and maybe 30 on Azure for years and they die or crash far more often than than the VM's hosted here, at colo, or our old bare metal machines. It's anecdotal but it seems like AWS doesn't really care about warning you before shutting off your machine. Azure is slightly better but still goes down regularly.
I know everyone says "it's okay! Just make your servers fault tolerant!". Well that works great for load balancers and frontend, but doesn't work at all for SQL databases. ACID compliant transactions require a single source of truth and a true multi master SQL database is impossible. Failover yes, but you always risk losing data in the switchover unless you use two phase commit which actually makes your multimaster database slower than a single system. In practice the failover almost always causes some data loss and log conflicts you have to diddle with later. And God help you if the replica falls behind more than a couple seconds.
Anyways, for SQL databases system reliability is as essential as ever and it's a lot easier to get high SLA numbers when you control the hardware and the power switch. The closest you can get to the Holy Grail is running KVM VMs locally and doing live machine migrations when hardware starts to fail, but even that won't keep your database running if something really bad happens.
Thanks for the info very useful, I didn't realise how unreliable AWS is for database servers. You're correct random unannounced shutdowns of db servers is just not acceptable for critical data. I will be launching a new business based on Postgres soon and the thought of this is terrifying. I'm not keen on the RDS type services or CoLo so this is an unexpected problem I need to overcome. Do you know whether VPS provider such as Digital Ocean or CloudSigma would be more reliable?
I would say colocation is most reliable. I'm sure dedicated VPS is better but most still reserve the right to pull the plug for hardware replacements. Colo isn't terribly expensive if you buy used equipment.
Really consider how important 100% uptime is though. Google and S3 have gone down multiple times without killing the internet or losing a ton of customers. Plenty of large SaaS providers still use maintenance windows. Heck, GitHub went down today. Not sure if you use ADP but that goes down for a couple days a week!
I know it's not the popular thing to do but you can get much better relative reliability by running a single database per tenant and running a limited number of tenants per VM.
I think you're being a bit rose-colored glasses here if you're implying that apache/cgi sites had some sort of incredible uptime record back in the time when "the slashdot effect" was still a thing.
> Most of my bare metal and colocated machines have uptime of many years.
Nobody cares about host uptime, that just means you're not applying security updates. You're lamenting that you have to worry about fault tolerance in the cloud when the entire point is to have throw away instances!
> Nobody cares about host uptime, that just means you're not applying security updates.
Um... Only kernel updates used to require a reboot, and production security-critical ones are fairly rare.
Solutions like ksplice, kexec, kpatch, or kgraft have existed since about 2011, and now the Linux kernel has first-class support for "no reboot" updates.
You're comparing the simplist possible scenario with a platform that can replace a large data center.
How do you handle segmentation of workload among thousands of servers? Provide firewall services? Meter bandwidth? Provide redundancy to protect against failure of a switch or any other single network component? Provide redundant transit via multiple ISPs?
The answer historically is that you would buy a bunch of stuff from Cisco, pay through the nose for TAC, and hire a team of network engineers who may or may not be idiots. One place I worked at lived with a 30 day eta in firewall changes.
Your server that has a long uptime is a risk in a large org. It's obviously not patched, nobody knows how to configure it again. Different customers have different needs.
You don't need anything but regular PC's these days...
2 front end IPVS servers running MAC layer direct return. Those can handle close to 40 gigabits each. If that's not fast enough hack something together with DPDK or netmap.
Firewall is on the front end load balancers. Linux is one of the best firewalls you can get if you configure it right.
Redundant L2 switches past that running RIPV2 or OSPF for routing. I've found that crappy consumer quality switches usually work fine, as long as you wire them in parallel. You can do dumb things like wire them quad redundant and it just works.
Redundant transit handled by datacenter level multihoming.
Extra redundancy using dual data centers if you want with DNS round robin.
Buy dedicated lines to avoid bandwidth costs.
You can rent a cage and do all this with used last gen dell workstations running Ubuntu. Get some last gen fibre NIC's off eBay too. Total cost of maybe 2k in hardware to run top 500 site levels of traffic...as long as you're not using something piss slow like PHP.
Historically how shitty your setup is depended on how good your network guys are, the cloud just took that out of the equation.
Now everybody can have a great setup as long as they pay out the nose.
Edit: Also you can patch Linux without reboots, even the kernel. I don't remember the last time I had to reboot from patching.
You do need to reboot for major version upgrades but if you stick with LTS versions you're usually good for 5-10 years
I don't see any claims about false advertising. Just complaints about one aspect of the service being very overpriced. Sounds like you consider it worth it for your application, and that's great. But it is simply not priced appropriately for a lot of services, which naturally go elsewhere.
It seems obvious to me that bandwidth is Azon's proxy measure for "utility". Lots of companies do this - Oracle uses core-count as a proxy for the "utility" you get from their software, for example. But like any proxy for a different variable, it is going to be wrong; sometimes a little, sometimes a lot.
And you hardly need to be a "BIG" company to choose something other than cloud hosting. My last two looked at and rejected them. Neither qualify as even medium-sized. Both are extremely cost-aware. Running our own data centers is cost-competitive at my current gig and was much, much cheaper at the last, which was very bandwidth-heavy.
If every cloud service arguably overprices one essential part of delivery, bandwidth usage in this case, my instinct (no more, no less and no pretension to analysis) is that there's a design intent to ensure a difficult to price component of the service provision is paid for.
I'm merely speculating, but if you have a idea what is the possible difficult to price component, can anyone help with any suggestions?
Edit to suggest a similitude: my job is to help people understand the price of (print) advertising. This is often intractable. To a certain extent, a good deal of people involved just throw costs into commission rates and other factors that make understanding the price models difficult and sometimes completely opaque. Is it impossible to imagine that the cost of cloud bandwidth is nothing less innocent than "put our non itemized expenses here'?
This is true, but I think there's another point that's even more significant: not all network access is the same. In particular, putting some machines in a rack in a colo and connecting to the nearest Internet exchange is really not comparable to Google's networking, which is essentially a worldwide second Internet that is faster and more reliable than the normal Internet, and which carries your packets as far as possible towards their destination before dropping them off at a POP.
Can't agree more. I balked at the cost (still sort of do as a GCP user) until I joined Google and was able to peek behind the curtain. The network quality is absurd. GCP customers pay a premium for having the same network reach as Google. Their network works extremely hard to get user traffic onto the internal network as early as possible.
Not only that, but Google's datacenters have 24/7 attention, lots of redundant providers, etc.. A bargain basement colo won't be nearly as reliable.
Yes, you're right, and you mentioned something else that I forgot to. A lot of people here are comparing Big 3 network costs to the cost of hiring a single network administrator. The correct comparison is to the cost of hiring a team big enough to staff a 24-hour on call rotation with a 5 minute response time SLA.
The OP has no fucking clue on this topic. If the goal is to get the cheapest bandwidth possible, by all means... I stopped reading after seeing Google Fiber in the comparison.....
>Their network works extremely hard to get user traffic onto the internal network as early as possible.
The benefit of this has can't be understated. In SE Asia connecting to their Taiwan region from most places I'm literally just at the mercy of a few local hops at level 3 due to their extensive remote peering. Staying on the Google network is also interesting to observe when connecting between Google Compute data centers.
Their business customers probably really do care. Companies run off of AWS and GCP. I'm sure that spotify and Snapchat and Netflix get considerable value from those guarantees.
While public bandwidth is indeed significantly more expensive on clouds, not all traffic needs to be public and different clouds charge different amounts.
I wrote a blog post on Google Cloud latency and pricing across zones and regions which may be useful for others:
It's suffice to say that the cloud providers have a different set of customers in mind. I have servers on both OVH/Linode variety of service providers as well as one single app running on AWS. For the products I run on OVH/Linode, I sell the service at less than $20/month. The one on AWS sells for $200+ per month. Again, it's because of the requirements/SLAs. Based on experience, AWS is a lot more robust for what I'm using it for.
Pitchfork mode on: Outgoing bandwidth should be even more expensive! Then maybe, just maybe my mobile data cap wouldn't be drained that quickly by bloated webpages and stuff.
This calculation is an all or nothing affair though. The reality is that you can avoid most AWS bandwidth charges by direct connecting to an internet exchange. It's still only worth the effort if you're looking at a 6 figure bandwidth bill, but it doesn't require new employees.
I guess what I'd like to know is: at what point is it worth seriously evaluating alternatives to just using cloud services and paying the sticker price -- which means evaluating the labor cost as well as the cost of the service, where this submission and many others I've seen will quantify the cost of the services and say that bare metal is so much cheaper while ignoring the cost of managing it.
The general break even point is circa 200k USD per month, assuming the company has spare capital, and the ability to commit to using the same gear for 3 years.
It's useful to keep in mind that in this type of scenario the goal is not to replace a cloud provider but to leverage capital to get better bang for the buck on select services. ex. using in-house compute farms and leveraging s3 for storage
50 rps is a very modest box you obviously will want at least 2 but at any rate the difference between that and AWS is not a full time employee it's more like 10h per month. Why would colo data transfer cost $30/TB :)?
The $30 number comes from the original article -- If I were to make the colo data transfer free and change nothing else, break even point is 100TB/mo.
For the employee cost I agree if you just have a couple boxes pumping out data your labor cost will be quite low. I'm imagining what else you might have happening on your back end using AWS services that you'd also have to replicate in your colocated installation to preserve the efficiency, which would take labor. For example, if you want some extra machines to do some load testing. Or if you need to store a bunch of data and want an S3-like system. Or if you want to wipe a machine and reinstall from an "image". The backup system, and backups testing. RDS, Aurora, DynamoDB, etc. Most of that can be done with a couple clicks on AWS.
it's $20 at DigitalOcean which is already heavily marked up :) compared to colo. The sweet spot for AWS is low traffic project that uses a very large number of AWS services or something with extremely spiky workload. It's a legend that you don't need ops people for AWS. We have a fairly complicated large scale project on AWS we have 4 full time ops people. If we were in colo we would need 5-6 instead of 4 and a bit of remote hands services.
This analysis is way too oversimplified. It completely ignores the shape of the traffic (real apps have peaks and valleys of usage - they don't pump exactly 100 Mbps every second of every day). Cloud providers charge the same amount regardless of how bursty your app is, and they have to provision capacity so that all customers get good performance even under unusual spikes (the more spiky your traffic, the better a deal per-MB pricing is for you). And of course it ignores all the ancillary networking HW and SW that supports these services, and all the labor you save by not having to manage that stuff yourself.
I've analyzed the cost of cloud services to death (I've worked for a couple of them) and the only way they aren't great deals is if you don't need high quality operations (i.e. if you can deal with slow-downs or occasional outages then you can do better elsewhere). Otherwise, if you're small-scale then these marginal cost differences don't matter, and if you're larger scale then call up these cloud providers and get yourself a discount off the list price.
If colocation actually cost that much, it would make sense for a connection that allows extreme bursting to charge 3x as much per byte.
The real number to compare to is the google data for business rate. You can do lots of colocation in that price range. And that is why the cloud prices are unreasonable.
This is a super irritating feature of "enterprise" vendor pricing. What almost everyone on these platforms is doing is moving most of their bandwidth out of one of the CDN services (like say cloudfront) and then negotiating custom pricing on that bandwidth that is often as much as a full decimal place cheaper as long as you sign a couple grand a month yearly commit. There's still this massive massive pricing cusp between using the cloud as a utility and jumping into the suits & drinks & lunches sales guy game.
The big cloud platforms offer a rich selection of different offerings, which (just like in every other industry) cross subsidize each other.
When I go to a restaurant, I don't expect that they will be making the same profit margin on every item on the final bill, and in fact, they almost never do. Drinks tend to have a very high profit margin, some labour intensive items may be a break even at best, and the complimentary bread sticks or chips and salsa (if offered) will certainly be a loss.
I guess I could write a very upset article about how my local mexican restaurant is SERIOUSLY SCREWING ME OVER with their drink prices, but if I don't write the companion piece about their cheap burritos (subsidized, of course, by the drink prices), it would only show half the picture.
The reality is that I'm buying a whole package (at AWS or a restaurant) and I should evaluate the whole picture. Yes, I can get bandwidth cheaper outside AWS (or a can of coke a lot cheaper from a big box retailer). But I can't really get the total package of integrated, managed services outside AWS (certainly not for the cost they charge), any more than I can get someone else to show up in my kitchen and cook a three piece meal and then do all the dishes. (Which is to say, I totally could hire a chef to do that, but it would cost me a lot more. I could BUILD an internal SQS clone if I had to, but my employer would never break even on the cost of getting me to do so.)
AWS is very cheap for some things and very expensive for others. Depending on your usage and workload it may or may not be economical to buy the package they offer. If it is, go for it. If not, don't. Just like, you know, every other good or service you purchase in both your personal and professional life.
There's a site on Neocities we host for free that would cost $560/mo to host on Amazon's CDN.
It costs me less than $5 with my current configuration (which is also a global CDN). It's way over our "soft limit", but it's an awesome site so I don't care. The important part is, I don't have to care.
This isn't about slightly more expensive tacos. It's about spending $560 on tacos instead of $5. "Meh" wouldn't exactly be my first reaction to getting that bill at the taco cart (or the fanciest taco place in the world, for that matter).
I can get IP transit in datacenters from $240-$600/Gb right now. So even his $960 transit cost for datacenters is off by quite a bit. He's comparing with a pretty high price and it still looks ridiculous.
I'm not going to attempt to save an inconsequential amount of money by hosting a site I'm hosting on another host.
But that strategy probably wouldn't end well. The $560 site uses 25x more bandwidth than Github's fairly low soft limit of 100GB (https://help.github.com/articles/what-is-github-pages/) and is likely above the 1GB site limit (which is also just the sum of any changes ever, because git) of a Github pages site.
From seeing who their CDN provider is (one that charges basically the same CDN rates as AWS), my guess is that Github is paying a lot more for CDN transit than I am. It's perhaps not a coincidence that the 100GB BW limit was quietly introduced after they started using the CDN.
Yeah Fastly (github's CDN) is ridiculously expensive and they charge for requests just like cloudfront which can become very expensive. They are the only major CDN provider other than cloud services which charges for requests.
We went from cloudfront -> edgecast -> keycdn and our bill dropped from $3000 -> $500 -> $120
Edgecast when you buy is through a reseller is quite cheap but we moved because we needed custom domain SSL which is quite expensive in edgecast
I realize this is far away from the point, but that infographic starts off stupid. Painting has a 6,000% markup? That could only possibly be true if you don't count the painter's time. Just because it's all labor doesn't mean it's high markup; the painter isn't selling you the paint he's selling you the service.
And mixed drinks are more like a 4x markup. Lower at the high end even. Not sure where they pulled that number from. If it's anything like the coffee total, they'er comparing the garbage ingredients people typically use at home against the highest quality ingredients in the best places.
I bet that site loses money for Neocities every month. But it's subsidized by the tens of thousands of $5 sites that don't consume their allotment of resources.
I look forward to your promised blog posts on how you've affordably built your scalable infrastructure.
Is the basic principle to use cheap cloud VPSes or dedicated hosting (e.g. of the Digital Ocean, Hetzner, or OVH ilk)? And then you can afford lots of servers, and have then geographically distributed?
Perhaps an even better analogy is popcorn and soda at the movie theater. Its 10x the price of buying it at costco (perhaps 100x), but you are paying for the movie. In the world of cloud computing this is things like TPUs. I think the lesson here is if you are merely looking for connected bandwidth, maybe cloud providers aren't for you. Shop elsewhere. The point of cloud computing is clearly moving in the direction of data-driven services, at large scale, for big businesses.
Agreed... this article makes a useless one-dimensional comparison of several different services based on their pricing for a single aspect of each of those services. I'm not saying there aren't price discrepancies, or opportunities for more competition that might benefit consumers and enterprises... but the comparisons as portrayed in this article are just not useful except to show how pointless such one-dimensional comparisons are.
> I guess I could write a very upset article about how my local mexican restaurant is SERIOUSLY SCREWING ME OVER with their drink prices
I notice that I drink less (too little, even, if I'm out for an evening) because of these prices. If each drink costs a few euros for 200ml glasses, I'd be crazy to drink a liter if at home the cost is somewhere between negligible (tap water) and 1.50 for that whole liter.
What I don't understand is why they can't spread out the profit margin. It'd make me feel a lot better about buying drinks and I'd probably spend more than I do now. Courses are a few euros in ingredients but I don't hesitate to pay €9-15 for it because I know labor is involved and someone has to pay rent. For drinks there is hardly any labor or ingredient cost (assuming all I ask is tap water, which is often since Dutch tap water tastes better than bottled water, and it's not cooled). Prices might also be higher because people don't order it in great quantities, which they might if restaurants wouldn't inflate drink prices to pay for rent (and other such costs), which would be so if you just pay per hour for a seat or something. I heard some places in Italy do that and people were surprised to be charged for it, but it's just itemizing costs and in my opinion more transparent.
That would be workable if humans would be perfectly rational, but they are not. If I set up a pizza shop with totally fair prices across the board, my pizza will cost 8 EUR and the drinks maybe 1 EUR. The pizza shop across the street will however put up a ginormous billboard advertising their 5 EUR pizza with a peace of free pizza bread (and have the 3 EUR drink prices in the small print menu). I get very few customers with my "normalized prices pizza and drinks" billboard. Most people would be just like: WTF? and go to the other shop.
People just don't behave rationally to this. Restaurants learned ages ago that people focus on entree pricing, as a result their is little to no margin on many main dishes (sometimes even negative). This is made up on wine, desserts, things that people have price elasticity on or poor information or both.
All that said, the operating margins on high end restaurants are ridiculously low. Franchises and fast food do better at this (economy of scale), but if it is a proper linen-table-cloths and local sourcing sort of place, they are playing with very thin margins at the best of times.
232 comments
[ 2.7 ms ] story [ 250 ms ] thread- http://www.gcping.com/
(From your browser to multiple Google zones)
For example: A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month.
IMHO the big cloud providers only make sense if you have a highly elastic load or if you are making extensive use of their as-a-service higher level offerings. If you're running baseline load bare metal destroys them.
And then magically maintain it for free?
Yeah, at the larger sizes it might make sense, but not for everyone.
I worked for a MSP for many years, and was well acquainted with our regional competitors. Our uptime and theirs was nowhere near AWS-level.
You're getting a different engineer every time you pick up the phone. They most certainly do not know your systems like an internal team would. I regularly saw cascading/circular issues caused by lack of familiarity and/or poor change management.
Owning and administering your own iron makes sense at a certain scale, but it's a much bigger scale than most companies will reach.
What do you do when they're on vacation/sick/sleeping?
Thing is, I really want us to host our own stuff. For one thing if you don't host anything you lose all competence and then you really are beholden to your cloud provider.
But it seems to be that we are in this situation until someone figures out how to comodify servers - and server orchestration - to the point where this stuff gets cheaper to manage again.
This pendulum always swings. No sooner will we control our own hardware again than someone will come up with a new way to centralize it.
With a bare metal server you have to manage all of that.
It's not a bed of roses in the physical world either, but you're simply wrong to say it's easy in the "Cloud" and hard in a DC/Bare Metal.
I find it amusing that people pretend without bare metal, you'll get all of that person's time back.
It really, really depends on on your use case, but there are some scenarios where bare metal just makes more sense. Also there are ones where cloud makes everything much easier.
North America-oriented B2B app that's a JavaScript SPA witch connects to some lightweight services fronting a database? I think that's a great contender for the cloud.
500-node Hadoop cluster running 24/7 with heavy load? You are going to see some huge savings by maintaining a Datacenter. You can use the extra 1.7MM/mo. to set up some robust networking equipment and have few people on-site.
Like every IT trend, "Cloud ops" is the trendy mind-slug that eats everyone's brain and makes them say that everyone who doesn't jump on the bandwagon is going to die broke and unloved in the gutter.
Remember how XML meant massively reduced integration costs and firing all your developers because GUI tools would let business managers connect pretty boxes and then kick back to watch the graphs go up and to the right as business exploded?
Yeah, so AWS is great, if you're building something that works well on AWS. And that's a large class of apps - if you're throwing up a Drupal front end to a CRUD line-of-business backend, need to push notifications and send some email, sure. You're dead in the middle of their target market.
Doing something interesting? Doesn't even have to be as massive as the parent suggests - if you're doing something that blows any of the billable parameters out of the sweet-spot, like bandwidth, storage, latency requirements, things that depend on system-locality, etc., you're much better off building, and using AWS for the pieces which can be broken off that don't hit the pain points.
Minimum wage => You get what you pay for.
Using EBS volumes with provisioned IOPS give you some advantages that you don't get with a bare SSD -- like snapshot support and the ability to easily detach it and move it to another server.
http://www.aerospike.com/benchmarks/amazon-ec2-i3-performanc...
Given the price of the server and the data you have on it, it's important enough to warrant a hot spare.
But if you are comparing it to a purchased server, you ought to be looking at Reserve pricing, a 1 year reserved instance is $16K/year, 3 years is $32K (or you can pay monthly if you want).
Still not as cheap as owning the server, but not quite as bad.
edit: I had selected a dedicated instance in order to compare apples to apples. By definition you don't share a colo server.
https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst...
For smaller instances, there is around a 10% increase for dedicated, for example, for an m4.4xl, the price goes to $0.88 for dedicated tenancy versus $0.80 for default.
There is a $2/hour fee for each region in which you have dedicated instances, but that gets lost in the noise if you're using significant AWS resources. (it's $2/hour regardless of how many instances you're running)
We haven't noticed any significant difference between performance on dedicated and non-dedicated instances, and only use dedicated instances where required for compliance/regulatory reasons.
My smallest physical servers have 8 physical cores.
It is the best extension.
2. Will Dell servers arrived with 0 problems on its components? Any time I ordered meaningful amount of servers, I usually get about 5% fail rate. VMs aren't perfect but you can destroy and recreate in different region almost real time.
3. Ever had to deal with difficult-to-work-with network admin? The cloud is significantly less pain in the ass.
There are network engineers in the world who don't live in caves and gnaw at the bones of administrative assistants.
And Amazon allowed to to achieve amazing things that never would have been possible and that's without the idea of things like Kinesis or Redshift that we couldn't have achieved easily/at all even without personnel problems. You simply could never get 3 new servers or drastically respec an existing one in under a day let alone minutes. We were a small shop so it's not like we had spare servers ready to go.
Networking is a great place for assholes to build empires and exert control. In 20 years of professional engagement with distributed and data center networks, most of these guys (and they are almost always guys) running network orgs are an impediment and spend more time helping out their vendor of choice than anything else.
I've run into awesome network guys in position of power only a few times, and they are 100x employees. The last major rollout of a service that I did was literally 3 months ahead of schedule solely because of the efforts of this awesome network dude snowman as recently put in charge.
Eventually replaced with someone helpful, it was amazing what we started to get done. Got lucky there we found the replacement and he stayed long enough to be there when needed.
Worst person I've ever worked with. When I tell stories at future jobs I fully expect people to not believe me. Hell, people who weren't there for earlier incidents often didn't believer them.
Just basic docker use (or something similar) would have made a HUGE difference. Parts of the department were sent to training but it didn't happen because it would have made him less indispensable.
At one point someone higher up (but not high enough) asked how many servers we had in Amazon and the answer like 10x our physical set.
"And only one of you manages them?"
"On the side, we don't usually have to touch them."
"Then why does it take 4 people for the other servers?"
(Collectively) "Yeah. Why IS that?"
Change control was a big problem too. "Random" changes, things that didn't change at all (even when we could prove it), stuff like that causes SO MANY issues, basically all avoidable.
Also, hiring people is a risk, and it doesn't happen instantly. When you have a gap of months without a sysadmin things are bad. Hiring a bad sysadmin is even worse. Not to mention the impossibility of hiring 1/2 or 3/4 of a sysadmin. There's scaling limits there. If you have the resources to DIY, that can be a key competitive advantage, but not everyone does.
It's no secret that AWS is a rip off on instance pricing. Use the competitors if you worry so much about pricing.
What are most people running anyway? Using this kind of behemoth server as a 'typical' example seems out of line.
AWS m4.16xlarge => $2592
Google base is slightly lower and it gives you 30% off automatically on instances that run 24/7 for the month.
That's typical of on premise deployments. You get big servers because they are so annoying to deploy you might as well fill everything possible in the box.
On the clouds, you would create a group of VM per service, with appropriate size for the service. It is significantly more efficient.
Cherry-picking a bit there. You're entering a 3-year commitment with the Dell lease, but not with the EC2 lease. Take a 3-year RI on that EC2 instance, and the price drops considerably. If you also take away the 'dedicated' requirement, the 3-year RI is about twice as expensive, not ten times.
Edit: correction from Johnny555 above: this top-tier server doesn't have a 'dedicated' option, so there's not an 'extra' cost for that. It's just twice the price, if you do a 3-year commitment (as you are doing for the dell)
It is interesting that they end up charging you this for _bandwidth_ though. Their staff time doesn't actually probably scale with bandwidth, but their prices do.
I think it's probably kind of a way of pricing based on a proxy for customer size, likely revenue, and ability/willingness to pay. Rather than as a % margin on actual costs.
1) remote hands
2) network switches
3) hardware components you'll need for redundancy, I've personally replaced network cards because fans burned out or HDDs because of usual failure rate.
4) when I'm done using a n1-highcpu-32 I stop using it and not pay for it, or when I need 5 of those for 2 hours I provision them and stop when I'm done running a workload.
5) upgrades to hardware or additional hardware, intel released skylake, should I buy those and try to sell my Haswell on eBay?
I'm not sure it becomes a simple price of server + colo cost comparison.
With GCP (I use daily), AWS (I used to use) or Azure, what I'm paying for is someone else to focus on things like colo leases, capacity expansion, redundancy, physical security and provide higher order services - load balancers, BigTable/BigQuery, CloudSQL/RDS, GKE, DataProc.
This allows me to run all of infra as close to capacity as possible while avoiding any wasted cycles
However I do miss going to data centers and hear my machines hum and blink, that is quite the feeling.
Timeline: 120 days procurement (including product evaluation and PoC), 30 days install activity, 10 days buildout. 10 days software build, 15 days testing.
That's a little over 6 months. A ton of vendor/build engineering/tester time. And we'll be at 50% capacity for another 4 months. Or... we could spin the whole thing up on AWS/GCE/Azure in < 30 days and only consume what we need when we need it.
Maybe worth renting an office in Provo just to get the deal.
I've done the math many times and it's orders of magnitude cheaper to colocate as long as you can afford an IT guy and the upfront cost of hardware.
What happens when a harddrive fails in your colo? Unless you have best practice backups, you will lose customer data and trust.
GCloud abstracts the issue of hardware so you can focus on real business development. And to some, that's worth the cost until they can properly afford all the hidden costs of independent hosting.
Really though in my experience the cloud is far less reliable than colocation. I've had AWS VMs "degrade" or randomly die countless times but I can't remember the last time my Colo boxes went down, probably not since I last upgraded the OS.
It may be. But for many people, it does not matter.
Let's say you work at the place (I do) where "compute" cost is like ~0.5% of the total costs. Almost all of the cost is employee's salaries and benefits.
BTW we get (and from what I hear, everyone does) offers from other cloud vendors to switch, and get 1 year free or whatever. And it gets ignored because it is a big risk for a tiny possible gain.
It's really just a new form of vertical integration. If you're working only at the high levels you're not only paying someone to do that rest for you, those people are making profit off of you. Profit that could be yours.
Still, recouping the cost of the NIC by making internal bandwidth free and Internet transfers really expensive is kind of a weird way to do it.
In any case, the hardware components are fixed costs. What's not fixed is the cost of applying routing and segmentation rules for each packet. There are scarce resources involved here: the hardware has limited memory and limited cycles. You're paying for the use of those scarce resources (e.g., having a memory resident stateful firewall rule for a TCP session). Charging for bandwidth isn't perfect, but it probably correlates pretty closely with the underlying resources and is a much more intuitive unit-of-value for customers.
The costs of doing those (well) yourself are not cheap.
Getting them from a provider that's certified to do them well while giving you software control also isn't cheap.
You're comparing cost of gas per gallon to to expense of miles driven per gallon. Pretty sure on your IRS or corporate expense report those aren't the same.
It takes times and skill to turn the ingredients into something useful.
Amazon, Google, and Microsoft are IaaS/PaaS, not ISPs.
Sell food at cost!
/s
When you put it all together, you're not paying for bandwidth, you're paying for the power and convenience of a robust, high-quality, high-availability solution that provides extraordinary benefits when looked at holistically.
[1] https://www.digitalocean.com/pricing/
[0] http://tldp.org/HOWTO/Traffic-Control-HOWTO/intro.html
I personally don't see the outrage. AmaGoogSoft overcharges for data transfers because they know they can get away with it and that lowering it won't attract more customers.
Customers with transfer-heavy applications will always buy their servers from providers with unlimited transfers like OVH[0][1], where you can do hundreds of terabytes a month with no extra charges (1.5 Gbps * 3600 * 24 * 30 = 486 TB). Even if AmaGoogSoft lowered their transfer prices by 100 fold their pricing still can't compete with OVH.
Companies with enough engineering resources can always go with the best of both worlds: transfer-heavy servers on OVH, and "regular" servers on AmaGoogSoft. The expensive data transfers will only hit smaller outfits, but these customers won't switch because it's not worth the hassle to split your hosting across two providers.
[0] https://www.ovh.com/us/private-cloud/options/bandwidth.xml
[1] https://www.ovh.co.uk/web-hosting/unlimited_traffic
I know a handful of scientists, myself included, who would consider cloud computing were it not for the expensive egress costs. The ease of spinning up lots of computing power for scientific modelling is useless if retrieving the vast quantities of raw output data is costly. I will have to investigate OVH though as a potential opportunity--thanks.
Scientific computing -- you've probably got a ton of data, and your 'revenue' from project-based grants doesn't really scale with the amount of data and other computing resource needs you have, at least not to the extent it does for ecommerce.
Possibly AWS could make money on scientific data at prices you could afford... but they probably make the _most_ money by targeting their pricing model at ecommerce, which has the most money to spend.
Possibly there's a business model for a cloud-provider focusing on scientific computing. I mean, there probably are such providers already I don't know about? Maybe? But it'd be a tough business, the people paying are pretty budgetarily constrained, their budgets don't neccesarily go proportionaly as their resource needs go up, they traditionally are (ironically) averse to innovation in IT, there might not be enough likely customers to give the provider the scale they need to make money, etc.
Capitalism!
In fact, the federal government in its current business-oriented mindset may even lean towards a contractor handled machine to replace ones like Yellowstone at NCAR or those at NOAA NCEP if the price was right.
Currently, I suspect a lot of IT-heavy grant-funded projects are basically relying on university infrastructure being provided without full (or in some cases any) project-based cost recoup, which universities are increasingly loathe to do. The indirect/overhead margin probably doesn't always really cover it. (And it's not like a university is going to let you use 'overhead' dollars for AWS, _even if_ it would actually be a cost savings compared to the local IT you are using instead!)
It's not really rational economic decisions being made all around.
Proper professional IT is expensive. People still think automating is supposed to save them money, but turns out, nope. Many industries are still relying on unprofessionally provided IT instead of paying for professional IT.
You're likely not running a model just for yourself. You have collaborators at other institutions that need data to compare or combine with something else.
Your grant may require you distribute the output for use by other researchers. That could either entail being hosted by you or by an agency or other entity. But you still have to get the data to them.
A reviewer for your publications may request the data.
That brings up another point that the review process can be upwards of a year.
Oftentimes you have to go through an exploratory data analysis, where you don't even know what your final analysis will entail.
Disagree. There's no false advertising here, they're making you pay for their service and convenience of using a combined [Paas, Iaas, Saas ..etc]. It's unfair to view these services as a singular function, you typically touch MANY features/products in production. The cost includes the convenience of offering everything under one roof, because, face it, doing everything by yourself at the SLAs provided by the giants is no trivial task.
Unless you're a BIG company that likes to distract itself with infrastructure instead of building and sharpening the core offerings, chances are that you will NEVER really build anything as reliable, inter-operable, configurable and manageable at cost.
A streaming video service, for example, might end up paying $0.10/user/hour. So a movie-a-night customer would cost $6. What multiple of reasonable do you think that is?
Primarily, until 2 years ago we did lean on S3 for all block storage, but most of the rest of the infrastructure (metadata storage, etc) ran in our own datacenters.
Your point I think you're getting at sounds like something I'd agree with though -- you can wait a bit the cost efficiency starts to be what is important/impactful to work on before shifting your usage away from some of these providers.
I don't know if it's true or not but I heard the story.
Fortunately, it's relatively easy to locate some of your very high egress services outside of AWS. I discourage that for casual optimization, but when you get to Netflix/20 scale, that makes sense to trifle with.
“The best way to express it is that everything you see on Netflix up until the play button is on AWS, the actual video is delivered through our CDN,” Netflix spokesperson Joris Evers said.[1]
[1] http://www.networkworld.com/article/3037428/cloud-computing/...
Let them charge for extra for the 'value adds', what is the justification for adding these costs to the bandwidth specifically?
I think people raising the issue are just seeking more transparent pricing.
Funny thing is every time Google added a new service you were opted in, until you turned it off, you didn't know there was a new service unless your monthly bill went up 10X, and went on like this for years.
99% of our biz was europe-usa, we had turned off all ad's outside those area's, but google kept creating new products and featuring our ad's all over Africa, where we had never shipped a product in our entire history.
Finally simply quit using ad-words, gave up with their games.
AWS is evil, Google is Evil.
If you can't touch it, you don't own it; That includes your data.
That includes your server's, that includes your pipe
Google is NSA - Putin
"Do no Evil"
Amazon is CIA
There's a lot of koolaid being thrown around by the companies with cloud to sell. Unfortunately these also happen to be the big "market leaders" so it's hard to deny what they're saying and get taken seriously.
It's like six sigma, agile, or stack ranking. The big guys are doing it so it must be the right thing to do... Right? Until everyone realized it's mostly a ploy to make money selling books and conference tickets, or in this case rent out a bunch of excess capacity for huge profit.
I disagree with reliability as well. Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two. With the cloud you have to worry a lot more about fault tolerance whereas with dedicated equipment simple offline backups are often enough to meet any reasonable SLA.
Before all of these providers existed:
1) The volume of internet traffic that exists today didn't exist then. Mobile devices didn't exist. Mobile devices that aren't always connected to the internet and consume hours of our day didn't exist. Large downloads (1GB) didn't exist. They couldn't exist because the infrastructure that exists now can properly support it...at scale.
2) Websites that had large volumes of traffic had top tier expensive admins to maintain them (surprise surprise, Amazon did and turned it into a service)
> Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two.
3) What's your definition of "most"? Sources for those numbers please?
I would guess a single server can handle thousands of times as many users as it could handle years ago. HAProxy, Netty, Nginx, and others can handle over a million (simple) HTTP requests per second. That's more requests than Google.com gets.
Most as in I've been watching over 100 AWS VM's and maybe 30 on Azure for years and they die or crash far more often than than the VM's hosted here, at colo, or our old bare metal machines. It's anecdotal but it seems like AWS doesn't really care about warning you before shutting off your machine. Azure is slightly better but still goes down regularly.
I know everyone says "it's okay! Just make your servers fault tolerant!". Well that works great for load balancers and frontend, but doesn't work at all for SQL databases. ACID compliant transactions require a single source of truth and a true multi master SQL database is impossible. Failover yes, but you always risk losing data in the switchover unless you use two phase commit which actually makes your multimaster database slower than a single system. In practice the failover almost always causes some data loss and log conflicts you have to diddle with later. And God help you if the replica falls behind more than a couple seconds.
Anyways, for SQL databases system reliability is as essential as ever and it's a lot easier to get high SLA numbers when you control the hardware and the power switch. The closest you can get to the Holy Grail is running KVM VMs locally and doing live machine migrations when hardware starts to fail, but even that won't keep your database running if something really bad happens.
Really consider how important 100% uptime is though. Google and S3 have gone down multiple times without killing the internet or losing a ton of customers. Plenty of large SaaS providers still use maintenance windows. Heck, GitHub went down today. Not sure if you use ADP but that goes down for a couple days a week!
I know it's not the popular thing to do but you can get much better relative reliability by running a single database per tenant and running a limited number of tenants per VM.
> Most of my bare metal and colocated machines have uptime of many years.
Nobody cares about host uptime, that just means you're not applying security updates. You're lamenting that you have to worry about fault tolerance in the cloud when the entire point is to have throw away instances!
Um... Only kernel updates used to require a reboot, and production security-critical ones are fairly rare.
Solutions like ksplice, kexec, kpatch, or kgraft have existed since about 2011, and now the Linux kernel has first-class support for "no reboot" updates.
How do you handle segmentation of workload among thousands of servers? Provide firewall services? Meter bandwidth? Provide redundancy to protect against failure of a switch or any other single network component? Provide redundant transit via multiple ISPs?
The answer historically is that you would buy a bunch of stuff from Cisco, pay through the nose for TAC, and hire a team of network engineers who may or may not be idiots. One place I worked at lived with a 30 day eta in firewall changes.
Your server that has a long uptime is a risk in a large org. It's obviously not patched, nobody knows how to configure it again. Different customers have different needs.
Firewall is on the front end load balancers. Linux is one of the best firewalls you can get if you configure it right.
Redundant L2 switches past that running RIPV2 or OSPF for routing. I've found that crappy consumer quality switches usually work fine, as long as you wire them in parallel. You can do dumb things like wire them quad redundant and it just works.
Redundant transit handled by datacenter level multihoming.
Extra redundancy using dual data centers if you want with DNS round robin.
Buy dedicated lines to avoid bandwidth costs.
You can rent a cage and do all this with used last gen dell workstations running Ubuntu. Get some last gen fibre NIC's off eBay too. Total cost of maybe 2k in hardware to run top 500 site levels of traffic...as long as you're not using something piss slow like PHP.
Historically how shitty your setup is depended on how good your network guys are, the cloud just took that out of the equation.
Now everybody can have a great setup as long as they pay out the nose.
Edit: Also you can patch Linux without reboots, even the kernel. I don't remember the last time I had to reboot from patching.
You do need to reboot for major version upgrades but if you stick with LTS versions you're usually good for 5-10 years
I don't see any claims about false advertising. Just complaints about one aspect of the service being very overpriced. Sounds like you consider it worth it for your application, and that's great. But it is simply not priced appropriately for a lot of services, which naturally go elsewhere.
It seems obvious to me that bandwidth is Azon's proxy measure for "utility". Lots of companies do this - Oracle uses core-count as a proxy for the "utility" you get from their software, for example. But like any proxy for a different variable, it is going to be wrong; sometimes a little, sometimes a lot.
And you hardly need to be a "BIG" company to choose something other than cloud hosting. My last two looked at and rejected them. Neither qualify as even medium-sized. Both are extremely cost-aware. Running our own data centers is cost-competitive at my current gig and was much, much cheaper at the last, which was very bandwidth-heavy.
I'm merely speculating, but if you have a idea what is the possible difficult to price component, can anyone help with any suggestions?
Edit to suggest a similitude: my job is to help people understand the price of (print) advertising. This is often intractable. To a certain extent, a good deal of people involved just throw costs into commission rates and other factors that make understanding the price models difficult and sometimes completely opaque. Is it impossible to imagine that the cost of cloud bandwidth is nothing less innocent than "put our non itemized expenses here'?
(Note: I worked for Google until 3 days ago.)
Not only that, but Google's datacenters have 24/7 attention, lots of redundant providers, etc.. A bargain basement colo won't be nearly as reliable.
The benefit of this has can't be understated. In SE Asia connecting to their Taiwan region from most places I'm literally just at the mercy of a few local hops at level 3 due to their extensive remote peering. Staying on the Google network is also interesting to observe when connecting between Google Compute data centers.
AWS and Azure behave considerably different.
Not everyone need a ritzy ultra reliable network. It costs way too much and hardly any customer is going to get comparable value out of it.
I wrote a blog post on Google Cloud latency and pricing across zones and regions which may be useful for others:
https://blog.elasticbyte.net/comparing-bandwidth-prices-and-...
;)
Take credit card churning and apply it to cloud data. Build tools to seamlessly move apps between cloud providers.
- assume $100/TB for cloud data transfer
- assume one employee full time equivalent to manage colo'd servers ($10,000/mo), plus $30/TB data transfer
The break even point for the colo'd setup from a networking perspective is:
At 1MB per "request" I believe this works out to about 50 requests per second average to reach this traffic level.Weaknesses of this model:
- Data transfer only. Depending on what else you're doing you could also save a lot on compute and storage.
- I don't know that much about how colocated data transfer would be priced ... i.e. do you need overprovision to guarantee availability, etc.
- one employee to handle servers to replicate the Amazon AWS experience ... could be highly variable depending on what AWS features you are using.
It's useful to keep in mind that in this type of scenario the goal is not to replace a cloud provider but to leverage capital to get better bang for the buck on select services. ex. using in-house compute farms and leveraging s3 for storage
For the employee cost I agree if you just have a couple boxes pumping out data your labor cost will be quite low. I'm imagining what else you might have happening on your back end using AWS services that you'd also have to replicate in your colocated installation to preserve the efficiency, which would take labor. For example, if you want some extra machines to do some load testing. Or if you need to store a bunch of data and want an S3-like system. Or if you want to wipe a machine and reinstall from an "image". The backup system, and backups testing. RDS, Aurora, DynamoDB, etc. Most of that can be done with a couple clicks on AWS.
I've analyzed the cost of cloud services to death (I've worked for a couple of them) and the only way they aren't great deals is if you don't need high quality operations (i.e. if you can deal with slow-downs or occasional outages then you can do better elsewhere). Otherwise, if you're small-scale then these marginal cost differences don't matter, and if you're larger scale then call up these cloud providers and get yourself a discount off the list price.
The real number to compare to is the google data for business rate. You can do lots of colocation in that price range. And that is why the cloud prices are unreasonable.
The big cloud platforms offer a rich selection of different offerings, which (just like in every other industry) cross subsidize each other.
When I go to a restaurant, I don't expect that they will be making the same profit margin on every item on the final bill, and in fact, they almost never do. Drinks tend to have a very high profit margin, some labour intensive items may be a break even at best, and the complimentary bread sticks or chips and salsa (if offered) will certainly be a loss.
I guess I could write a very upset article about how my local mexican restaurant is SERIOUSLY SCREWING ME OVER with their drink prices, but if I don't write the companion piece about their cheap burritos (subsidized, of course, by the drink prices), it would only show half the picture.
The reality is that I'm buying a whole package (at AWS or a restaurant) and I should evaluate the whole picture. Yes, I can get bandwidth cheaper outside AWS (or a can of coke a lot cheaper from a big box retailer). But I can't really get the total package of integrated, managed services outside AWS (certainly not for the cost they charge), any more than I can get someone else to show up in my kitchen and cook a three piece meal and then do all the dishes. (Which is to say, I totally could hire a chef to do that, but it would cost me a lot more. I could BUILD an internal SQS clone if I had to, but my employer would never break even on the cost of getting me to do so.)
AWS is very cheap for some things and very expensive for others. Depending on your usage and workload it may or may not be economical to buy the package they offer. If it is, go for it. If not, don't. Just like, you know, every other good or service you purchase in both your personal and professional life.
It costs me less than $5 with my current configuration (which is also a global CDN). It's way over our "soft limit", but it's an awesome site so I don't care. The important part is, I don't have to care.
This isn't about slightly more expensive tacos. It's about spending $560 on tacos instead of $5. "Meh" wouldn't exactly be my first reaction to getting that bill at the taco cart (or the fanciest taco place in the world, for that matter).
I can get IP transit in datacenters from $240-$600/Gb right now. So even his $960 transit cost for datacenters is off by quite a bit. He's comparing with a pretty high price and it still looks ridiculous.
And soda has a 1,150% markup [1], so it actually is like that.
[1] http://www.businessinsider.com/products-high-markups-2014-7
P.S. You could host it on Github for even less, if you were really price conscious.
> P.S. You could host it on Github for even less.
I'm not going to attempt to save an inconsequential amount of money by hosting a site I'm hosting on another host.
But that strategy probably wouldn't end well. The $560 site uses 25x more bandwidth than Github's fairly low soft limit of 100GB (https://help.github.com/articles/what-is-github-pages/) and is likely above the 1GB site limit (which is also just the sum of any changes ever, because git) of a Github pages site.
From seeing who their CDN provider is (one that charges basically the same CDN rates as AWS), my guess is that Github is paying a lot more for CDN transit than I am. It's perhaps not a coincidence that the 100GB BW limit was quietly introduced after they started using the CDN.
We went from cloudfront -> edgecast -> keycdn and our bill dropped from $3000 -> $500 -> $120
Edgecast when you buy is through a reseller is quite cheap but we moved because we needed custom domain SSL which is quite expensive in edgecast
And mixed drinks are more like a 4x markup. Lower at the high end even. Not sure where they pulled that number from. If it's anything like the coffee total, they'er comparing the garbage ingredients people typically use at home against the highest quality ingredients in the best places.
Neocities isn't just for static websites? How much traffic are getting that it would $560 on s3?
Is the basic principle to use cheap cloud VPSes or dedicated hosting (e.g. of the Digital Ocean, Hetzner, or OVH ilk)? And then you can afford lots of servers, and have then geographically distributed?
I notice that I drink less (too little, even, if I'm out for an evening) because of these prices. If each drink costs a few euros for 200ml glasses, I'd be crazy to drink a liter if at home the cost is somewhere between negligible (tap water) and 1.50 for that whole liter.
What I don't understand is why they can't spread out the profit margin. It'd make me feel a lot better about buying drinks and I'd probably spend more than I do now. Courses are a few euros in ingredients but I don't hesitate to pay €9-15 for it because I know labor is involved and someone has to pay rent. For drinks there is hardly any labor or ingredient cost (assuming all I ask is tap water, which is often since Dutch tap water tastes better than bottled water, and it's not cooled). Prices might also be higher because people don't order it in great quantities, which they might if restaurants wouldn't inflate drink prices to pay for rent (and other such costs), which would be so if you just pay per hour for a seat or something. I heard some places in Italy do that and people were surprised to be charged for it, but it's just itemizing costs and in my opinion more transparent.
All that said, the operating margins on high end restaurants are ridiculously low. Franchises and fast food do better at this (economy of scale), but if it is a proper linen-table-cloths and local sourcing sort of place, they are playing with very thin margins at the best of times.