There was another article that suggested they were manufactured specifically to amazon's standards for things like cooling and power stability. Instead of having to run in a wide range of environments, they can be designed to run in just the narrow range of Amazon's environment.
Interesting trend. If you could spec out the whole thing including cooling for use in a data center, you could probably do some really damn fast many-core chips that wouldn't be likely to sell much in the desktop and a la carte server markets.
SSD instance storage will always be significantly faster than EBS storage, even if SSD-backed. I'm disappointed that Amazon dropped it from their C4 offering.
For Elasticsearch, we use R3 (memory-optimized) instances which still have SSD instance storage. And boy is it ever fast.
I don't care about disk speed nearly as much as I do reliability. EBS combines inconsistency with unreliability. It is the worst of both worlds for building a reliable, redundant infrastructure. I'm pretty bummed.
Agreed, and I think this isn't getting enough play. EBS is one of the more failure-prone parts of the AWS stack. I full-stop don't use EBS, anywhere, on any machine, because of the reliability concerns. (Since I am at least sort of competent, I design for redundancy and don't care if I lose a node anyway, so instance stores are fine.)
These machines sound cool, but C3's will do me just fine.
There's no question that what you sacrifice in performance with EBS, you get back in easy durability. But if you design your persistence layer with replicated instances, you can have the performance of instance storage. You'll need to make backups yourself, so it is a bit of extra work over making EBS snapshots, but IMO the performance gains are well worth it.
EBS disks are more durable, sure. When you can reach them. You should be designing for failure, and EBS fails not infrequently relative to the rest of AWS. The problem is, when it fails, your instances go with it.
Individual machines' durability is almost irrelevant. Your service's durability is very relevant. This is the most critical consideration in almost all cases in the cloud. EBS makes service durability much harder.
Besides my database servers, all of my machines are ephemeral[1]. I can lose machines and spin up new ones automatically (and therefore scale elastically).
EBS adds an unnecessary additional point of failure that I want to avoid at all costs.
Having run production systems in EC2 since 2010, I've survived numerous EBS outages as other services have crashed and burned by sticking to this philosophy.
[1]: I don't use EBS for my database servers either. I use a replicated DB that can lose multiple machines simultaneously without loss of data (and is backed up to S3 in case of a catastrophic event).
Ephemeral give you extremely high IOPS for no additional cost. It's perfect if you don't need a ton of disk space. A great example is Graphite. Very high IOPS requirement but (usually) a low disk space requirement (assuming you configure relatively low retention) The c3.2xlarge boxes are great for that.
The equivalent on EBS on the other hand will cost you an extra $250/mo or so in addition to your server cost. And you will cap out at 4000 IOPS.
In case anyone else was wondering what "ability to control P-States and C-States" is about, I think this is a useful reference [1]. I guess with the C4-8Xlarge you control the entire physical machine and you can enable additional P-states to get more power out of the cpu while using more electricity.
In some cases, your workload might not need all 18 of the cores (each of which
runs two hyperthreads, for a total of 36 vCPUs on c4.8xlarge). To tune your
application for better performance, you can manage the power consumption on a
per-core basis. This is known as C-state management, and gives you control over
the sleep level that a core may enter when idle. Let’s say that your code needs
just two cores. Your operating system can set the other 16 cores to a state that
draws little or no power, thereby creating some thermal headroom that will give
the remaining cores an opportunity to Turbo Boost. You also have control over
the desired performance (CPU clock frequency); this is known as P-state
management. You should consider changing C-state settings to decrease CPU
latency variability (cores in a sleep state consume less power, but deeper sleep
states require longer to become active when needed) and consider changing
P-state settings to adjust the variability in CPU frequency in order to best
meet the needs of your application.
I question the real world utility/practicality of this, but thanks for the explanation.
To elaborate: I start with (e.g.) a 2 cpu c4 instance. But my single thread performance isn't fast enough. So I'm going to spend a king's random to upgrade to a 36 cpu instance, just so I can disable 35 of those cpus? There are several other options I'd probably investigate first.
Amazon seems to offer more network bandwidth though, so they might have an advantage for CPU-heavy and network-heavy loads. But from the testing I've done Amazon comes in almost dead last for cost/performance on most metrics. I suppose they could make sense if you make heavy use of all their services, and maybe their cost/performance improves with scale?
You should be aware though that integrating with AWS services that aren't also provided by AWS' competition, you're increasing the cost of switching away from AWS in the future.
I'm not saying don't do it, I'm just saying be aware that you may be making it harder for your company to change providers down the road.
As far as I know, the biggest reason not to rely unconditionally on AWS-specific offerings is the same reason to avoid vendor lock-in elsewhere: They've got you over a barrel when it comes time to re-negotiate pricing. Amazon has a pretty wonderful history of price cuts, but when Google comes in with a better deal (or you want to hedge your bets with a hybrid cloud-type thing), it's waaay harder to find analogues of things like Kinesis and SES and Elastic Beanstalk.
There are definitely some PaaS-type offerings they have that aren't as scary (RDS/MySQL, for example, and ElastiCache/Memcached, etc.), but if you're trying to bounce systems between clouds, or deploy the exact same configuration everywhere you're running, the only real answer is to rely on compute resources (basically EC2) and build your own stuff. All the clouds have Linux servers.
I agree with your general point, but switching between Kinesis and Kafka isn't too crazy. SES can be replaced by Mailgun, Postmark, so on and so forth.
The AWS services I use without hesitation are EC2, Route 53, S3, SNS (to mobile devices, not to my applications), and SES. RDS and ElastiCache can be okay for early, pre-scaling situations, and Dynamo is okay for proving out a problem before cranking out a Cassandra cluster (but I wouldn't want to scale on Dynamo, gets real pricey real fast).
I will also heavily recommend Redshift, because it's fantastic for a lot of different offline workflows and it's a lot nicer than maintaining your own. It works well without being tied to AWS for your actual infrastructure.
As far as I am aware, there is nothing out there that can rival the cost and functionality of S3. I'm not going to avoid using it simply because there is a lack of open alternative - it's that much better that I will use it now and redevelop to use different technology if and when I have to.
After all, most of these services all have pre-made client libraries. Rewriting code to use a different service isn't likely to be world-shattering.
For all means, if you have LOTS more money than time, you should pay someone else to do this. Outside of certain sheltered silicon valley startups, this is often not the case though. :)
You know, I'm pretty sure that almost everyone is better off spending a few cents per gigabyte per month on S3 for inert file storage, a functionality that's performance-inelastic except at the far end of the curve, than maintaining a Riak cluster.
Source: I've maintained a Riak cluster.
There are lots of AWS services I avoid (I won't use SQS, I won't use Dynamo except during prototyping, etc.), but S3 is so utterly portable across any cloud you go to (and latency is already not something you are worrying about if you're using S3) that it's laughable to suggest that you're going to roll your own with anywhere near the reliability at anywhere near the cost.
We rolled our own proprietary version of what became known as S3 and scaled it up to several thousand nodes. At any given time, less than half a FTE spent any time on this, and I think we've had roughly equal (probably better) availability than that of S3, at a fraction of the cost.
Assuming you pay your engineers a paltry $50k/yr, half of a FTE is $25k/yr. Quick calculation based on the public S3 pricing says that's 847TB of data you could be storing just with their salary. That doesn't include all the actual costs associated with running thousands of nodes - acquisition, hardware replacements, utilities and other operating costs, space, etc.
(To be honest, with that many nodes I have a hard time believing that you're not spending half a FTE just replacing dead drives.)
Not to mention the additional costs if you're actually interested in the same level of durability (target eleven 9s (99.999999999%), but last time I talked to an engineer they're still at 100% since leaving beta). Multiple DCs, transit between them...
I'm sure there's a scale at which it's worth it, but it's pretty far up there and for almost everyone S3 is a cheaper option.
I think you're overestimating how expensive S3 is. You'd have to be storing a lot of data before running your own infrastructure becomes more affordable (especially factoring in employee costs)
Not to say it never happens, but 90% of people are better off just using S3.
True. Those are better if you want to grab a server and sit on it for a long time as a dedicated box at a data center. It's more like old school rack hosting but you don't have to actually screw with hardware. They'd be awful if you want elasticity, which I suppose is why "elastic" is in Amazon's name. Duh. :)
I have some stuff at Kimsufi (BHS data center). The performance and stability are really excellent but I'd have to wait for availability to add more, which sucks.
Most of the servers on that page aren't even running ECC RAM. The only one listing ECC RAM is the E5504 which dates back to 2009 whereas C4 is running the fastest Xeon model available today. The rest of the list is Intel Atom and i5 and i7 desktop chips. 60% of the server types on that page are also currently out of stock.
That assumes that there's only one chip in the server (unlikely with 36 vcpus listed for C4), that you're happy with using a desktop chip with no ECC for Hetzner, and your pricing doesn't consider reserved instance pricing for C4 which costs much less than $1300.
AWS reserved instances are buying hours. If you can commit to running a particular instance type for ~720 hours every month for a year or 3, then it makes sense to go RI. You don't have to use for one machine running 24x7, you can burst with loads - 2 machines 12x7, etc etc.
Does that server also provide 36 cores, 60GB of RAM, 10Gbps network and the fastest Intel CPU available anywhere? You're comparing a single low-end chip vs a server with multiple cutting edge chips. Everyone knows you pay more for the fastest chips. If budget is the only criterion then there are many less expensive instances.
72 comments
[ 3.5 ms ] story [ 141 ms ] threadhttp://ark.intel.com/products/81706
I don't think anyone but Amazon has this specific model. I don't see it available for sale anywhere.
The extra features of EBS SSDs are pretty useful (ie. snapshotting)
But having attached disks was super fast.
Last time we spec-ed up an ElasticSearch cluster the SSD EBS didn't exist yet, so on-board SSDs was an obvious win.
I should revisit it again some time.
For Elasticsearch, we use R3 (memory-optimized) instances which still have SSD instance storage. And boy is it ever fast.
C3 SSDs were crazy faster than even the fastest EBS option listed, plus you could raid0 them...
Fortunately for me, I don't need that much disk space and C3's aren't going anywhere for a while.
These machines sound cool, but C3's will do me just fine.
Individual machines' durability is almost irrelevant. Your service's durability is very relevant. This is the most critical consideration in almost all cases in the cloud. EBS makes service durability much harder.
EBS adds an unnecessary additional point of failure that I want to avoid at all costs.
Having run production systems in EC2 since 2010, I've survived numerous EBS outages as other services have crashed and burned by sticking to this philosophy.
[1]: I don't use EBS for my database servers either. I use a replicated DB that can lose multiple machines simultaneously without loss of data (and is backed up to S3 in case of a catastrophic event).
The equivalent on EBS on the other hand will cost you an extra $250/mo or so in addition to your server cost. And you will cap out at 4000 IOPS.
[1] http://www.xenserver.org/partners/developing-products-for-xe...
To elaborate: I start with (e.g.) a 2 cpu c4 instance. But my single thread performance isn't fast enough. So I'm going to spend a king's random to upgrade to a 36 cpu instance, just so I can disable 35 of those cpus? There are several other options I'd probably investigate first.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processor...
I am about to reserve some R3 instances so maybe that will speed it up :-)
http://www.kimsufi.com/en/
Amazon seems to offer more network bandwidth though, so they might have an advantage for CPU-heavy and network-heavy loads. But from the testing I've done Amazon comes in almost dead last for cost/performance on most metrics. I suppose they could make sense if you make heavy use of all their services, and maybe their cost/performance improves with scale?
I'm not saying don't do it, I'm just saying be aware that you may be making it harder for your company to change providers down the road.
There are definitely some PaaS-type offerings they have that aren't as scary (RDS/MySQL, for example, and ElastiCache/Memcached, etc.), but if you're trying to bounce systems between clouds, or deploy the exact same configuration everywhere you're running, the only real answer is to rely on compute resources (basically EC2) and build your own stuff. All the clouds have Linux servers.
The AWS services I use without hesitation are EC2, Route 53, S3, SNS (to mobile devices, not to my applications), and SES. RDS and ElastiCache can be okay for early, pre-scaling situations, and Dynamo is okay for proving out a problem before cranking out a Cassandra cluster (but I wouldn't want to scale on Dynamo, gets real pricey real fast).
I will also heavily recommend Redshift, because it's fantastic for a lot of different offline workflows and it's a lot nicer than maintaining your own. It works well without being tied to AWS for your actual infrastructure.
As far as I am aware, there is nothing out there that can rival the cost and functionality of S3. I'm not going to avoid using it simply because there is a lack of open alternative - it's that much better that I will use it now and redevelop to use different technology if and when I have to.
After all, most of these services all have pre-made client libraries. Rewriting code to use a different service isn't likely to be world-shattering.
(I haven't done this particular pricing calculation, but everything else in AWS seems awfully overpriced, so why not S3?)
Source: I've maintained a Riak cluster.
There are lots of AWS services I avoid (I won't use SQS, I won't use Dynamo except during prototyping, etc.), but S3 is so utterly portable across any cloud you go to (and latency is already not something you are worrying about if you're using S3) that it's laughable to suggest that you're going to roll your own with anywhere near the reliability at anywhere near the cost.
You might not be in "almost everyone", but almost everyone else is.
(To be honest, with that many nodes I have a hard time believing that you're not spending half a FTE just replacing dead drives.)
Not to mention the additional costs if you're actually interested in the same level of durability (target eleven 9s (99.999999999%), but last time I talked to an engineer they're still at 100% since leaving beta). Multiple DCs, transit between them...
I'm sure there's a scale at which it's worth it, but it's pretty far up there and for almost everyone S3 is a cheaper option.
Not to say it never happens, but 90% of people are better off just using S3.
I have some stuff at Kimsufi (BHS data center). The performance and stability are really excellent but I'd have to wait for availability to add more, which sucks.
Their CPU has a passmark score of 16k[1] and they ask $1300 USD/mo for it.
Hetzner will rent you one with passmark 10k[2] for $60/mo...
[1] http://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2680+v...
[2] http://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-4770+%...
That is a crazy difference though if you can take advantage of it.
3 years reserved c4.8xlarge costs $501/mo at EC2.
If you can commit for 3 years, why would you pay a 6x markup for "elasticity"? (11x if you commit for only 1 year...)
The docs say that it is a VF but I wonder since they are offering the full 10G of bandwidth.
We could relocate our Snabb Lab to AWS if we got PFs. https://github.com/SnabbCo/snabbswitch/wiki/Snabb-Lab
c4.2xlarge https://gist.github.com/karlseguin/a659ef87b3a4a5d590e9
c4.8xlarge https://gist.github.com/karlseguin/0f38b9ad87ebd54375da
i7-4770 https://gist.github.com/karlseguin/5a6a45ace2048545b6c3
https://aws.amazon.com/blogs/aws/now-available-new-c4-instan...