Also gives you 224 physical CPU cores (448 logical cores).
Requires a 3-year reservation, and "the effective hourly rate for the All Upfront 3-Year Reservation for a u-12tb1.metal Dedicated Host in the US East (N. Virginia) Region is $30.539 per hour."
Works out to $267,512.88 per year, or $802,538.64 for the 3-year term. I wonder how that compares to building your own on-premise host with that much RAM (obviously, there's operational costs to consider as well).
Also, don't miss the last line: "We’re not stopping at 12 TiB, and are planning to launch instances with 18 TiB and 24 TiB of memory in 2019."
HANA in memory database that can serve both OLTP and OLAP workloads with sub second response times. The cost of the server is dwarfed by the licence cost.
I know you're joking, but that operation is often disk-io bound, not cpu or memory bound... and the disk is still EBS, you'll not get much better performance on that than on an m5.xlarge I bet, unless you use a grep regex that needs to gobble lots of data into memory as part of the search.
> $802,538.64 for the 3-year term. I wonder how that compares to building your own on-premise host with that much RAM
This was discussed a couple days ago in a different thread [1], when 4TB was the EC2 limit.
$400k for the current-gen (224 cores) or, as a sibling comment [2] notes, $250k for the previous CPU generation (which can use twice as many DIMMs of half the density and only 192 cores of that).
> obviously, there's operational costs to consider as well
We can estimate an upper bound on this, given that the current-gen system has N+2 5x1600W 96% efficient PSUs, so 5kW max. If you're paying a colo $.50/kWh, that's another $66k over 3 years worst case.
Realistically, though, CPUs with a max TDP of 1640W, DIMMs (generously) 700W, leaves plenty of room for fans, SSDs and other overhead before getting to 3kW or $40k, and that's still assuming running full-bore the whole time.
There are obviously also ancillary costs to AWS, such as data transfer and EBS.
If you're already running your own hardware, it sure does look more attractive to pay $400k now and $40k over 3 years than to pay AWS $800k All Up Front (now). If not, perhaps it's more attractive to hand-wave away that $360k as saving the "hassle" of hiring someone who knows how to run equipment in a datacenter.
One thing missing from this analysis is the networking cost. The EC2 instance comes with 14gb/sec of dedicated EBS bandwidth and 25gb/sec of network bandwidth. That implies a 40gb NIC, which will require a 40gb top of rack switch and enough cross-sectional bandwidth in the datacenter fabric to handle however many of these machines they plan for (at whatever oversubscription ratio, if any).
That's a pretty non-trivial amount of expense to setup a network with enough capacity to support a machine like this fi you want to do it yourself.
It may imply that but only outright requires 40Gb/s total, which is provided by the "4-port 10GBase-T Ethernet SIOM (Intel® X550 controller)" of a SuperServer 7089P-TR4T.
The EBS bandwidth can be elided entirely by using local storage instead. Local storage bandwidth can be much cheaper than ethernet, especially considering how many NVMe ports are already included.
As 10GE is maybe $200/port (including SFPs), at scale, network cost doesn't "move the needle" on a $400k server.
I would like to see if you can build such a monster yourself, and how much would it cost? Did Amazon designed their own motherboards or used some off the shelf boards?
The processors they're using appear to be well in excess of $10,000 each. To get 224 physical cores, you'd need to run them in a 8-CPU configuration, so you've got ~$100,000 invested just in the processors alone.
Memory is absurdly expensive right now, and I'd be shocked if 12TB of it cost any less than another $100,000.
You have to sign a multi-year reservation, and with a cost of $267,512.88 per year (as calculated from another comment in the thread), I assume their profit margins for the first year are nearly non-existent. However, over the course of the remaining two years of your reservation, they're making a great deal of money on each reservation.
So yea, I imagine someone could build something like this, but such an individual would need to have very deep pockets.
12TB is only about $7000 in chips. I buy them for my own products. There’s a lot of ancillary things you need but raw dimms aren’t anywhere near that experience.
You'll pay more for the highest density chips, but there are back-planes that support more of the lower density parts. I buy the raw chips in reels not DIMMs. The $7k was based on my costs.
ECC DIMMS at the density required for this probably going to run you substantially more.List price for an ECC DDR4 128GB is like close to $10K US each
Nobody pays list, but still.. I’m guessing they have something like 48x256gb dimms — that is serious coin.
Once you go beyond dual socket boards (and even then, they're not widely available), you're firmly in the territory of custom form factors, and extensibility usually happens via proprietary daughterboards.
At this point it's not really consumer-grade hardware though, so I wouldn't really classify it under "off the shelf".
The "custom" and "proprietary" are largely irrelevant, in that they're merely internal implementation details.
These systems (from SuperMicro or Lenovo, for example) still fit in a standard 19" rack, and still take standard CPUs, standard DIMMs, and standard PCIe cards.
The main limitation is that the complete "barebones" system is indivisible, unlike with most 1S-2S systems, where motherboards can be mixed and matched among a wider variety of chasses. This limitation also applies to high-density offerings like blades and SM's "Twin" line. I have yet to encounter a situation where such a limitation has ever even entered the conversation.
> The "custom" and "proprietary" are largely irrelevant, in that they're merely internal implementation details.
Agreed. I was merely trying to convey that they wouldn't be the normal [E]ATX form factor boards.
The parent was asking for something "off the shelf", and I took this to mean purchasing a motherboard by itself. You could get something like that from SM[0], but I doubt you'd be able to find a compatible chassis for the motherboard from anyone other than SM. At that point it probably makes more sense to just buy the system[1].
> The parent was asking for something "off the shelf", and I took this to mean purchasing a motherboard by itself.
The parent did ask about a mobo, specifically, but I think it's a stretch that the question was about the component on its own, at least in the context of the broader conversation of feasibility.
I took the "off the shelf" question to be synonymous with "commodity". That is, the opposite of (truly) custom. One doesn't have to be Google or Facebook and design a board layout to get 8 sockets. One can just order it from the same vendor as all the other commodity hardware.
> it probably makes more sense to just buy the system
I'm asserting that this is what every commercial purchaser does, anyway.
> cost of $267,512.88 per year (as calculated from another comment in the thread), I assume their profit margins for the first year are nearly non-existent
NB that this was calculated based on the "All Up Front" pricing, so, for that situation, it's safer to say that their profit for the first year is $300k+ [1] and merely non-existent or negative for the second and third years.
I would suspect they are building their own HW for this, but this is speculation based on the fact that AWS is building DCs for the IC/Gov and as such are working to accommodate where that is heading, and addressing how the ICs/Gov would want control over such HW.
Interesting that you can't simply spin one up on your own; you have to contact AWS to get the process started. Maybe it's simply because of the amount of money you're committing to spend, but I find the possibility that they're now offering an instance type that requires them to physically provision it for you intriguing.
If you're at the level of spend where paying almost a million dollars for an instance, contacting them isn't too hard. You usually have the email address and phone number of a bunch of people who can get it done in a minute or two. And they don't usually provision it -- they just flip a bit to allow you to use the API to provision it yourself.
I think they want the phone call in case you want 10 of them.
Sadly, it just used 6TB as a fixed cutoff, so keeping it up-to-date would be a manual task.
Today, only a few years later, that number is 24TB. (Even more sadly, that's for the last generation of Intel CPU, whereas the current 8S generation tops out at 12TB and isn't even scheduled to have models that would get to 24TB until next year).
One of the neat things of instances w/ this scale of memory is that we can once again start avoiding the perils of distributed computation. Because you have immense data locality, a lot of computations can be parallelized much more efficiently if they involve large sequences of trivially parallelizable tasks that are short-lived.
Under 150% gross profit margin (without electricity, fibre or real-state) over 3 years. I'd say the closer figure is ~ $300-400k per box for single company-scale servers, leading to a closer-to-net profit Amazon profit of around 100%
Although, it's possible to keep a server beyond its lifecycle and run it into the ground once it's already paid-for, as opposed to getting nothing at the end of the Amazon lease.
There's trade-offs for both cases; some people would rather pay more to not have to deal with quotes, vendors or shipping issues.
Anyone paying more than a negligible amount (per server) for any of these is needlessly over-paying. For a server this expensive, it had better be way below 1%. (Tech support is arguable, but it costs a pretty penny from AWS, too).
> RAM 64GB: $866.23 x 192 = $166,316.16
These CPUs can't support 192 DIMMs, only 96, so they're limited to the much more expensive 128GB modules. That means you're looking at closer to $250k for the RAM alone.
If someone wants to throw more money on RAM to be slightly faster, that's a solution design-decision; it's doubtful Amazon would do that. If they are, great.
66 comments
[ 1.5 ms ] story [ 174 ms ] threadRequires a 3-year reservation, and "the effective hourly rate for the All Upfront 3-Year Reservation for a u-12tb1.metal Dedicated Host in the US East (N. Virginia) Region is $30.539 per hour."
Works out to $267,512.88 per year, or $802,538.64 for the 3-year term. I wonder how that compares to building your own on-premise host with that much RAM (obviously, there's operational costs to consider as well).
Also, don't miss the last line: "We’re not stopping at 12 TiB, and are planning to launch instances with 18 TiB and 24 TiB of memory in 2019."
[1] https://www.thinkmate.com/system/superserver-7088b-tr4ft
>> SAP HANA in Minutes
> The EC2 High Memory instances are certified by SAP for ⟨…⟩
[SAP HANA keeps everything in RAM.]
the first time.. 12TB can cache a lot of logs.
This was discussed a couple days ago in a different thread [1], when 4TB was the EC2 limit.
$400k for the current-gen (224 cores) or, as a sibling comment [2] notes, $250k for the previous CPU generation (which can use twice as many DIMMs of half the density and only 192 cores of that).
> obviously, there's operational costs to consider as well
We can estimate an upper bound on this, given that the current-gen system has N+2 5x1600W 96% efficient PSUs, so 5kW max. If you're paying a colo $.50/kWh, that's another $66k over 3 years worst case.
Realistically, though, CPUs with a max TDP of 1640W, DIMMs (generously) 700W, leaves plenty of room for fans, SSDs and other overhead before getting to 3kW or $40k, and that's still assuming running full-bore the whole time.
There are obviously also ancillary costs to AWS, such as data transfer and EBS.
If you're already running your own hardware, it sure does look more attractive to pay $400k now and $40k over 3 years than to pay AWS $800k All Up Front (now). If not, perhaps it's more attractive to hand-wave away that $360k as saving the "hassle" of hiring someone who knows how to run equipment in a datacenter.
[1] https://news.ycombinator.com/item?id=18041486
[2] https://news.ycombinator.com/item?id=18089267
That's a pretty non-trivial amount of expense to setup a network with enough capacity to support a machine like this fi you want to do it yourself.
It may imply that but only outright requires 40Gb/s total, which is provided by the "4-port 10GBase-T Ethernet SIOM (Intel® X550 controller)" of a SuperServer 7089P-TR4T.
The EBS bandwidth can be elided entirely by using local storage instead. Local storage bandwidth can be much cheaper than ethernet, especially considering how many NVMe ports are already included.
As 10GE is maybe $200/port (including SFPs), at scale, network cost doesn't "move the needle" on a $400k server.
Memory is absurdly expensive right now, and I'd be shocked if 12TB of it cost any less than another $100,000.
You have to sign a multi-year reservation, and with a cost of $267,512.88 per year (as calculated from another comment in the thread), I assume their profit margins for the first year are nearly non-existent. However, over the course of the remaining two years of your reservation, they're making a great deal of money on each reservation.
So yea, I imagine someone could build something like this, but such an individual would need to have very deep pockets.
Where can you get 12tb for $7000?
At this point it's not really consumer-grade hardware though, so I wouldn't really classify it under "off the shelf".
These systems (from SuperMicro or Lenovo, for example) still fit in a standard 19" rack, and still take standard CPUs, standard DIMMs, and standard PCIe cards.
The main limitation is that the complete "barebones" system is indivisible, unlike with most 1S-2S systems, where motherboards can be mixed and matched among a wider variety of chasses. This limitation also applies to high-density offerings like blades and SM's "Twin" line. I have yet to encounter a situation where such a limitation has ever even entered the conversation.
Agreed. I was merely trying to convey that they wouldn't be the normal [E]ATX form factor boards.
The parent was asking for something "off the shelf", and I took this to mean purchasing a motherboard by itself. You could get something like that from SM[0], but I doubt you'd be able to find a compatible chassis for the motherboard from anyone other than SM. At that point it probably makes more sense to just buy the system[1].
[0] http://www.supermicro.com/products/motherboard/Xeon/C600/X10...
[1] http://www.supermicro.com/products/system/2U/2048/SYS-2048U-...
The parent did ask about a mobo, specifically, but I think it's a stretch that the question was about the component on its own, at least in the context of the broader conversation of feasibility.
I took the "off the shelf" question to be synonymous with "commodity". That is, the opposite of (truly) custom. One doesn't have to be Google or Facebook and design a board layout to get 8 sockets. One can just order it from the same vendor as all the other commodity hardware.
> it probably makes more sense to just buy the system
I'm asserting that this is what every commercial purchaser does, anyway.
They also have 4 sockets motherboards for a "general public", and I suspect might have something for a not-so-general public.
NB that this was calculated based on the "All Up Front" pricing, so, for that situation, it's safer to say that their profit for the first year is $300k+ [1] and merely non-existent or negative for the second and third years.
[1] Assuming $400k purchase cost as per https://news.ycombinator.com/item?id=18090058
I think they want the phone call in case you want 10 of them.
I don't have any experience with it but it's the one single example that's always used in every vendor post about high-memory machines.
[0]: https://news.ycombinator.com/item?id=9581862
Maybe their data was in memory, and the machine powered off.
Sadly, it just used 6TB as a fixed cutoff, so keeping it up-to-date would be a manual task.
Today, only a few years later, that number is 24TB. (Even more sadly, that's for the last generation of Intel CPU, whereas the current 8S generation tops out at 12TB and isn't even scheduled to have models that would get to 24TB until next year).
CPU 8176M: $11,805.00 USD x 8 = $94,440.00
RAM 64GB: $866.23 x 192 = $166,316.16
Chassis + 2x 10 GbE NICs + SSD boot device: ~$8000
Total: ~$269k USD
AWS price: $803k
Under 150% gross profit margin (without electricity, fibre or real-state) over 3 years. I'd say the closer figure is ~ $300-400k per box for single company-scale servers, leading to a closer-to-net profit Amazon profit of around 100%
Although, it's possible to keep a server beyond its lifecycle and run it into the ground once it's already paid-for, as opposed to getting nothing at the end of the Amazon lease.
There's trade-offs for both cases; some people would rather pay more to not have to deal with quotes, vendors or shipping issues.
Anyone paying more than a negligible amount (per server) for any of these is needlessly over-paying. For a server this expensive, it had better be way below 1%. (Tech support is arguable, but it costs a pretty penny from AWS, too).
> RAM 64GB: $866.23 x 192 = $166,316.16
These CPUs can't support 192 DIMMs, only 96, so they're limited to the much more expensive 128GB modules. That means you're looking at closer to $250k for the RAM alone.
6 channels per socket (2 controllers with 3 lanes each), 8 sockets, and 64 GB LR DIMMs.
128 DIMMs are $2-3k a pop.
192 sockets, 8 way, up to 24TB of ram w 128GB or 12TB with 64GB sticks:
https://www.supermicro.com/products/system/7U/7088/SYS-7088B...
If someone wants to throw more money on RAM to be slightly faster, that's a solution design-decision; it's doubtful Amazon would do that. If they are, great.
That's previous generation:
>Intel® Xeon® processor E7-8800 v4/v3 family
The articles states:
> All three sizes are powered by the latest generation Intel® Xeon® Platinum 8176M (Skylake) processors
You mention the 8176Ms in your original comment, too. The current SuperMicro product that supports those is https://www.supermicro.com/products/system/7U/7089/SYS-7089P... and that only has 96 DIMM slots.
> Intel® Xeon® Scalable Processors, 8S-3 UPI up to 10.4GT/s
> 96x 288-pin DDR4 DIMM slots
> Supports up to 12TB DDR4 ECC 3DS LRDIMM in 96 DIMM slots
I think you'll find that this is due to the limitations of the CPU itself:
> Max Memory Size (dependent on memory type) 1.5 TB
according to https://ark.intel.com/products/120505/Intel-Xeon-Platinum-81...