52 comments

[ 3.5 ms ] story [ 1241 ms ] thread
This article talks about the underlying technical differences between various classes of EC2 computer... is this documented by Amazon anywhere? I would totally care more about replacing stuff if I knew "ok, there is some new virtualization difference in this new set of machines" that I could analyze against my use cases instead of "I have no clue what if anything is different, other than the pricing is a little different".
Some of this is fairly well known if you happen to be quite familiar with Xen.

If your use case is "I don't want to understand Xen", then fine, just use AWS and pick the newest instances, they're usually the best.

AWS does have some words about PV vs HVM here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualiz...

But really, they don't say much and that's fine. If you just want to run stuff, you probably don't need to understand the deep details of Xen or PV drivers or whatever; the cpu speed, memory, and "disk io"/"cpu credits" they document quite well will be what matters significantly more.

If you are at the scale of netflix, then it does make sense to figure out an extra several percent disk performance.

> I would totally care more about replacing stuff if I knew ... instead of "I have no clue what if anything is different

They almost always have blog posts accompanying each new instance launch which roughly outline the performance improvements and reasons that instance type is cool vs previous ones, even if they don't go into the depth of virtualization differences so often (which again, usually doesn't matter! for nitro, the C5 blog did talk about the visualization stuff in depth https://aws.amazon.com/blogs/aws/now-available-compute-inten...)

I knew about PV vs. HVM, as that is documented. However, there is a ton of nuance in this article about every instance type and specific changes to what parts of the system are virtualized (and in particular stuff with EBS, which I stress heavily). I guess you are saying that people need to essentially find all of the old blog posts from AWS to figure out any of this, but why isn't it just documented? I clearly am someone who cares a lot about these things, or I wouldn't have posted the question I did: but Amazon's instances are essentially just black boxes. I am hoping that there is somewhere in the documentation that I don't know about that just shows the differences between all of these instance types in a few centralized pages.
Does this allow nested virtualization?

ie the ability to run VMs within an EC2 instance?

Definitely on the pure-hardware one. VMWare's running their stuff on that.
C5 does not support nested virtualization but i3.metal allows using virtualization technology without nested virtualization.

Both i3.metal and c5 use the same underlying Nitro technology.

Wait, so even the "bare metal" i3.metal sits under a hypervisor? It would seem inaccurate to call it "bare metal" in that case.
No, Nitro refers to the whole technology stack as well as the hypervisor. *.metal instances utilize most of the Nitro technology stack, but not the Nitro hypervisor.
Am super curious to see some benches on AWS Nitro compared with GCP's Skylake VMs.
Turns out Nitro is not only an improved KVM but also work exclusively with their Nitro Custom Silicon. [1]

And we are finally here, the true cloud, elastic computing with little to no performance overhead.

[1]: https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_a...

We were here back in 2005 with the introduction of Solaris zones, which now live on in SmartOS. Took GNU/Linux “only“ 12 years to begin to catch up. This might be new to you, but it’s nothing new to those of us who built private clouds back in 2005/2006. Just don’t think it’s new because it isn’t.
Zones are not VMs. If you need to vent your Linux hate at least save it for a post about Docker.
I don’t need to be patronized on the difference between zones and VM’s as I’m also one of those VMware users from 1999 in Brendan’s article.

Zones however are superior to VM’s because they don’t require hard pre-allocation of resources like virtual machines do, so they are a superior choice in environments where Windows isn’t desired or required.

As for my Linux hate, if Linux were so great and well made, I wouldn’t hate it. I like well made things.

> Zones however are superior to VM’s because they don’t require hard pre-allocation of resources like virtual machines do, so they are a superior choice in environments where Windows isn’t desired or required.

In public cloud that's not a problem, that's a requirement, so one VM wouldn't trash all its neighbors.

That is what containers are for. On SmartOS a container is used to constrain resource limits of a zone, but if a zone is using less than the limits it’s more efficient in terms of hardware. However, if one is not using this technology, one is forced to hard partition with virtual machines.
You're arguing one edge of a very much double-edged sword, and I must say (as someone that was deploying the linux-vserver patches by at least 2004) it's incredibly tiring to watch.
No less tiring than reading about re-inventing virtualization in Linux. For both of us the fix is the same: either don’t read it or remark how tiresome it is.

Where we both agree is that it’s tiresome, but for very different reasons.

Yes there is definitely a way to praise alternatives without railing against the current discussion topic.

Some may appreciate being made aware of another technology; unfortunately this specific example reinforces the importance of marketing and pursuing mindshare.

You do understand how unsuitable zones are in a public cloud like AWS, right? Why are you even having this discussion.
Perhaps it is because given technical capability, I cannot understand why Amazon is even being discussed when it is technically inferior to what Joyent offers and Amazon is the most expensive provider.
> I cannot understand why...

Sounds to me that your understanding is the real problem here. There are a lot of resources you can use to read up on the value proposition, and while I do think AWS is pretty expensive and perhaps priced a bit too high, it's a great platform that goes way beyond just the speed of the VM tech.

> As for my Linux hate, if Linux were so great and well made, I wouldn’t hate it.

With Linux running on ~10^11 devices, everything from watches to supercomputers to the Falcon 9 rocket, the market disagrees with you to a pretty comical extent.

Historia est magistra vitae. When have the masses ever been correct?
I dunno, most of the time? They some years ago decided learning Latin was pointless, for example.
Masses have always been wrong, to the extent that they burned and killed innocent people for one reason or another. No mass has ever produced any innovation; if you look at revolutionary inventions, they were always invented by single persons or very small, focused group of individuals, but never masses. Masses don't have the competence to judge whether that which they are following is good or bad, correct or incorrect. That's why they follow. It's what defines them as a mass.

As for Latin, this is the same reasoning of "learning" Linux and not understanding UNIX and mis-configuring the OS or breaking Linux backwards compatibility. Or re-inventing the wheel all the time while basic functionality like booting or the fiberchannel stack still do not work correctly after 20+ years.

Thank you masses for all the sleepless nights I was forced to endure because Linux has the same problems Solaris solved decades ago. Because masses know what's good and what's bad.

You know the author of this piece is Brendan Gregg, right? DTrace author, performance guru, ex-Sun engineer and lead performance engineer at Joyent. And now a full time Linux guy. He doesn't think it's so bad apparently!
Yes, I know that; we even corresponded privately. However, call to authority does not magically validate Amazon, nor does it magically invalidate my points. Nothing Brendan wrote contradicts the points either of us made.
What is your point, exactly? That we shouldn't care about hardware-backed virtualization because we have zones? That AWS should go home and re-write EC2 to just be zones instead of VMs?
What is your point, exactly? That we shouldn't care about hardware-backed virtualization because we have zones?

Yes!

That AWS should go home and re-write EC2 to just be zones instead of VMs?

There is no reason for AWS to exist.

Zones aren't a replacement for VMs. With VMs you can have different OSes, for example, which is not generally possible with a zone (you can do some stuff with syscall routing to do this in certain ways, but requires support from the root container). In a zone you can't do a variety of things that require higher level kernel access, like installing custom kernel modules. Zones also have a higher attack surface, which matters quite a lot when you are dealing with a multi-tenancy situation. But I'm pretty sure you knew that already.
If you need another OS besides SmartOS for serving data, you have far bigger problems.

Kernel modules are only meant to be installed in the global zone, on the hardware. This is by design. If you need access to that particular device, it can be delegated to that zone with the add subcommand to zonecfg(1M).

Zones do not increase attack surface, I don't know what would make you think that. They reduce the attack surface because one can set up read-only zones, good luck changing anything in there. In a zone, attacker lands in a jail, assuming they actually manage to break in!

What strange times we live in when there is so much misconception. So much knowledge lost! No wonder AWS and Linux rule the day!

Zones are a revolutionary technology in terms of network security. What makes you think zones increase attack surface?

Zones increase attack surface compared to a hypervisor because you have a shared kernel, and every syscall is a potential attack surface. Compare that to a hypervisor, which a much simpler API exposed with far less functionality. Hypervisor vulnerabilities are far less common than kernel vulnerabilities for that reason. This is fairly well known in the security space.

In fact, if you look into Joyent, you'll see they don't use regular zones for multi-tenancy, but rather a KVM-inspired hypervisor implementation of zones.

This is cookamamie. A hypervisor also has syscalls and in this particular case the global zone is the hypervisor. They even call it "type 1 hypervisor".

Apropos "KVM-inspired hypervisor": the zones still function in exactly the same way as they did when I first used them in 2006; in fact when running lx and KVM Joyent is exploiting an obscure capability called a brand (branded zones), but that capability has always been there, since day one back in 2006.

Finally, there has only been one known case of managing to break out of a zone and this was through a kernel driver exposed to it, exactly the kind of scenario you'd have on VMware. In the decade since, there have been no known cases, so not only is your argumentation cookamamie, it's in the realm of pure hypothesis.

SmartOS provides Zones, and if you need it a KVM instance within a Zone.

Hypervisors have their own security issues. They're just programs. That's why there've been all the patching over the years for Xen and for KVM.

Last year, an independent researcher did find an issue related to Dtrace that Joyent patched. However, the security and engineering that backs Zones is way more than with other kernels.

https://www.joyent.com/blog/dtrace-conf-16-wrap-up

DTrace exploitation: http://slides.com/benmurphy/deck#/

http://benmmurphy.github.io/blog/2017/01/06/arbitrary-kernel...

https://vimeo.com/173300650

I for one agree with you, but the main issue with SmartOS, Opensolaris, Solaris etc. is not the operating system. It's that some things are half baked and enterprisy.

What do I mean by that? Brendan Gregg did mention in his article about Solaris migration certain things like amount of developers out of box performance etc.

Here are a few issues that the Solaris crowd failed to address and if they don't address it soon imho they will fortify their own demise:

Hardware availability, if you have a blade center with 10G intel hardware everything will work fine, but want to run it on a ryzen? tough luck. This already disqualifies a bunch of curious developers.

KVM support? Sure, but it's not well documented the tooling if you don't use the enterprise management tools by Joyent is horrible. The standard tooling `vmadm` in SmartOS will leave logadm residues that will spam your notifications it's been an unfixed bug for multiple years.

UEFI? Nope. The tooling and documentation is the worst. Want to migrate from libvirt? You have to shut it down, dd the image to the zfs zone and hope that it works. If it didn't well you can't access vnc before the host is already booted into a failed state.

The actual killer:

Documentation. The SmartOS confluence wiki starts off by giving you a taste and the typical enterprise way they offer you training.

That works just fine so long as you're one of select few choices to pick from(Jira loves that model).

But when you fail to attract the crowd of developers, you're bound to fall behind. There's exactly 3 books on OpenSolaris, 0 on SmartOS and a bunch of technical old documents on Solaris from Oracle.

There's probably been at least 15 Docker books this year alone. Whole bunch to manage kubernetes. For creating network topologies in linux you pick up a bunch of books that aren't outdated.

I love the idea of SmartOS and I'll run my machines as long as I feel it makes sense and you might argue that the complexity is a lot lower than what linux provides, but that's besides the point.

What the point in crossbow, HVM, etc. when the only place of information is:

a) an old oracle tech paper

b) an old oracle/sun/joyent consultant

c) one of 5 blog posts that explain how to run it in your home NAS

d) a wiki with so basic information that it's guaranteed not to work on your system

At least the complexity of Linux is visible, and you can find a solution somewhere. Rather than having to find a consultant that guards that solution with his life.

The standard tooling `vmadm` in SmartOS will leave logadm residues that will spam your notifications it's been an unfixed bug for multiple years.

So let's get this straight: when Linux zealots 15-20 years ago answered "you have the source! Fix it yourself!" that was okay, but when it comes to SmartOS, it's out of the question?

Sure, but it's not well documented the tooling

Read. The. Manual. Pages. It's a real UNIX and that means high quality documentation! No "Stack Overflow" vomit!

The actual killer: Documentation.

Repeat after me: I will read the manual pages. There was just a discussion on this subject recently here.

b) an old oracle/sun/joyent consultant

so old Solaris guys are no good?

At least the complexity of Linux is visible, and you can find a solution somewhere.

Linux has incredible kludges and brokenness under the hood, it's just that one needs to be an old UNIX guy to understand just how much of garbage "solutions" places like "Stack Overflow" offer.

Rather than having to find a consultant that guards that solution with his life.

I've always been happy to teach all that I know about UNIX and never had a complaint or heard an unkind word about my teaching efforts, on the contrary. Find yourself a mentor. "Stack Overflow" isn't one and could never substitute one.

Dude that elitest crap you're spouting is exactly what killed Solaris.

Repeat after me: I read the manual pages and it's still crap.

Your best source of documentation is the oracle documents they don't map 1:1 to SmartOS. Wanna learn about network topologies in SmartOS. Here's the 425 pages Oracle document that provides a tiny intro to how you "could" "maybe" get started [1]

When I asked Joyent in 2013 for information needed to improve the wifi stack there was exactly zero interest.

SmartOS supports docker images in a really interesting manner. But again it's half baked. It may work fine in the Joyent UI but that's meaningless. This is what a normal person has to do to get it running [2]

Judging by the way you talk I really seriously doubt you have any meaningful architecture running on SmartOS that's self managed. If you want to set up a small private cloud that's SmartOS based you're better off trying Danube cloud, but who knows how long that's going to last [3]

Compare that to AWS, you have a proper fully documented VPC tool that's Cloudformation, you have a whole bunch of recipes, you have tooling around working around the nasties of CloudFormation. It's just not attractive.

You're the zealot here. If I may ask, have you ever contributed to illumos codebase in any way yet?

[1]: https://docs.oracle.com/cd/E23824_01/pdf/821-1458.pdf

[2]: https://blog.jasper.la/docker-on-smartos-the-harder-way/

[3]: https://docs.danubecloud.org/user-guide/esdc/index.html

Dude that elitest crap you're spouting is exactly what killed Solaris.

I am not farm help and there is no need to insult me. It won't help you.

Repeat after me: I read the manual pages and it's still crap.

Then you need to go back to the basics and learn UNIX. Start with "The art of UNIX programming" (it's freely available on-line). If you don't grok UNIX (not Linux) manual pages, it's time for back to school. It'll help you being better on Linux too, because you'll grok the overarching concepts.

When I asked Joyent in 2013 for information needed to improve the wifi stack there was exactly zero interest.

That is completely understandable to me; Joyent doesn't want to finance the development of a driver for your laptop because their resources are constrained and their focus is on serving data across datacenters. In the UNIX mantra: "do one thing and do it well".

They will however help you to learn how to write device drivers but you obviously have no interest in that, otherwise you'd have done so like for example Murayama-san did.

Here's the 425 pages Oracle document that provides a tiny intro to how you "could" "maybe" get started

You only needed to read the manual pages for dladm(1M) and ifconfig(1M) and to have looked at the EXAMPLES sections. Crossbow, the underlying technology in both Solaris and illumos remains unchanged or at the very least backward compatible.

Judging by the way you talk I really seriously doubt you have any meaningful architecture running on SmartOS that's self managed.

If you equate "self managed" and "architecture" with clicking around on little VM buttons in a (web) GUI or editing Docker YAML files, then you are correct. I do my (configuration) management with operating system packages. When configuration overlay packages are installed, systems talk to each other automatically. If a human has to click on anything anywhere, it's a bug and gets logged and corrected as such. I on the other hand suspect this is a concept so alien to you that you probably think it's absolutely crazy. Fact is, it worked for 70,000+ systems for the past 12 years and continues to do so.

You're the zealot here. If I may ask, have you ever contributed to illumos codebase in any way yet?

Yes I am because it's excellent technology; I don't hide it and I don't pretend otherwise. And yes I have.

> Took GNU/Linux “only“ 12 years to begin to catch up.

What does Linux have to do with this? This is a new hypervisor backed by hardware virtualization, it isn't remotely related to Linux. Sure, you can run Linux on EC2, but you can also run literally any other OS as well.

Solaris Zones were great, but AWS Nitro is better.
It’s also proprietary, runs only on AWS and costs more than zones since AWS is the most expensive.

How does AWS Nitro help solve the hard resource preallocation problem?

What about CPU bursting capability of the mighty illumos kernel, how does AWS Nitro compare?

Can you beat zones on pricing (in terms of resources required) with AWS Nitro?

To be fair the overhead of virtualization was within tolerable in last ~5-7 years already. With these new additions you can finally move the extremely performance (mostly latency) sensitive use cases that was not possible to run on the top of virtualization. I think these use cases are probably less than 5% of the cloud usage. Still nice to have though.
I wonder what measures are in place in the bare-metal edition to protect themselves from firmware attacks. What protects them from possible attacks against BIOS flash or potentially infect the CPU and HDD chips?
There is a Nitro security chip that is integrated into the motherboard. SPI and I2C buses are routed through the security chip. During system reset, the security chip holds motherboard components in reset while firmware is inspected out of band. So all contents of flash are verified.

Local NVMe storage is also handled through our Nitro hardware, and the underlying physical flash devices are protected from firmware modifications.

https://www.youtube.com/watch?v=o9_4uGvbvnk provides an overview.

Can you use a regular AMI on the bare metal instances?
AMIs that work on c5 instances should work fine on bare metal instances. NVMe and ENA support will be the big two things you need.