- I have nodejs server for the APIs and its running on m5.xlarge instance. I haven't done much research on what instance type should I go for. I looked up and it seems like c5n.xlarge(mentioned in the article) is meant compute optimized. That cost difference isn't much between m5.xlarge and c5n.xlarge. So, I'm assuming that switching to c5 instance would be better, right?
- Does having ngnix handle the request is better option here? And setup reverse proxy for NodeJS? I'm thinking of taking small steps on scaling an existing framework.
The c5 instance type is about 10-15% faster than the m5, but the m5 has twice as much memory. So if memory is not a concern then switching to c5 is both a little cheaper and a little faster.
You shouldn't need the c5n, the regular c5 should be fine for most use cases, and it is cheaper.
Nginx in front of nodejs sounds like a solid starting point, but I can't claim to have a ton of experience with that combo.
m5 has more memory, if you application is memory bound stick with that instance type.
I'd recommend just using a standard AWS application load balancer in front of your Node.js app. Terminate SSL at the ALB as well using certificate manager (free). Will run you around $18 a month more.
Very nice round-up of techniques. I'd throw out a few that might or might not be worth trying: 1) I always disable C-states deeper than C1E. Waking from C6 takes upwards of 100 microseconds, way too much for a latency-sensitive service, and it doesn't save you any money when you are running on EC2; 2) Try receive flow steering for a possible boost above and beyond what you get from RSS.
Would also be interesting to discuss the impacts of turning off the xmit queue discipline. fq is designed to reduce frame drops at the switch level. Transmitting as fast as possible can cause frame drops which will totally erase all your other tuning work.
Does C-state tuning even do anything on EC2? My intuition says it probably doesn't pass through to the underlying hardware -- once the VM exits, it's up to the host OS what power state the CPU goes into.
Okay, so it looks as though it only applies to certain large instance types -- presumably ones which are large enough that it makes sense for the host to statically allocate CPU cores (or even sockets) to a guest. Interesting.
I suspect that the web server's CPU usage will be pretty high (almost 100%), so C-state tuning may not matter as much?
EDIT: also, RSS happens on the NIC. RFS happens in the kernel, so it might not be as effective. For a uniform request workload like the one in the article, statically binding flows to a NIC queue should be sufficient. :)
AWS doesn't let you mess with c-states for instances smaller than a c5.9xlarge[1]. I did actually test it out on a 9xlarge just for kicks, but it didn't make a difference. Once this test starts, all CPUs are 99+% Busy for the duration of the test. I think it would factor in more if there were lots of CPUs, and some were idle during the test.
> Try receive flow steering for a possible boost
I think the stuff I do in the "perfect locality" section[2] (particularly SO_ATTACH_REUSEPORT_CBPF) achieves what receive flow steering would be trying to do, but more efficiently.
> Would also be interesting to discuss the impacts of turning off the xmit queue discipline
Yea, noqueue would definitely be a no-go on a constrained network, but when running the (t)wrk benchmark in the cluster placement group I didn't see any evidence of packet drops or retransmits. Drop only happened with the iperf test.
That is one hell of a comprehensive article. I wonder how much impact would such extreme optimizations on a real-world application, which for example does DB queries.
This experiment feels similar to people who buy old cars and remove everything from the inside except the engine, which they tune up so that the car runs faster :).
Yes, my experience (not much) is that what makes YouTube or Google or any of those products really impressive is the speed.
YouTube or Google Search suggestion is good, and I think it could be replicable with that amount of data. What is insane is the speed. I can't think how they do it. I am doing something similar for the company I work on and it takes seconds (and the amount of data isn't that much), so I can't wrap my head around it.
The point is that doing only speed is not _that_ complicated, and doing some algorithms alone is not _that_ complicated. What is really hard is to do both.
A lot of this is just spending more money and resources to make it possible to optimize for speed.
With sufficient caching with and a lot of parallelism makes this possible. That costs money though. Caching means storing data twice. Parallelism means more servers (since you'll probably be aiming to saturate the network bandwidth for each host).
Pre-aggregating data is another part of the strategy, as that avoids using CPU cycles in the fast-path, but it means storing even more copies of the data!
My personal anecdotal experience with this is with SQL on object storage. Query engines that use object storage can still perform well with the above techniques, even though querying large amounts of data from object is slow. You can bypass the slowness of object storage if you pre-cache the data somewhere else that's closer/faster for recent data. You can have materialized views/tables for rollups of data over longer periods of time, which reduces the data needed to be fetched and cached. It also requires less CPU due to working with a smaller amount of pre-calculated data.
Apply this to every layer, every system, etc, and you can get good performance even with tons of data. It's why doing machine-learning in real- is way harder than pre-computing models. Streaming platforms make this all much easier as you can constantly be pre-computing as much as you can, and pre-filling caches, etc.
Of course, having engineers work on 1% performance improvements in the OS kernel, or memory allocators, etc will add up and help a lot too.
One interesting thing to note is that there are lots of internal tools (CLI, web UI, etc.) that are REALLY slow. Things that are heavily used in the fast-path for development (e.g. code search, code review, test results) are generally pretty quick, but if there's a random system that has a UI, it's probably going to be very slow - because there's no budget for speeding them up, and the only people it annoys are engineers from other teams.
I've had them take seconds for suggestions before when doing more esoteric searches. I think there's an inordinate amount of cached suggestions and they have an incredible way to look them up efficiently.
Related, I worked on an embedded linux device that bit-banged an unusual communication protocol. I wanted to offload this bit banging to some dedicated microcontrollers built into the SOC, to help free up the main CPU for some other use.
I spent some time writing a nice little device driver that used DMA to send the transaction to the bit banger. The main CPU usage went from 60% down to 2%! Success! Unfortunately, the transactions/second was now a fraction of what it was before.
After some digging, it turned out the improved CPU usage was low enough that the CPU scaling governor was staying at the lowest frequency. Disabling the governor and forcing the CPU to max frequency resulted in a pretty huge transactions/second improvements. Obvious in retrospect, of course.
This comprehensive level of extreme tuning is not going to be directly useful to most people; but there are a few things in there like SO_ATTACH_REUSEPORT_CBPF that I hope to see more servers and frameworks adopt. Similarly I think it is good to be aware of the adaptive interrupt capabilities of AWS instances, and the impacts of speculative execution mitigations, even if you stick to the defaults.
More importantly it is about the idea of using tools like Flamegraphs (or other profiling tools) to identify and eliminate your bottlenecks. It is also just fun to experiment and share the results (and the CloudFormation template). Plus it establishes a high water mark for what is possible, which also makes it useful for future experiments. At some point I would like to do a modified version of this that includes DB queries.
Wow, I haven't seen SO_ATTACH_REUSEPORT_CBPF before, I didn't even know it existed. That is a pretty ingenious and powerful primitive to cut down on cross-NUMA chatter. I always like it when folks push things to the extreme, it really shows exactly what is going on under the hood.
He's cheating by assuming all http responses fit in one TCP packet, but you could use FreeBSD which is already tuned like this and has optimizations like ACCEPT_FILTER_HTTP not mentioned in this article.
I'm of two minds with regards to this: This is cool but unless you have no authentication, data to fetch remotely or on disk this is really just telling you what the ceiling is for everything you could possibly run.
As for this article, there are so many knobs that you tweaked to get this to run faster it's incredibly informative. Thank you for sharing.
Fantastic article. Disabling spectre mitigations on all my team's GCE instances is something I'm going to check out.
Regarding core pinning, the usual advice is to pin to the CPU socket physically closest to the NIC. Is there any point doing this on cloud instances? Your actual cores could be anywhere. So just isolate one and hope for the best?
Pinning to the physically closest core is a bit misleading. Take a look at output from something like `lstopo` [https://www.open-mpi.org/projects/hwloc/], where you can filter pids across the NUMA topology and trace which components are routed into which nodes. Pin the network based workloads into the corresponding NUMA node and isolate processes from hitting the IRQ that drives the NIC.
Yea, I looked it wrk2 but it was a no-go right out the gate. From what I recall the changes to handle coordinated omission use a timer that has a 1ms resolution. So basically things broke immediately because all requests were under 1ms.
I didn't make any coordinated omission changes (I really didn't make many changes in general), so twrk does what wrk does. It attempts to correct it after the fact by looking for requests that took twice as long as average and doing some backfilling[1].
I am no expert where coordinated omission is concerned, but my understanding is that it is most problematic in scenarios where your p90+ latency is high. Looking at the results for the 1.2M req/s test you have the following latencies:
If you were to apply wrk's coordinated omission hack to these result, the backfilling only starts for requests that took longer than p50 x 2 (roughly) = 406us, which is probably somewhere between p99.999 and pMAX; a very, very small percentage.
I am not claiming that wrk's hack is "correct", just that I don't think coordinated omission is a major concern for *this specific workload/environment*
If I understand correctly, coordinated omission handling only matters if the benchmark is done with a fixed rate RPS right? In this case, it looks like a closed model benchmark where a fixed number of client threads just go as fast as they can.
edit: Oh, perhaps wrk2 still relies on the timer even when not specifying a fixed rate RPS.
I really like that wrk2 allows to configure fixed framerate, latency measurement works much better in this case.
But wrk2 itself has bugs that doesn't allow it to use in more complicated cases, e.g. lua scripts are not working properly.
There is no published limit, but based on my tests the network device for the c5n.xlarge has a hard limit of 1.8M pps (which translates directly to req/s for small requests without pipelining).
There is also a quota system in place, so even though that is the hard limit, you can only operate at those speeds for a short time before you start getting rate-limited.
It is a statically generated site created with vitepress[1] and hosted on Cloudflare Pages[2]. The only dynamic functionality is the contact form which sends a JSON request to a Cloudflare Worker[3], which in turn dispatches the message to me via SNS[4].
It is modeled off of the code used to generate Vue blog[5], but I made a ton of little modifications, including some changes directly to vitepress.
Keep in mind that vitepress is very much an early work in progress and the blog functionality is just kinda tacked on, the default use case is documentation. It also definitely has bugs and is under heavy development so wouldn't recommend it quite yet unless you are actually interested in getting your handa dirty with Vue 3. I am glad I used it because it gave me an excuse to start learning Vue, but unless you are just using the default theme to create a documentation site, it will require some work.
On the other hand you could probably make the table of content be always visible when the screen size allows it. Clicking on the burger in the site menu to get a page-specific sidebar is a bit counter-intuitive.
Since it's CPU-bound and spends a lot of time in the kernel would compiling the kernel for the specific CPU used make sense? Or are the CPU cycles wasted on things the compiler can't optimize?
Wow. Such impressive bpftrace skill! Keeping this article under my pillow ;)
Wonder where the next optimization path leads? Using huge memory pages. io_uring, which was briefly mentioned. Or kernel bypass, which is supported on c5n instances as of late...
How can you be sure the estimated max server capability is not actually just a limitation in the client, i.e, the client maxes out at sending 224k requests / second.
I see that this is clearly not the case here, but in general how can one be sure?
You parallelize the load from multiple clients (running on separate hardware). There are some open source projects that facilitate this sort of workload (and the subsequent aggregation of results/stats.)
I don't have anything to add to the conversation other than to say that this is fantastic technical writing (and content too). Most of the time, when similar articles like this one are posted to company blogs, they bore me to tears and I can't finish them, but this is very engaging and informative. Cheers
Thanks, that actually means a lot. It took a lot of work, not just on the server/code, but also the writing. I asked a lot of people to review it (some multiple times) and made a ton of changes/edits over the last couple months.
Some of these things could be fixed upstream and everyone see real perf gains...
For example, having dhclient (a very popular dhcp client) leave open an AF_PACKET socket causing a 3% slowdown in incoming packet processing for all network packets seems... suboptimal!
Surely it can be patched to not cause a systemwide 3% slowdown (or at least to only do it very briefly while actively refreshing the DHCP lease)?
I would also love to see that dhclient issue resolved upstream, or at least a cleaner way to work around it. But we should also be mindful that for most workloads the impact is probably way, way less.
Some of these things really only show up when you push things to their extremes, so it probably just wasn't on the developer's radar before.
I believe systemd-networkd has its own implementation of DHCP and therefore doesn't use dhclient. But I wonder if it's behavior is any better in this respect.
systemd-networkd keeps open that kind of socket for LLDP but apparently not for the DHCP client code. wpa_supplicant also keeps open this type of socket on my local system. and the dhcpd daemons on my routers have some of those too for each interface...
i wonder if the slow path here could be avoided by using separate network namespaces in a way these sockets don't even get to see the packets...
Specifically on EC2 I don't think you actually need to keep dhcp client running anyways, afaik EC2 instance ips are static so you can just keep using the one you got on boot.
Great work, thanks for sharing! Systems performance at its best. Nice to see the use of the custom palette.map (I forget to do that myself and I often end up hacking in highlights in the Perl code.)
BTW, those disconnected kernel stacks can probably be reconnected with the user stacks by switching out the libc for one with frame pointers; e.g., the new libc6-prof package.
Thank you for sharing all your amazing tools and resources brendangregg! I wouldn't have been able to do most of these optimizations without FlameGraph and bpftrace.
I actually did the same thing and hacked up the perl code to generate the my custom palette.map
Thanks for the tip re: the disconnected kernel stacks. They actually kinda started to grow on me for this experiment, especially since most of the work was on the kernel side.
Is libc6-prof just glibc recompiled with -fno-omit-frame-pointer? I did that a couple times and found that while that fixes a few system calls, it doesn't fix all of them. I think the main issue was several syscalls being called from asm, which wasn't unsurprisingly isn't affected by -fno-omit-frame-pointer.
Right, it is. It fixed my hot-path syscalls on x86 (via read/write functions, pthread_mutex functions, etc.). But if you have syscalls called via asm outside of libc (by who?) then they need frame pointers as well.
When is it advisable to turn off spectre/meltdown mittigations in practice? My guess is that if you are on a server and not running any user supplied code then you are on the safe side; on condition that you could exclude buffer overuns by running managed code/java or by using Rust.
So the unspoken part of your question is when is it useful to turn off mitigations. The answer to that is when your application makes a lot of syscalls / when syscalls are a bottleneck beyond the actual work of the syscalls.
This case, where it's all connection handling and serving a small static piece of data is a clear example; there's almost no userland work to be done before it goes to another syscall so any additional cost for the user/kernel barrier is going to hurt.
Then the question becomes who can run code on your server; also condidering maybe there's a remote code execution vulnerability in your code, or library code you use. Is there a meaningful barrier that spectre/meltdown mitigations would help enforce? Or would getting RCE get control over everything of substance anyway?
> Disabling [spectre] mitigations gives us a performance boost of around 28%
Every couple months these last several years there always seems to be some bug where the fix only costs us 3% performance. Since those tiny performance hits add up over time, security is sort of like inflation in the compute economy. What I want to know is how high can we make that 28% go? The author could likely build a custom kernel that turns off stuff like pie, aslr, retpoline, etc. which would likely yield another 10%. Can anyone think of anything else?
Most of these mitigations are worse than useless in an environment not executing untrusted code. Simply put, if you have a dedicated server and you aren't running user code, you don't need them.
But of course other exploits (e.g. in your webapp) might lead to "running user code" where you didn't expect it and then the mitigations could prevent privilege escalation, couldn't they?
But if you have a dedicated server for your web app, if there's some kind of exploit in it allowing for random code to be run, said code already has access to everything it needs, right?
The interesting data will probably be whatever secrets the app handles, say database credentials, so the attacker is off to the races. They probably don't care about having root in particular.
All those points are true - though I'd argue this is stretching the "one app per VM" thing -, but I guess this is just the usual case of understanding your situation and realizing there's no one size fits all.
My take on this question is rather that there shouldn't be any dogma around this, such as disabling mitigations should not be considered absolutely, 100% harmful and never, ever, ever disabled.
In the context of the OP, where the application is running on AWS, backups, email, etc are all likely to be handled either externally (say EBS snapshots) in which case there's no issue, or via "trusting the machine", so getting credentials via the instance role which every process on the VM can do, so no need for privilege escalation.
So I guess if you trust EC2 or Task roles or similar (not familiar with EKS) to access sensitive data and only run a "single" application, there's likely little to no reason to use the mitigations.
But, yeah, if you're running an application with multiple components, each in their own processes and don't use instance roles for sensitive access, maybe leave them on. Also, maybe, this means you're not running a single app per vm?
Like everything in security, it's about tradeoffs.
> Also, maybe, this means you're not running a single app per vm?
This is an argument for unikernels.
Instead, on 99.9% of your services you want to run multiple independent processes, especially in a datacenter environment: your service, web server, sshd, logging forwarder, monitoring daemon, dhcp client, NTP client, backup service.
Often some additional "bigcorp" services like HIDS, credential provider, asset management, power management, deployment tools.
Yes, but I was using my initial post's parent's terminology. But I agree, in my mind, the subject was one single "service", as in process (or a process hierarchy, like say with gunicorn for python deployments).
> This is an argument for unikernels.
It is. And I'm also very interested in the developments around Firecraker and similar technologies. If we'd be able to have the kind of isolation AWS promises between ec2 instances on a single physical machine, while at the same time being able to launch a process in an isolated container as easy as with docker right now, I'd consider that really great. And all the other "infrastructure" services you talk about could just live their lives in their dedicated containers.
Not sure how all this would compare, performance-wise, with just enabling the mitigations.
The puzzling thing was that spectre V2 mitigations were cited as the main culprit. They were responsible by themselves for a 15-20% slowdown, which is about an order of magnitude worse than in my experience. I wonder if the system had IBRS enabled instead of using retpolines at the mitigation strategy?
I am not full deep in SecOps these days and would gladly hear opinion of some expert:
Can disabling these mitigations bring any risks assuming the server is sending static content to the Internet over port 80/443 and it is practically stateless with read-only file system?
I am not an expert but you shall have my take either way. The most important question here is "Am I executing arbitrary untrusted code?". HTTP servers will parse the incoming requests so they are executed to some extent. But I would not worry about it unless there is some backend application doing more involved processing with the data. repl.it should not disable mitigations.
149 comments
[ 2.4 ms ] story [ 218 ms ] thread- I have nodejs server for the APIs and its running on m5.xlarge instance. I haven't done much research on what instance type should I go for. I looked up and it seems like c5n.xlarge(mentioned in the article) is meant compute optimized. That cost difference isn't much between m5.xlarge and c5n.xlarge. So, I'm assuming that switching to c5 instance would be better, right?
- Does having ngnix handle the request is better option here? And setup reverse proxy for NodeJS? I'm thinking of taking small steps on scaling an existing framework.
The c5 instance type is about 10-15% faster than the m5, but the m5 has twice as much memory. So if memory is not a concern then switching to c5 is both a little cheaper and a little faster.
You shouldn't need the c5n, the regular c5 should be fine for most use cases, and it is cheaper.
Nginx in front of nodejs sounds like a solid starting point, but I can't claim to have a ton of experience with that combo.
I'd recommend just using a standard AWS application load balancer in front of your Node.js app. Terminate SSL at the ALB as well using certificate manager (free). Will run you around $18 a month more.
As in all things, check the results on your own workload!
Would also be interesting to discuss the impacts of turning off the xmit queue discipline. fq is designed to reduce frame drops at the switch level. Transmitting as fast as possible can cause frame drops which will totally erase all your other tuning work.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processo...
EDIT: also, RSS happens on the NIC. RFS happens in the kernel, so it might not be as effective. For a uniform request workload like the one in the article, statically binding flows to a NIC queue should be sufficient. :)
> I always disable C-states deeper than C1E
AWS doesn't let you mess with c-states for instances smaller than a c5.9xlarge[1]. I did actually test it out on a 9xlarge just for kicks, but it didn't make a difference. Once this test starts, all CPUs are 99+% Busy for the duration of the test. I think it would factor in more if there were lots of CPUs, and some were idle during the test.
> Try receive flow steering for a possible boost
I think the stuff I do in the "perfect locality" section[2] (particularly SO_ATTACH_REUSEPORT_CBPF) achieves what receive flow steering would be trying to do, but more efficiently.
> Would also be interesting to discuss the impacts of turning off the xmit queue discipline
Yea, noqueue would definitely be a no-go on a constrained network, but when running the (t)wrk benchmark in the cluster placement group I didn't see any evidence of packet drops or retransmits. Drop only happened with the iperf test.
1. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processo...
2. https://talawah.io/blog/extreme-http-performance-tuning-one-...
This experiment feels similar to people who buy old cars and remove everything from the inside except the engine, which they tune up so that the car runs faster :).
YouTube or Google Search suggestion is good, and I think it could be replicable with that amount of data. What is insane is the speed. I can't think how they do it. I am doing something similar for the company I work on and it takes seconds (and the amount of data isn't that much), so I can't wrap my head around it.
The point is that doing only speed is not _that_ complicated, and doing some algorithms alone is not _that_ complicated. What is really hard is to do both.
With sufficient caching with and a lot of parallelism makes this possible. That costs money though. Caching means storing data twice. Parallelism means more servers (since you'll probably be aiming to saturate the network bandwidth for each host).
Pre-aggregating data is another part of the strategy, as that avoids using CPU cycles in the fast-path, but it means storing even more copies of the data!
My personal anecdotal experience with this is with SQL on object storage. Query engines that use object storage can still perform well with the above techniques, even though querying large amounts of data from object is slow. You can bypass the slowness of object storage if you pre-cache the data somewhere else that's closer/faster for recent data. You can have materialized views/tables for rollups of data over longer periods of time, which reduces the data needed to be fetched and cached. It also requires less CPU due to working with a smaller amount of pre-calculated data.
Apply this to every layer, every system, etc, and you can get good performance even with tons of data. It's why doing machine-learning in real- is way harder than pre-computing models. Streaming platforms make this all much easier as you can constantly be pre-computing as much as you can, and pre-filling caches, etc.
Of course, having engineers work on 1% performance improvements in the OS kernel, or memory allocators, etc will add up and help a lot too.
It's hard to measure, so nobody does.
Throughput is easy to measure, so everybody does.
Latency is hard to buy, so few people try.
Throughput is easy to buy, so everybody does.
Latency is what matters to every user.
Throughput matters only to a few people.
Turn on SR-IOV. Disable ACPI C-states. Stop tunnelling internal traffic through virtual firewalls. Use binary protocols instead of JSON over HTTPS.
I've seen just those alone improve end-user experience tenfold.
Related, I worked on an embedded linux device that bit-banged an unusual communication protocol. I wanted to offload this bit banging to some dedicated microcontrollers built into the SOC, to help free up the main CPU for some other use.
I spent some time writing a nice little device driver that used DMA to send the transaction to the bit banger. The main CPU usage went from 60% down to 2%! Success! Unfortunately, the transactions/second was now a fraction of what it was before.
After some digging, it turned out the improved CPU usage was low enough that the CPU scaling governor was staying at the lowest frequency. Disabling the governor and forcing the CPU to max frequency resulted in a pretty huge transactions/second improvements. Obvious in retrospect, of course.
More importantly it is about the idea of using tools like Flamegraphs (or other profiling tools) to identify and eliminate your bottlenecks. It is also just fun to experiment and share the results (and the CloudFormation template). Plus it establishes a high water mark for what is possible, which also makes it useful for future experiments. At some point I would like to do a modified version of this that includes DB queries.
As for this article, there are so many knobs that you tweaked to get this to run faster it's incredibly informative. Thank you for sharing.
That's a useful piece of info to know when performance tuning a real world app with auth / data / etc.
Regarding core pinning, the usual advice is to pin to the CPU socket physically closest to the NIC. Is there any point doing this on cloud instances? Your actual cores could be anywhere. So just isolate one and hope for the best?
https://github.com/giltene/wrk2
Maybe you duplicated some of these fixes?
I am no expert where coordinated omission is concerned, but my understanding is that it is most problematic in scenarios where your p90+ latency is high. Looking at the results for the 1.2M req/s test you have the following latencies:
If you were to apply wrk's coordinated omission hack to these result, the backfilling only starts for requests that took longer than p50 x 2 (roughly) = 406us, which is probably somewhere between p99.999 and pMAX; a very, very small percentage.I am not claiming that wrk's hack is "correct", just that I don't think coordinated omission is a major concern for *this specific workload/environment*
1. https://github.com/wg/wrk/blob/a211dd5a7050b1f9e8a9870b95513...
edit: Oh, perhaps wrk2 still relies on the timer even when not specifying a fixed rate RPS.
There is also a quota system in place, so even though that is the hard limit, you can only operate at those speeds for a short time before you start getting rate-limited.
Presented this way may help noobs like me with capacity planning.
It is modeled off of the code used to generate Vue blog[5], but I made a ton of little modifications, including some changes directly to vitepress.
Keep in mind that vitepress is very much an early work in progress and the blog functionality is just kinda tacked on, the default use case is documentation. It also definitely has bugs and is under heavy development so wouldn't recommend it quite yet unless you are actually interested in getting your handa dirty with Vue 3. I am glad I used it because it gave me an excuse to start learning Vue, but unless you are just using the default theme to create a documentation site, it will require some work.
1. https://vitepress.vuejs.org/
2. https://pages.cloudflare.com/
3. https://workers.cloudflare.com/
4. https://aws.amazon.com/sns/
3. https://github.com/vuejs/blog
1. https://lwn.net/Articles/830300/
Wonder where the next optimization path leads? Using huge memory pages. io_uring, which was briefly mentioned. Or kernel bypass, which is supported on c5n instances as of late...
I see that this is clearly not the case here, but in general how can one be sure?
Thanks again to my reviewers!
For example, having dhclient (a very popular dhcp client) leave open an AF_PACKET socket causing a 3% slowdown in incoming packet processing for all network packets seems... suboptimal!
Surely it can be patched to not cause a systemwide 3% slowdown (or at least to only do it very briefly while actively refreshing the DHCP lease)?
Some of these things really only show up when you push things to their extremes, so it probably just wasn't on the developer's radar before.
This has piqued my interest.
i wonder if the slow path here could be avoided by using separate network namespaces in a way these sockets don't even get to see the packets...
Looks like LLDP can be switched off in the network config.
https://systemd.network/systemd.network.html
BTW, those disconnected kernel stacks can probably be reconnected with the user stacks by switching out the libc for one with frame pointers; e.g., the new libc6-prof package.
I actually did the same thing and hacked up the perl code to generate the my custom palette.map
Thanks for the tip re: the disconnected kernel stacks. They actually kinda started to grow on me for this experiment, especially since most of the work was on the kernel side.
This case, where it's all connection handling and serving a small static piece of data is a clear example; there's almost no userland work to be done before it goes to another syscall so any additional cost for the user/kernel barrier is going to hurt.
Then the question becomes who can run code on your server; also condidering maybe there's a remote code execution vulnerability in your code, or library code you use. Is there a meaningful barrier that spectre/meltdown mitigations would help enforce? Or would getting RCE get control over everything of substance anyway?
Did you use io_uring? Is its performance much better than or comparable with using aio_read/aio_write for block io? (i did use async io for block io).
Every couple months these last several years there always seems to be some bug where the fix only costs us 3% performance. Since those tiny performance hits add up over time, security is sort of like inflation in the compute economy. What I want to know is how high can we make that 28% go? The author could likely build a custom kernel that turns off stuff like pie, aslr, retpoline, etc. which would likely yield another 10%. Can anyone think of anything else?
The interesting data will probably be whatever secrets the app handles, say database credentials, so the attacker is off to the races. They probably don't care about having root in particular.
On the same host there could be SSL certificates, credentials in a local MTA, credentials used to run backups and so on.
Or the application itself could be made of multiple components where the vulnerable one is sandboxed.
My take on this question is rather that there shouldn't be any dogma around this, such as disabling mitigations should not be considered absolutely, 100% harmful and never, ever, ever disabled.
In the context of the OP, where the application is running on AWS, backups, email, etc are all likely to be handled either externally (say EBS snapshots) in which case there's no issue, or via "trusting the machine", so getting credentials via the instance role which every process on the VM can do, so no need for privilege escalation.
So I guess if you trust EC2 or Task roles or similar (not familiar with EKS) to access sensitive data and only run a "single" application, there's likely little to no reason to use the mitigations.
But, yeah, if you're running an application with multiple components, each in their own processes and don't use instance roles for sensitive access, maybe leave them on. Also, maybe, this means you're not running a single app per vm?
> there shouldn't be any dogma around this
Like everything in security, it's about tradeoffs.
> Also, maybe, this means you're not running a single app per vm?
This is an argument for unikernels.
Instead, on 99.9% of your services you want to run multiple independent processes, especially in a datacenter environment: your service, web server, sshd, logging forwarder, monitoring daemon, dhcp client, NTP client, backup service.
Often some additional "bigcorp" services like HIDS, credential provider, asset management, power management, deployment tools.
Yes, but I was using my initial post's parent's terminology. But I agree, in my mind, the subject was one single "service", as in process (or a process hierarchy, like say with gunicorn for python deployments).
> This is an argument for unikernels.
It is. And I'm also very interested in the developments around Firecraker and similar technologies. If we'd be able to have the kind of isolation AWS promises between ec2 instances on a single physical machine, while at the same time being able to launch a process in an isolated container as easy as with docker right now, I'd consider that really great. And all the other "infrastructure" services you talk about could just live their lives in their dedicated containers.
Not sure how all this would compare, performance-wise, with just enabling the mitigations.
Many security changes also help you find memory corruption bugs, which is good for developer productivity.
Can disabling these mitigations bring any risks assuming the server is sending static content to the Internet over port 80/443 and it is practically stateless with read-only file system?
The last time I looked I found a lot of waffle but no simple way I can just turn that stuff off...