Kind of weird that NOP actually slows down the pipeline, as I'd think that would be the easiest thing to optimize out of the pipeline, unless instruction fetch is one of the main limiting factors. Is it architecturally defined that NOP will slow down execution?
In my opinion, NOP and MOV, which are recommended in TFA for slowing down, are the worst possible choices.
The authors have tested a rather obsolete CPU, with a 10-year-old Skylake microarchitecture, but more recent Intel/AMD CPUs have special optimizations for both NOP and MOV, executing them at the renaming stage, well before the normal execution units, so they may appear to have been executed in zero time.
For slowing down, one could use something really slow, like integer division. If that would interfere with the desired register usage, other reliable choices would be add with carry or perhaps complement carry flag. If it is not desired to modify the flags, one can use a RORX instruction for multiple bit rotation (available since Haswell, but not in older Atom CPUs), or one could execute BSWAP (available since 1989, therefore it exists in all 64-bit CPUs, including any Atom).
There's a whole niche in cryptography called verifiable delay functions (VDFs) if you want a big rabbit hole to go down.
The idea is that these are unavoidably slow and preferably non-parallelizable to compute one way, but fast or near instantaneous to verify the result. Examples include the Weslowski VDFs based on similar math to RSA, MIMC, and the use of zero knowledge proofs to provide proof of slow computations.
From what I understand if you really want to mess with some performance stuff just throw a random AVX or even better AVX512 workload in the critical path of all cores. Could be something really simple, like expanding out a normal cmov, jmp statement, or just regular integer addition to be needlessly AVX involved. Depending on the CPU this can apparently downclock the core, change the power delivery, and potentially result in pipeline stalls on other cores on the same die.
Causal profiling is something I infrequently use but I have been able to apply it outside of micro benchmarks in an orchestration system.
If you have a business process that is complex and owned by many different groups then causal profiling can be a good tool for dealing with the complexity. For large workflows in particular this can be powerful as the orchestration owner can experiment without much coordination (other than making sure the groups know/agree that some processing might be delayed).
> we can estimate whether an optimization is beneficial before implementing it. Coz simulates it by slowing down all other program parts. The part we think might be optimizable stays at the same speed it was before, but is now virtually sped up, which allows us to see whether it gives enough of a benefit to justify a perhaps lengthy optimization project.
Really interesting idea. I suppose the underlying assumption is that speeding up a function might reveal a new performance floor that is much higher than the speedup would suggest. Spending time to halve a function's time only to discover that overall processing time only decreased slightly because another function is now blocking would indeed be quite bad.
Not sure how this handles I/O, or if it is enough to simply delay the results of I/O functions to simulate decreased bandwidth or increased latency.
When doing web development I will occasionally connect my local code base to a remote SQL server via SSH.
This adds enough latency to be noticeable and I’ve found pages that were “OK” in prod that were unbearable in my local environment. Most of the time it was N+1 queries. Sometimes it was a cache that wasn’t working as intended. Sometimes it simply was a feature that “looked cool” but offered no value.
I’m not sure if there is a proxy that would do this locally but I’ve found it invaluable.
macOS has embedded feature that allows slowing down network calls. Similar to the IP/netfilter suite Linux has. PF handles most of the magic behind the scenes.
Even better because there is an UI to configure it. You just need to download Xcode additional tools package, there is a NetworkConditioner.prefpane inside. Install that and various settings will show up in the regular System Settings/Preferences...
Taking this approach seems to be effective at surfacing issues that otherwise wouldn't show up in regular testing. I could see this being useful if it was standardized to help identify issues before they hit production.
I fondly remember using a program called MoSlow to slow down older DOS games that were used to using every cycle they could get, and so were unplayable when I got a Pentium computer. I joked about how I wish I could find the corresponding "MoFast" program.
After a recent upgrade, I've been pondering how to spot test optimizations on my workstation, which is now much faster than the average target system. (Keeping a slow computer around and continually moving my code to it would be effective, but inconvenient.)
My first idea is to taskset the code to a particular CPU core, and see if linux will let me put that core in a low frequency mode. Has anyone here done this on AMD hardware? If so, were the available frequencies slow enough to be useful for this purpose?
When I had to do this, I just incremented a register.
xor rcx,rcx
loop:
inc rcx
cmp rcx, -1
jne loop
Something like that roughly. Except, I didn't count all the way to -1, just roughly half way through the 64bit space. It stalls for a couple of minutes.
Now, I wasn't doing this for stressing profilers or anything fancy like that. I was looking into malware delayed execution techniques. Sleep() and other "Dear system, I'm gonna nap for a bit, wake up later" routines have the downside of being api or system service calls, and detecting branching decisions after sleeping is a tell-tale red flag for anti-malware systems.
Sure, you can detect techniques like this as well, but I figured I could insert idempotent instructions in the loop's inner basic block that makes it look like it's crunching data for legitimate reasons instead. It just needs to delay execution long enough to fool sandbox automated analysis. I've thought about delaying/frustrating human analysis too, and I'm sure better minds than mine have thought of better solutions, but making this part of an unpacking routine that relies on the computed value to decrypt malicious code seems to be the obvious thing to do. Which again, I'm sure it's been done, but doing it yourself and figuring out the anti-anti-analysis techniques is fun.
Kinda related: sending a SIGSTOP to your datastore is a nice way to discover one of the failure modes of your architecture. (Use SIGCONT to make the server continue.)
19 comments
[ 3.1 ms ] story [ 43.8 ms ] threadLater on the early PCs we had a Turbo Button, but since everyone had it in Turbo mode all the time it essentially was a way to slow down the machine.
EDIT: Found an image of what I remember as the "C64 Snail". It is called "BREMSE 64" (which is German for brake) in the image.
https://retroport.de/wp-content/uploads/2018/10/bremse64_rex...
The authors have tested a rather obsolete CPU, with a 10-year-old Skylake microarchitecture, but more recent Intel/AMD CPUs have special optimizations for both NOP and MOV, executing them at the renaming stage, well before the normal execution units, so they may appear to have been executed in zero time.
For slowing down, one could use something really slow, like integer division. If that would interfere with the desired register usage, other reliable choices would be add with carry or perhaps complement carry flag. If it is not desired to modify the flags, one can use a RORX instruction for multiple bit rotation (available since Haswell, but not in older Atom CPUs), or one could execute BSWAP (available since 1989, therefore it exists in all 64-bit CPUs, including any Atom).
The idea is that these are unavoidably slow and preferably non-parallelizable to compute one way, but fast or near instantaneous to verify the result. Examples include the Weslowski VDFs based on similar math to RSA, MIMC, and the use of zero knowledge proofs to provide proof of slow computations.
If you have a business process that is complex and owned by many different groups then causal profiling can be a good tool for dealing with the complexity. For large workflows in particular this can be powerful as the orchestration owner can experiment without much coordination (other than making sure the groups know/agree that some processing might be delayed).
Really interesting idea. I suppose the underlying assumption is that speeding up a function might reveal a new performance floor that is much higher than the speedup would suggest. Spending time to halve a function's time only to discover that overall processing time only decreased slightly because another function is now blocking would indeed be quite bad.
Not sure how this handles I/O, or if it is enough to simply delay the results of I/O functions to simulate decreased bandwidth or increased latency.
This adds enough latency to be noticeable and I’ve found pages that were “OK” in prod that were unbearable in my local environment. Most of the time it was N+1 queries. Sometimes it was a cache that wasn’t working as intended. Sometimes it simply was a feature that “looked cool” but offered no value.
I’m not sure if there is a proxy that would do this locally but I’ve found it invaluable.
Even better because there is an UI to configure it. You just need to download Xcode additional tools package, there is a NetworkConditioner.prefpane inside. Install that and various settings will show up in the regular System Settings/Preferences...
- Making Postgres slower (https://news.ycombinator.com/item?id=44704736)
Taking this approach seems to be effective at surfacing issues that otherwise wouldn't show up in regular testing. I could see this being useful if it was standardized to help identify issues before they hit production.
https://en.wikipedia.org/wiki/Turbo_button#Purpose
My first idea is to taskset the code to a particular CPU core, and see if linux will let me put that core in a low frequency mode. Has anyone here done this on AMD hardware? If so, were the available frequencies slow enough to be useful for this purpose?
xor rcx,rcx
loop:
inc rcx
cmp rcx, -1
jne loop
Something like that roughly. Except, I didn't count all the way to -1, just roughly half way through the 64bit space. It stalls for a couple of minutes.
Now, I wasn't doing this for stressing profilers or anything fancy like that. I was looking into malware delayed execution techniques. Sleep() and other "Dear system, I'm gonna nap for a bit, wake up later" routines have the downside of being api or system service calls, and detecting branching decisions after sleeping is a tell-tale red flag for anti-malware systems.
Sure, you can detect techniques like this as well, but I figured I could insert idempotent instructions in the loop's inner basic block that makes it look like it's crunching data for legitimate reasons instead. It just needs to delay execution long enough to fool sandbox automated analysis. I've thought about delaying/frustrating human analysis too, and I'm sure better minds than mine have thought of better solutions, but making this part of an unpacking routine that relies on the computed value to decrypt malicious code seems to be the obvious thing to do. Which again, I'm sure it's been done, but doing it yourself and figuring out the anti-anti-analysis techniques is fun.
for i = 1 to 100
nextof course with modern cpus you gotta adjust your i to be something more reasonable, like
for i = 1 to 1000000000
works every time
Why: 100x more users instantly