> What happens if you grab the jq source code from Launchpad, then configure and rebuild it with no flags at all? Even that is about 2-4% faster than the Ubuntu binary package.
Debug symbols do not need to be paged in so shouldn’t make much/any difference. Compiling with -O3 can increase code size a lot due to inlining, which can be a little bad.
Mario 64 gets flak for some files not being compiled with optimizations on. There is a YouTuber, whose name I forget at the moment, who has been doing optimization and improvements on it and constantly talks about how this is for the better due to the low instruction cache size and slow speed on the console. Compiling with optimizations explodes code size (unrolling links, inlining, etc) to the point it’s a net loss after factoring in the increased load on the IC.
The IC is the instruction cache. Bloated code needs more slow loads from the IC, and even if the performance is faster per cache line, the 50x overhead of added cache loads diminishes all -O3 advantages.
Right, my humor there was that I'm assuming people specifically did non-debug flags for the build as an optimization. Only for them to still fall short.
And understood a little on -O3 possibly increasing code size. I had thought that was more of a concern for tight environments than for most systems? Of course, I'd have assumed that -march=native would be more impactful, but the post indicates otherwise.
I said in a top level, but it seems the allocator makes the biggest impact for this application? Would be interesting to see which applications should use different allocators. Would be amazing to see a system where the more likely optimal allocator was default for different applications, based on their typical allocation patterns.
> Right, my humor there was that I'm assuming people specifically did non-debug flags for the build as an optimization.
It used to be the case that the presence of debug symbols would affect GCC code generation. Nowadays that should be fixed. I think it still affects the speed of compilation so if you're building the whole system from source you might want to avoid it.
Right, I meant the humor to be that folks to not use debug flags, with my naive presumption that they did this as an optimization, only for that not to really help.
Ubuntu is still building for CPUs with the x86-64-v1 feature set. On experimental images built for x86-64-v3 (roughly the feature set of 10 year old CPUs, a bit newer on low-power CPUs) they have observed 60% improvement on some benchmarks (and much less in some others). Some distros have switched to -v3, Ubuntu is still holding out to support older hardware. The author is compiling to the actual feature set of their specific CPU, which is nearly guaranteed to be even more than -v3.
Another thing the article adds is LTO, which in my experience also makes a huge difference: it makes your software a lot faster in certain cases, but also makes build times a lot worse. Spending a bit more time in the build process should be an easy call, but might be harder at the scale of a distro.
I was under the, evidently naive, understanding that there were some shenanigans done in the libc to pick optimal code paths based on the architecture. This would obviously have a startup cost on the first run, but presumably the dynamic linking would work to this advantage by having it in a fast state for most calls?
Not sure where I got that idea, though. :(. Will have to look into that later.
Even if libc does it, other packages wouldn’t. The compiler is what generates code and they generally don’t try to automatically generate for different instruction sets because they don’t know what code is what hotpath and worth optimizing for. You could probably try to build for multiple different CPU features at the top level and do a universal executable like Apple did for their CPU transitions but that’s a lot of expense to pay both in terms of compilation time AND in terms of size.
Using preload on the stock Ubuntu binary, to give it mimalloc, they got a 44% speedup.
By rebuilding the binary with different compiler options, but not changing malloc, they got an 20% speedup.
If we naively multiply these speedups, we get 1.78: 78% faster.
How it goes to 1.9 is that when you speed up only the program, but leave malloc the same, malloc matters to its performance a lot more.
When the faster malloc is applied to a program that is compiled better, it will make a better contribution than the 44% seen when the allocator was preloaded into the slower program.
To do the math right, we would have to look into how much time was saved with just the one change, and how much time was saved with the other. If we add those times, and subtract them from the original, slow time, we should get a time that is close to the 1.9 speedup.
It's not a 90% speedup, it's ~50% (still quite impressive).
The author seems to be confused, because the original jq is 1.9x slower than the optimized one.
That depends on how you're representing the speedup.
To travel 10 miles, at 60 MPH, takes 10 minutes. Make it 100% faster, at 120 MPH, and that time becomes 5 minutes. Travel just as far in 50% of the time. Or travel just as far 100% faster. The 90% speedup matches the reduction of the time it takes to nearly half (a 90% (projected) speedup, or about a 45% time reduction, as mathed out by kazinator `Projected speedup from both: 4.631/2.431 = 1.905`). Your claim that its closer to 50% is correct from a total time taken perspective, just coming at it from the other direction.
It's been awhile since I looked into this, but it's not necessarily an easy change. glibc malloc has debugging APIs; a distro can't easily replace it without either emulating the API or patching programs that use it.
No need to even patch it out. It's relatively easy to change the default, rebuild the world (distros have a flow for this), and restore glibc for the tiny, tiny handful of individual programs that actually rely on glibc debugging APIs.
From a end developer perspective: I have no particular familiarity with mimalloc, but I know jemalloc has pretty extensive debugging functionality (not API compatible with glibc malloc of course).
If I had the time to fork jq, I would convert the relevant part to C++ so it doesn't spend literally all of its time dynamically allocating strings it is just about to discard.
jaq is nice. It just loses, performance-wise, to the final step in this article, and it can't do the second mentioned workload (yet) so I didn't include it.
It bombs out on the jq program I use for the 2nd corpus that I mentioned. On further investigation, the show-stopping filter is strftime. In the jaq readme this is the only not-yet-checked box in the compatibility list, so perhaps some day soon.
Apologies, I did my first post on my phone, so was rather quick. I had found the repository and it is a good read. What I'd love to see is a comprehensive dive on the different options and a good general discussion on when each is superior.
I realize, of course, that this could just be overall optimal. Feels more likely that the allocation patterns of the application using it will impact which is the better allocator?
How would you do the things mentioned in the article using Guix?
You could write your own custom package definitions, extending the default to change up compile flags and allocators, but then you need to do this for every single package (and maintain them all). I'm not sure Guix gives you much here, though maybe that's fine for one or two packages.
The most pain-free option I can think of is the --tune flag (which is similar to applying -march=native), but packages have to be defined as tunable for it to work (and not many are).
If you want it to be permanent, then you can use a guix home profile (that's a declarative configuration of your home directory) with a patch function in the package list there:
You can also write a 10 line guile script to automatically do it for all dependencies (I sometimes do--for example for emacs). That would cause a massive rebuild, though.
>The most pain-free option I can think of is the --tune flag (which is similar to applying -march=native), but
> packages have to be defined as tunable for it to work (and not many are).
We did it that way on purpose--from prior experience, otherwise, you would get a combinatorial explosion of different package combinations.
If it does help for some package X, please email us a 2 line patch adding (tunable? . #t) to that one package.
If you do use --tune, it will tune everything that is tuneable in the dependency graph. But at least all dependents (not dependencies) will be just grafted--not be rebuilt.
O3 is a bad idea if the package is important. While that’s a dated opinion based on older experiences (Gentoo, various server software, etc), I suspect it’s still the easiest way to get exposed to compiler bugs.
If you're going to offer unsupported FUD, I would prefer you direct the FUD at NDEBUG and the fear that it could have disabled a load-bearing assertion.
Just because it's turned off by default doesn't mean it isn't load bearing. If you didn't ever encounter an ASSERT with side effects that were crucial to proper functioning, you just haven't seen enough ASSERT statements.
Rule #1 of programming: If it can go wrong, it will.
That's before we even consider instances where assert was used to check a condition that should always be verified. In my personal projects I define a "require" macro that I use liberally.
If acknowledging my opinion was dated based on older experiences constitutes “unsupported FUD”, well, have fun with the life experiences that attitude produces for you.
The undefined behaviour is already in the C standard.
But you are right that enabling -O3 is generally going to make your compiler more aggressive about exploiting the undefined behaviour in the pursuit of speed.
In general, if you want to avoid being the guy to run into new and exciting bugs, you want to run with options that are as close as possible to what everyone else is using.
The benchmarking tool being used in this post accounts for that using multiple runs for each invocation, together with a warmup run that is not included in the result metric.
I wish I had seen this earlier. My mental model was close but not quite there to the point where I needed to think too hard about how to solve problems.
Yeah after this post I sought a couple youtube videos explaining it. It's starting to make a bit more sense now. But the lightbulb hasn't gone off just yet. Appreciate the link.
I think jq's syntax is pretty sweet, once you get used to it (or are already familiar with point-free style, as it's in Haskell). The man page is subpar, however.
Misleading title, it's 90% of the faster time. It's about 45% faster.
It's actually a little bit interesting, if you are interested in how we use language. You could argue that now you now get 90% more work done in the same amount of time, and that would align with other 'speed' units that we commonly use (miles per hour, words per minute, bits per second). However, the convention in computer performance is to measure time for a fixed amount of work. I would guess that this is because generally we have a fixed amount of work and what might vary is how long we wait for it (and that is absolutely true in the case of this blog post) so we put time in the numerator.
It's a very interesting post and very well done, but it's not 90% faster.
I feel like using units of rate (90% faster) and not units of time makes more sense here.
Plus, if you were using units of time, you wouldn’t use the word “faster.” “Takes 45% less time” and “45% faster” are very different assertions, but they both have meaning, both in programming and outside it.
It comes down to convention. When talking about proportional differences, we can fix the unitary example to be the smaller, larger, earlier, or later, object or subject.
I think, generally, we fix on the earlier when talking about the change over time of a characteristic. "This stock went up 100%, this stock went down 50%". In both cases it's the earlier measurement that is taken as the unit. That makes this a 45% reduction in time to do the work, and that's actually what they measured.
When talking about comparisons between two things that aren't time dependent it depends on if we talk in multiples or percents I think. A twin bed is half as big as a king bed. A king bed is twice as big as a twin bed. Both are idiomatic. A king bed is 100% bigger than a twin bed. Yes, you could talk like this. A twin bed is 100% smaller than a king bed. Right away you say wait, a twin bed isn't 0 size! Because we don't talk in terms of the smaller thing when talking about decreasing percents, only increasing. A twin bed is 50% smaller than a king bed (iffy). A twin bed is 50% as big as a king bed. There, that's idiomatic again.
I think you're right. I think you could say the package (as in the code) is 45% faster or that the package increases parsing rate by 90%. But mixing the two is confusing.
I think it's an interpretation issue. When you say it's 45% faster I interpret that as "the new package handles work at a rate of 145% when compared to the original, that is, 1.45 times as fast".
I would rephrase your comment as "the package takes 45% less time to process a given data set".
Thanks for this, as an average HN user I didn't click the link and just skimmed the comments thinking how is it possible that they reduced the runtime to 10% of the original. This post clarifies that for me (now on to actually read the blog post).
Great read. I hadn't even considered that people might interpret "90% faster" as "10 times as fast", i.e., it will take 100-90=10% of the original time. It seems like a completely incorrect interpretation to me, but obviously there are people who read it with this understanding. Huh.
It is unfortunate (because how do you interpret 100% faster) but a common interpretation of X% faster implies it takes x% less time than before. One easy way is to have chatgpt give a numeric example for a statement like this, and in all cases I tried it gives an example of sorts if the job took 100 seconds before it takes 10 seconds now (a 10x increase), and I'm assuming it is a representation of common usage based on how much data is used to train it.
After thinking about it more, I think I know why it seems misleading. They are talking about the change in the larger value as a % of the smaller value. That's what is misleading. They are saying it is reduced by 90% (of some other thing). When you say "We reduced THING by N%" it's just assumed that the N% is N% of THING, not OTHER THING.
For kicks, I downloaded the file and compared duckdb with the spatial extension against jq. jq was about 2x as fast for interactive use, but if you had multiple queries to run, paying a small cost to create a duckdb database and then querying it would be vastly faster
I also was nerdsniped into trying this and found that after extracting the features array into a newline delimited json file, DuckDB finishes the example query in 500 ms (M1 Mac), querying the 1.3 GB json file directly with read_json!
Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON.
This is certainly true. Also, by replacing the allocator and changing compiler flags, you're possibly immunizing yourself from attacks that rely on some specific memory layout.
ASLR is not security through obscurity though. It forces attacker to get a pointer leak before doing almost anything (even arbitrary read and arbitrary write primitives are useless without a leak with ASLR). As someone with a bit of experience in exploit dev, it makes a world of a difference and is one of the most influential hardenings, next to maybe stack cookies and W^X.
ASLR obscures the memory layout. That is security by obscurity by definition. People thought this was okay if the entropy was high enough, but then the ASLR⊕Cache attack was published and now its usefulness is questionable.
I'm genuinely curious what was so undesirable about this sibling comment that it was removed:
"ASLR obscures the memory layout. That is security by obscurity by definition. People thought this was okay if the entropy was high enough, but then the ASLR⊕Cache attack was published and now its usefulness is questionable."
Usually when a comment is removed, it's pretty obvious why, but in this case I'm really not seeing it at all. I read up (briefly) on the mentioned attack and can confirm that the claims made in the above comment are at the very least plausible sounding. I checked other comments from that user and don't see any other recent ones that were removed, so it doesn't seem to be a user-specific thing.
I realize this is completely off-topic, but I'd really like to understand why it was removed. Perhaps it was removed by mistake?
I assume people downvoted it because “ASLR obscures the memory layout. That is security by obscurity by definition” is just wrong (correct description here: https://news.ycombinator.com/item?id=43408039). It does say [flagged] too, though, so maybe that’s not the whole story…?
No, that other definition is the incorrect one. Security by obscurity does not require that the attacker is ignorant of the fact you're using it. Say I have an IPv6 network with no firewall, simply relying on the difficulty of scanning the address space. I think that people would agree that I'm using security by obscurity, even if the attacker somehow found out I was doing this. The correct definition is simply "using obscurity as a security defense mechanism", nothing more.
No, I would not agree that you would be using security by obscurity in that example. Not all security that happens to be weak or fragile and involves secret information somewhere is security by obscurity – it’s specifically the security measure that has to be secret. Of course, there’s not a hard line dividing secret information between categories like “key material” and “security measure”, but I would consider ASLR closer to the former side than the latter and it’s certainly not security by obscurity “by definition” (aside: the rampant misuse of that phrase is my pet peeve).
> The correct definition is simply "using obscurity as a security defense mechanism", nothing more.
This is just restating the term in more words without defining the core concept in context (“obscurity”).
I'm inclined to agree and would like to point out that if you take a hardline stance that any reliance on the attacker not knowing something makes it security by obscurity then things like keys become security by obscurity. That's obviously not a useful end result so that can't be the correct definition.
It's useful to ask what the point being conveyed by the phrase is. Typically (at least as I've encountered it) it's that you are relying on secrecy of your internal processes. The implication is usually that your processes are not actually secure - that as soon as an attacker learns how you do things the house of cards will immediately collapse.
What is missing from these two representations is the ability for something to become trivially bypassable once you know the trick to it. AnC is roughly that for ASLR.
I'd argue that AnC is a side channel attack. If I can obtain key material via a side channel that doesn't (at least in the general case) suddenly change the category of the corresponding algorithm.
Also IIUC to perform AnC you need to already have arbitrary code execution. That's a pretty big caveat for an attacker.
You are not wrong, but how big of a caveat it is varies. On a client system, it is an incredibly low bar given client side scripting in web browsers (and end users’ tendency to execute random binaries they find on the internet). On a server system, it is incredibly unlikely.
I think the middle ground is to call the effectiveness of ASLR questionable. It is no longer the gold standard of mitigations that it was 10 years ago.
ASLR is not purely security through obscurity because it is based on a solid security principle: increasing the difficulty of an attack by introducing randomness. It doesn't solely rely on the secrecy of the implementation but rather the unpredictability of memory addresses.
Think of it this way - if I guess the ASLR address once, a restart of the process renders that knowledge irrelevant implicitly. If I get your IPv6 address once, you’re going to have to redo your network topology to rotate your secret IP. That’s the distinction from ASLR.
I don't like that example because the damaged cause by and the difficulty of recovering from a secret leaking is not what determines the classification. There exist keys that if leaked would be very time consuming to recover from. That doesn't make them security by obscurity.
I think the key feature of the IPv6 address example is that you need to expose the address in order to communicate. The entire security model relies on the attacker not having observed legitimate communications. As soon as an attacker witnesses your system operating as intended the entire thing falls apart.
Another way to phrase it is that the security depends on the secrecy of the implementation, as opposed to the secrecy of one or more inputs.
You don’t necessarily need to expose the IPv6 address to untrusted parties though in which case it is indeed quite similar to ASLR in that data leakage of some kind is necessary. I think the main distinguishing factor is that ASLR by design treats the base address as a secret and guards it as such whereas that’s not a mode the IPv6 address can have because by its nature it’s assumed to be something public.
Huh. The IPv6 example is much more confusing that I initially thought. At this point I am entirely unclear as to whether it is actually an example of security through obscurity, regardless of whatever else it might be (a very bad idea to rely on it for one). Rather ironic given that the poster whose claims I was disputing provided it as an example of something that would be universally recognized as such.
I think it’s security through obscurity because in ASLR the randomized base address is a protected secret key material whereas in the ipv6 case it’s unprotected key material (eg every hop between two communicating parties sees the secret). It’s close though which is why IPv6 mapping efforts are much more heuristics based than ipv4 which you can just brute force (along with port #) quickly these days.
I'm finding this semantic rabbit hole surprisingly amusing.
The problem with that line of reasoning is that it implies that data handling practices can determine whether or not a given scheme is security through obscurity. But that doesn't fit the prototypical example where someone uses a super secret and utterly broken home rolled "encryption" algorithm. Nor does it fit the example of someone being careless with the key material for a well established algorithm.
The key defining characteristic of that example is that the security hinges on the secrecy of the blueprints themselves.
I think a case can also be made for a slightly more literal interpretation of the term where security depends on part of the design being different from the mainstream. For example running a niche OS making your systems less statistically likely to be targeted in the first place. In that case the secrecy of the blueprints no longer matters - it's the societal scale analogue of the former example.
I think the IPv6 example hinges on the semantic question of whether a network address is considered part of the blueprint or part of the input. In the ASLR analogue, the corresponding question is whether a function pointer is part of the blueprint or part of the input.
> The problem with that line of reasoning is that it implies that data handling practices can determine whether or not a given scheme is security through obscurity
Necessary but not sufficient condition. For example, if I’m transmitting secrets across the wire in plain text that’s clearly security through obscurity even if you’re relying on an otherwise secure algorithm. Security is a holistic practice and you can’t ignore secrets management separate from the algorithm blueprint (which itself is also a necessary but not sufficient condition).
Consider that in the ASLR analogy dealing in function pointers is dealing in plaintext.
I think the semantics are being confused due to an issue of recursively larger boundaries.
Consider the system as designed versus the full system as used in a particular instance, including all participants. The latter can also be "the system as designed" if you zoom out by a level and examine the usage of the original system somewhere in the wild.
In the latter case, poor secrets management being codified in the design could in some cases be security through obscurity. For example, transmitting in plaintext somewhere the attacker can observe. At that point it's part of the blueprint and the definition I referred to holds. But that blueprint is for the larger system, not the smaller one, and has its own threat model. In the example, it's important that the attacker is expected to be capable of observing the transmission channel.
In the former case, secrets management (ie managing user input) is beyond the scope of the system design.
If you're building the small system and you intend to keep the encryption algorithm secret, we can safely say that in all possible cases you will be engaging in security through obscurity. The threat model is that the attacker has gained access to the ciphertext; obscuring the algorithm only inflicts additional cost on them the first time they attack a message secured by this particular system.
It's not obvious to me that the same can be said of the IPv6 address example. Flippantly, we can say that the physical security of the network is beyond the scope of our address randomization scheme. Less flippantly, we can observe that there are many realistic threat models where the attacker is not expected to be able to snoop any of the network hops. Then as long as addresses aren't permanent it's not a one time up front cost to learn a fixed procedure.
Function pointer addresses are not meant to be shared - they hold 0 semantic meaning or utility outside a process boundary (modulo kernel). IPv6 addresses are meant to be shared and have semantic meaning and utility at a very porous layer. Pretending like there’s no distinction between those two cases is why it seems like ASLR is security through obscurity when in fact it isn’t. Of course, if your program is trivially leaking addresses outside your program boundary, then ASLR degrades to a form of security through obscurity.
I'm not pretending that there's no distinction. I'm explicitly questioning the extent to which it exists as well as the relevance of drawing such a distinction in the stated context.
> Function pointer addresses are not meant to be shared
Actually I'm pretty sure that's their entire purpose.
> they hold 0 semantic meaning or utility outside a process boundary (modulo kernel).
Sure, but ASLR is meant to defend against an attacker acting within the process boundary so I don't see the relevance.
How the system built by the programmer functions in the face of an adversary is what's relevant (at least it seems to me). Why should the intent of the manufacturer necessarily have a bearing on how I use the tool? I cannot accept that as a determining factor of whether something qualifies as security by obscurity.
If the expectation is that an attacker is unable to snoop any of the relevant network hops then why does it matter that the address is embedded in plaintext in the packets? I don't think it's enough to say "it was meant to be public". The traffic on (for example) my wired LAN is certainly not public. If I'm not designing a system to defend against adversaries on my LAN then why should plaintext on my LAN be relevant to the analysis of the thing I produced?
Conversely, if I'm designing a system to defend against an adversary that has physical access to the memory bus on my motherboard then it matters not at all whether the manufacturer of the board intended for someone to attach probes to the traces.
I think that's why the threat model matters. I consider my SSH keys secure as long as they don't leave the local machine in plaintext form. However if the scenario changes to become "the adversary has arbitrary read access to your RAM" then that's obviously not going to work anymore.
> The correct definition is simply "using obscurity as a security defense mechanism", nothing more.
Also stated as "security happens in layers", and often obscurity is a very good layer for keeping most of the script kiddies away and keeping the logs clean.
My personal favorite example is using a non-default SSH port. Even if you keep it under 1024, so it's still on a root-controlled port, you'll cut down the attacks by an order of magnitude or two. It's not going to keep the NSA or MSS out, but it's still effective in pushing away the common script kiddies. You could even get creative and play with port knocking - that keeps under-1024 ports logs clean.
Some people use the "flag" button as a "disagree" button or even a "fuck this guy" button. Unfortunately, constructive but unpopular comments get flagged to death on HN all the time.
I had thought that flagging was basically a request for a mod to have a look at something. But based on this case I now suspect that it's possible for a comment to be removed without a mod ever looking at it if enough people flag it.
My point was more that, at least in this case, it looks like a post was hidden without any moderator intervention.
If this is indeed what happened, it seems like a bad thing that it's even possible. Since many, perhaps most people probably don't have showdead enabled, it means that the 'flag' option is effectively a mega-downvote.
I believe most people see security through obscurity has an attempt to hide an insecurity.
ASLR/KASLR intends to make attackers lives harder by having non consistent offsets of known data structures. Its not obscuring a security flaw, instead its raises an attacks 'single run' effectivness.
The ASLR attack that i believe is being referenced is specific to abuse within the browser, and running with a single process. This single attack vector does not mean that KASLR globally is not effective.
Your quote has some choice words, but its contextually poor.
That attack does not require a web browser. The web browser being able to do it showed it was higher severity than you would think than if the proof of concept had been in C, since web browsers run untrusted code all of the time.
The 'attack' there does require you to be able to run code and test within a single process with a single randomized address space, which is the exact vector that the web browser provides.
Most times in C, each fork() (rather than thread) has a differential address space, so it's actually less severe than you think.
The kernel address space is the same regardless of how many fork() calls have been done. I would assume the exploitation path for a worst case scenario would be involve chaining exploits to do: AnC on userspace, JavaScript engine injection to native code, sandbox escape, AnC on kernel space, kernel native code injection. That would give complete control over a user’s machine just by having the user visit a web page.
I am not sure why anyone would attempt what you described, for the exact reason you stated. It certainly is not what I had in mind.
Its been a few days and a thousand kilometers since I've read the paper, I thought it referenced userspace. How is it able to infer kernel addresses that are not mapped in that process ?
Except I do know what security by obscurity is and you are out of date on the subject. When you have attacks that make ASLR useless, then it is security by obscurity. Your thinking would have been correct 10 years ago. It is no longer correct today. The middle ground is to say that the benefits of ASLR are questionable, like I said in the comment you downvoted.
ASLR is by definition security through obscurity. That doesn't make it useless, as there's nothing wrong with using obscurity as one layer of defenses. But that doesn't change what it fundamentally is: obscuring information so that an attacker has to work harder.
Is having a secret password security by obscurity? What about a private key?
Security by obscurity is about the bad practice of thinking that obscuring your mechanisms and implementations of security increases your security. It's about people that think that by using their nephew's own super secret unpublished encryption they will be more secure than by using hardened standard encryption libraries.
Security through obscurity is when you run your sshd server on port 1337 instead of 22 without actually securing the server settings down, because you don’t think the hackers know how to portscan that high. Everyone runs on 22, but you obscurely run it elsewhere. “Nobody will think to look.”
ASLR is nothing like that. It’s not that nobody thinks to look, it’s that they have no stable gadgets to jump to. The only way to get around that is to leak the mapping or work with the handful of gadgets that are stable. It’s analogous to shuffling a deck of cards before and after every hand to protect against card counters. Entire cities in barren deserts have been built on the real mathematical win that comes from that. It’s real.
With attacks such as AnC, your logic fails. They can figure out the locations and get plenty of stable gadgets.
Any shuffling of a deck of cards by Alice is pointless if Bob can inspect the deck after she shuffles them. It makes ASLR not very different from changing your sshd port. In both cases, this describes the security:
okay, sure, ASLR can be defeated by hardware leaks. The first rowhammer papers were over ten years ago, it's very old news. It's totally irrelevant to this thread. The fact that there exist designs that have hardware flaws which make them incapable of hosting a secure PRNG does not have any relevance to a discussion about the merits or lack thereof of a PRNG-based security measures. The systems you're referring to don't have secure PRNGs.
Words have meaning, god damn it! ASLR is not security through obscurity.
Edit: I was operating under the assumption that “AnC” was some new hotness, but no, this is the same stuff that’s always been around, timing attacks on the caches. And there’s still the same solution as there was back then: you wipe the caches out so your adversaries have no opportunity to measure the latencies. It’s what they always should have done on consumer devices running untrusted code.
ASLR is technically a form of security by obscurity. The obscurity here being the memory layout. The reason nobody treated it that way was the high entropy that ASLR had on 64-bit, but the ASLR⊕Cache attack has undermined that significantly. You really do not want ASLR to be what determines whether an attacker takes control of your machine if you care about having a secure system.
The defining characteristic of security through obscurity is that the effectiveness of the security measure depends on the attacker not knowing about the measure at all. That description doesn’t apply to ASLR.
It produces a randomization either at compile time or run time, and the randomization is the security measure, which is obscured based on the idea that nobody can figure it out with ease. It is a poor security measure given the AnC attack that I mentioned. ASLR randomization is effectively this when such attacks are applicable:
You are confusing randomization, a legitimate security mechanism, with security by obscurity. ASLR is not security by obscurity.
Please spend the time on understanding the terminology rather than regurgitating buzz words.
I understand the terminology. I even took a graduate course on the subject. I stand by what I wrote. Better yet, this describes ASLR when the AnC attack applies:
UBSAN is usually a debug build only thing. You can run it in production for some added safety, but it comes at a performance cost and theoretically, if you test all execution paths on a debug build and fix all complaints, there should be no benefit to running it in production.
I think it's time for the C/C++ communities to consider a mindset shift and pivot to having almost all protectors, canaries, sanitizers, assertions (e.g. via _GLIBCXX_ASSERTIONS) on by default and recommended for use in release builds in production. The opposite (i.e, the current state of affairs) should be discouraged and begrudginly accepted in select few cases.
https://www.youtube.com/watch?v=gG4BJ23BFBE is a presentation that best represents my view on the kind of mindset that's long overdue to become the new norm in our industry.
I do not think things like the time command need to be compiled with such things. It is pointless, but your suggestion here is to do it anyway. Why bother?
Assertions in release builds are a bad idea since they can be fairly expensive. It is a better idea to have a different variety of assertion like the verify statements that OpenZFS uses, which are assertions that run even in release builds. They are used in situations where it is extremely important for an assertion to be done at runtime, without the performance overhead of the less important assertions that are in performance critical paths.
Why would I want potentially undefined behaviour in 'time'? I expect it to crash anytime it's about to enter UB. Sure, you may want to minimize such statements between the start/stop of the timer, but I expect any processing of stdout/stderr of the child process to be UB-proofed as much as possible.
I think it's a philosophical difference of opinions and it's one of the things that drive Rust, Go, C# etc. ahead - not merely language ergonomics (I hope Zig ends up as the language that replaces C). The society at large is mostly happy to take a 1-3% perf hit to get rid of buffer overflows and other UB-inducing errors.
But I agree with you on not having "expensive" asserts in releases.
By hardwiring the allocator you may end up with binaries that load two different allocators. It is too fun to debug a program that is using jemalloc free to release memory allocated by glibc. Unless you know what you are doing, it is better to leave it as is.
Indeed, there are many methods to have a custom build and still get security updates, including at least one method that is native to Ubuntu and doesn’t need any external tooling. However my warning refers to the method presented in the article, where this isn’t the case.
This is generally true but specifically false. The builds described in the gist are still linking onigurama dynamically. It is in another package, libonig5, that would be updated normally.
The gist uses the orig tarball only, so skips the distro patch that selects the distro packaged libonig in favor of the "vendored" one. At least that's how it appears to me. I only skimmed the situation.
Or do you see something deeper that ensures that the distro libonig is actually the one that gets used?
But isn't there still the kernel of an idea here for a package management system that intelligently decides to build based on platform? Seems like a lot of performance to leave on the table.
Rebuilding from scratch also takes longer than installing a prebuilt package. So while it might be worth it for a heavily used application, in general I doubt it.
Also I think in earlier days the argument to build was so you can optimize the application for the specific capabilities of your system like the supported SIMD instruction set or similar. I think nowadays that is much less of a factor. Instead it would probably be better to do things like that on a package or distribution level (i.e. have one binary distribution package prebuilt by the distribution for different CPU capabilities).
I'm curious how applicable these are, in general? Feels like pointing out that using interior doors in your house misses out on the security afforded from a vault door. Not wrong, but there is also a reason every door in a bank is not a vault door.
That is, I don't want to devalue the CVE system; but it is also undeniable that there are major differences in impact between findings?
Sure, but jq is very much a "front door" in your analogy. You'd have to look at each individual CVE to assess the risk for your specific case, but for jq, claimed security vulnerabilities are worth paying attention to.
The normal way is to use dpkg to rebuild and patch, and use dch to increase the patch version with a .1 or something similar, so that the OS version always takes precedence, and then rebuild.
The Zircon kernel does not support signals, so basic C is not going to work well.
"It is heavily inspired by Unix kernels, but differs greatly. For example, it does not support Unix-like signals, but incorporates event-driven programming and the observer pattern."
As long as there's Bash and Python support, Gentoo Prefix could as well. That's kind of the only things that are hard needed for Gentoo as Portage is written in Python and Ebuilds are Bash scripts. Bigger issue would be that the Fuchsia kernel likely isn't POSIX complete.
I used Gentoo for a while, but the temptation to endlessly fiddle with everything always let me to eventually break the system. (It's not Gentoo's fault, it's mine.)
Afterwards I moved to ArchLinux, and that has been mostly fine for me.
If you are using a fairly standard processor, then Gentoo shouldn't give you that much of an advantage?
Gentoo lets you do all of the tweaks mentioned here within the system package manager, so you still get security updates for your tweaked build. You can also install Gentoo on top of another system via Gentoo Prefix for use as a userland packages manager:
These are the Arch packages built for x86-64-v2, x86-64-v3 and x86-64-v4, which are basically names for different sets of x86-64 extensions. Selecting the highest level supported by your processor should get you most of the way to -march=native, without the hassle of compiling it yourself.
I've broken Arch but never broken Gentoo. I think this more due to the fact I ran Arch first and you then Gentoo first, rather than any real difference between them.
Gentoo is more stable than Arch by default, though. It's not actually a bleeding edge distro, but you can choose to run it that way if you wish. Gentoo is about choice.
> I think this more due to the fact I ran Arch first and you then Gentoo first, rather than any real difference between them.
I can believe that.
> Gentoo is more stable than Arch by default, though. It's not actually a bleeding edge distro, but you can choose to run it that way if you wish. Gentoo is about choice.
I actually had way more trouble with stuff breaking with Ubuntu. That's because every six months, when I did the distro upgrade, lots of stuff broke at once and it was hard to do root cause analysis.
With a rolling distribution, it's usually only one thing breaking at a time.
> Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase.
That's true but worth noting that "optimize" here doesn't necessarily refer to performance.
I've been using Gentoo for 20 years and performance was never the reason. Gentoo is great if you know how you want things to work. Gentoo helps you get there.
Long time ago when I was using it I preferred Gentoo because of ergonomics and better exposition to supply chain.
Slackware was very manual and some bits were drowned in its low level and long command chains. Gentoo felt easy but highlighted dependencies with a hard cost associated with compilation times.
Being a newb back then I enjoyed user friendliness with access to the machinery beneath.
Satisfaction of a 1s boot time speedu, a result of 48h+ compilation, was unparalleled, too ;)
Reducing the dependency tree gets a bit more complicated once you consider that now you have to satisfy not only runtime dependencies for all packages but also build-time dependencies. There may be ways of cleaning that up after a build, but next time you want to emerge a new package you'll just end up having to re-build the build-time dependencies, so in practice you'll just end up leaving them there. There is an ability to emerge packages to a separate part of the filesystem tree (ROOT="/my/chroot" emerge bla), so that you have one build-time system act as a kind of incubator for a runtime system that gets to be minimal. But you'll end up encountering problems that most other Gentoo users wouldn't encounter, having to do with the separation between build-time dependencies and runtime dependencies not being correctly made in the recipes. Personally, I had been relying on this feature for roughly the last 10 years, but there has been steady deterioration there over the years and I eventually gave up late last year.
This is a good point. I've been using Gentoo since early 2004 (the dreaded Pentium IV era, Lol). Lately, I run into this with dev-lang/tcl only being need to build dev-db/sqlite. I actually think it's pretty weird that software intended to be as widely used as sqlite with as much of a free base of supporting devs doesn't just do the extra effort to use a Makefile.
Never seen the HN version of the 'install gentoo' meme before, more sophisticated definitely.
> The goal of Gentoo is to have an operating system that builds all programs from source, instead of having pre-built binary packages. While this does allow for advanced speed and customizability, it means that even the most basic components such as the kernel must be compiled from source. It is known through out the Linux community as being a very complex operating system because of its daunting install process. The default Gentoo install boots straight to a command prompt, from which the user must manually partition the disk, download a package known as a "Stage 3 tarball", extract it, and build the system up by manually installing packages. New or inexperienced users will often not know what to do when they boot in to the installer to find there is no graphical display. Members of /g/ will often exaggerate the values of Gentoo, trying to trick new users in to attempting to install it.
Where does that blurb come from, chatgpt? I don't think it's true anymore, last time I checked I think Gentoo had a "normal" liveCD installation for the base system, which you could then recompile on your own if wanted.
GRP (Gentoo packages) existed at least 20 years ago, from my memory, as that's the last time I really used it in anger. I remeber packages being available and not having to rice everything, for sure.
I had had Gentoo continuously in use since 2003, and only very recently moved off of it (late 2024) when I tried Void Linux. On Void, buildability from source by end users is not a declared goal nor architectural feature, but you have a pretty decent chance of being able to make it work. You can expect one or two hiccups, but if you have decent all-round Linux experience, chances are you'll be able to jump into the build recipes, fix them, make everything work for what you need it to do, and contribute the fixes back upstream. This is what you get from a relentless focus on minimalism and avoiding overengineering of any kind. It's what I had been missing in Gentoo all those years. With Gentoo, I always ended up having to fiddle with use flags and package masks in ways that wouldn't be useful to other users. The build system is so complex that it had been just too difficult for me, over all these years, to properly learn it and learn to fix problems at the root cause level and contribute them upstream. Void should also be an ideal basis for when you don't want to build the entire system from source, but you just want to mix & match distro-provided binaries with packages you've built from source (possibly on the basis of a modified build recipe to better match your needs or your hardware).
Kidding... honestly that was a pretty fun distribution to play around with ~20 years ago. The documentation was really good and it was a great way to learn how a lot of the pieces of a Linux distribution fit together.
I was never convinced that the performance difference was really noticeable, though.
Gentoo was the primary source of heating for my living quarters back in the early 2000s. My tower was highly constrained on memory, and I was on a relentless quest to pare out any modules or dependencies I wasn't actually using. Performance gains were primarily from being able to stay out of slow HDD swap space memory. I doubt there were any gains once amortizing the compilation times, but I ran my compile batches at night, and they kept me nice and warm.
-march=native + mimalloc (or jemalloc) should be sufficient without causing significant undefined behavior like -O3 or most extra optimization related compiler arguments.
Nope, I'm not sure about it. I remember when I was using Gentoo about 10 years ago, this was the common reason given for using -O2 instead of -O3 in your build flags, and I'm just speaking from that memory.
> Can’t one recompile the same exact Ubuntu packages you already have on your system with optimal flags for your specific hardware?
Well, in principle yes. But you'd also need to figure out what you mean by 'optimal' flags? The flags might differ between different packages, and not all packages will be compatible with all the flags.
Even worse, in the example in the article they got most of their speedup out of moving to a different allocator, and that's not necessarily compatible with all packages (nor an improvement for all packages).
However, if you still want to do all these things, then Gentoo Linux is the distribution for you. It supports this tinkering directly and you'll have a much happier time than trying to bend Ubuntu to your will.
Waste minutes of your time just so anonymous people on the internet can read your comment :).
Not everything in life is an optimization problem. Fun little projects like this is what makes us human (and, arguably, are both educational and entertaining).
I see what you're saying but I am also talking about how computers originally saved humans time and that the OP seems to now be slaving away to save the CPU time. Doesn't that seem a little backwards?
If we're talking about micro-seconds of difference, the trade-off doesn't seem worth it. Even on a mass scale where this is somehow adopted, nobody is going to notice the difference. Maybe if this were in something like eCommerce and web browing where the lag translates to profit lost? Or perhaps game engines?
IDK, I just consider humans time more precious than a slow package (that already runs blazingly fast on a CPU that it barely matters.)
Longer waiting times can decrease productivity a lot more than just the runtime increase. If a program takes long enough that the user switches context mentally, or goes to get another coffee or looks at a few posts on social media that now means far more wasted time. Humans are not robots and won’t just keep staring at the screen.
Also a lot of simple optimizations could save seconds or minutes in the lives of millions of people, and across multiple devices/programs that adds up. Microsoft once sped up the booting process of their consoles by 5 seconds, by simply making the boot animation shorter by 5 seconds. Those consoles were sold to millions of people and it took MS 8 years or so to make the fix. That’s many lifetimes wasted once you add it all up.
Today it's 2.2GB of JSON files, tomorrow it's tens of petabytes of archived data that you need to look through for work. A speedup of 2x is nothing to scoff at.
As a broader comment though, this is how we learn and discover things. Just because the specific outcome here is "trivial" doesn't mean the approach or the lessons learnt aren't valuable.
I found it an interesting read despite not have any stake in the outcome.
I haven't kept up with Gentoo. How long does it take these days to bootstrap a functional desktop from source?
I remember when it was my main driver back in early '00s, running on a horribly underpowered PII at 266MHz. It would often be compiling 24/7 to keep up with the updates.
I'd be curious how the performance compares to this Rust jq clone:
cargo install --locked jaq
(you might also be able to add RUSTFLAGS="-C target-cpu=native" to enable optimizations for your specific CPU family)
"cargo install" is an underrated feature of Rust for exactly the kind of use case described in the article. Because it builds the tools from source, you can opt into platform-specific features/instructions that often aren't included in binaries built for compatibility with older CPUs. And no need to clone the repo or figure out how to build it; you get that for free.
jaq[1] and yq[2] are my go-to options anytime I'm using jq and need a quick and easy performance boost.
As a bonus that people might not be aware of, in the cases where you do want to use the repo directly (either because there isn't a published package or maybe you want the latest commit that hasn't been released), `cargo install` also has a `--git` flag that lets you specify a URL to a repo. I've used this a number of times in the past, especially as an easy way for me to quickly install personal stuff that I throw together and push to a repo without needing to put together any sort of release process or manually copy around binaries to personal machines and keep track of the exact commits I've used to build them.
374 comments
[ 3.4 ms ] story [ 290 ms ] threadWhy wouldn't that be identical?
I recall trying with march=native and seeing some improvement, but not enough to care at a system level.
Upstream is stripped actually. My final build here is 43x larger than the distro binary, partly due to having plentiful debug info.
Love to see why they wouldn't have more optimizations.
I'm assuming this isn't the case for more packages? Feels too good.
What impact does that have?
[0] https://www.youtube.com/@KazeN64
And understood a little on -O3 possibly increasing code size. I had thought that was more of a concern for tight environments than for most systems? Of course, I'd have assumed that -march=native would be more impactful, but the post indicates otherwise.
I said in a top level, but it seems the allocator makes the biggest impact for this application? Would be interesting to see which applications should use different allocators. Would be amazing to see a system where the more likely optimal allocator was default for different applications, based on their typical allocation patterns.
It used to be the case that the presence of debug symbols would affect GCC code generation. Nowadays that should be fixed. I think it still affects the speed of compilation so if you're building the whole system from source you might want to avoid it.
Increasing code size too much can result in hot functions not fitting in the icache, and that ultimately can make your program slower.
Debug symbols have 0 runtime penalty, just storage. They're just another section of the binary, referenced by debuggers, and which the loader skips.
In any case, all distros break out the symbols into separate files so that they can have their (storage) cake and (debug) eat it too.
Another thing the article adds is LTO, which in my experience also makes a huge difference: it makes your software a lot faster in certain cases, but also makes build times a lot worse. Spending a bit more time in the build process should be an easy call, but might be harder at the scale of a distro.
Not sure where I got that idea, though. :(. Will have to look into that later.
By rebuilding the binary with different compiler options, but not changing malloc, they got an 20% speedup.
If we naively multiply these speedups, we get 1.78: 78% faster.
How it goes to 1.9 is that when you speed up only the program, but leave malloc the same, malloc matters to its performance a lot more.
When the faster malloc is applied to a program that is compiled better, it will make a better contribution than the 44% seen when the allocator was preloaded into the slower program.
To do the math right, we would have to look into how much time was saved with just the one change, and how much time was saved with the other. If we add those times, and subtract them from the original, slow time, we should get a time that is close to the 1.9 speedup.
Original time: 4.631
Better compiler options alone: 3.853 (-0.778)
Better allocator alone (preload): 3.209 (-1.422)
Add time saved from both: 2.200
Projected time: 4.631 - 2.200 = 2.431
Projected speedup from both: 4.631/2.431 = 1.905
Bang on!
When an AST interpreted languages gets a VM, gets native code, each step reveals GC to be slow.
You might go from 1% time spent in the GC to 15% to 60% (numbers of out thin air).
To travel 10 miles, at 60 MPH, takes 10 minutes. Make it 100% faster, at 120 MPH, and that time becomes 5 minutes. Travel just as far in 50% of the time. Or travel just as far 100% faster. The 90% speedup matches the reduction of the time it takes to nearly half (a 90% (projected) speedup, or about a 45% time reduction, as mathed out by kazinator `Projected speedup from both: 4.631/2.431 = 1.905`). Your claim that its closer to 50% is correct from a total time taken perspective, just coming at it from the other direction.
From a end developer perspective: I have no particular familiarity with mimalloc, but I know jemalloc has pretty extensive debugging functionality (not API compatible with glibc malloc of course).
jaq runs 5x faster on my machine in some cases
Enables fun & safe math optimizations!
we're using it in feldera and it's been giving us (slightly) better performance than jemalloc which is what we've used before mimalloc
I realize, of course, that this could just be overall optimal. Feels more likely that the allocation patterns of the application using it will impact which is the better allocator?
You could write your own custom package definitions, extending the default to change up compile flags and allocators, but then you need to do this for every single package (and maintain them all). I'm not sure Guix gives you much here, though maybe that's fine for one or two packages.
The most pain-free option I can think of is the --tune flag (which is similar to applying -march=native), but packages have to be defined as tunable for it to work (and not many are).
Is there another option?
You can also write a 10 line guile script to automatically do it for all dependencies (I sometimes do--for example for emacs). That would cause a massive rebuild, though.
>The most pain-free option I can think of is the --tune flag (which is similar to applying -march=native), but
> packages have to be defined as tunable for it to work (and not many are).
We did it that way on purpose--from prior experience, otherwise, you would get a combinatorial explosion of different package combinations.
If it does help for some package X, please email us a 2 line patch adding (tunable? . #t) to that one package.
If you do use --tune, it will tune everything that is tuneable in the dependency graph. But at least all dependents (not dependencies) will be just grafted--not be rebuilt.
Even Design by Contract in Eiffel turns off assertions in production.
Rule #1 of programming: If it can go wrong, it will.
The undefined behaviour is already in the C standard.
But you are right that enabling -O3 is generally going to make your compiler more aggressive about exploiting the undefined behaviour in the pursuit of speed.
In general, if you want to avoid being the guy to run into new and exciting bugs, you want to run with options that are as close as possible to what everyone else is using.
In all seriousness though, are you sure some of this isn’t those blocks being loaded into some kind of file system cache the second and third times?
How about if you rebooted and then ran the mimalloc version?
See this: https://en.wikipedia.org/wiki/Tacit_programming#jq
This is much more intuitive now.
It's actually a little bit interesting, if you are interested in how we use language. You could argue that now you now get 90% more work done in the same amount of time, and that would align with other 'speed' units that we commonly use (miles per hour, words per minute, bits per second). However, the convention in computer performance is to measure time for a fixed amount of work. I would guess that this is because generally we have a fixed amount of work and what might vary is how long we wait for it (and that is absolutely true in the case of this blog post) so we put time in the numerator.
It's a very interesting post and very well done, but it's not 90% faster.
Plus, if you were using units of time, you wouldn’t use the word “faster.” “Takes 45% less time” and “45% faster” are very different assertions, but they both have meaning, both in programming and outside it.
I think, generally, we fix on the earlier when talking about the change over time of a characteristic. "This stock went up 100%, this stock went down 50%". In both cases it's the earlier measurement that is taken as the unit. That makes this a 45% reduction in time to do the work, and that's actually what they measured.
When talking about comparisons between two things that aren't time dependent it depends on if we talk in multiples or percents I think. A twin bed is half as big as a king bed. A king bed is twice as big as a twin bed. Both are idiomatic. A king bed is 100% bigger than a twin bed. Yes, you could talk like this. A twin bed is 100% smaller than a king bed. Right away you say wait, a twin bed isn't 0 size! Because we don't talk in terms of the smaller thing when talking about decreasing percents, only increasing. A twin bed is 50% smaller than a king bed (iffy). A twin bed is 50% as big as a king bed. There, that's idiomatic again.
I would rephrase your comment as "the package takes 45% less time to process a given data set".
https://simdjson.org/
> * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corruption and stack buffer overflow.
(that one was for CVE-2017-9224, CVE-2017-9226, CVE-2017-9227, CVE-2017-9228 and CVE-2017-9229)
I don't need to outrun the bear. I just need to outrun you.
"ASLR obscures the memory layout. That is security by obscurity by definition. People thought this was okay if the entropy was high enough, but then the ASLR⊕Cache attack was published and now its usefulness is questionable."
Usually when a comment is removed, it's pretty obvious why, but in this case I'm really not seeing it at all. I read up (briefly) on the mentioned attack and can confirm that the claims made in the above comment are at the very least plausible sounding. I checked other comments from that user and don't see any other recent ones that were removed, so it doesn't seem to be a user-specific thing.
I realize this is completely off-topic, but I'd really like to understand why it was removed. Perhaps it was removed by mistake?
> The correct definition is simply "using obscurity as a security defense mechanism", nothing more.
This is just restating the term in more words without defining the core concept in context (“obscurity”).
It's useful to ask what the point being conveyed by the phrase is. Typically (at least as I've encountered it) it's that you are relying on secrecy of your internal processes. The implication is usually that your processes are not actually secure - that as soon as an attacker learns how you do things the house of cards will immediately collapse.
Also IIUC to perform AnC you need to already have arbitrary code execution. That's a pretty big caveat for an attacker.
I think the middle ground is to call the effectiveness of ASLR questionable. It is no longer the gold standard of mitigations that it was 10 years ago.
Think of it this way - if I guess the ASLR address once, a restart of the process renders that knowledge irrelevant implicitly. If I get your IPv6 address once, you’re going to have to redo your network topology to rotate your secret IP. That’s the distinction from ASLR.
I think the key feature of the IPv6 address example is that you need to expose the address in order to communicate. The entire security model relies on the attacker not having observed legitimate communications. As soon as an attacker witnesses your system operating as intended the entire thing falls apart.
Another way to phrase it is that the security depends on the secrecy of the implementation, as opposed to the secrecy of one or more inputs.
The problem with that line of reasoning is that it implies that data handling practices can determine whether or not a given scheme is security through obscurity. But that doesn't fit the prototypical example where someone uses a super secret and utterly broken home rolled "encryption" algorithm. Nor does it fit the example of someone being careless with the key material for a well established algorithm.
The key defining characteristic of that example is that the security hinges on the secrecy of the blueprints themselves.
I think a case can also be made for a slightly more literal interpretation of the term where security depends on part of the design being different from the mainstream. For example running a niche OS making your systems less statistically likely to be targeted in the first place. In that case the secrecy of the blueprints no longer matters - it's the societal scale analogue of the former example.
I think the IPv6 example hinges on the semantic question of whether a network address is considered part of the blueprint or part of the input. In the ASLR analogue, the corresponding question is whether a function pointer is part of the blueprint or part of the input.
Necessary but not sufficient condition. For example, if I’m transmitting secrets across the wire in plain text that’s clearly security through obscurity even if you’re relying on an otherwise secure algorithm. Security is a holistic practice and you can’t ignore secrets management separate from the algorithm blueprint (which itself is also a necessary but not sufficient condition).
I think the semantics are being confused due to an issue of recursively larger boundaries.
Consider the system as designed versus the full system as used in a particular instance, including all participants. The latter can also be "the system as designed" if you zoom out by a level and examine the usage of the original system somewhere in the wild.
In the latter case, poor secrets management being codified in the design could in some cases be security through obscurity. For example, transmitting in plaintext somewhere the attacker can observe. At that point it's part of the blueprint and the definition I referred to holds. But that blueprint is for the larger system, not the smaller one, and has its own threat model. In the example, it's important that the attacker is expected to be capable of observing the transmission channel.
In the former case, secrets management (ie managing user input) is beyond the scope of the system design.
If you're building the small system and you intend to keep the encryption algorithm secret, we can safely say that in all possible cases you will be engaging in security through obscurity. The threat model is that the attacker has gained access to the ciphertext; obscuring the algorithm only inflicts additional cost on them the first time they attack a message secured by this particular system.
It's not obvious to me that the same can be said of the IPv6 address example. Flippantly, we can say that the physical security of the network is beyond the scope of our address randomization scheme. Less flippantly, we can observe that there are many realistic threat models where the attacker is not expected to be able to snoop any of the network hops. Then as long as addresses aren't permanent it's not a one time up front cost to learn a fixed procedure.
> Function pointer addresses are not meant to be shared
Actually I'm pretty sure that's their entire purpose.
> they hold 0 semantic meaning or utility outside a process boundary (modulo kernel).
Sure, but ASLR is meant to defend against an attacker acting within the process boundary so I don't see the relevance.
How the system built by the programmer functions in the face of an adversary is what's relevant (at least it seems to me). Why should the intent of the manufacturer necessarily have a bearing on how I use the tool? I cannot accept that as a determining factor of whether something qualifies as security by obscurity.
If the expectation is that an attacker is unable to snoop any of the relevant network hops then why does it matter that the address is embedded in plaintext in the packets? I don't think it's enough to say "it was meant to be public". The traffic on (for example) my wired LAN is certainly not public. If I'm not designing a system to defend against adversaries on my LAN then why should plaintext on my LAN be relevant to the analysis of the thing I produced?
Conversely, if I'm designing a system to defend against an adversary that has physical access to the memory bus on my motherboard then it matters not at all whether the manufacturer of the board intended for someone to attach probes to the traces.
My personal favorite example is using a non-default SSH port. Even if you keep it under 1024, so it's still on a root-controlled port, you'll cut down the attacks by an order of magnitude or two. It's not going to keep the NSA or MSS out, but it's still effective in pushing away the common script kiddies. You could even get creative and play with port knocking - that keeps under-1024 ports logs clean.
If this is indeed what happened, it seems like a bad thing that it's even possible. Since many, perhaps most people probably don't have showdead enabled, it means that the 'flag' option is effectively a mega-downvote.
ASLR/KASLR intends to make attackers lives harder by having non consistent offsets of known data structures. Its not obscuring a security flaw, instead its raises an attacks 'single run' effectivness.
The ASLR attack that i believe is being referenced is specific to abuse within the browser, and running with a single process. This single attack vector does not mean that KASLR globally is not effective.
Your quote has some choice words, but its contextually poor.
Most times in C, each fork() (rather than thread) has a differential address space, so it's actually less severe than you think.
I am not sure why anyone would attempt what you described, for the exact reason you stated. It certainly is not what I had in mind.
Security by obscurity is about the bad practice of thinking that obscuring your mechanisms and implementations of security increases your security. It's about people that think that by using their nephew's own super secret unpublished encryption they will be more secure than by using hardened standard encryption libraries.
Security through obscurity is when you run your sshd server on port 1337 instead of 22 without actually securing the server settings down, because you don’t think the hackers know how to portscan that high. Everyone runs on 22, but you obscurely run it elsewhere. “Nobody will think to look.”
ASLR is nothing like that. It’s not that nobody thinks to look, it’s that they have no stable gadgets to jump to. The only way to get around that is to leak the mapping or work with the handful of gadgets that are stable. It’s analogous to shuffling a deck of cards before and after every hand to protect against card counters. Entire cities in barren deserts have been built on the real mathematical win that comes from that. It’s real.
Any shuffling of a deck of cards by Alice is pointless if Bob can inspect the deck after she shuffles them. It makes ASLR not very different from changing your sshd port. In both cases, this describes the security:
https://web.archive.org/web/20240123122515if_/https://www.sy...
Words have meaning, god damn it! ASLR is not security through obscurity.
Edit: I was operating under the assumption that “AnC” was some new hotness, but no, this is the same stuff that’s always been around, timing attacks on the caches. And there’s still the same solution as there was back then: you wipe the caches out so your adversaries have no opportunity to measure the latencies. It’s what they always should have done on consumer devices running untrusted code.
I used to think this, but hearing about the AnC attack changed my mind. I have never heard of anyone claiming to mitigate it.
https://web.archive.org/web/20240123122515if_/https://www.sy...
https://web.archive.org/web/20240123122515if_/https://www.sy...
https://www.youtube.com/watch?v=gG4BJ23BFBE is a presentation that best represents my view on the kind of mindset that's long overdue to become the new norm in our industry.
Assertions in release builds are a bad idea since they can be fairly expensive. It is a better idea to have a different variety of assertion like the verify statements that OpenZFS uses, which are assertions that run even in release builds. They are used in situations where it is extremely important for an assertion to be done at runtime, without the performance overhead of the less important assertions that are in performance critical paths.
I think it's a philosophical difference of opinions and it's one of the things that drive Rust, Go, C# etc. ahead - not merely language ergonomics (I hope Zig ends up as the language that replaces C). The society at large is mostly happy to take a 1-3% perf hit to get rid of buffer overflows and other UB-inducing errors.
But I agree with you on not having "expensive" asserts in releases.
Can you explain a little more? Search has failed me on this one.
Or do you see something deeper that ensures that the distro libonig is actually the one that gets used?
Also I think in earlier days the argument to build was so you can optimize the application for the specific capabilities of your system like the supported SIMD instruction set or similar. I think nowadays that is much less of a factor. Instead it would probably be better to do things like that on a package or distribution level (i.e. have one binary distribution package prebuilt by the distribution for different CPU capabilities).
That is, I don't want to devalue the CVE system; but it is also undeniable that there are major differences in impact between findings?
You could, though. It's 99.9% stuff like this!
After initial setup, it’s pretty simple and easy to use, I remember making a ton of friends at matrix’s Gentoo Linux channel, was fun times.
https://www.gentoo.org/
Fun fact, initial ChromeOS was basically just custom Gentoo Linux install, I’m not sure if they still use Gentoo Linux internally.
https://en.m.wikipedia.org/wiki/Microsoft_POSIX_subsystem
The Zircon kernel does not support signals, so basic C is not going to work well.
"It is heavily inspired by Unix kernels, but differs greatly. For example, it does not support Unix-like signals, but incorporates event-driven programming and the observer pattern."
https://en.m.wikipedia.org/wiki/Fuchsia_(operating_system)#K...
Afterwards I moved to ArchLinux, and that has been mostly fine for me.
If you are using a fairly standard processor, then Gentoo shouldn't give you that much of an advantage?
https://wiki.gentoo.org/wiki/Project:Prefix
Yes, Gentoo is great. I'm just saying that for me it was too much of a temptation.
These are the Arch packages built for x86-64-v2, x86-64-v3 and x86-64-v4, which are basically names for different sets of x86-64 extensions. Selecting the highest level supported by your processor should get you most of the way to -march=native, without the hassle of compiling it yourself.
It also enables -O3 and LTO for all packages.
[1]: https://github.com/an0nfunc/ALHP
LTO is great, but I have my doubts about -O3 (vs the more conservative -O2).
UPDATE: bah, ALHP repos don't support the nvidia drivers. And I don't want to muck around with setting everything up again.
Another update: I moved to nvidia-open, so now I can try the suggested repos.
Seems to work so far, but nothing 'feels' faster either.
doing this should allow you to use as many optimized packages as possible while still being able to install packages not supported by the alhp
Gentoo is more stable than Arch by default, though. It's not actually a bleeding edge distro, but you can choose to run it that way if you wish. Gentoo is about choice.
I can believe that.
> Gentoo is more stable than Arch by default, though. It's not actually a bleeding edge distro, but you can choose to run it that way if you wish. Gentoo is about choice.
I actually had way more trouble with stuff breaking with Ubuntu. That's because every six months, when I did the distro upgrade, lots of stuff broke at once and it was hard to do root cause analysis.
With a rolling distribution, it's usually only one thing breaking at a time.
I found Arch Linux to be more stable than Gentoo, but that is just my own experience.
That's true but worth noting that "optimize" here doesn't necessarily refer to performance.
I've been using Gentoo for 20 years and performance was never the reason. Gentoo is great if you know how you want things to work. Gentoo helps you get there.
Slackware was very manual and some bits were drowned in its low level and long command chains. Gentoo felt easy but highlighted dependencies with a hard cost associated with compilation times.
Being a newb back then I enjoyed user friendliness with access to the machinery beneath. Satisfaction of a 1s boot time speedu, a result of 48h+ compilation, was unparalleled, too ;)
A new hobby
> The goal of Gentoo is to have an operating system that builds all programs from source, instead of having pre-built binary packages. While this does allow for advanced speed and customizability, it means that even the most basic components such as the kernel must be compiled from source. It is known through out the Linux community as being a very complex operating system because of its daunting install process. The default Gentoo install boots straight to a command prompt, from which the user must manually partition the disk, download a package known as a "Stage 3 tarball", extract it, and build the system up by manually installing packages. New or inexperienced users will often not know what to do when they boot in to the installer to find there is no graphical display. Members of /g/ will often exaggerate the values of Gentoo, trying to trick new users in to attempting to install it.
https://www.shlomifish.org/humour/by-others/funroll-loops/Ge...
Kidding... honestly that was a pretty fun distribution to play around with ~20 years ago. The documentation was really good and it was a great way to learn how a lot of the pieces of a Linux distribution fit together.
I was never convinced that the performance difference was really noticeable, though.
Which is why I'm in support of always building your own software with -O3.
Can’t one recompile the same exact Ubuntu packages you already have on your system with optimal flags for your specific hardware?
Plus, the configuration could be automated - just push a button to (lazy, in background) recompile.
Priority for the ones profiled to be performance critical.
Well, in principle yes. But you'd also need to figure out what you mean by 'optimal' flags? The flags might differ between different packages, and not all packages will be compatible with all the flags.
Even worse, in the example in the article they got most of their speedup out of moving to a different allocator, and that's not necessarily compatible with all packages (nor an improvement for all packages).
However, if you still want to do all these things, then Gentoo Linux is the distribution for you. It supports this tinkering directly and you'll have a much happier time than trying to bend Ubuntu to your will.
Not everything in life is an optimization problem. Fun little projects like this is what makes us human (and, arguably, are both educational and entertaining).
If we're talking about micro-seconds of difference, the trade-off doesn't seem worth it. Even on a mass scale where this is somehow adopted, nobody is going to notice the difference. Maybe if this were in something like eCommerce and web browing where the lag translates to profit lost? Or perhaps game engines?
IDK, I just consider humans time more precious than a slow package (that already runs blazingly fast on a CPU that it barely matters.)
Also a lot of simple optimizations could save seconds or minutes in the lives of millions of people, and across multiple devices/programs that adds up. Microsoft once sped up the booting process of their consoles by 5 seconds, by simply making the boot animation shorter by 5 seconds. Those consoles were sold to millions of people and it took MS 8 years or so to make the fix. That’s many lifetimes wasted once you add it all up.
As a broader comment though, this is how we learn and discover things. Just because the specific outcome here is "trivial" doesn't mean the approach or the lessons learnt aren't valuable.
I found it an interesting read despite not have any stake in the outcome.
Try this (the placement of FROM was incorrect):
I remember when it was my main driver back in early '00s, running on a horribly underpowered PII at 266MHz. It would often be compiling 24/7 to keep up with the updates.
My first install was 2002 and it took my a good 24 hours to get X to boot.
It did catch a faulty memory DIM later on because the linking kept failing intermittently.
cargo install --locked jaq
(you might also be able to add RUSTFLAGS="-C target-cpu=native" to enable optimizations for your specific CPU family)
"cargo install" is an underrated feature of Rust for exactly the kind of use case described in the article. Because it builds the tools from source, you can opt into platform-specific features/instructions that often aren't included in binaries built for compatibility with older CPUs. And no need to clone the repo or figure out how to build it; you get that for free.
jaq[1] and yq[2] are my go-to options anytime I'm using jq and need a quick and easy performance boost.
[1] https://github.com/01mf02/jaq
[2] https://github.com/mikefarah/yq
Every once in a while I test jaq against jq and gojq with my jq solution to AoC 2022 day 13 https://gist.github.com/oguz-ismail/8d0957dfeecc4f816ffee79d...
It's still behind both as of today