Thanks yeah just woke up, finished making my way through the single-article website and still unclear if other libcs do the same thing (the article says libxcrypt picked it up, but only after musl did it).
Pedantic, but more "author's understanding of the spec" which isn't always correct.
> assume nonstandard behavior
I go back and forth on this one (spec vs standard behavior). It feels like at some point the most widely used impl -is- the spec, as we've seen in other places (web standards).
> Upon successful completion, crypt() shall return a pointer to the hashed password [...]. Otherwise, it shall return a null pointer and set errno to indicate the error.
To the best of my knowledge, only OpenBSD and Fedora have a crypt(3) that returns NULL. Everybody else returns a string starting with "*" on error.
For example on Debian the manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an _invalid_ hash to the output field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
The spec only defines a single error state - ENOSYS, that is basically only for systems that don't implement crypt at all.
Error handling is underspecified. The spec doesn't give you enough well-defined behaviour, meaning everyone lists pages of error-handling extensions they have added.
Apparently because some devs didn't check for NULL which resulted in crashes. Weird choice to fix other peoples' bad programming by deviating from the spec.
To be fair (as the sshole site mentions) it is also necessary to fake a password hash check to avoid timing side channels revealing to remote clients whether users exist, so returning a sentinel value is not a horrifically unreasonable solution to make sure even naive code does approximately the right thing.
Though I am surprised that all of the code examples do not use constant-time strcmp(). Actually, I wonder if even constant-time strncmp() would still leak information about whether the account is disabled because in that case both arguments are much shorter? Food for thought.
But yeah, the crypt(3) API is very 80s C, the man page even features a length description of a long-deprecated and utterly broken hashing algorithm that is even implied to be broken in the text, which so old it refers to "parallel computers" as though they are a new concept.
Constant time strcmp doesn't fix it. The problem is that the user doesn't have a password at all (the hash is *). But which hashing algorithm is used is encoded in the password. So they pick a different user's hash to not leak the information that there is no password.
Alternate solution would be to hardcode a specific algorithm, but that could also leak info in many cases, I guess.
No, I get that -- my first point was meant to allude to that but maybe it wasn't phrased particularly well. The bit about constant-time strcmp() was about comparing the hashes afterwards, which was a classic attack to extract the password hash for offline cracking (though salts probably make it untenable?).
In any case, the fact you need access to an existing hash to not select something stupid like DES is all due to crypt(3) being a wacky API and while callers could work around it by selecting more reasonable defaults it's a pretty big foot gun.
This is either a bad engineering choice or an intentional bug. An empty string would have been better. Why '*' instead of, say, a tilde, @ symbol, or something that generally isn't used?
i think it's just unbelievable there isn't a bigger push to move away from c to languages that can prevent stuff like this from happening. this and just the pure unsafety. it's 2026. it's like pure negligence at this point, the equivalent of building a bridge with matchsticks because you like using it more.
How would using a language different than C help here? At it's core this bug is a logic bug, where "*" == crypt(x), because crypt in musl started returning "*" in some cases some years ago. This could be Rust. This could be Go.
I write firmware where the program has to run for years unattended.
You are right, C with manual memory management, no lifetime management, lack of phat pointers and a culture of leet has problems with deranged maddening bugs that other languages don't have.
But this bug first shows how inband signaling is bad news usually. But worse it's a failure to adhere to the spec properly. If you implement a library and you don't adhere to the spec you're going to bone someone.
Perhaps even worse is obviously the spec doesn't come with any standard testing framework. That was also the root of Apples infamous goto bug.
Cars without seatbelts and airbags chopping off driver's heads in relatively slow collisions? Well guess what, that could happen in a modern car as well if the driver sticks their head out the window while going through a very tight tunnel!
Rust would not save you here as this is a logic bug. Nor would, well, really any other language. I suppose someone might claim that Rust's result type would save you but it really wouldn't: I can easily make the error type a String or pretty much any other type with little consequence.
Rust would absolutely have saved you here. Even with `Result<String, String>`, which is pretty poor design, there's no confusing `Ok(s)` with `Err(s)`. Sure you can go out of your way to write something like
let hash = match result {
OK(s) => s,
Err(s) => s,
};
but at that point I feel like the library is forgiven (except for the poor error design).
@my-huge-pony does make a good point elsewhere in the thread: if the potential error was not foreseen and the code was written originally to return `String` instead of `Result<String, Error>` then even Rust would be in a pickle.
both alpine and musl are great in my opinion, and as far as I know theres no empirical evidence that it is more or less secure overall than alternatives. would you like to elaborate your opinion?
I have to admit that I am a bit disappointed in this kind of thinking. There's vulnerabilities everywhere as especially the past few weeks have shown.
One occurrence is not a statistic and should not be used as evidence - it's just an example of a vulnerability that due to various bad circumstances lead to a particularity bad outcome. This logic is in my opinion equally flawed as calling alpine safe now because this particular vulnerability has been fixed.
I'm not a big fan of blaming and I think it's just a result of bad circumstances, but it's interesting to see how some seem to blame musl, but one could argue further that musl documented the behavior of their API which was ignored by openssh and so on
IMO only good thing about alpine is that it's a good environment to build stuff that links musl statically.
Never understood why Alpine is used to actually run anything in a container - if you statically link binary, then you don't need alpine as a base and if you don't what is the point? Don't @ me about image size, even without layer caching we're talking about tiny difference that is often dwarfed by size of your runtime.
Musl itself is a great example of taking "simple and lightweight" so seriously that everyone else gets to deal with the complexity. Slow allocator, underoptimized memcpy/string routines, limited locale support, no NSS, and endless glibc compatibility headaches.
But hey, at least libc is small. %%Proceeds to pull in megabytes of <any other>malloc, and other libraries to replace the slow parts of musl.%%
And now sshhole.com is a particularly spectacular example of what happens when "simple" meets the rest of the software ecosystem.
Static linking makes distribution easier, but it's an absolute security nightmare.
With dynamic linking, you just upgrade the library on your system and you're done. With static linking, every binary that uses the library has to be rebuilt.
IMO this is where languages like Rust shine. The design patterns built into the language for handling errors (Result) - force you to handle errors / function results with sentinels, null pointers, or errnos are so much more error prone than handling a Result<T,E> using a match statement.
crypt() returning `*` is insane especially since these kinds of sentinels are all over /etc/passwd
I guess this is where the culture around rust shines. You can write crypt that returns a string in rust just fine, just like you can write crypt that returns a tagged union in C. Cultural differences and syntactic sugar change which version is idiomatic.
I suppose if people started writing lots of code that crashed on error in Rust some enterprising developer might improve the function to always return a non-error value and set it to ‘*’ in error cases.
The culture is downstream of the design IMO. In general, people write the API that is obvious. In C, returning a string is obvious, while defining a custom OptionalString type is high friction. In Rust, returning Option<String> is as easy and obvious as returning a String.
Sure, you could write deliberately bad code in Rust by unwrapping everything everywhere, but the point is that it is a distinct choice to do so. You can’t forget to handle errors - you have to make the choice to disregard them. If you forget to check for NULL in C it won’t complain and it’s dependent upon you to remember to do so.
All the standard Rust libs use results. The standard C libs use errorno, return 0, return -1, return nullptr, or segfault. Others dream up things like returning "*". C++ has exceptions in some areas but not others, and if you turn them off, they're segfaults. I actually like C, at least more than C++, but the error handling is the worst part.
I agree but I think was also because Java didn't have sum types (or still doesn't? years I haven't read Java) and rethrowing is super easy, while Rust kinda nudges you into the right direction (if you need to rethrow at all, think if you want to rethrow verbatim or wrap with a semantic error type).
The question is _where_ the error needs to be handled. It's almost never directly at the call site. And then there's a few issues with checked exceptions and Java language things (RMI, closures, etc) which make checked exceptions cumbersome. An exception (checked or not) might be worse than sum types, but they're still a thousand times better then returning an error in-band.
Go doesn't require you to handle or pass an error upwards. You can call something that returns err and just ignore it, compiler won't care. The proper "if err != nil" thing is annoying too. Rust made the right compromise with !
> The question is _where_ the error needs to be handled. It's almost never directly at the call site.
Which is why exceptions in Java can bubble up the stack? You don't have to catch it in a particular method you can let it go to the caller of the method, and so on.
The problem with them all always comes when the ever-steady march of abstractions, features, and edge cases results in one routine throwing the same two error classes but in different places, and how you handle them is different despite being the same wrapper. Add a thousand developers and several years of churn, and poof you get extremely complex bugs that result from errors seeming to come from one place, but actually coming from another.
Having been on the receiving end of debugging that a few times, I no longer mind the in-band error handling, although admittedly I'm inexperienced enough to be confident its better. Just for the random code I've had to read where I just need to know exactly how something I didn't write and will never read again works, and whether the error is handled and how, i love reading in-band error handling, even if I too dislike the way an error laden routine must be written. Over time I've come to appreciate elegance "in the large" as opposed to the code I see in any individual function.
Amen, people saw the symptoms of "needs more syntactic sugar" and misinterpreted it into "error handling is bad".
The same way that layered code should wrap/translate return types, it should also be wrapping/translating exceptions. They're both results of computation.
In the vast majority of programs, errors and non-results are irrelevant at the call site. You want to program for the happy path and not wrap every single call with error checking code or `try/catch/finally`.
h = file_open(path)
data = file_read(h)
file_close(h)
return data
is the code you want to read. This essentially is:
return file_open(path).success(h -> h, file_read(h)).success(h, data -> file_close(h); data)
A language with thoughtful error handling primitives will support such code.
returning a non-success you either handle it or pass it up the chain, potentially with not a lot of information about what originally caused it. with an exception you can retain the full information about what happened by either rethrowing it, throwing your own exception with it embedded as the cause or declaring that you don't handle it by saying you can throw it.
Yeah, checked exceptions are weird. Almost everything has some way to fail. The exact type of exception is usually meaningless, more like there are a couple of broad categories. HTTP did this well with 4xx basically meaning client's fault and 5xx meaning server's fault.
Well, for example, if you had opened a file to put the data into, you might delete the file or restore from a backup. But even if the answer is nothing, bubbling it all the way up to main is not the wrong answer. The point isn't whether you can do something, it's whether someone can do something.
This is the root of a lot of the evil. Yes, humans want to program for the happy path. It is a lot more fun. However, that bias leads to so much lack of diligence to properly deal with the unhappy path.
Syntax is not going to solve logic errors. No amount of checked exceptions or forced unwrapping of a result will help with TOCTOU or any number of similar vulnerabilities in code.
You ideally want to attack the problem at two ends:
- make the syntax noise-free so you can see what is happening
- build primitives at the syscall and stdlib level for common footguns
You would still need to deal with domain-specific logic errors, but you have at least reduced the surface area exposed to casual attacks.
It is not. Being extremely restrictive in the happy path, throwing all over the place, but have generic error handling code at the top level (e.g. HTTP entry point) makes for some pretty robust, stable, easy to read software. Albeit it typically has somewhat rough user experience.
The thing is, that's a pretty good place to start. From there on, you can handle exceptions one by one, surgically, and make things nicer.
I keep hearing how diligent it is to handle errors at the call site, always by people who end up making services that either crash for preventable reasons or bury errors. Python is actually safer than Golang, and I don't mean with type hints.
I always thought the ideas was great, but type system was holding it back. If you could have expressed that your thrown exceptions are the same as some other method's (e.g. a method of a class given as a parameter), and some other classes, then it could have been a lot less annoying. Powerful, even! But Java did take its sweet time to get even type parameters.
I have written very little Java, though, so this is more of an armchair analyst position.
Better design breeds better habits. In Java, you can end up handling a DNS exception 20 layers up the stack, even if your call never actually triggers a DNS lookup, simply because some method deep in the call stack declares it as possible. The culture is to propagate or wrap exceptions rather than translate them into domain-specific errors near the source.
Rust makes handling and mapping `Result` locally more ergonomic, even though `?` makes propagation easy too. Rust, however, also makes it easy to lose the context during propagation and end up with useless `Error`.
Unfortunately this is also an area where a Rust programmer would likely just reach for the platform libc, because you really want to be compatible with exactly how the platform is hashing passwords. Once you know about the inconsistency, you can make predictable wrappers around it, but platform legacy is gonna keep throwing these curveballs.
It's nice to see how quickly this is being addressed, and likely monitored for any unintended side effects.
Looking at the unpatched Ubuntu things in my life, and I'm pretty close to asking Claude to update things to see how quickly Alpine can get close, assuming responsiveness can stay this fast and quality.
Not really related to the topic, but AI-popular sentences and language structure stand out to people right away these days right? I noticed some.
"If you cannot update yet, at least disable password logins: set PasswordAuthentication no in your sshd configuration and restart sshd. Either one closes the hole." (only AI would characterize a critical SSH implementation vulnerability as a hole closing game)
An attacker who can reach your SSH port can log in without knowing any password. It takes one connection, works every time, and needs no retries. (It sounds like the writer is bragging about the exploit instead of describing it. AI does this)
The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash. The attacker then sends a very long password (257 to 1024 bytes). One user with a password is enough: it makes every account without one reachable. (same as above made stronger with the 'statement: statement2' format)
"Users coming from a directory service through musl-nscd (nslcd, winbind, sssd), since they have no shadow entry." (a human would be more clear about what a shadow entry is and give more context, I think)
Hmm the rest of the article (the remaining 70-80%) seems quite human.
Anyway, not sure why I spent 10+ minutes on this. It just jumps at me right away.
An aside but one nice thing I realize about not using LLMs is, I guess, I am not tortured by this kind of thing, because presumably I am not seeing it all the time. I can't even imagine quite being in the mode of reading this kind of text so closely like this, really. Its a security advisory, I don't really have my critical prose-reviewer hat on.
As a grad student I read and graded a lot of student philosophy papers (well before LLMs), and I just remember the exhaustion of it. It made me appreciate how much I could otherwise just read something, get information or enjoyment out of it, and not focus on each sentence construction.
Just to say, I do believe you, I get that if you have to put up with it so much in general, it must prime you to find it everywhere. You have to read so much more now, I'm sure, so you're trained to be in that mode. It makes sense! I am just sorry its like this for you and most everyone else now. It sounds exhausting.
> set PasswordAuthentication no in your sshd configuration and restart sshd.
Funnily enough, they don’t recommend checking pubkey auth is enabled and working. By blindly (or agenticly) following those instructions the risk is getting locked out of some machine.
113 comments
[ 3.1 ms ] story [ 69.6 ms ] threadhttps://news.ycombinator.com/item?id=50377221
Pedantic, but more "author's understanding of the spec" which isn't always correct.
> assume nonstandard behavior
I go back and forth on this one (spec vs standard behavior). It feels like at some point the most widely used impl -is- the spec, as we've seen in other places (web standards).
> Upon successful completion, crypt() shall return a pointer to the hashed password [...]. Otherwise, it shall return a null pointer and set errno to indicate the error.
For example on Debian the manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an _invalid_ hash to the output field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
Error handling is underspecified. The spec doesn't give you enough well-defined behaviour, meaning everyone lists pages of error-handling extensions they have added.
Though I am surprised that all of the code examples do not use constant-time strcmp(). Actually, I wonder if even constant-time strncmp() would still leak information about whether the account is disabled because in that case both arguments are much shorter? Food for thought.
But yeah, the crypt(3) API is very 80s C, the man page even features a length description of a long-deprecated and utterly broken hashing algorithm that is even implied to be broken in the text, which so old it refers to "parallel computers" as though they are a new concept.
In any case, the fact you need access to an existing hash to not select something stupid like DES is all due to crypt(3) being a wacky API and while callers could work around it by selecting more reasonable defaults it's a pretty big foot gun.
Not every C vuln is a memory safety problem.
Easier to formulate invariants and check/prove them. Better language correlates with safer software.
You are right, C with manual memory management, no lifetime management, lack of phat pointers and a culture of leet has problems with deranged maddening bugs that other languages don't have.
But this bug first shows how inband signaling is bad news usually. But worse it's a failure to adhere to the spec properly. If you implement a library and you don't adhere to the spec you're going to bone someone.
Perhaps even worse is obviously the spec doesn't come with any standard testing framework. That was also the root of Apples infamous goto bug.
@my-huge-pony does make a good point elsewhere in the thread: if the potential error was not foreseen and the code was written originally to return `String` instead of `Result<String, Error>` then even Rust would be in a pickle.
There's a good chance similar flaws will be discovered elsewhere.
One occurrence is not a statistic and should not be used as evidence - it's just an example of a vulnerability that due to various bad circumstances lead to a particularity bad outcome. This logic is in my opinion equally flawed as calling alpine safe now because this particular vulnerability has been fixed.
I'm not a big fan of blaming and I think it's just a result of bad circumstances, but it's interesting to see how some seem to blame musl, but one could argue further that musl documented the behavior of their API which was ignored by openssh and so on
Never understood why Alpine is used to actually run anything in a container - if you statically link binary, then you don't need alpine as a base and if you don't what is the point? Don't @ me about image size, even without layer caching we're talking about tiny difference that is often dwarfed by size of your runtime.
Musl itself is a great example of taking "simple and lightweight" so seriously that everyone else gets to deal with the complexity. Slow allocator, underoptimized memcpy/string routines, limited locale support, no NSS, and endless glibc compatibility headaches.
But hey, at least libc is small. %%Proceeds to pull in megabytes of <any other>malloc, and other libraries to replace the slow parts of musl.%%
And now sshhole.com is a particularly spectacular example of what happens when "simple" meets the rest of the software ecosystem.
With dynamic linking, you just upgrade the library on your system and you're done. With static linking, every binary that uses the library has to be rebuilt.
crypt() returning `*` is insane especially since these kinds of sentinels are all over /etc/passwd
I suppose if people started writing lots of code that crashed on error in Rust some enterprising developer might improve the function to always return a non-error value and set it to ‘*’ in error cases.
One is to return NULL for errors.
Another is to make the string result a parameter and use a return code to signal error.
A third, less common but still out there, is an error parameter.
Failure to observe these is on the caller. Forcing the caller to consider the error is probably where rust shines.
But a good C programmer is thinking about these idioms when they encounter a function signature for the first time, or they write one.
Right idea at the wrong time.
Which is why exceptions in Java can bubble up the stack? You don't have to catch it in a particular method you can let it go to the caller of the method, and so on.
Having been on the receiving end of debugging that a few times, I no longer mind the in-band error handling, although admittedly I'm inexperienced enough to be confident its better. Just for the random code I've had to read where I just need to know exactly how something I didn't write and will never read again works, and whether the error is handled and how, i love reading in-band error handling, even if I too dislike the way an error laden routine must be written. Over time I've come to appreciate elegance "in the large" as opposed to the code I see in any individual function.
The same way that layered code should wrap/translate return types, it should also be wrapping/translating exceptions. They're both results of computation.
Past rant/discussion: https://news.ycombinator.com/item?id=42946597
I do prefer Maybe/Option/Either style returns in some cases though. Makes for a cleaner API.
So each method has its uses.
This is the root of a lot of the evil. Yes, humans want to program for the happy path. It is a lot more fun. However, that bias leads to so much lack of diligence to properly deal with the unhappy path.
You ideally want to attack the problem at two ends:
- make the syntax noise-free so you can see what is happening
- build primitives at the syscall and stdlib level for common footguns
You would still need to deal with domain-specific logic errors, but you have at least reduced the surface area exposed to casual attacks.
It is not. Being extremely restrictive in the happy path, throwing all over the place, but have generic error handling code at the top level (e.g. HTTP entry point) makes for some pretty robust, stable, easy to read software. Albeit it typically has somewhat rough user experience.
The thing is, that's a pretty good place to start. From there on, you can handle exceptions one by one, surgically, and make things nicer.
I have written very little Java, though, so this is more of an armchair analyst position.
Rust makes handling and mapping `Result` locally more ergonomic, even though `?` makes propagation easy too. Rust, however, also makes it easy to lose the context during propagation and end up with useless `Error`.
Another Rust self-suck session rendered completely irrelevant. Again.
Looking at the unpatched Ubuntu things in my life, and I'm pretty close to asking Claude to update things to see how quickly Alpine can get close, assuming responsiveness can stay this fast and quality.
"If you cannot update yet, at least disable password logins: set PasswordAuthentication no in your sshd configuration and restart sshd. Either one closes the hole." (only AI would characterize a critical SSH implementation vulnerability as a hole closing game)
An attacker who can reach your SSH port can log in without knowing any password. It takes one connection, works every time, and needs no retries. (It sounds like the writer is bragging about the exploit instead of describing it. AI does this)
The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash. The attacker then sends a very long password (257 to 1024 bytes). One user with a password is enough: it makes every account without one reachable. (same as above made stronger with the 'statement: statement2' format)
"Users coming from a directory service through musl-nscd (nslcd, winbind, sssd), since they have no shadow entry." (a human would be more clear about what a shadow entry is and give more context, I think)
Hmm the rest of the article (the remaining 70-80%) seems quite human.
Anyway, not sure why I spent 10+ minutes on this. It just jumps at me right away.
As a grad student I read and graded a lot of student philosophy papers (well before LLMs), and I just remember the exhaustion of it. It made me appreciate how much I could otherwise just read something, get information or enjoyment out of it, and not focus on each sentence construction.
Just to say, I do believe you, I get that if you have to put up with it so much in general, it must prime you to find it everywhere. You have to read so much more now, I'm sure, so you're trained to be in that mode. It makes sense! I am just sorry its like this for you and most everyone else now. It sounds exhausting.
I'm not sure this gets the point across when (according to https://sshhole.com/) it effectively means "any other user with a real password"
Funnily enough, they don’t recommend checking pubkey auth is enabled and working. By blindly (or agenticly) following those instructions the risk is getting locked out of some machine.