484 comments

[ 2.9 ms ] story [ 56.7 ms ] thread
The real name of this is sshole - the is the "star" of the show, after all.
*sshole, the * is the star of the show

You can use \ to escape formatting.

(comment deleted)
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?

As in: "Actually out there"

The hypothetical Alpine exists, yes, but what runs that like that?

Agent sandboxes today probably
I really like creativity in the name, the way website built and the content

"tl;dr" was a bit of a downgrade though, but that's fine

it's a nice way to promote the company behind it

That's a well-made site, yet I feel nothing.
What is dead can never die. All blueteamers are ironborn these days.
Maybe that's on you though. I enjoyed it, though the discrepancy between the `*sshole` name and the `sshhole.com` domain made me twitch. How is anyone[0] obsessive enough to find this and write it up and yet not obsessive enough to make the amount of `h` consistent?

[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.

Look closely. The second h is behind the word and almost as dark as the background
I thought it was a great write up. It did feel a bit AI-coded, but whether it actually was or not, it's still a good bit of work imo.
It would be weird to spend this much effort doing the site by hand, for a bug that the author says was found by a swarm of agents.
Looks good overall, but I was wondering if eg. "the timer learns nothing" and "no X, no Y, no Z" was slop rather than a deliberate choice.
Slop, no question. There's at least 10 Claudisms on the page, and I'm not that good at smelling them. Some of the sentences might have been human-written.
fwiw pangram believes the text is AI as well
I do actually. My guess is that this was not just one-prompted, but someone took the time to refine it based on their tastes and expectations until it matched something they would actually like.

However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.

It has a crap load of text but says very little and is very bad at conveying the details because it thinks it has to tell some weird story.
I had this classmate that had their ChatGPT instructions to respond in this same laidback, lowercase, gen-z style. i recognized the output as a lot of llm writing almost immediately
It has very low information density. This would read 10x if it were just a few paragraphs of text. Instead we need to render an animation every few sentences. Not a fan.
I think at some point I'm going to get banned from HN for repeatedly saying this, but here we go:

This thread highlights why I often say that at some point in the future, the obsession with whether something was written by AI or not will be classified as a mental disorder.

It is tiring to virtually every day see this discussion play out in some HN thread or other. It's also just noise. I don't think your comment, nor the thread it inspired, added anything of value to the discussion.

I left after trying to scroll to a concise description of the issue. I don't want a theatrical presentation.
> except a lot of programs never check for NULL. they just compare whatever crypt() gives back with the stored field, hit the NULL, and crash

> so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead.

Considering it's been accepted that crashing if malloc returns a NULL is your fault, did musl really feel the need to accommodate bad implementations like this or is this just AI dribble?

> did musl really feel the need to accommodate bad implementations like this

I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.

it's not "glibc behavior", returning null on error is in the spec
To be fair the API is older than the POSIX spec.
"unfortunately, a large portion of programs which call crypt are not prepared for its failure and do not check that the return value is non-null before using it. thus, always "succeeding" but giving an unmatchable hash is reportedly a better behavior than failing on error." [1]

[1] https://git.musl-libc.org/cgit/musl/commit/?id=b3c4cc121f70f...

Such a questionable decision! "Rather than let a bug in a caller (not handling the valid null output and thus crashing) surface, let's just silence it and pretend that the caller is correct! A potential clash in some situations (resulting in this vulnerability) is better than a crash!"

Maybe I should update my plea to fail loudly (https://alejo.ch/3gk) to list this as an example.

It's always a people problem. Doing a bad patch on my side is easier than bothering anyone else to fix their code
I liked that this site explained the vulnerability in an easy-to-understand way.
I can't understand it. All this scrolling and pictures are distracting me, can't concentrate really.
Yep, scrolling through my phone and reading slowly to understand. I'm seeing distracting animations below what I'm reading. And when I go to them, there is no way to restart them. So it completely misses the point.
Yet another clankered finding creating a clankered website with clankered explanations. To be fair, this one's a lot better than the other clankered equivalents, but I wonder whether these people even know enough about their bug to have their own write-up?

Also, what's with this damn trend of typing in lowercase only? Why is this a thing?

> oh did i say "i"? sorry. i had agent swarms. a lot of them. let's not get into it

Still a cool finding... it's just the communication is becoming a lot worse these days - I want to understand the findings please, not be impressed by your webpage! Biggest gripe in recent memory is https://github.com/xoreaxeaxeax/skitter-creek-bath-salts

> Also, what's with this damn trend of typing in lowercase only?

Anticapitalism.

You made my morning. Got a new dad joke.
sigh, it's not a new trend. it's just yet another pop-cultural appropriation from the gay community, now mainstream thanks to ai. as trump's gay dementia progresses, pretty soon even he will be truthing about other men's asses in lowercase, just to get a rise out of them!!! however, the *sshole project's shibboleth punctuation and anal asterisk do suggest its authors may be bona fide members of the lgbtq+ community.

why do gay people love typing in lowercase?

https://thetastefultwink.substack.com/p/the-queer-lowercase

> Let’s sit and twink about the data. 24.6% of my 19,000+ respondents abandoned capitalization entirely. If you control for age, queer people were 1.5x more likely than straight people to type in lowercase. Lesbians out-lowercased every other sexuality at 28.3%, followed by bisexuals at 24.9%. Honorable mention to the lowercase enbies, at 29.2%.

New and Changing Social Evaluations of All-lowercase and Exclamation Points

https://repository.upenn.edu/server/api/core/bitstreams/edbc...

> both exclamation points and lowercase usage are associated with queer identity.

One more reason to build your (own) Docker images on 'FROM scratch' (basically nothing).
No one runs sshd in containers, at least I hope so. What would even be the point when you have docker/kubectl commands available?
Using LXC (not the same as docker) and running openssh inside the container.
Can we just have a site with the text and images without the awkward scrolling crap? Hijacking basic features is an instant click away from sites for me

Edit: It seems to work fine on mobile. I'm not sure why the scrolling behaviour was added on the desktop version of the site

Above all, I’m a little sad that the musl developers decided to make a lot of “well it’s convenient” changes to core crypto functions — at least without the level of auditing that would have caught this! It’s tarnished their reputation in my eyes a little.
I think everyone is over fixated on AI finding complex bugs beyond a typical person's understanding either to locate or exploit.

I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.

I think that's the case where LLMs might indentify most.

It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head

Finding more and more that these complex bugs people gloss over like you are now are being used in exploit chains and the end result is still my clients getting hacked
This one might be hard to find using the author’s approach of throwing everything at various builds of ssh.

It is not really that hard to find in the wild if you had a device built this way, when overflowing buffers on embedded devices is a standard attack vector.

A little?!? I’m sorry, but musl is a joke, this is yet another proof of that fact, and I have no idea why anyone uses it in production, they have only themselves to blame after the many warning signs.
In many ways each individual decision makes some sense; wanting to stop loads of programs being affected by a (remotely triggerable?) crash wanting to avoid time based attacks wanting to be able to set accounts to not have a password

Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.

> Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.

But that's not a perfectly valid argument, and it never was.

The real problem here is the C language. Our industry should have dumped it a long time ago and we’re now paying the price of prevarication.
Which of the several flaws in this bug chain would have been materially different with a language other than C?
For the other part of the issue (the truncation in SHA512), in Rust you could've had:

    struct Input<'a>(&'a [u8]);
    

    impl<'a> Input<'a> {
      fn new(input: &'a [u8]) -> Result<Self, Error> {
        if input.len() >= 256 {
           return Err(Errro...);
        }
        
        Ok(Self(input))
      }
    }
Now the truncation is a terrible idea in the first place, but at least with this it would have been entirely obvious to the caller that it was there.
In C you could do just the same: https://uecker.codeberg.page/2026-10-24.html Whether you should is a different question.

In any case, you can not change the API of an existing function. But returning NULL for error has required by POSIX would have been just fine.

I understand that other languages could help avoiding the semi-predicate problem. That being said, in C there are better ways around semi-predicates, as inelegant as they may be, is possble, is not an inherent problem of the language, but may a bad habit/idiom of it.
I would be more concerned about the lack of explicit return types, one of the arguments for strongly typed languages is that you should not have ambiguous data and yet…

Granted I mostly work with high level programming languages so I may be missing something.

Yes this is C, C is not strongly typed at all and the type system is very primitive. NULL ptr is the return value you should probably use if something goes wrong. But you can’t, say, return a sum type with the result and an error. It’s not uncommon at all to use magic sentinel values in C to indicate error conditions.
If they had just all used a different garbage symbol would this have been avoided?
I guarantee that every single developer of musl in 2012 had at least account on their machine with "*" as the password hash. Also crypt() is a standardized function and the standard states that a null pointer is returned on error.

That being said, OpenSSH did mismanage their attempts at handling a garbage return-value from crypt(), so there's definitely some blame to go around.

You must be new here. While I applauded the idea of musl when it first came out (coincidentally, glibc wasn't having the best time in its development then), the developer's arguments why <some behavior drastically different from every libc under the sun> was akchyually correct, which was why you had to fix a program that hadn't been broken for 30 years and now suddenly was, made me put him under the "clowns" category and never look at musl seriously again.

When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.

And now we have this.

Uh, isn't this the case of the opposite? The musl devs trying to accomodate broken programs?
Honestly the ideas around musl are interesting, especially being able to statically link, but given the sort of nonsense around how they handled pthreads, and generally other trade offs, it's hard to imagine wanting to use it over glibc (which everyone shits on) if you don't absolutely need static linking.
I am sorry, a little? I am deeply disappointed that libc decided to do the same just with another sequence.

And why because some dude was complaining sshd or whatever was crashing because it didn't handle nulls? Just handle the null my guy you already know why your bullshit is crashing.

Let's be real here, the main culprit is C. People love to say "but you can still get logic errors with Rust", and no shit of course you can. But it's much less likely. You wouldn't have gotten this bug.
Did anyone get it to work? I tried on the official alpine docker image, enabled password login for root. The shadow entry has ! as password by default, this didn't work with a > 256 character password (i also did passwd -l which also does !). I manually changed it to * and it still didn't work.

Am I doing anything wrong here?

OpenSSH_10.3p1, OpenSSL 3.5.9 29 Sep 2026

PRETTY_NAME="Alpine Linux v3.24"

Nope, wasn’t able to reproduced it in my firecracker VM, which isn’t publicly accessible anyway.
Did you try one incorrect standard password fail first?
As far as I understand, it wont work on default setups. You need musl-c, no pam and openssh.
In Alpine, these would all be the case for an out of the box install with openssh added unless you specifically install openssh-server-pam and configure sshd_config to use PAM (Alpine is already musl-c based and doesn't default to PAM).
Doesn't it also need a user to be added with a password entry?

I only ever see Alpine Linux used for containers which don't need accounts with interactive user logins, which is not to say it doesn't exist, just that it doesn't seem like Alpine comes with all preconditions met out of the box

Isn't it the opposite, accounts which have no password and hence had * instead of a hash? I.e. the bug is:

> accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.

But the correct behavior was supposed to be:

> a * isn't a hash, so no password can match it. unix still does this today'

> and if the field is just *? there's nothing to match. type whatever you want, it's never getting you in

Also didn't have much luck on Alpine 3.20.10 with OpenSSH_9.7p1 which I've freshly installed from an official ISO.

Last two entries in /etc/shadow:

    test:*:20762:0:99999:7:::
    q3k:$6$snip.GgDQNTrqqX.A0:20762:0:99999:7:::
SSHing into test@ with one 'wrong' password then one 'long' password doesn't work. I have UsePAM no and PasswordAuthentication yes in my sshd_config.
No, didn't work for me open OpenWRT x64 running in a container but that is probably 'cause I'm using dropbear instead of openssh-server. That's what OpenWRT uses by default so the impact there will be minimal assuming that most installations don't change to openssh-server - it isn't installed by default.
On top of OpenWRT on my router (running 25.12) running dropbear instead of openssh-server, I also do not have any accounts in /etc/passwd|/etc/shadow with "*" for a password hash and an actual shell (all are set to /bin/false).
It's because alpine has updated all openssh-server packages already. Even from EOL alpine versions.
I can't reproduce it either. I used alpine:3.24.1 and openssh is openssh-server-10.3_p1-r2

I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.

Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.

(comment deleted)
at this point my only reaction is "are you fucking kidding me"
Wait so musl, famously a stickler for standards, just changed the behaviour of a POSIX function?
I avoided using alpine-based containers ever since finding out a lot of Linux server software is actually Linux + SystemD software. The union set API of both Linux and SystemD seems to be the default "Linux API" today.
I use alpine-based containers whenever I can just to keep the world safer from SystemD rotting Linux.

Never enabled sshd there... except on my PostmarketOS PineBook Pro... oh shit!

but alpine famously does not use systemd but openrc?
That's their point. The software they want to use often assumes its being run on a platform with systemd, so running that software on Alpine which makes a point to not use systemd causes challenges for them compared to using a different distro.
Container images rarely ship with an init service of any kind (systemd or otherwise); the expectation is that your application is the root process. This includes Alpine-based images.
Of all the named vulnerabilities, I appreciate the history lesson and screenshots of how it got there
This is a super duper annoying site for people who just want to know if their systems are vulnerable, after patching like a dozen critical vulnerabilities in other software over the last week and are almost at the breaking point of resigning.

Can we have a “don’t treat me like five/toddler” / “idgaf about the explanation” quick link to a section for adults who need to patch systems?

(And if your reaction is “just scroll down, stupid”, I’m happy to report that with the scroll jacking, I scroll not enough / past the critical sections nine out of ten times.)

Edit: You may want to read https://sshhole.com/#exposure.

To be fair, I have less than zero sympathy for anyone who runs SSH on any platform without `PasswordAuthentication=no` in 2026.

Its not asking for trouble, it is there will be trouble, not if but when.

As for the people who say "well, I run SSH on port 942329053052". Well guess what, the bots make short work on that. We tried it at $work as an experiment, multiple bots found it in less than a couple of hours. That's why your SSH config must be hardened, preferably running on a hardened OS too.

How is password auth weaker than key auth? Bugs in auth code can happen in both pathways. And apart from stupidly short passwords both should be equally impossible to brute-force.

In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?

What people think of as a very strong password often has <100 "security bits" of entropy.

Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.

Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.

But the SSH keys are based on asymmetric crypto, not shared secrets.
That makes it impossible to steal the key if you MITM, but there is still the same chance of guessing it. (Assuming password has as many bits of pure randomness, such as machine generated password.)
> What people think of as a very strong password often has <100 "security bits" of entropy.

The best password in the world will still be defeated by a simple keylogger or breach of your password manager.

Password gets sent to the server. Private key does not get sent to the server, and authentication protocol can be forwarded safely, enabling things like ssh-agent, FIDO SSH keys. This is solvable with passwords (see: augmented PAKE), but `PasswordAuthentication` isn’t that. But there are also settings where none of this really matters, yes, to be fair if that’s what you’re referring to.

SSH certificates are cool too.

password doesn’t get sent to the server, hash does.

but I guess that the hash is always the same unlike with pub priv key mechanism.

No, you are wrong. The password gets sent to the server. That said, I still enable passwords on servers, but don't ever share passwords between trust domains.
Password reuse is the problem. The password hash is stolen from some other site and brute forced offline.
you claimed my favorite game's character name!
I run SSH with password authentication on but the port binds to a ULA address. It’s fine. No need to be so absolutist.
> Its not asking for trouble, it is there will be trouble, not if but when.

Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.

> Did any of those bots brute force a password?

The point is they found the security by obscurity so-called "hidden" port in less than a couple of hours.

At that point, what they did is irrelevant. The point is that security by obscurity does not work in 2026.

> Because a 20 character random password is comparable in brute force complexity to an SSH key.

You can't be serious with your comment.

No its not.

And even then, SSH key is asymmetric, you're not sending the private key to the server. But with a password you are, which leaves you vulnerable to many attack vectors that simply do not exist with keys.

Other points are fine, but why do you say a 20-character random password isn’t comparable in brute force complexity to an SSH key?
> > Because a 20 character random password is comparable in brute force complexity to an SSH key.

> You can't be serious with your comment. \n\n No its not.

If you differ in opinion, maybe provide evidence? Especially for a claim like long randomly generated value 1 being absurdly ("can't be serious"-level) weak as compared to long randomly generated value 2

If you mean it's log(62^20,2)≈119 bits of entropy instead of 128+ and so it's theoretically weaker when we build a Dyson sphere around the sun to brute force that key (after somehow obtaining the shadow file, since an online attack would be way too slow), then sure

I mean there are 65k possible ports so it's the equivalent of guessing an 16 bit password, i.e. two (extended) ascii characters. It's not very surprising that 'multiple bots' found it: a single `nmap` command will give you the open ports in a few seconds.
Running on that port would be perfectly safe as it's outside the range of valid port numbers.
The bugs and commits that "fix" them reveal somewhat of a shitshow for how musl is managed. All projects have bugs, but I can't help but question why the wheel was reinvented here. Many toes have been stubbed along the way.
In-band signaling strikes again. * should not have been used as special value, neither for "no password" in /etc/passwd nor "error while hashing the password" in crypt. I'm afraid we'll find many more of these abuses, specially in string manipulations.
(comment deleted)
Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem. Silent compatibility hacks are the worst, they guarantee problems never get fixed and the stack of hacks never gets smaller.

Microsoft has historically been one of the leaders on the wrong side of this one, but they have also demonstrated how effective it is to call out badly behaved software and make it annoy the users with UAC. When Vista came out, so many applications needed elevated privileges to run because they were badly designed to write user data in system locations, but now their application was more annoying to use and came with a scary dialog about it making changes to your computer that other better-designed software didn't get.

More often than not if a program didn't actually intend to change system settings it was just a matter of badly chosen default paths which are pretty trivial to fix, but so many vendors just would not do it despite years of complaints from NT admins wanting to not have to spend hours with Process Explorer figuring out specific permissions to grant limited users enough access. Once their failures were put front and center, calling their application out as the one getting things wrong every time the user ran it, they suddenly had motivation to fix it.

For security it's better, but in many cases if you're not writing security-oriented software it can be worse from a user perspective. I remember the first job I had over 20 years ago was working on VoD streaming and one of the "tricks" to improve stability was mmap'ing the 0 page so that NULL dereferences wouldn't immediately segfault. Ugly as hell but when you were working with all sorts of more or less well coded media codecs and security wasn't a concern at all (worst case scenario the user would stream movies without paying) it made some amount of sense.

I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.

More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.

That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.

> because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great.

Story time?

There's not much else to say honestly, for these very important events there will usually be several layers of redundancy for the most critical feeds, so the production is always able to send something to the satellite uplink even while our devices reboot due to a crash, but brutally losing 90% of your inputs for ~1 minute while the machine reboots is not a pleasant experience for anyone...
> because having a critical piece of live media production reboot mid

It's not great, but these days a trickster with AI will find some way to start broadcasting porn to a billion people because of the flaw, so may end up being the better option.

> if you're not writing security-oriented software

It's hard to define what's security oriented and what is not. For example, is this HN web app security oriented? Not in the way of sshd but it must keep our accounts data separated and it must not let people exfiltrate data from the db, where for example it is storing the emails of some of us.

It depends on the time. Twenty or thirty years most software was not security-oriented. Nowadays, everything is online and therefore everything needs to be vigilant about security.
Some applications, such as servers, often want to continue, even if a bug is detected in one part of the system, and the potential worst-case fallout from that specific kind of bug is limited. Though such bugs are often also logged, at the very least.

Different situations, different requirements, different options, different trade-offs.

Consider also the use cases in https://github.com/rust-lang/rust/issues/43596 , despite Rust defaulting to panicking. Also, despite Rust not having exceptions, some configurations allow for "catching" panics, meaning that some Rust programs have to be "unwind-safe", similar to C++ exception safety.

> Microsoft has historically been one of the leaders on the wrong side of this one

Objective-C ahem nil messaging...

> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option

Sadly, we will never learn this because failing fast runs counter to intuition, and a class of person will never override it.

Moralizing leaves them trapped in epistemic closure.

They will look at problems caused by failing late and they'll see proof that failing fast is bad.

> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem.

100%

Agreed. I feel like this attitude has been holding Python back, too. It's one thing to criticize "move fast and break things", but it's also not good to ignore already existing breakage just because the pieces still fit together neatly, and it's not good to refuse to move at all.
> letting badly programmed software die when unexpected things happen

Go ahead and program a flight controller this way, I'm sure nothing bad will happen.

I'd still rather have the software crash and alert a human that to continue in a faulty state. The way to add robustness in this case is to add backup systems, not "ON ERROR RESUME NEXT".
(comment deleted)
This is a side-effect of Unix-y "it's all plain text" design that leans extremely heavily upon in-band signalling (among other design decisions).

If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.

With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.

Yes, absolutely. Take for example this cURL command:

  NO_PROXY=1 curl -qs --data @./my\ message.txt -- \
  "https://user:password@example.com/user/${userid}/my%20message.txt"
Random parts of it will be interpreted by the shell, others by cURL, or the filesystem, or the remote server. They each have their own encoding rules, nested. I count ten distinct separators and nine ways of escaping/tagging. It's one command. No, just no.

We have seriously overused strings and we're now paying the price.

Also: free text is the input format of LLMs...
I'd say another UNIX-y thing - "C is good enough" - is part of the problem. C makes programming painful which results in all sorts of hacks.

E.g. in Pascal you can can define a record and then defined file of record to have typed I/O. I.e. it comes with built-in serialization format supported by compiler. Not a good one - but perhaps a step in a good direction.

OTOH C people starving for serialization often believe that `struct` byte layout made by a specific version of a compiler is a serialization format - a view not endorsed by the standard.

Pascal's record typed files have the exact same problems regarding endianness and word size as directly writing structs to files in C, so I don't see how they're any better. They're also a complete deadend and unusable for pretty much any file format from the last 40 years+, since you can't even handle a header or magic bytes at the start.
[delayed]
No, that's just more in-band signaling!

Some badly designed password hashing algorithms (cough bcrypt cough) have maximum password lengths, ignoring everything after X characters. So it could happen that I set my password to a 100-character string, it gets saved "==DANGEROUS_CRYPT_ERROR_length==", and then anybody can log in to my account by entering an unrelated long string.

If a function promises a string result, it should only provide a string if the function succeeded. The null was much better than magic strings.

>but the fact that crypt() doesn't have any way to signal failure

https://pubs.opengroup.org/onlinepubs/009604599/functions/cr...

>Upon successful completion, crypt() shall return a pointer to the encoded string. The first two characters of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.

returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.

OP's sshhole link specifically notes that it _should_ return NULL, but some MUSL devs decided they wanted it to return trash instead, and the trash they returned instead of NULL caused this.

All because some devs were too lazy to check the result and another dev decided to be clever and "fix" NULL errors from crypt by sending back 'unmatchable' trash banking on devs not checking NULL.

If this is true, then the entire thread branch here on HN is entirely wrong. What is going on??
People don't RTFM, either on HN or when implementing "standards compliant" stuff like MUSL.

No error handling scheme can fix that problem. For instance, if a Rust API says I should return a Result<String, ENOSYS>, I can always write code like `if(ENOSYS) { return Ok("*") } instead of returning Err(ENOSYS).

Then my code would be compatible with MUSL.

> I can always write code like `if(ENOSYS) { return Ok("*") } instead of returning Err(ENOSYS)

You can do that but it's going to feel bad to write that and it's going to stand out like a sore thumb to anyone who reads it. But "errno" is just ambient state that's completely invisible unless you happen to have the manpage fresh in your memory.

They had to write

   if(ENOSYS) {
     return "*";
   }
after seeing that other implementations use different sentries like "*0" and that the spec says they MUST return NULL. That must have felt bad too.
Sure, but if you didn't do that, and returned the tagged union (in Rust or whatever language), your users would necessarily be forced to handle it in order to process the result.

The only reason this library did the insane thing of returning "*" is because it was extremely easy to cause crashes or false-positives (NULL == NULL) by returning NULL+errno. The reason that was easy is because C lacks a means of making error handling mandatory (exceptions, tuple returns, tagged unions, whatever).

If you return a tagged union, users might mis-handle it, it's true (https://blog.cloudflare.com/18-november-2025-outage/). But they have to handle it, which is a requirement that C APIs are incapable of providing.

You have to handle NULL in C, or your program crashes (though I suppose a misguided optimization of strcmp could do "a == b || really_strcmp(a, b)", which gets us back to your bug).

Also, in things like rust you can just do result.unwrap() if you want. In general, language constructs don't really save you from bad programmers.

> a misguided optimization of strcmp could do "a == b || really_strcmp(a, b)",

In practice, that's less of a risk than:

    char* x = function_that_might_return_null_and_error();
    dangerous_function();
    if (x == NULL) {...}
Or code for which pointer identity is relevant (e.g. accessing slices/subparts of a data structure):

    char* x = returns_null_or_a_slice_pointer(data_structure);
    char* y = other_function_that_returns_null_or_a_slice_pointer(data_structure);
    if (x == y) { puts("you're fucked now!"); }
And for sure there are much better ways to write both kinds of code! But people routinely do not do that.
A lazy attitude like "if we just used <n> language, all our problems would be fixed".
Except it's not trash. It's a backdoor funded by our glorious leader Kim Jong Un.
> returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.

> OP's sshhole link specifically notes that it _should_ return NULL

I meant there's no mechanism besides returning NULL that would make it hard to ignore the issue (exceptions need to be caught, results need to be unwrapped and their failure handled, an integer return value makes it clearer that there's something there that you need to check). I also missed errno codes when glancing at the manpage earlier which should illustrate the problem.

And yes, whoever first had the brilliant idea to start returning garbage never should've done that, I'm not disputing that.

This is true, and "trash" is putting it mildly.

But all those sentinels were added for the reason that, when your code is a public API intended to be widely used by who-knows-who, it needs to be really difficult to misuse it. NULL-plus-errno is easy to misuse. Yes, it's a dangerous mistake and your own fault if you do it. But people have been making that mistake for years and don't show any signs of stopping.

The root cause here is that C is fundamentally a rotten choice for public APIs of any kind. I griped more here: https://news.ycombinator.com/item?id=50381800

Yup, that's the way to explain it in order to blame musl.

Here's how to explain it in order to blame openssh:

openssh processes passwords and doesn't want to reveal through timing whether a user has a valid password (or exists at all). So far so good.

What they do if they hit a user with no valid password (i.e. "*") is that they look for another user with a valid password, to use the seed from. A bit weird already, but OK from a perspective of avoiding implementation specifics on seeds and hash algorithms. They know at this point that the result must be to deny access even if the hashed value matches, because that has is for another user.

So they put the password received over ssh into crypt() with this other user's hashed value to get a seed. And then accept it if it matches, apparently?

Homework: what happens if the password actually matches another user? Can I log in as root if openssh randomly picks my user for the fake salt, and I type my own password correctly?

The spec implies a string of at least two characters in the success case, am I reading that right?
Personally I think there was nothing wrong with returning NULL. "But if people don't check for it it crashes their programs!" Good! Let it crash loudly! Then they notice and fix the issue.

Also I think it could have returned ":". Given the file format of /etc/shadow and /etc/passwd I don't think it's possible to have a : as a value for anything.

> Also I think it could have returned ":". Given the file format of /etc/shadow and /etc/passwd I don't think it's possible to have a : as a value for anything.

I wonder if somebody else would have thought the exact same thing in a different (non-/etc/*) context, and then you get the same effect in a more obscure manner.

Indeed. Let's keep NULL.
> The issue isn't the config

That's a meaningless statement. There isn't a singular "the" issue like you said. Any one of several different technical decisions would have completely prevented the issue. Including, to my point, /etc/shadow being a JSON file where an account being locked is an `"active": false` field instead of an asterisk with no intrinsic meaning.

At this point I’m resigning myself to think that we’ve just gotta wait for some people to die. It feels very dramatic and glib (no pun intended lol) and whatever to say, but at this point it feels like there’s a certain hostility in some parts to changing anything from the platonic ideals of however computing happened to work in 2004. “Oh we don’t need to mess with it!”
How did you arrive at that conclusion? If anything, this vulnerability happened because someone decided to change the behaviour of crypt() without fully understanding the consequences. This specific vulnerability did not exist in 2004.
I agree with him. I only managed to get asterisks enabled by default in sudo because sudo-rs isn't owned by people who have decided that the 70s was the perfect decade and nothing should ever change. And look at the insane pushback when it did!

I can't even get an obvious UX improvement through PAM because they like it broken.

https://github.com/linux-pam/linux-pam/issues/778

A few years back I un-bashed my life as much as possible in favor of nushell (though it's not the only shell to encourage structure). I feel like it has really helped me notice a lot of this-needs-more-structure type problems outside of my shell that I otherwise wouldn't have.

The elegance of the unix-y approach is actually that it's all bits. You can just join things together and shove the bits through, that's a wild simplification compared with having vendor-specific kinds of plugs. But we could have that kind of compatibility while also benefiting from structure.

So I encourage everybody to give a shell with structured output a try, let it rewire your brain, and then ask yourself: how is the OS holding this back from being everything it could be? Because if we build that OS, it won't encourage these problems.

When I imagine it, it would probe either side of a pipe to see if they support negotiation. If not, send the bits through in the same old unixy way. If so, use the newest schema for structured data that is supported by both sides.

At Amazon we had RecordStream which implemented most of the standard Unix tools on structured streams. It was pretty easy to write filters, aggregators, and formatted output in something not much more complicated than sed/awk/cut/join/sort pipelines. There was a convenient structured log format for services that was often used as input.

https://github.com/benbernard/RecordStream

That looks really cool. I've been heading the opposite direction: posix shell for everything.

I have a bunch of management scripts, etc, that need to run on everything from a potato to a T-1000, and that's the only language that's always available. (OpenBSD sh, FreeBSD sh, busybox, bash, dash, zsh all can cope with it).

I'd love something that's at least JSON-typed (or equivalent), but that can be bootstrapped up on least-common-denominator posix.

Has anyone built something similar? (Yes, I'm being picky -- bun / ts is already a pretty low bar for cross platform support, but these scripts do stuff like install node, so I'm not totally crazy for asking.)

> "it's all plain text" design that leans extremely heavily upon in-band signalling

Honestly I've been shaking my fist at these since the XOn/XOff days

That's not specifically bash though, that's process exit codes
Yes. But it's still contradictory
The question you are asking when querying a process exit code is "did this program exit with an error?". The default assumption is "no, it completed successfully", so 0 (false) is the logical expected default return.

A function is expected to produce a result - so the question is "what was the result?". !0 (something) is the logical expected default return.

I've never found it to be contradictory.

Error code 0 means success in a lot more places than just bash.
Returning “*” instead of NULL to signal failure is the real footgun. Don’t blame the format.

For historical reasons, we could’ve ended up JSON as “hash”: “*” just the same. The /etc/shadow format lets you skip values completely, like “::”, so your argument doesn’t hold.

Terminating an app because of a NULL dereference is the other footgun. Sadly, we still have “panic” in modern languages like Go and Rust.

"panic" is a good thing. There's always some error condition that your code cannot handle. At some point, the least bad thing to do is crash.

For instance, what if your error handling code throws?

One spectacular example of panic saving the day was the Apollo 11 1202 error. It was basically a watchdog timer that fired if the landing computer was starved of CPU time. A misconfiguration caused the first moon landing to hit CPU overload. When that happened, the computer displayed 1202 and rebooted, then ran things in priority order. Apollo 11 landed in the middle of a tight 'panic, reboot, adjust thrusters' loop. Thanks to core memory, most of the heap survived the reboot, so the reboot step was fast enough for it to mostly land without human interaction. The pilot had to make ~1 adjustment to get it to actually hit the landing site, but the panic-ridden computer handled the rest of the descent.

https://dpma.de/english/our_office/publications/ingeniouswom...

sshole-type APIs < panicking APIs < required error handling.

Panic is fine, but fundamentally a whole-program crash is undesirable in a really wide range of use-cases. I don't want other SSH connections to crash (or the server to stop accepting valid logins until someone restarts sshd) if one person tries to log in with a credential that triggers a panicking bug in crypt(). Same for crashes in request handlers and so on.

The language/runtime itself doesn't (and probably cannot, at least if it's a general purpose language) distinguish between local and global error conditions, so panics (just like "throws" in Java or process boundaries in Erlang) end up being either unnecessarily damaging, or languages like Go end up normalizing panic "localization"/handling in dangerous ways.

The better solution is required error handling. If user code must engage with the possibility of an error state in order to access a result, that's a huge step forward. They can do the error handling wrong, but if you make most people have to do it at all most of the problems will go away.

"I forgot to check for an error" should not be implicated in bugs in 2026; we have had a large menu of possible alternatives (checked exceptions, unchecked exceptions, tagged unions, etc.) for decades. But we're still over here writing C (and Go), languages pathologically incapable of providing this ability.

SSH specifically has separate processes per connection the last time I checked, so a panic does the right thing: The attacker's connection drops uncleanly.

This is also why you can upgrade SSH over SSH without having the connection drop mid-stream.

Again: You cannot avoid "I forgot to check for an error". What happens if you get a sigpipe on stdout when printing the "I crashed" line? If Linux overcommits memory? If you turn off overcommit, port all software to check for + cleanly handle NULL from malloc (a substantial effort!), but then get a machine check exception from a double bit ECC error during an infallible read?

PSA: you can typically declare different types of string so that your type system objects to mixing data of different types even though deep down we know they're both strings.
If you mean type aliases, that's a good idea within a single codebase, but the distinction between different types of strings doesn't survive serialization so it wouldn't have prevented this error. What they need is a way to signal errors out-of-band, like Result<T> or exceptions.
In python, aliases are separate from typing.NewType, but you're right: it's only enforced within a codebase and then only if the dev cares to run the typechecker.

So yeah, I guess it either needs to happen at runtime, or we just do signaling and user data on separate file descriptors.

This is presuming that the type system is nomitive rather than structural and there is not serialisation/deserialisation or type refinement occurring. A structural difference is much more preferable.
Even using "no_password" and "error_while_hashing" would have prevented it. The problem was picking a generic, meaningless signal string.
That would've happened to fix it, because "error_while_hashing" and "no_password" happen to not be valid password hashes. The right way would've been to have different syntax for hashes and non-hashes; you could've made it so that a hash starts with a '#' and if it doesn't start with a '#', the rest of the string should be ignored.

Or, for backwards compatibility, start non-hash strings with a character which can't appear in a hash, such as '@'. Your examples would then become '@no_password' and '@error_while_hashing'.

Well neither * nor ! (the two characters I've seen to indicate "no password") will appear in a valid hash. The problem was using a character that could also be returned as an error indicator, IIUC.
Yeah but it's not forwards compatible with future hashes where '*' is suddenly a legal hash, which is what happened here. You need a grammar which lets you tell the difference at the parsing stage between "this is a password hash" and "this account has no password". That's not the same as letting the value '*' flow through the system as normal and hope it doesn't happen to match the hash.
True, but if they would have picked ?, or #, or any character other than * with its designated special meaning in the /etc/shadow file, then we would not be in this pickle.
hashes can't start with * and there's no reason to think they ever will
Technically they did use a character which can't appear in a hash, '*'.
Right but not in a "rigorous" way. They kept the format of the file the same: the string between the colon characters is a hash. They exploited the fact that an asterisk happens to never appear in a hash, and just let the system believe the asterisk was supposed to be a hash to be matched again.

In a sense, they should've changed the parsing logic to return a struct with a field 'password_hash: Option<String>', and defined the grammar of the field to be: "a hash string, or '*' for no login". The bug is, in part, caused by the haphazard way they don't change any formal grammar or contract and just relied on the fact that '' happens to* never be a valid hash.

It's the difference between "happens to work" and "defined to work by construction".

Exceptions are almost never the right choice. They create extremely complicated control flow extremely quickly, and then you have to test all the different control flows. Java software is an example of how this goes sideways fast. There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor), and objects are default initialized, so all fields (even static ones that are assigned in the constructor) can be null.

So, any field de-reference in Java can produce a null pointer exception. Try testing all paths in something like:

try { foo.sideEffect1().sideEffect2()...sideEffectN(); } catch { bar.sideEffect1().sideEffect2()...sideEffectM(); } finally { quuz.sideEffect1().sideEffect2()...sideEffectP(); }

There are M * N * P paths. The only way to write testable code with exceptions is to panic on uncaught exception, and to never catch unexpected exceptions. Even then, you're relying on nothing unexpectedly throwing the exception type you caught.

I'd argue Result<T> >> NULL >> sentries >> exceptions for this reason. Sentries are terrible, but at least they're not computationally hard to use correctly.

I'd argue that's the fault of having default nullability, not the exceptions. You would have the same problem if every attribute access returned a Result<T> too.

And I'm not a fan of breaking control flow, but I still find it preferable to nulls. A null may travel far from the place where it was generated before blowing up, making troubleshooting harder, or it might come from a function you called for its side effect but forgot to check the return value. I gotta say I'm also a careful enthusiast of checked exceptions, it's great knowing which ways a function can fail.

But I think the consensus is growing that Result<T> (plus warnings for unused return values) is simply the superior choice.

C has what you are asking for, mostly:

   int foo f() __attribute__((warn_unused_result))
As long as an error condition on the return value causes a crash on use, then it's mostly the same as result. (The main remaining difference being that you do an implicit `.unwrap()` in C, and in things like rust that's an explicit code smell.)
> There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor)

Which is why the pattern of double-checked locking doesn't work.

https://en.wikipedia.org/wiki/Double-checked_locking

It's wrong. The caller doesn't get the reference until the constructor runs. Although it looks like two operations in bytecode, the only thing you can legally do with an uninitialised object reference is call its constructor.
> never catch unexpected exceptions

Yes, broad exception catch clauses are simply incorrect code.

Panic is maybe an overreaction, "scratch everything and orderly abort" is sufficient.

> There's no way to create a class that prevents callers from reading the object before the constructor runs

That can only happen if you leak the reference (given we are talking about java language level semantics and not byte code), which I give you is unfortunately not that hard to do.

As for exceptions, I think they would pair beautifully with software transactional memory, which would work well with effects. In that case (and no side effects) it would not matter what code path we took, we could just go back to a known safe point.

Because ergonomy-wise I still think exceptions win (even though I do like Maybe types, and Rust's ? macro is plenty good): - they let one focus on the actual happy path code - no error is ever swallowed, unless explicitly written so (looking at you go with useless if errs that fail to properly handle the error), they are always handled somewhere - they automatically "unwrap" the Ok result, and automatically bubble up the Err side. - bubbling up is especially useful because most errors can't be handled at their source (e.g. IO error is meaningless, how you handle it depends on what you want to do in the first place, which is done several layers higher) - they automatically attach a stackframe/source, while with result types you may get a list of Ok Ok Ok Err and have no idea how that Err got there.

Exceptions work fine and enable a clean separation of business logic and error handling. Visibility into partially-initialized objects is a language design issue independent of exceptions. In a non-exceptional language, the corresponding access to uninitialized fields would just be bugs, not errors you can see.

Nothing in your post actually supports the idea that exceptions are "computationally hard". The CFG error edges exist however the errors are spelled

Unfortunately we’re moving to llm based systems which rely entirely on in-band signaling.
Not the worst site, but it's clearly a publicity machine from wakefield.co, whom I haven't heard of. Also, exploitation conditions are super narrow, albeit, any password is wild.
Wow. I wonder which of the common fuzzers would have found this a decade ago. If literally any password over 256 chars works, that means nobody has ever fuzzed openssh on musl linux