OK, I think I see what’s going on with this, in this configuration.
The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
I think the conclusion that can be drawn from this is that the behavior of crypt() is howling-at-the-moon insane, and it should no longer be considered for any system built in this century, at least not without extensive wrappers that correct and normalize all these edge cases.
In this particular case, it is type safe. NULL, by the C standard does not refer to an object. It means that every pointer in C is effectively an option type, which mean that a function that returns a pointer can return NULL for "no result", which, in the case of crypt() is perfectly adequate.
It is the opposite that is unsafe, that is, when the returned value can't be NULL, but it stills returns a pointer. C++ has references in an attempt to fix this problem.
Because C tends to prioritize performance over safety, a NULL pointer dereference is considered undefined behavior and the compiler can invoke nasal demons if it makes the code go faster, but with the right options, you can have it checked and invoke what in other languages would call a panic handler, but on UNIX is the SIGSEGV signal. You can then attach a debugger or read the core file and see what went wrong.
Note: I am not talking about "char *" for strings, that one is unsafe, but NULL is not. It is just a special case of an option type, or a result type, or an error union. Modern languages just extended this idea to objects other than pointers, and with more than one type of error.
The implicitness is exactly the problem. The workaround that caused this vulnerability was done because people assumed that crypt() would never return null.
You don't even need a fancy result type. A simple Option<String> would've forced downstream users to consider the failure case, avoiding this situation in the first place.
Exactly, whoever program in C knows that every function that returns a pointer may return NULL (even if not documented) and thus every pointer must be checked.
However a better API (to me) that I would have written was:
That way btw the function doesn't need to allocate memory inside, which is (being a mostly embedded programmer) something I don't like at all. Plus returning an error enum makes it clear instead of having to look at errno.
But POSIX API is not something well designed, it dates back to the '80 so... probably today would be written completely differently
In theory, having newtons and pounds force be represented as `float` is type safe. In practice, it destroyed a multi-million-dollar NASA spacecraft because of human error.
If NASA were to use the equivalent of `pub struct Newtons(f32); pub struct PoundsForce(f32);`, where the types Newton and PoundsForce were not interchangeable, this would never have happened[1].
[1] I'm aware that that story is a bit more complicated than that, that CSV files were involved (which are, again, not type safe), and that, in that particular scenario, more would have to be done to preserve type safety across formats, vendors and codebases, but my point still stands.
> It is just a special case of an option type, or a result type, or an error union.
These are not the same, and your next sentence suggests why. Option has but a single error type, but result types and error unions allow arbitrary types of error.
That depends on how you interpret the pointer type, but in any case there are multiple interpretation and just the fact that these exist will create confusion and type unsafety.
> “When the compiler encounters [a given undefined construct] it is legal for it to make demons fly out of your nose” (the implication is that the compiler may choose any arbitrarily bizarre way to interpret the code without violating the ANSI C standard)
However this is limited by the compiler's actual abilities and code. The C standard doesn't forbid nasal demons, but the compiler isn't programmed to invoke nasal demons and isn't running on hardware with that capability, so it won't happen.
I've heard nasal demons invoked as a literal possibility. No they are not a literal possibility.
Maybe your compiler isn't, but GCC is designed to summon every nasal demon it can. It especially loves to elide null pointer checks. There was a PR that almost made it to Linux mainline that did some weird stuff in kernel space where memory mapped to address 0 is valid and contains important stuff. It got a pointer, read data from it, then checked if pointer is null before giving root permissions for some operation. Reading the code, everything seems alright. Even stepping through it with debugger shows that it works. But when compiled with -O2 or higher, GCC removed the null check and now everybody got root permissions unconditionally - because reading from null pointer is UB regardless of circumstances, and GCC treats UB as dead code even if it is reachable.
Sure, and char[] is no string because you can't tie anything with it. And don't get me started on the insects that allegedly live inside digital source files - that's an absurd concept, no self respecting computer specialist should be talking like that.
Wait, so if you're running on embedded, you're not allowed to store anything at address 0, requiring you to waste that little bit of RAM?
That's... a bizarre choice, considering how religious C is about not making any assumptions about what a machine can and cannot do, even going to the trouble of working with non-IEEE floats and int types that are not a multiple of 8 bits.
You might consider it type safe according to C's rules, but it is not type safe according to contemporary opinion because the success type and error type are the same.
> [...] because the success type and error type are the same.
That's a bit weird? In Rust, if you have an thing that returns Result<String, String> that's still type safe, even though success and error types are the same.
I think I understand what you want to express, maybe. You might say it's in-band vs out-of-band error signalling?
Nit: the type constructor is `Result`: given two types `A` and `B` (not instances of those types, but actual types) it gives you a new type `Result<A, B>`.
`Ok` and `Err` are called "variants" and can be used to construct a new _instance_ of `Result<A, B>`.
Soundness means that the program behaves in accordance with what its types express. You're right that C may be unsound due to undefined behaviour, but I don't think this is about soundness. What you're saying is that papers on types show how you can use types to express more constraints.
You may be right in general, but soundness comes from formal logic, where they're pretty fastidious with their definitions. Soundness certainly doesn't mean the expressivity of the types (i.e. what constraints they can represent); it's about the relationship between the runtime behaviour of a program and the types in the code.
Again I think you're trying to push a specific definition as more correct than OP's somehow. Why not just ask them what they meant to understand their intent/context instead of saying they are wrong?
Unless the point of your argument isn't to understand OP but simply to "win."
I think I understand their point and even said, "What you're saying is that papers on types show how you can use types to express more constraints," and that's correct. I merely pointed out that that's not soundness, which is something I think they'd like to know. There may be some leeway in the definition of soundness and how it's similar to or different from type safety, but either way, type expressivity is something different altogether (and C certainly has issues with both expressivity and type safety or soundness, but they're different issues). So the point isn't to win - the argument was right and I'm not claiming that C's type system is expressive - but to explain the different terms and their meaning in programming languages and formal semantics.
That's not the same as asserting that soundness requires adherence to the current fashion, sum types over results and errors. TIMTOWTDI.
You don't have to think C is sound or good to distinguish a disfavored API style from mathematical unsoundness. There are infinitely many type safe APIs and languages that are, despite their soundness, ugly and hard to use.
Let's not project every dimension along which a language can be good or bad onto the axis of safety.
Opinion among people who know better than to try to authoritatively explain subjective words with rigid dogma, because that just makes them look silly.
The idea of trying to provide a workaround for buggy code calling crypt() is terrifying. For sure, trying to grease the skids for buggy code calling printf(), go for it. But it is hard to imagine cases where slightly-incorrect usage of crypt() is a good idea.
I don't think so. Even in a language with a good type system like Rust, somebody could have made this API:
/// Hash the phrase with the given setting.
fn crypt(phrase: &[u8], setting: &[u8]) -> [u8; CRYPT_OUTPUT_SIZE];
Then, in a later version, `crypt` needs to become fallible, so they need to signal errors somehow. But because they really want to preserve backward compatibility, they can't change the output type to `Result<..., ...>`. And so they end up doing the same thing that was done in libcrypt - they output some error value and hackily make sure it's not equal to the `setting` string, under the mistaken assumption that the output of `crypt` would only ever be compared to the `setting` string. And then someone creates the exact same bug.
Maybe I'm wrong, but it looks to me like the exact same incentives and means to screw things up exist in C and in Rust for this particular bug.
IMO type safety requires both language features and a mindset where you take efforts to keep things type safe. Your example has only the former.
For example, in a Rust web app, you can choose to use String for encoded URLs, unencoded URL fragments, escaped html, unescaped HTML, usernames, etc. Or you can choose to use newtypes and carefully design all APIs to only take and return the correct newtypes.
The problem is not that you can write bad stuff in rust, the problem is that this particular usage pattern can't be sanely coded in C. This is a public API that should inform users and enforce certain properties. A Results type does exactly that, you would have to go way out of your way to force invalid stuff through that type alone.
I don't think so. In both Rust and C, the solution is the same - make the function explicitly fallible, instead of trying to preserve backward compatibility. In C, the function would probably just look like this, returning zero on success and negative on failure:
int crypt2(const char *phrase, const char *setting, char *out);
You can call such a signature "insane" if you want, and that's fine, but it would have averted the bug that is the topic of discussion.
what you are suggesting is a form of type safety. There are still degrees of type safety even in C, and this bug resulted from deciding that backwards compatibility is more important than type safety, which I think we agree on.
This isn't good either. Crypt is a pure computation that can't fail for arbitrary reasons. The former implementation could fail if the salt was invalid, which is something you could guarantee. Now it also fails if the password is invalid. That's new behaviour and no matter which way you wrap it up it has the potential to be incompatible. They should have added the maximum length recommendation to the man page, and let DoS attacks continue until the calling software was updated, not the crypt library.
But in your version, you can't preserve backwards compatibility, making an infallible function fallible is a breaking change. For something comparable to libc you might want to (eg Rust std still has mem::uninitialized), but even with security issues, I feel like people would be more inclined to deprecate for a new function (or panic, which does have its own problems).
A good type system also means things are written with better forwards compatibility, and can be safely changed later. The Rust equivalent to `crypt` likely wouldn't exist: a low-level library would just provide hash functions, leaving callers to mitigate DOS etc; a broader library might return some opaque handle with equality/serialization, which you couldn't try to compare to a passwordless user; a high-level library would parse the account store itself. None of these really need to change in incompatible ways, they have all the information to add features and fixes at their own level of abstraction.
You wouldn't be concerned about breaking backward compatibility with rust though, would you? A recompile would be needed anyway and the change would be detected and enforced. The rationale for the bad API here (two types in one variable) was that users were ignoring NULL - that's the part which wouldn't happen in rust.
Specifically, that the lack of tagged unions (or really, structured return types in general; you can pretend those are a tagged union if you believe hard enough) in C library interop conventions is a terrible foundation on which to build widely-used APIs.
And that just ends up indicting the C language itself. Can't send errors down a separate channel; folks might not inspect errno. Can't return NULL on error, folks might compare it blindly to another result. No tagged unions, so can't return one. Can't crash on error because there's no scoped error handling. Can't return a struct for checking .error/.result separately, because alignment and equality (and tearing and accidental clobbering and more) make that brittle as hell for widely-used APIs.
C is not a good language for library APIs. Like, really not. It has its uses/strengths, sure, but without even getting into the typical gripes people have about C, libraries--one of the things that it is used for the most--are one of the things for which it is the least well-suited.
Passing and returning structs by value has been part of C since C89. Most compilers pass small structs in individual registers, and nowadays with LTO they'll even elide the copy of bigger ones if they can. What's definitely the case though is that by-value structs weren't a common thing when most of the Unix API was written.
Another example that C programmers love: using 0 to mean infinity. Eg when you have a timeout setting, and 10 means ten seconds, but 0 means infinity seconds instead of zero seconds.
In general it's a problem of sentinel values. Some languages let you abstract this - I think in Rust and C++ you can define an "intrusive optional" which is like optional but uses an invalid value pattern to store the not-present state. In memory it's the same as a sentinel value but the compiler forces you to unwrap it like an optional.
Amusingly, Win API of all places generally evades this pattern: 0 timeout means "no waiting", and to wait indefinitely, you use timeout of INFINITE which is actually UINT_MAX or, equivalently, -1 (in Win API, timeouts are unsigned, not time_t).
That’s just a very bad API. You could do the same in Rust? There are nice C APIs that would use a nice struct that describes the timeout you want… eg GCD.
Is really the solution to a bug where a function had a wrapper that tried to solve its documented issue of returning *, to make yet another wrapper to try to solve the documented issue of returning true?
musl is actually safe against this use. OpenSSL takes a bizarre code path, detects that $hash is too short, calls crypt($pw, $other_string), forgets that it detected that the hash was too short, and checks $hash == return value.
I did note that musl would normally work just fine here. Though I appreciate that your other comment goes into much better detail about what was actually going on in the code, as my analysis basically stopped once I understood the high-level flow of what was going on.
Except OpenSSH qould have worked just fine if musl behaved correctly. Instead musl ignored the spec and tried to hack around badly written programs but ended up creating a security hole in a correctly written one!
Yep, the linked page doesn't /feel/ AI written (maybe it's using a grok or muse model whose tone I don't quite recognize yet?) but it kinda just states a lot of facts that make sense on their own but don't cohere into a whole vulnerability like the comment above does.
I think it's just Claude prompted to write in a casual, conversational, no caps style.
The surface-level hints are gone, but the attempt to make every sentence pithy, the decisions on what information to present and what to assume the audience knows, and the over-emotionalization are all still there.
It feels very AI written, it's just not the blatantly AI default style. Pretty much any prompt with a "write X in this style: [example text]" will produce writing like this.
and what same human would embed two linux systems (??) in a blog post just for a demo that prints "pass"/"fail" that hardly anyone will bother to interact with?
Because Claude wrote the linked page. Claude is impressively bad at communicating these days. with gpt-5.6+ approaching Claude level like 30% of time and Gemini much clearer but also the most annoying 7 year old ever, it’s a pretty lame environment especially for people who r new to everything
I wondered about this myself as the explanations and documents I have been maintaining are becoming increasingly 'slop', but a different kind of slop.
In the past I could get an explanation, where I was able to track the line of thought. Now when I read through the explanation it is like someone has taken two or three explanations and merged them into each other, like there is a combining of iterations without clarity.
example:
The tree is blue.
The tall tree has the color of blue.
the tree, while tall, is blue.
That finally gets squished into: the blue tall tree has height and is evident of blue.
Is this double speak or injected double effort of words?
Until Opus 5.0 I'd have agreed with you, but Opus 5.5 is honestly one of the best I've ever sparred with, vastly better than the rest, including Astra imho. Especially so when prompting for clear top-down, first-principles communication.
Disagree, I review a lot of Opus 5.5 code and its comments and docs are still pretty inane and Claudish. Is it better than the 4.x and 5.0 era, sure. But it's still nouning the verbs, using colons everywhere, a-not-b, "a foos the b" etc. A lot of lingual knots that need a human hand to untie.
Opus 5.5 is much better, but IME I always have to force it to review its writing (Opus 5.5) to get it to put the most important facts first (and not bury the lede). It loves to park the main point far away after intermediate clauses. It always writes “premise, so consequence.” It takes a prompt asking to review against rules in context for it to fix it.
The ‘concise’ output style setting is bad with Opus 5.5, as it will often spit out a cryptic (is this really English?!), allergic-to-commas concise response, which takes more time to parse.
I agree. The “excuse” for implementing those behaviors is just providing plausible deniability. The commit that lets crypt return “*”, followed closely by a change that lets you easily trigger that behavior sounds just like how somebody would sneakily introduce a back-door into tons of linux systems while making it look as innocent as possible.
Well, it was intentional, but not in the way I think you mean.
musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
That was a misfeature and should never have been implemented. But I bet it was an easy band-aid change to make!
But choosing "*" as the character to return is an insane choice for anyone who knows the basics of linux sysadmin, or hell anyone who's ever used a search field in any context and thus knows it is ubiquitously used as a wildcard operator.
But it fits the common Linux implementation of the function... e.g. the Debian manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an invalid hash to the _output_ field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
It's almost sensible if you only think of one side. It's like if /etc/passwd put "NO_PASSWORD" in the field for disabled logins and someone was like, oh, I'll be clever and say crypt() returns "NO_PASSWORD" on error because that's obviously invalid and won't match. But that's exactly what *does* make it match.
> musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
Of all the things to mitigate...
1) crashing should be an option - it noisily indicates "there's a problem" and if it faults and dumps core, you've got a guided tour of your problem.
2) Is testing NULL as a passable option likely? I'd guess a more likely idiom is:
Wise words. I think early in my career I tended towards coding as defensively[1] as possible in a very specific way of "don't crash", as I definitely was not working in codebases I felt I had a full picture of. "I don't know who calls this method, and I don't want to get a call in the middle of the night that I broke prod" so I would say, return an empty string when asked to format a datetime except I'm passed a string or an int. (Yes, I was working in dynamic, untyped languages, too).
Later on, even in untyped languages, I came around more to "Throw loud and specific exceptions when asked to do things that are nonsensical" - it requires more overall cojones though because it may cause "your code" to crash. But the realization I came to was that it's better to crash ('a MaliciouslyLongPasswordError was uncaught') because "AuthManager asked PasswordService to check a password that was 2MB long" than to paper that over until something like this comes out. Or, the more common situation, you're passed garbage, you return something that won't choke the caller at the time, but what it saves to the database in that scenario ends up being corrupt data. 6 months later someone notices, and has to sort out a data mess that may or may not be recoverable.
[1] note that most "code defensively" advice is good. What I was doing could be thought of as a foolish misinterpretation of the phrase.
I think the issue is that a lot of callers were not aware that crypt() could error at all and all the issues are downstream of that. If OpenSSH knew that crypt() could error they would have handled it (possibly by inspecting errno).
I don't think that's the problem. The man page for crypt clearly says that erroring is a possibility and that it's not standardized how it errors:
> The behavior of crypt on errors isn't well standardized. Some implementations simply can't fail (except by crashing the program), others return a null pointer or a fixed string. Most implementations don't set errno, but some do. POSIX specifies returning a null pointer and setting errno, but it defines only one possible error, ENOSYS, in the case where crypt is not supported at all.
In my opinion the problem is that the behavior of crypt is underspecified - it isn't even specified how it signals failure, which is shocking.
Another disturbing decision that crypt's signature takes in the hash its output is supposed to be compared against - if it weren't for that, this half-baked "on error the output won't compare equal to the input" measure that crypt and crypt_* use wouldn't have been possible.
You can't check errno to see if there was a failure because, as the man page says, the value of errno is unspecified when crypt succeeds.
This specific manifestation of "C is terrible for shared code APIs" is rare.
But issues caused by folks forgetting to check errno, miscomparing NULL to NULL when calling multiple APIs, dereferencing external-API-returned NULL due to missed error checking, mishandling inherited errno after successful action, or--the most common--mishandling negative integer return sentinels meant to indicate errors? Those are incredibly common.
This specific library call is quite poorly designed.
Standard C api calls return an integer, which is 0 on success or nonzero on error. Ignoring it is an intentional choice. `errno` is the hack that shouldn't really exist because it's trivially easy to forget to check that and also has issues with threading (patched by making it a thread local) and more recently, coroutines (which admit no general purpose patch because they're a userspace construct).
Another reason why this is terrible is that it's unclear what the returned pointer's lifetime is. Is it until the next call to crypt? If so, is it thread-local like errno? If it's passed on to the user, is it safe to call free() on it, or is there a crypt_free function that should be called instead?
But all of this, and I mean all of it pales in comparison to returning '*' on error.
> But all of this, and I mean all of it pales in comparison to returning '*' on error.
100% agree. This is an egregious failure. But C's error-handling story would be bad even without this, and the badness of C's error handling is what motivated the (bad) decision to return *.
It'd be great if we wrote systems software in a language that supported advanced features like...
checks notes
...returning multiple values.
Hell, even in VBA you had to at least spend the keystrokes to type "on error resume next".
It's just that every other string than "*" would have been a better choice, including even the empty string (which would have been the next terrible choice).
It's not a C thing, it's a Linux thing. The asterisk is relevant to how crypt() is used for /etc/shadow.
If the password field contains some string that is not a valid result of crypt(3), for instance ! or *, the user will not
be able to use a unix password to log in (but the user may log in the system by other means).
No, it's a C thing. They previously returned NULL for invalid passwd lookups, but because there's no way in C to force API users to decide what to do if an error occurrs, that was causing program crashes due to NULL dereferences.
So they made the (very bad, but motivated by C's broken design/conventions) decision to return a magic string instead.
> but because there's no way in C to force API users to decide what to do if an error occurrs,
0. There are tons of libc functions that return NULL on failure, it is conventional way in C to indicate failure.
1. There's no much to decide, just fail to login if crypt() fails with NULL.
2. errno is standard way in C to check error code of a failing function, but in case of login you hardly need this, just fail!
I totally agree that fixing this issue in musl was a pretty dubious solution. Don't fix what's not broken, fix broken application instead. Publish a CVE if you cannot fix it.
I looked through FreeBSD source code for how they use crypt(), it's checked for NULL everywhere!
Nothing you said is incorrect, but I believe they meant that there’s no way to force a C program to check the NULL correctly—the compiler will happily compile code that gets a pointer back from a function but doesn’t check for NULL before dereferencing it.
In something like Rust the type system won’t let you touch the value until you’ve done something to deal with the error. Even if you just want to ignore it, you are forced to acknowledge that the function is fallible by the type system—it’s a compile error if you don’t.
Returning NULL on error is actually an excellent choice because most programs will crash if they try to use it without checking. A program, any program, crashing is better than this bug. Of course, modern programming languages have better constructs, but this was rare to nonexistent when C was started and if you think C is going away anytime soon you're delirious. musl's decision to change the return value to "*" is ununderstandable and likely intentionally evil.
> musl's decision to change the return value to "*" is ununderstandable and likely intentionally evil.
Dude, they documented their rationale. It was dumb and wrong but totally relatable. Irresponsible users were getting crashes, and I’m sorry to tell you that for a lot of library maintainers
> Returning NULL on error is actually an excellent choice because most programs will crash if they try to use it without checking.
…is not something they want to write in responses to bug reports, even though it is quite true. It’s human nature to want to prevent your stuff causing problems for others, even if the problems are fully their fault. Most people cannot and will not be Steve “so don’t hold it like that” Jobs, for fear that their decision will be just as short-sighted as his was.
In this case, that was obviously the dead wrong call. But I’ve been in that seat, and “closed: user error” for the 100th time feels bad to write. Add to that a touch of misplaced cleverness, and you get “hey…what if we just returned an impossible string instead of an error?”
Evil requires motive or malice, and I’m having trouble coming up with those in this case. These patches weren’t concealed or miscategorized a la xz, the behavior was discussed in the open. They were just, in hindsight, really poor decisions made for ordinary reasons. Hanlon’s Razor and all that.
That’s why I’ve been harping on C being trash for shared-code APIs in these comments: bad decisions are gonna happen, and irresponsible users are forever, so can we maybe use tools that require you to put on a seat belt (mandatory error handling) before starting the car?
> That’s why I’ve been harping on C being trash for shared-code API…
Agreed, but I can’t think of any other widespread ABIs that are better. Rust’s is not stable both C++ and Rust have concepts that don’t map into other languages well, so using them wholesale is a no go. So it would probably need to be a subset of those. Honestly just tagged unions would probably be enough. Maybe the solution is to get native tagged unions into C…
Of all the possible strings this seems like the absolute worst choice. If I were typing that string I'd pause to think "wait, what if the caller uses this with a regex?"
One has to remember the context here. The only error possible according to the SUS is ENOSYS, literally that crypt() is not actually implemented in the library. The traditional AT&T Unix crypt() was always implemented, and its encryption did not have failure modes, so there was never any error return to handle. This is why programs were written to never check the result of crypt(). It was only ever NULL on systems where one couldn't use the mechanism at all in the first place.
So the knock-on effect is that libraries that do have errors other than ENOSYS to accommodate, have to return magic 'impossible' values instead of setting errno and returning NULL, because of the weight of existing applications that expect nothing but the standard, traditional AT&T Unix (and indeed old BSD), behaviour.
Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Also check FreeBSD's OpenPAM for a case in point. Even today, it does not check for a NULL return from crypt() and leaps straight to strcmp().
If you want to see real history here, check out the 1992 comment by Eric Andrew Young, that is still in the source for OpenBSD today (in des/des_fcrypt.c), on how SSLeay's crypt() had to be adjusted so that any single character password field, including "*", did not end up successfully matching itself.
It's worth pointing out that musl tried to do the same as SSLeay, and deliberately returns something different to the original salt so that they will not compare equal. It does not return "*" if the input is actually "*" in the first place, contrary to the headlined cartoon explanation. (The locus of the problem is the caller calling it with a salt that is not "*" and then comparing it with "*".)
> Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Although that's a fine distinction for the point being made, as it's still the manual page for the crypt() function that one gets on the operating system, which that other manual page is not.
I suggest finding a code path where FreeBSD crypt() ever returns NULL, which would mean one of its internal functions returning non-zero, before getting too excited. They all end in return (0) . (-:
As I said, the callers of crypt() have for a long time been using the knowledge that crypt() on their platform simply never returns NULL. Unix has never been returning NULL from crypt() as far back as 7th Edition, when it simply always returned a pointer to a single internal static buffer.
The reality is that all C library people have to work with a corpus of Unix applications that do not check for crypt() returning NULL that goes back almost 50 years, and an error return mechanism that barely exists outwith the pages of the SUS.
That includes callers in single-user mode in process 1 in some configurations.
Well, crypt_blowfish() has a number of returns with (-1). I'm not familiar with Blowfish, so it's hard for me to trace the possibility of failure here. But what if in a distant future someone adds more hashing algorithms to the list without possessing historical knowledge of this "never return not 0" ?
Failing to check result of crypt() in init cannot do much harm, doing so is equivalent of going system reboot (crash init :-)). Besides, whoever has access to system console can do whatever they want with the system, no password can prevent from that.
I have no doubt, despite that Marshall Kirk McKusick is still committing in 2026, that FreeBSD developers will come along that do not know history. But they will get pointed to the manual page that (amusingly) says that returning NULL is 'not a standard behaviour' and that AT&T Unix never did that.
NetBSD's manual is more detailed on the subject, and explains the bind that library implementors have been in for decades, thanks to almost 50 years of applications knowing what crypt() 'really' does.
And this despite the SUS trying to shoehorn an error return mechanism into crypt(3) in order to accommodate the problems that operating systems had exporting encryption code, two decades afterwards. (The 1997 GoSolo book explains that this is why there's an ENOSYS error.)
The most charitable interpretation here is that musl authors are incompetent.
There is simply no universe in which a security-conscientious reviewer looks at the code, spots * being a valid return value, remembers that * is the default value in /etc/shadow for "no password login please" and then is all like:
This is fine, you check for * and return + instead.
There are more 32-long character sequences than atoms in the universe. Pick one, any of them instead.
Returning a string on error honestly is defensible (especially in a language like C where the type system doesn’t have exceptions or result types), the question is why the string was “*” instead of like “crypt ERROR: input password cannot be longer than 256 characters” lol
"*" is what's been historically used to represent an invalid/no value in passwd/shadow, so it makes some sense.
Of course given the chance that one might do a direct string comparison between crypt's return and the passwd/shadow file's hash field it was still a not well thought out choice.
In-band signaling[1] strikes again. * should not have been used as special value, neither for "no password" in /etc/passwd nor "error while hashing the password" in crypt. I'm afraid we'll find many more of these abuses, specially in string manipulations. The null was ok, an exception (or equivalent) better, and Result<T> (or equivalent) best.
Exceptions are almost never the right choice. They create extremely complicated control flow extremely quickly, and then you have to test all the different control flows. Java software is an example of how this goes sideways fast. There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor), and objects are default initialized, so all fields (even static ones that are assigned in the constructor) can be null.
So, any field de-reference in Java can produce a null pointer exception. Try testing all paths in something like:
There are M * N * P paths. The only way to write testable code with exceptions is to panic on uncaught exception, and to never catch unexpected exceptions. Even then, you're relying on nothing unexpectedly throwing the exception type you caught.
I'd argue Result<T> >> NULL >> sentries >> exceptions for this reason. Sentries are terrible, but at least they're not computationally hard to use correctly.
I'd argue that's the fault of having default nullability, not the exceptions. You would have the same problem if every attribute access returned a Result<T> too.
And I'm not a fan of breaking control flow, but I still find it preferable to nulls. A null may travel far from the place where it was generated before blowing up, making troubleshooting harder, or it might come from a function you called for its side effect but forgot to check the return value. I gotta say I'm also a careful enthusiast of checked exceptions, it's great knowing which ways a function can fail.
But I think the consensus is growing that Result<T> (plus warnings for unused return values) is simply the superior choice.
As long as an error condition on the return value causes a crash on use, then it's mostly the same as result. (The main remaining difference being that you do an implicit `.unwrap()` in C, and in things like rust that's an explicit code smell.)
> There's no way to create a class that prevents callers from reading the object before the constructor runs
That can only happen if you leak the reference (given we are talking about java language level semantics and not byte code), which I give you is unfortunately not that hard to do.
As for exceptions, I think they would pair beautifully with software transactional memory, which would work well with effects. In that case (and no side effects) it would not matter what code path we took, we could just go back to a known safe point.
Because ergonomy-wise I still think exceptions win (even though I do like Maybe types, and Rust's ? macro is plenty good):
- they let one focus on the actual happy path code
- no error is ever swallowed, unless explicitly written so (looking at you go with useless if errs that fail to properly handle the error), they are always handled somewhere
- they automatically "unwrap" the Ok result, and automatically bubble up the Err side.
- bubbling up is especially useful because most errors can't be handled at their source (e.g. IO error is meaningless, how you handle it depends on what you want to do in the first place, which is done several layers higher)
- they automatically attach a stackframe/source, while with result types you may get a list of Ok Ok Ok Err and have no idea how that Err got there.
> There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor)
Which is why the pattern of double-checked locking doesn't work.
It's wrong. The caller doesn't get the reference until the constructor runs. Although it looks like two operations in bytecode, the only thing you can legally do with an uninitialised object reference is call its constructor.
You should never ignore exceptions! The least you can do is to crash the program. Then you fix the bugs via battle hardening until it no longer crashes. Making the program not crash by ignoring the errors will just mess with the state of the program until you have no idea what caused it. It's better to let it crash, load the dump into your debugger tool to see what went wrong, then fix the bug, and repeat until you have a stable program. Rather then just live by the motto "ignorance is a bliss".
Exceptions work fine and enable a clean separation of business logic and error handling. Visibility into partially-initialized objects is a language design issue independent of exceptions. In a non-exceptional language, the corresponding access to uninitialized fields would just be bugs, not errors you can see.
Nothing in your post actually supports the idea that exceptions are "computationally hard". The CFG error edges exist however the errors are spelled
PSA: you can typically declare different types of string so that your type system objects to mixing data of different types even though deep down we know they're both strings.
If you mean type aliases, that's a good idea within a single codebase, but the distinction between different types of strings doesn't survive serialization so it wouldn't have prevented this error. What they need is a way to signal errors out-of-band, like Result<T> or exceptions.
In python, aliases are separate from typing.NewType, but you're right: it's only enforced within a codebase and then only if the dev cares to run the typechecker.
So yeah, I guess it either needs to happen at runtime, or we just do signaling and user data on separate file descriptors.
This is presuming that the type system is nomitive rather than structural and there is not serialisation/deserialisation or type refinement occurring. A structural difference is much more preferable.
This is a side-effect of Unix-y "it's all plain text" design that leans extremely heavily upon in-band signalling (among other design decisions).
If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.
With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.
A few years back I un-bashed my life as much as possible in favor of nushell (though it's not the only shell to encourage structure). I feel like it has really helped me notice a lot of this-needs-more-structure type problems outside of my shell that I otherwise wouldn't have.
The elegance of the unix-y approach is actually that it's all bits. You can just join things together and shove the bits through, that's a wild simplification compared with having vendor-specific kinds of plugs. But we could have that kind of compatibility while also benefiting from structure.
So I encourage everybody to give a shell with structured output a try, let it rewire your brain, and then ask yourself: how is the OS holding this back from being everything it could be? Because if we build that OS, it won't encourage these problems.
When I imagine it, it would probe either side of a pipe to see if they support negotiation. If not, send the bits through in the same old unixy way. If so, use the newest schema for structured data that is supported by both sides.
At Amazon we had RecordStream which implemented most of the standard Unix tools on structured streams. It was pretty easy to write filters, aggregators, and formatted output in something not much more complicated than sed/awk/cut/join/sort pipelines. There was a convenient structured log format for services that was often used as input.
That looks really cool. I've been heading the opposite direction: posix shell for everything.
I have a bunch of management scripts, etc, that need to run on everything from a potato to a T-1000, and that's the only language that's always available. (OpenBSD sh, FreeBSD sh, busybox, bash, dash, zsh all can cope with it).
I'd love something that's at least JSON-typed (or equivalent), but that can be bootstrapped up on least-common-denominator posix.
Has anyone built something similar? (Yes, I'm being picky -- bun / ts is already a pretty low bar for cross platform support, but these scripts do stuff like install node, so I'm not totally crazy for asking.)
The issue isn't the config, but the fact that crypt() doesn't have any way to signal failure. In better designed languages you would have exceptions, result types, or something. Even in C you have "return integer indicating success or error" which would've prevented the confusion.
Now the public API is hard to change but you could still avoid the issue by returning a longer token like "==DANGEROUS_CRYPT_ERROR_length==" or "==DANGEROUS_CRYPT_ERROR_unsupported_alg==" which would never match the password field unless someone was really trying to induce the issue.
>Upon successful completion, crypt() shall return a pointer to the encoded string. The first two characters of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.
OP's sshhole link specifically notes that it _should_ return NULL, but some MUSL devs decided they wanted it to return trash instead, and the trash they returned instead of NULL caused this.
All because some devs were too lazy to check the result and another dev decided to be clever and "fix" NULL errors from crypt by sending back 'unmatchable' trash banking on devs not checking NULL.
edit: apparently crypt implementations have been trash ignoring the NULL return for a long time. this isn't only on MUSL, but it's still because people ignored and tried to work around the existing out-of-band error mechanism.
But all those sentinels were added for the reason that, when your code is a public API intended to be widely used by who-knows-who, it needs to be really difficult to misuse it. NULL-plus-errno is easy to misuse. Yes, it's a dangerous mistake and your own fault if you do it. But people have been making that mistake for years and don't show any signs of stopping.
> returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.
> OP's sshhole link specifically notes that it _should_ return NULL
I meant there's no mechanism besides returning NULL that would make it hard to ignore the issue (exceptions need to be caught, results need to be unwrapped and their failure handled, an integer return value makes it clearer that there's something there that you need to check). I also missed errno codes when glancing at the manpage earlier which should illustrate the problem.
And yes, whoever first had the brilliant idea to start returning garbage never should've done that, I'm not disputing that.
It's subtly wrong, just like most of the rest of the commentary in this comment section. Yes, crypt is specified to return NULL on failure but if you read the rest of that page it is only permitted to fail with ENOSYS. The spec does not allow for failure for any other reason.
So if you're strictly adhering to the spec in this case then unfortunately NULL is not an acceptable way to signal a generic error condition. Hence all the seemingly bizarre special values. (In a strange turn of events all those experts doing things that looked weird turned out to know what they were doing. Go figure.)
People don't RTFM, either on HN or when implementing "standards compliant" stuff like MUSL.
No error handling scheme can fix that problem. For instance, if a Rust API says I should return a Result<String, ENOSYS>, I can always write code like `if(ENOSYS) { return Ok("*") } instead of returning Err(ENOSYS).
Sure, but if you didn't do that, and returned the tagged union (in Rust or whatever language), your users would necessarily be forced to handle it in order to process the result.
The only reason this library did the insane thing of returning "*" is because it was extremely easy to cause crashes or false-positives (NULL == NULL) by returning NULL+errno. The reason that was easy is because C lacks a means of making error handling mandatory (exceptions, tuple returns, tagged unions, whatever).
If you return a tagged union, users might mis-handle it, it's true (https://blog.cloudflare.com/18-november-2025-outage/). But they have to handle it, which is a requirement that C APIs are incapable of providing.
You have to handle NULL in C, or your program crashes (though I suppose a misguided optimization of strcmp could do "a == b || really_strcmp(a, b)", which gets us back to your bug).
Also, in things like rust you can just do result.unwrap() if you want. In general, language constructs don't really save you from bad programmers.
> I can always write code like `if(ENOSYS) { return Ok("*") } instead of returning Err(ENOSYS)
You can do that but it's going to feel bad to write that and it's going to stand out like a sore thumb to anyone who reads it. But "errno" is just ambient state that's completely invisible unless you happen to have the manpage fresh in your memory.
Yup, that's the way to explain it in order to blame musl.
Here's how to explain it in order to blame openssh:
openssh processes passwords and doesn't want to reveal through timing whether a user has a valid password (or exists at all). So far so good.
What they do if they hit a user with no valid password (i.e. "*") is that they look for another user with a valid password, to use the seed from. A bit weird already, but OK from a perspective of avoiding implementation specifics on seeds and hash algorithms. They know at this point that the result must be to deny access even if the hashed value matches, because that hash is for another user.
So they put the password received over ssh into crypt() with this other user's hashed value to get a seed. And then accept it if it matches, apparently?
Homework: what happens if the password actually matches another user? Can I log in as root if openssh randomly picks my user for the fake salt, and I type my own password correctly?
Personally I think there was nothing wrong with returning NULL. "But if people don't check for it it crashes their programs!" Good! Let it crash loudly! Then they notice and fix the issue.
Also I think it could have returned ":". Given the file format of /etc/shadow and /etc/passwd I don't think it's possible to have a : as a value for anything.
> Also I think it could have returned ":". Given the file format of /etc/shadow and /etc/passwd I don't think it's possible to have a : as a value for anything.
I wonder if somebody else would have thought the exact same thing in a different (non-/etc/*) context, and then you get the same effect in a more obscure manner.
That's a meaningless statement. There isn't a singular "the" issue like you said. Any one of several different technical decisions would have completely prevented the issue. Including, to my point, /etc/shadow being a JSON file where an account being locked is an `"active": false` field instead of an asterisk with no intrinsic meaning.
Some badly designed password hashing algorithms (cough bcrypt cough) have maximum password lengths, ignoring everything after X characters. So it could happen that I set my password to a 100-character string, it gets saved as "==DANGEROUS_CRYPT_ERROR_length==", and then anybody can log in to my account by entering an unrelated long string that returns the same "error".
If a function promises a string result, it should only provide a string if the function succeeded. The null was much better than magic strings.
I know, it's bad. I agree that it should've stayed NULL. But if you really really really think you have an acceptable reason to do something like this (and you probably don't), the least you can do is use a non-trivial token.
If I was really desperate and could only return a valid string, I would probably use a message and add a random component to avoid ever matching two errors. Like SQL, where null != null.
Returning “*” instead of NULL to signal failure is the real footgun. Don’t blame the format.
For historical reasons, we could’ve ended up JSON as “hash”: “*” just the same. The /etc/shadow format lets you skip values completely, like “::”, so your argument doesn’t hold.
Terminating an app because of a NULL dereference is the other footgun. Sadly, we still have “panic” in modern languages like Go and Rust.
"panic" is a good thing. There's always some error condition that your code cannot handle. At some point, the least bad thing to do is crash.
For instance, what if your error handling code throws?
One spectacular example of panic saving the day was the Apollo 11 1202 error. It was basically a watchdog timer that fired if the landing computer was starved of CPU time. A misconfiguration caused the first moon landing to hit CPU overload. When that happened, the computer displayed 1202 and rebooted, then ran things in priority order. Apollo 11 landed in the middle of a tight 'panic, reboot, adjust thrusters' loop. Thanks to core memory, most of the heap survived the reboot, so the reboot step was fast enough for it to mostly land without human interaction. The pilot had to make ~1 adjustment to get it to actually hit the landing site, but the panic-ridden computer handled the rest of the descent.
sshole-type APIs < panicking APIs < required error handling.
Panic is fine, but fundamentally a whole-program crash is undesirable in a really wide range of use-cases. I don't want other SSH connections to crash (or the server to stop accepting valid logins until someone restarts sshd) if one person tries to log in with a credential that triggers a panicking bug in crypt(). Same for crashes in request handlers and so on.
The language/runtime itself doesn't (and probably cannot, at least if it's a general purpose language) distinguish between local and global error conditions, so aborts end up being either unnecessarily damaging, or languages like Go end up normalizing panic "localization"/handling in dangerous ways. Isolation using threads isn't robust enough. Isolation using processes is far too fiddly, slow, and opaque.
The better solution is required error handling. If user code must engage with the possibility of an error state in order to access a result, that's a huge step forward. They can do the error handling wrong, but if you make most people have to do it at all most of the problems will go away.
There are other good solutions too, like Erlang's process boundaries, but those tend to have drawbacks that make them inapplicable to a lot of programming areas. Tagged unions, on the other hand, are only inapplicable in C library code because of language defects and terrible community conventions around ABIs.
"I forgot to check for an error" should not be implicated in bugs in 2026; we have had a large menu of possible alternatives (checked exceptions, unchecked exceptions, tagged unions, etc.) for decades. But we're still over here writing C (and Go), languages pathologically incapable of providing this ability.
SSH specifically has separate processes per connection the last time I checked, so a panic does the right thing: The attacker's connection drops uncleanly.
This is also why you can upgrade SSH over SSH without having the connection drop mid-stream.
Again: You cannot avoid "I forgot to check for an error". What happens if you get a sigpipe on stdout when printing the "I crashed" line? If Linux overcommits memory? If you turn off overcommit, port all software to check for + cleanly handle NULL from malloc (a substantial effort!), but then get a machine check exception from a double bit ECC error during an infallible read?
Sort of. It does some of the validation/auth code in the sshd parent daemon.
It has to: if it did 100% of the post-accept(2) work in a subprocess it would be pretty easy to remotely forkbomb/PID-exhaust a system by spamming invalid logins.
> You cannot avoid "I forgot to check for an error"
Yes you can. Checked exceptions and tagged unions both force calling code to be written with error-awareness.
In SSH's case, "I forgot to check" might lead to crashes (which might knock over sshd so new connections are impossible), or it might lead to false-positives via comparing NULL == NULL.
I'd much rather that boneheaded API users have to write code like
This contains a false snuck premise - that one and only one thing was the footgun. This is categorically not true. An obtuse, poorly-understood, purpose-built text-based DSL is absolutely a footgun and if it was replaced with a JSON and inactive accounts had been indicated by `"active": false` instead of a freaking asterisk then this would absolutely would not have happened.
> For historical reasons, we could’ve ended up JSON as “hash”: “” just the same.
That's dishonest, because it intentionally conceals the fact that that's a vanishingly* unlikely scenario.
People have brains that, factually, are far more likely to see a single context-less character "" and not realize the implications of it than to see `"hash": ""` and think "oh, that's normal".
This is just straight-up psychologically true.
> The /etc/shadow format lets you skip values completely, like “::”
Totally irrelevant.
> so your argument doesn’t hold.
False `::` is both (1) still harder to understand than `"active": false` and (2) even though the syntax existed it wasn't used here because everyone thought they understood the system.
Don't make excuses for this insane text syntax. It is, factually, harder to understand than a decently-written JSON format.
Random parts of it will be interpreted by the shell, others by cURL, or the filesystem, or the remote server. They each have their own encoding rules, nested. I count ten distinct separators and nine ways of escaping/tagging. It's one command. No, just no.
We have seriously overused strings and we're now paying the price.
At this point I’m resigning myself to think that we’ve just gotta wait for some people to die. It feels very dramatic and glib (no pun intended lol) and whatever to say, but at this point it feels like there’s a certain hostility in some parts to changing anything from the platonic ideals of however computing happened to work in 2004. “Oh we don’t need to mess with it!”
How did you arrive at that conclusion? If anything, this vulnerability happened because someone decided to change the behaviour of crypt() without fully understanding the consequences. This specific vulnerability did not exist in 2004.
I agree with him. I only managed to get asterisks enabled by default in sudo because sudo-rs isn't owned by people who have decided that the 70s was the perfect decade and nothing should ever change. And look at the insane pushback when it did!
I can't even get an obvious UX improvement through PAM because they like it broken.
I think this is a bit unfair. The way I see it, a lot of people are saying that changing breaks lots of old stuff, and risks complete fragmentation. Those are real costs to change. It might still be worth it, but let's not pretend that everyone is resisting change just because it's change.
The question you are asking when querying a process exit code is "did this program exit with an error?". The default assumption is "no, it completed successfully", so 0 (false) is the logical expected default return.
A function is expected to produce a result - so the question is "what was the result?". !0 (something) is the logical expected default return.
"False pretends to have an error" is where it gets weird. It's not that they got booleans backwards, it's that they're stretching a tenuous analogy a bit too far.
I'd say another UNIX-y thing - "C is good enough" - is part of the problem. C makes programming painful which results in all sorts of hacks.
E.g. in Pascal you can can define a record and then defined file of record to have typed I/O. I.e. it comes with built-in serialization format supported by compiler. Not a good one - but perhaps a step in a good direction.
OTOH C people starving for serialization often believe that `struct` byte layout made by a specific version of a compiler is a serialization format - a view not endorsed by the standard.
Pascal's record typed files have the exact same problems regarding endianness and word size as directly writing structs to files in C, so I don't see how they're any better. They're also a complete deadend and unusable for pretty much any file format from the last 40 years+, since you can't even handle a header or magic bytes at the start.
That would've happened to fix it, because "error_while_hashing" and "no_password" happen to not be valid password hashes. The right way would've been to have different syntax for hashes and non-hashes; you could've made it so that a hash starts with a '#' and if it doesn't start with a '#', the rest of the string should be ignored.
Or, for backwards compatibility, start non-hash strings with a character which can't appear in a hash, such as '@'. Your examples would then become '@no_password' and '@error_while_hashing'.
Well neither * nor ! (the two characters I've seen to indicate "no password") will appear in a valid hash. The problem was using a character that could also be returned as an error indicator, IIUC.
Yeah but it's not forwards compatible with future hashes where '*' is suddenly a legal hash, which is what happened here. You need a grammar which lets you tell the difference at the parsing stage between "this is a password hash" and "this account has no password". That's not the same as letting the value '*' flow through the system as normal and hope it doesn't happen to match the hash.
True, but if they would have picked ?, or #, or any character other than * with its designated special meaning in the /etc/shadow file, then we would not be in this pickle.
Right but not in a "rigorous" way. They kept the format of the file the same: the string between the colon characters is a hash. They exploited the fact that an asterisk happens to never appear in a hash, and just let the system believe the asterisk was supposed to be a hash to be matched again.
In a sense, they should've changed the parsing logic to return a struct with a field 'password_hash: Option<String>', and defined the grammar of the field to be: "a hash string, or '*' for no login". The bug is, in part, caused by the haphazard way they don't change any formal grammar or contract and just relied on the fact that '' happens to* never be a valid hash.
It's the difference between "happens to work" and "defined to work by construction". The former approach breaks once a hash function is invented where '*' is a legal hash (such as this new musl crypt() hash function where all strings longer than 256 bytes hash to '*'). The latter keeps working regardless because the asterisk is treated as "no password" already at the parser stage.
The vulnerable path in musl is only a little bit vulnerable, because "*" is not actually a sha256 encrypted configuration and should not reach this code: musl's crypt(..., "*") will not call that code! Turns out that sshd detects that the password hash is too short and does not forward the salt!
Look, no else statement! If that if doesn't get taken, then salt remains NULL. And the sshd code is really quite messy and doesn't keep track of the fact that this code path is inappropriate for validating an existing password hash.
A workaround in the sha256 path in musl wouldn't even have prevented the bug!
I'm going to give some credit to musl for unwisely using "*" and some credit to OpenSSH for having nonsensical control and data flow.
> (hey, this workaround is wrong if the salt is "x"!)
When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
> Turns out that sshd detects that the password hash is too short and does not forward the salt!
I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
> When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
I think you're right. That's what I get for doing a very very quick read.
> I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
I don't care what the intent of the check is. If you're doing a fake password hash to obscure timing, remember that it's fake and don't let the user log in! Don't deliberately pass nonsense into the live check and rely on the nonsense check failing to prevent the fake check from accidentally working. And don't have a code path that thinks it's checking the real password but passes in the wrong second parameter to crypt().
Or, better yet, have an entirely separate function fake_check_password() that does the fake check, returns void, and cannot possibly be confused with the actual password check code.
> If you're doing a fake password hash to obscure timing, remember that it's fake and don't let the user log in!
I think this is the crux. Instead of doing the right thing, they tried to codegolf to get the desired behavior and it got real subtle, real fast.
this is a little flip: maintaining a security-critical program naturally pushes one to minimize code changes because a 30-line change is usually easier to review than a 100-line change. Except when it's not.
My general rule of thumb is: "no magic strings". This is much easier to implement in higher-level languages and when writing application/web code than system programming.
Don't return empty string/asterisk/anything else. Instead, return a failure value.
When absolutely needed, choose the magic string with more caution:
One string showing the SHA-256 of a non-existing hash? It's a good idea for it to be a valid hash. Now we have a new risk: if it's broken once, it's broken for all. So it's a good idea for it to be a random one for each machine and/or account. Even this might have problems I can't think of, but it seems to be as secure as an account with a real password.
If this is not a good decision for some reason, choose something unmistakable: "--ACCOUNT-HAS-NO-PASSWORD--". This is much less likely to be mistakenly used in a different context, is much easier to look up in a code base than "*", and is self-documenting.
One potential point of disagreement: it's easy to implement structural prohibition of magic strings in low-level languages too if you consider Rust a low-level language (which I've heard before, even though I didn't really consider it to be one for our application of it).
I'm not a Rust expert but do run a few thousand lines of it in prod for some perf/correctness critical code, and I wish it didn't have the reputation of being so difficult to comprehend. There's so much resistance to introducing it or new languages in our large org, that I'm surprised we were able to introduce our thin slice at all. I am not sure I'd use it where you need easy async primitives like goroutines, but it prevents whole swathes of bugs that are repeated over and over again with devastating impact, like the OP's bug, via thoughtful language design.
Maybe this resistance is an artifact of where I work more than large organizations generally, though. Curious about others' experiences. I'm not really an evangelist for the language but would really like to work in a more polyglot environment one day.
Above all, I’m a little sad that the musl developers decided to make a lot of “well it’s convenient” changes to core crypto functions — at least without the level of auditing that would have caught this! It’s tarnished their reputation in my eyes a little.
I think everyone is over fixated on AI finding complex bugs beyond a typical person's understanding either to locate or exploit.
I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.
I think that's the case where LLMs might indentify most.
It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head
Finding more and more that these complex bugs people gloss over like you are now are being used in exploit chains and the end result is still my clients getting hacked
This one might be hard to find using the author’s approach of throwing everything at various builds of ssh.
It is not really that hard to find in the wild if you had a device built this way, when overflowing buffers on embedded devices is a standard attack vector.
It might be hard to root cause, but it would not be that hard to find.
Honestly the ideas around musl are interesting, especially being able to statically link, but given the sort of nonsense around how they handled pthreads, and generally other trade offs, it's hard to imagine wanting to use it over glibc (which everyone shits on) if you don't absolutely need static linking.
In many ways each individual decision makes some sense;
wanting to stop loads of programs being affected by a (remotely triggerable?) crash
wanting to avoid time based attacks
wanting to be able to set accounts to not have a password
Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
I would be more concerned about the lack of explicit return types, one of the arguments for strongly typed languages is that you should not have ambiguous data and yet…
Granted I mostly work with high level programming languages so I may be missing something.
Yes this is C, C is not strongly typed at all and the type system is very primitive. NULL ptr is the return value you should probably use if something goes wrong. But you can’t, say, return a sum type with the result and an error. It’s not uncommon at all to use magic sentinel values in C to indicate error conditions.
I guarantee that every single developer of musl in 2012 had at least account on their machine with "*" as the password hash. Also crypt() is a standardized function and the standard states that a null pointer is returned on error.
That being said, OpenSSH did mismanage their attempts at handling a garbage return-value from crypt(), so there's definitely some blame to go around.
Any use of sum types would have prevented this immediately.
The whole thing exists because APIs that return a NULL value that "nobody checks", plus a string that you shouldn't use if you actually did check the null value.
In languages with sum types, this cannot happen.
Either you get your hash, or no result at all.
You couldn't get eg NULL and *.
This argument is weak. C, and specifically libc, has errno. The error code is there. Not in band.
But there has to be something in the payload field when it is not applicable. A null is perfectly fine.
The argument for changing that to "*", if I understand the above linked discussion correctly, was that some users of this api threw away errors and a dummy value would be harder to ignore. Maybe one could argue that these behaviours are less common in Rust or Go, but not because of a technical difference.
The world is not perfect and sometimes you really need to work around bad users. Even in Rust or Go. I am not convinced this was such a case. If you are convinced it was it would still have been better to select a less trivial dummy value (such as "*:ERROR:*").
> But there has to be something in the payload field when it is not applicable
Not in languages with ergonomic sum types. A `Result` or `Either` will have no payload field on the error type. The compiler won't let you dereference it.
It is extremely unidiomatic to use tagged unions in C for error handling. It's definitely possible, but awkward, and the compiler doesn't check that you checked the tag.
Because usually NULL is good enough and you need to have domain use case to make library for it.
> * Return an integer error with any return values as out parameters.
You must first standarize all that (usually negative) integers, make docs, check for it, do not make mistakes in your if's or case's...
And errno is just that - integers with a meaning. errno is even thread local now and quite safe to use but people on internet scream loudly when someone try to use that not app global varible :) I guess their llms statistically have less new texts feeded in ;)
At some point it is just better to return pointer to struct with all info you can imagine.
And how you think other languages implemented their Error type returns ? Just a struct or union with tag. Because current CPU's operate on simple type registers there is no other sane way to do it.
Yet the authors of crypt did not use them. The C language is not designed in a way to encourage people to write code that correctly uses tagged unions (or nullable pointers for that matter).
because POSIX dictates the shape of crypt from a function written in 1973 and unions/structs were only introduced to the C language in 1976.
the fault for sshhole isn't just "C bad", it's a series of poor choices in workarounds falling together to create a vulnerability.
being condescending about decisions made 50 years ago, before any of the better languages you're mad the developer didn't use even existed, is silly.
if you want to lay blame, lay it at the feet of more recent devs that started returning values instead of NULL because people were using crypt incorrectly. likely sha512 returned a fake value instead of a real one because prior hashes couldn't fail. the interface should have been updated then, but instead they worked around it.
Nowhere. It's the same argument as upthread. A language that forces downstream consumers of an API to correctly deal with all possible outcomes would have prevented this.
Goalposts are fine. If crypt did return union type, C still allows cowboy programmers to straight extract the char* without any check whether it's valid. So we'd end up in exactly the same situation.
Because in fact all the changes were about changing the function signatures. The function signature is enforced by the linker. The semantic of strings that go through isn't.
Returning 2 values is possible in C (via pointers input parameters). The main issue here is that libc is a strong API contract based on C function signatures, that nobody can change, only augment with new symbols. Such strong API compatibility constraints happen in any programming language, and workaround to fit new behaviours in old APIs happens everywhere.
I don't think nh2 is saying that this is how the "issue" should've been solved here, but that it's a fatal flaw of C itself (yes I know, unions, but no one uses them).
But also, its not like there were many alternative languages when the POSIX standards were designed so it's fair that it works like this.
Even C has a better type system than C: All modern C compilers support an extension that lets you flag a function's return-value as requiring a null check.
means that most ways of returning an error from a function are not available in C. Because of this, many C APIs, including `crypt` in this case, resort to in-band error signalling by using a sentinel value (like NULL, a negative integer, "*" etc.).
Most other languages lack one or two of the above, but very few lack all three, and so most have developed a different culture around error reporting (Result in Rust, multiple return values in Go, exceptions in Java, Python etc.).
One of the small but important significances of Rust is that it very often gets to have the same machine code as a correct "in-band error signalling" strategy from C but with source code that the maintainers can actually understand.
And really often when your "in-band error signalling" in Rust is more expensive you realise (perhaps to your horror) that your C wouldn't be correct, so you just dodged a bullet. For example Option<OwnedFd> is, like OwnedFd, a 32-bit value, Unix has always reserved -1 to mean invalid, so Rust can treat that as a niche. But the Windows equivalent Option<OwnedHandle> can't do this because, well, cue Raymond Chen: https://devblogs.microsoft.com/oldnewthing/20040302-00/?p=40...
> One of the small but important significances of Rust is that it very often gets to have the same machine code as a correct "in-band error signalling" strategy from C but with source code that the maintainers can actually understand.
But aside from NonZero- and NonNull-based layout optimizations, I don't think Rust is doing anything special to obtain that machine code - it's just relying on LLVM to optimize things.
Well, all of the niche optimisations, so that includes every normal C-style user-defined enumeration because (unlike C or C++) Rust says those aren't just the integers wearing a funny hat.
But sure, for the most part LLVM will do The Right Thing™ here without much help and Rust is just giving better ergonomics to the same backend.
not in a way that forces the user to actually handle them properly. C is a weakly typed language and as such is pretty badly suited to systems programming.
What is a weakly typed language to you? To me, implicit type conversions are a big part of the definition, and C is flush with those (void* converts to any pointer, all kinds of number conversions, arrays decay to pointers, functions convert to pointers, lvalue conversion strips const/volatile etc.).
However, I also think "weakly typed" is not a well-defined concept, everyone seems to apply it differently. In particular, it seems to mostly mean "playing it loose with types in a way that I don't like".
Yes, it is not well-defined. But most conversions, function calls, and accesses are strongly typed in C, so I would not call it weakly typed.
Implicit conversion could be seen as weak typing, but it is mostly limited to specific arithmetic operations in C. I would see implicit conversions as problematic if they allowed operations that make no sense. Decay of arrays and function to pointer conversion also do not seem problematic to me for this reason.
Void pointers are also weakly typed, but they are specifically designed for this. So if you do not want this behavior, one would simply not use them. It is a bit like using "unsafe" in Rust.
There are certainly stronger typed languages, but on the full range C is more on the strongly than weakly typed side in my opinion. A language that allows arbitrary implicit conversions between pointers, would be weakly typed, or which allows calling functions with wrongly typed arguments (which was true in the past for C a long time ago)
C is definitely not the most weakly typed (e.g. Javascript/PHP are both weaker), but IMO C is weaker than python in that python will give you a runtime type error if you do something invalid, while in C you cast something and just get UB. Most higher level strongly typed languages are able to make their casts checked because they have generics to remove the need for 99% of them, but C is missing those features and thus has to rely much more heavily on unchecked and unsafe runtime casts.
Casts are escape hatches just like "unsafe" in Rust. You can write safe generics in C that either do not use casts or wrap all unsafe operations behind type-safe interfaces.
C unions are sum types if you just "draw the rest of the fucking owl". C has sum types the way x86 assembly has function calls. You see, there are "call" and "ret" instructions. Just figure out calling convention and the stack yourself, it's easy. And yes I can stick the tag next to the union myself in C and write a bunch of functions that hopefully deal with it correctly. I can even write some macros that look like pattern matching and maybe convince myself that it's a language feature, but it just isn't.
And it's very evident why the difference matters. People gravitate to what's readily available and ergonomic. That's why every C API under the sun (most definitely including libc itself) does janky stuff like returning -1 or "*" on error instead of building their own bespoke Result-style abstraction. A programming language is not just the things that are in some way possible within it, but more importantly the things that are easy in it.
While I agree that this should come out of the box, the "rest of the fucking owl" is a tiny macro or function, so not exactly something difficult to do complex as "drawing an owl" would imply.
Sure, when I last wrote a sizeable chunk of C, I wrote those macros and with a bit of help from new-ish compiler warnings, I got a reasonable error handling system in C. But whenever I interacted with anyone else's code, they apparently didn't get the memo that this is possible and desirable.
A language should encourage writing robust/secure code, not just make it possible.
While I see the point, I am not entirely sure I agree. I kind of like that C is not an opinionated language which tells you exactly how to write things. This comes at the cost that many people will do things in a bad way, and this is difficult to fix, but still probably easier than getting them to move to a different language.
In any case, I do not think the issue here is caused by not having a pre-defined result type. Checking for null would have been fine. The issue was that to accommodate old and broken code a compromise was made. But as long such compromises are done even where fixing such old code would be quite easy, it is clear that changing to another language is completely unrealistic. So what are we even talking about? The real issue is that nobody wants to spend money or effort maintaining the old code. No language can fix this.
There are different amounts of effort you can pour in to this depending on your requirements, and we can argue about those, but to me the evidence is clear: It's too much of a hassle for library authors to actually go and do it. Everyone just uses sentinel values instead.
Rust and go still make it pretty easy to not check results if you don’t care about the success value (lots of code just does `let _ = foo()`), but if you do want the value (like with a hashing function), Rust makes it really hard to not check the error too. There’s no syntax to say “assume this worked and give me the success value, including whatever garbage is there if it’s not set”:
- result.unwrap() will panic if it failed
- result.ok() throws away the error but gives you a None value instead (still can’t use it as if it’s a success)
- matching on the Ok variant and doing nothing on the Err variant, you only get to use the success value inside the Ok arm
Etc etc.
The problem is when the compiler says “NULL is a valid instance of T”, which go lets you do, and rust doesn’t (without using the unsafe keyword and intentionally casting a null pointer, which you don’t do by accident.) Go lets you write code like `var hash string; hash, _ = hashfn()` and it compiles and now you have null instead of a valid hash.
Minor nit, strings are never nil in Go, so that would be "" instead. Usually nil is only a valid instance of pointer-to-T. The point stands though, a common pattern is "defer foo.Close()", the Close() signature almost always returns an error and almost no one checks it...which is probably fine, but not automatically fine.
I understand that other languages could help avoiding the semi-predicate problem. That being said, in C there are better ways around semi-predicates, as inelegant as they may be, is possble, is not an inherent problem of the language, but may a bad habit/idiom of it.
You must be new here. While I applauded the idea of musl when it first came out (coincidentally, glibc wasn't having the best time in its development then), the developer's arguments why <some behavior drastically different from every libc under the sun> was akchyually correct, which was why you had to fix a program that hadn't been broken for 30 years and now suddenly was, made me put him under the "clowns" category and never look at musl seriously again.
When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.
I’m also reminded of the clowns pushing “it’s smaller!” container images based on social media vibes rather than analysis where, yes, not using Debian+glibc saved tens of megabytes on the base OS but did so at the expense of measurable performance reductions & compatibility issues like the one you mentioned, and literally every single one of those dudes I worked with was shipping containers pushing a gigabyte so the size reduction had no measurable impact on deployments.
I am sorry, a little? I am deeply disappointed that libc decided to do the same just with another sequence.
And why because some dude was complaining sshd or whatever was crashing because it didn't handle nulls? Just handle the null my guy you already know why your bullshit is crashing.
Let's be real here, the main culprit is C. People love to say "but you can still get logic errors with Rust", and no shit of course you can. But it's much less likely. You wouldn't have gotten this bug.
A little?!? I’m sorry, but musl is a joke, this is yet another proof of that fact, and I have no idea why anyone uses it in production, they have only themselves to blame after the many warning signs.
MITRE scored this as a 8.1 with a high attack complexity. [0]
I fail to see how the attack complexity of this vulnerability is high, as all it requires is attempting an ssh login with a password longer than 256 characters to an affected alpine image with port 22 open. Am I in the wrong?
In my last org I was in charge of internal testing and qualifying software releases for network equipment, and designing test harnesses and tests for these. Significant install base. All the major vendors. Many years. Basically doing the vendor QA for them, because we couldn't afford things to fail. As you do - I wish.
This was not security testing as such - mostly latency, performance and hardcore network "can we break it" and "does it comply" stuff, but also a lot of cable and module / line card pulls, pluggable and fiber breaks, random reboots and pre-empting a lot of possible operator inventiveness. At one point we considered SEU bit flip testing on rented cyclotron time.
And always we battled against the top brass urging us to speed up the process and don't bother with "really basic" testing, because we observed software quality going down over the years as the push for release and time to market stared to dominate.
So we always started with sanity testing - and that included oversized password and CLI input strings. Always. And we caught things more than once, but never like this.
This is why I don’t fully buy the argument that software that’s been around for a long time is more secure and reliable because it’s been battle-tested. There’s been a lot of time for cruft to accumulate until no-one can ever fix it because everyone is expecting it to function in the broken manner it’s always functioned. And some of the more obscure configuration options get a lot less scrutiny.
This is not an obscure configuration option. Your password file almost certainly has at least one account with "*" as the password hash.
Integration of different software is the stage most likely to introduce uncaught bugs. That is what happened here: musl intentionally broke the documented API of a function used by OpenSSH. Then Alpine integrated this incorrect implementation with OpenSSH.
I was able to reproduce this on Alpine 3.23.7 with OpenSSH_10.2p1.
Luckily PermitRootLogin [0] hasn't allowed logins with passwords for root since 2015 [1]. I'm guessing that's at least part of why this went undiscovered for so long.
Did anyone get it to work? I tried on the official alpine docker image, enabled password login for root. The shadow entry has ! as password by default, this didn't work with a > 256 character password (i also did passwd -l which also does !). I manually changed it to * and it still didn't work.
Ahh! Wasn't working on 3.16, which does indeed have an updated package.
But, got it to work on 3.15 - which requires changing the ! to a * in the shadow file in addition to other steps (making another user with a hash, setting PermitRootLogin to yes, etc.).
In Alpine, these would all be the case for an out of the box install with openssh added unless you specifically install openssh-server-pam and configure sshd_config to use PAM (Alpine is already musl-c based and doesn't default to PAM).
Doesn't it also need a user to be added with a password entry?
I only ever see Alpine Linux used for containers which don't need accounts with interactive user logins, which is not to say it doesn't exist, just that it doesn't seem like Alpine comes with all preconditions met out of the box
Isn't it the opposite, accounts which have no password and hence had * instead of a hash?
> accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.
But yeah, Alpine for persistant installs isn't unheard of - just not as popular as the container use case. They offer official VM, Xen, Pi, and bare metal images users deploy as well. I actually had a fleet of VMs to update!
Edit: The Alpine notice made it clearer to me that at least one other account has to have a password, it's just the target account that doesn't. So if you truly have no users with passwords in your container deployment then those passwordless users aren't vulnerable to this bug, but any of the image types which have a user will likely have root exposed.
> The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash.
No, didn't work for me open OpenWRT x64 running in a container but that is probably 'cause I'm using dropbear instead of openssh-server. That's what OpenWRT uses by default so the impact there will be minimal assuming that most installations don't change to openssh-server - it isn't installed by default.
On top of OpenWRT on my router (running 25.12) running dropbear instead of openssh-server, I also do not have any accounts in /etc/passwd|/etc/shadow with "*" for a password hash and an actual shell (all are set to /bin/false).
I can't reproduce it either. I used alpine:3.24.1 and openssh is openssh-server-10.3_p1-r2
I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.
Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.
edit: can reproduce after `git revert 384e25e6bd1f2e7b30095900e48b570d46953510` in aports and rebuilding openssh without the fix. It also accepts the password on the 2nd+ attempt, NOT the first.
Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem. Silent compatibility hacks are the worst, they guarantee problems never get fixed and the stack of hacks never gets smaller.
Microsoft has historically been one of the leaders on the wrong side of this one, but they have also demonstrated how effective it is to call out badly behaved software and make it annoy the users with UAC. When Vista came out, so many applications needed elevated privileges to run because they were badly designed to write user data in system locations, but now their application was more annoying to use and came with a scary dialog about it making changes to your computer that other better-designed software didn't get.
More often than not if a program didn't actually intend to change system settings it was just a matter of badly chosen default paths which are pretty trivial to fix, but so many vendors just would not do it despite years of complaints from NT admins wanting to not have to spend hours with Process Explorer figuring out specific permissions to grant limited users enough access. Once their failures were put front and center, calling their application out as the one getting things wrong every time the user ran it, they suddenly had motivation to fix it.
For security it's better, but in many cases if you're not writing security-oriented software it can be worse from a user perspective. I remember the first job I had over 20 years ago was working on VoD streaming and one of the "tricks" to improve stability was mmap'ing the 0 page so that NULL dereferences wouldn't immediately segfault. Ugly as hell but when you were working with all sorts of more or less well coded media codecs and security wasn't a concern at all (worst case scenario the user would stream movies without paying) it made some amount of sense.
I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.
More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.
That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.
> if you're not writing security-oriented software
It's hard to define what's security oriented and what is not. For example, is this HN web app security oriented? Not in the way of sshd but it must keep our accounts data separated and it must not let people exfiltrate data from the db, where for example it is storing the emails of some of us.
It depends on the time. Twenty or thirty years most software was not security-oriented. Nowadays, everything is online and therefore everything needs to be vigilant about security.
> because having a critical piece of live media production reboot mid
It's not great, but these days a trickster with AI will find some way to start broadcasting porn to a billion people because of the flaw, so may end up being the better option.
There's not much else to say honestly, for these very important events there will usually be several layers of redundancy for the most critical feeds, so the production is always able to send something to the satellite uplink even while our devices reboot due to a crash, but brutally losing 90% of your inputs for ~1 minute while the machine reboots is not a pleasant experience for anyone...
> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem.
Agreed. I feel like this attitude has been holding Python back, too. It's one thing to criticize "move fast and break things", but it's also not good to ignore already existing breakage just because the pieces still fit together neatly, and it's not good to refuse to move at all.
I'd still rather have the software crash and alert a human that to continue in a faulty state. The way to add robustness in this case is to add backup systems, not "ON ERROR RESUME NEXT".
Is there a test or check I can use to determine that software is 'good'?
Oh there isn't? Good/bad is subjective? We need to scape goating hacks as bad software, as if anyone is capable of perfect code. Perfect code would require knowing the system inside and out, the stack up and down - no one knows all that, no one is perfect. Sufficiently advanced AI though can rip through all of that, find the weakest spot and zero day it, bringing the entire house of cards down.
Software you thought was good can still be hacked. There is no way to create provably unhackable software.
Which means all software is packed with vulnerabilities just waiting to be discovered, no matter how good you think it is.
Now everything installs to a user local folder, runs as the user, there are no UAC prompts, and malware can just freely tamper with other applications, act as the user, and harvest all data the user has access to.
Hey, but at least it can’t touch the System32 folder.
- (note root won't work by default because of PermitRootLogin prohibit-password default)
- attempt twice, first attempt fails second long password is accepted (that took me a while to understand)
In practice passwd -l seems to lock with a ! prefix, and new account without passwords are created with a single ! which is also not affected, I'm not sure how big the impact actually will be in practice and don't understand the recommendation to upgrade urgently right now... Just don't use `*`?
EDIT: ah, some default container images apparently have a default alpine user with that hash, so I guess that's bad for them. This makes a bit more sense now.
> some default container images apparently have a default alpine user with that hash
Do you have an example? Looking at Alpine's advisory, I can only see that their cloud images (i.e. what you use to run a VM, not a container) have a default alpine user, no container images. And even with those, cloud images with cloud-init are unaffected with default config due to `PasswordAuthentication no` being inserted by cloud-init into sshd_config. So that leaves... Cloud images that use Tiny Cloud for provisioning. That's an instance bootstrapper authored by Alpine from the looks of it so I'm not sure how many people would prefer that over cloud-init (which a lot of VM/bare metal management software may already make easy to use) or just a non-cloud image.
Well but, alas, a non-zero number of user probably do use it, though and "upgrade X" is a clearer and better instruction to communicate than "change this setting where maybe you don't even know what it means and it might break your workflow, but saves you an upgrade".
In some contexts (e.g. embedded) you'd be surprised at how reluctant some people are to upgrade when they can run through a dozen of hoops for workarounds such as changing hash instead...
It's amazing how similar this exploit payload is to the 20+ year-old Quake 3 vulnerabilities found by Luigi Auriemma (aluigi), which also used the exact same 256-to-1024 character range and nothing more than a long string of AAAA.... Here you're feeding >256 A's up to a 1024-byte limit to trip musl's defensive length cap, whereas back in the mid-2000s, Luigi was famous for publishing one-line .cfg scripts where the entire exploit was literally "say aaaaa..." repeated a few hundred times overflowing server's memory.¹ ²
A great example is Jedi Academy, a 2003 Quake 3 engine game whose multiplayer is still actively played today (its open-source engine port, openjk, is even in the Debian repos btw). Luigi had two classic scripts for it on his site: Jedi Academy <= 1.011 in-game server buffer-overflow (jamsgbof)¹, which crashed the server just by pushing a string of as past the 1024-byte console buffer, and Quake 3 engine in-game players kicking (q3msgboom)², where a wall of as hit the engine's 1022-char truncation limit and the leftover bytes confused the parser into disconnecting everyone on the server.
Kind of wild that 20 years later, the exact same payload goes from crashing Quake 3 matches from the tilde console to getting root on some SSH servers.
Offtopic: this is a very cool way to present commits and code excerpts. Love classic Windows styling.
Were those pictures with links generated using a publicly available tool that we could use, or were it custom (vibe)coded specifically to create this page?
To be honest, I love this sort of vibe-coding. Explains it really well, in an interesting way. If this is vibe-coding, then the AI has done a great job.
It was fun to see this type of style around the release of Opus 4.6, but its really tired now. I've seen countless landing pages like this...its basically ritual
There's nothing great about this site. It should give you a succinct summary of the vulnerability right up front, so you can determine if it applies to you. The history lesson is not relevant to the vast majority of visitors.
Slop, no question. There's at least 10 Claudisms on the page, and I'm not that good at smelling them. Some of the sentences might have been human-written.
I had this classmate that had their ChatGPT instructions to respond in this same laidback, lowercase, gen-z style. i recognized the output as a lot of llm writing almost immediately
Maybe that's on you though. I enjoyed it, though the discrepancy between the `*sshole` name and the `sshhole.com` domain made me twitch. How is anyone[0] obsessive enough to find this and write it up and yet not obsessive enough to make the amount of `h` consistent?
[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.
Yes, but if you select the text and copy paste it it comes out with a single h, and also the <title> of the page has it with a single h and a double h.
<title>*sshole · sshhole.com</title>
Strange and confusing.
I mean I understand that in “*sshole” it’s referencing the * that is central to the issue and putting it so it looks like a censored version of “asshole” while also referencing that it’s an ssh hole. And the domain is more simply ssh hole instead of sshole. Is a way maybe it does makes sense to do both. But it did seem at least at first strange to me that the domain is one thing and the name is a slightly different but almost same thing. Like there was a typo when registering the domain, or like the author changed their mind about the name slightly after registering the domain.
It has very low information density. This would read 10x if it were just a few paragraphs of text. Instead we need to render an animation every few sentences. Not a fan.
I do actually. My guess is that this was not just one-prompted, but someone took the time to refine it based on their tastes and expectations until it matched something they would actually like.
However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.
I think at some point I'm going to get banned from HN for repeatedly saying this, but here we go:
This thread highlights why I often say that at some point in the future, the obsession with whether something was written by AI or not will be classified as a mental disorder.
It is tiring to virtually every day see this discussion play out in some HN thread or other. It's also just noise. I don't think your comment, nor the thread it inspired, added anything of value to the discussion.
Yes, correct for OpenWrt. OpenWrt defaults to dropbear for both server and client. However, more advanced users may optionally choose to replace it with OpenSSH, or install OpenSSH side-by-side, to gain additional features of OpenSSH.
To be fair, I have less than zero sympathy for anyone who runs SSH on any platform without `PasswordAuthentication=no` in 2026.
Its not asking for trouble, it is there will be trouble, not if but when.
As for the people who say "well, I run SSH on port 942329053052". Well guess what, the bots make short work on that. We tried it at $work as an experiment, multiple bots found it in less than a couple of hours. That's why your SSH config must be hardened, preferably running on a hardened OS too.
> Its not asking for trouble, it is there will be trouble, not if but when.
Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.
The point is they found the security by obscurity so-called "hidden" port in less than a couple of hours.
At that point, what they did is irrelevant. The point is that security by obscurity does not work in 2026.
> Because a 20 character random password is comparable in brute force complexity to an SSH key.
You can't be serious with your comment.
No its not.
And even then, SSH key is asymmetric, you're not sending the private key to the server. But with a password you are, which leaves you vulnerable to many attack vectors that simply do not exist with keys.
This is a sampling error - we're only hearing about the public disclosures.
(almost) No company actually has secret source code, only source code not publicly viewable. And the LLMs crawling those don't belong to people who make public disclosures.
> LLMs crawling those don't belong to people who make public disclosures
Using LLMs is a great way to empower the governments behind them (US and PRC) to cultivate a garden of undisclosed bugs in your closed commercial codebases.
> > Because a 20 character random password is comparable in brute force complexity to an SSH key.
> You can't be serious with your comment. \n\n No its not.
If you differ in opinion, maybe provide evidence? Especially for a claim like long randomly generated value 1 being absurdly ("can't be serious"-level) weak as compared to long randomly generated value 2
If you mean it's log(62^20,2)≈119 bits of entropy instead of 128+ and so it's theoretically weaker when we build a Dyson sphere around the sun to brute force that key (after somehow obtaining the shadow file, since an online attack would be way too slow), then sure
How is password auth weaker than key auth? Bugs in auth code can happen in both pathways. And apart from stupidly short passwords both should be equally impossible to brute-force.
In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?
Password gets sent to the server. Private key does not get sent to the server, and authentication protocol can be forwarded safely, enabling things like ssh-agent, FIDO SSH keys. This is solvable with passwords (see: augmented PAKE), but `PasswordAuthentication` isn’t that. But there are also settings where none of this really matters, yes, to be fair if that’s what you’re referring to.
> password doesn’t get sent to the server, hash does.
Kind of irrelevant what gets sent to the server and what happens on the server.
Password is already easily defeatable long before it even leaves your computer, thanks to keyloggers and/or password manager breaches and/or old-school shoulder surfing (assisted in the 2026 era by covert filming and AI).
All of which are not vulnerabilities with hardware-backed keys which you don't even need a Yubikey for these days since most modern computers and smartphones have secure enclaves and/or TPMs. (Yubikeys are still great though).
What people think of as a very strong password often has <100 "security bits" of entropy.
Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.
Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.
That makes it impossible to steal the key if you MITM, but there is still the same chance of guessing it. (Assuming password has as many bits of pure randomness, such as machine generated password.)
Worth noting that if you are running a remotely recent version of SSH it has everything you need built-in. No need for Fail2Ban on SSH these days if your config is fully modernised.
There is regular bot traffic attacking SSH, but it's not "saturate your downlink" level of traffic. If you're getting attacked that heavily then you're being specifically targeted.
Huh? I run ssh (OpenSSH) on port 22 with root login enabled and no fail2ban on a single-core laptop cpu from fifteen years ago. The load average is 0.00 even with the small mail and web server that runs on it
What cpu and especially what disk load are you talking about? Also not running out of sockets, not sure if that's RAM-dependent (the box has 2GB)
I believe this depends on your exposure level. If you have one, or maybe no, domain names pointed at your server, you are probably fine. If you have tens, or hundreds, of domains pointed at your server (i.e. you are a web site hoster), you will probably need something like fail2ban.
On another server with the same setup, there's various services and websites running, so load average is never zero there (due to scrapers) but ssh has yet to catch my eye in htop. Whenever I've looked, even the amount of cpu that a Windows VM uses constantly while idle is many times higher than what ssh uses
Working as a sysadmin, and with disk encryption the constant log writes generated from ssh was enough to cause heavy load on severs. Long lasting connections from bots also forced us to increase the maximum number of sockets, as some scan waves was enough to hit the maximum and block other connections. Fail2ban has mitigated those issues to the point where neither has been an issue.
Same issue with key reuse. Unless you have a policy to rotate them and verify which host they login from, someone's unencrypted hard drive from 5 years ago is more likely to have a key than password.
> someone's unencrypted hard drive from 5 years ago is more likely to have a key than password.
Even 5+ years ago, hardware-backed SSH keys were a thing (Yubikey). The only difference today is that its 100% easier to do keys properly because of the widespread commoditisation of secure enclaves in desktops and smartphones to support FIDO, its basically free of charge now because you're not obliged to buy a Yubikey.
Even if you don't want to go all the way to SSH certificates, SSH public-key authentication is laughably easy to do.
I also don't buy the argument "but its on a ULA address". Attacker's old favourite these days is finding a way in (North–South) and then using that as a stepping stone (East–West). Also the general concept of "lower security is OK because its on the LAN" is so 1997.
Yeah because if attackers are already finding a way in, they can easily read my ~/.ssh/id_ed25519. A password is strictly better, although not by much because lateral movement is very easy anyways. The only exception is if the private key is actually on a HSM, which is the case for all my SSH private keys on iOS.
Your comment is self-contradictory. In one sentence you say “these days” and in another you say “never, ever.” These are contradictory. What if your computer is so old that it doesn’t have a TPM?
I was already pointing out that if you have secure enclave for your private key you should use key-based authentication. That’s what I do on iOS. Otherwise a password is better than your private key in plain text on disk. So much malware steal your private SSH keys (along with your other saved passwords and browser cookies); no malware that I know of performs social engineering and tricks you into giving up your SSH passwords.
The exploit works for a user account that has no password. This is often confused with turning off password auth, especially in containers that have only one usable account.
It's not just the parent commenter. I'm sure somebody out there is going to remove their account password instead of editing their sshd_config in an attempt to "patch" this vulnerability, actually exacerbating the problem.
Bug > are password logins still on? they are by default
You > This particular bug works even with password authentication turned off.
No. The bug is related to password authentication. If you do not have password authentication then you are not affected.
What freaks some people out is they chose parameters where they do have password authentication but they expect to always fail. That's a bad choice, like OK you did fit a cheap manual keyhole, but you threw the keys away thinking now nobody can open that - you realise somebody with a cheap jiggler can open it right ?
I mean there are 65k possible ports so it's the equivalent of guessing an 16 bit password, i.e. two (extended) ascii characters. It's not very surprising that 'multiple bots' found it: a single `nmap` command will give you the open ports in a few seconds.
The terrifying decision for the API to return "*" on failure is a second-order consequence of memory unsafety, because it was a half-baked attempt to mitigate a common source of UB that people were encountering in the wild.
That was a contributing factor, but there are plenty of other instances of C APIs that make use of in-band signaling like this. That the API was changed from returning one in-band signal to another is because the original value was causing programs to crash unpredictably due to UB.
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?
Possibly public SFTP servers. I have allowed null passwords in the past for anonymous public SFTP and I run Alpine Linux exclusively for my servers both VM's and physical. The shadow package (PAM) did not get installed by default. I installed it but I had to specify it in the installation so there are likely people without it. There is probably a way to find them on shodan.io.
I will activate one for people to play around with for today:
sftp hn@ping.ohftp.org
There's no firewall and no password. Hack away HN.
Edit: Sure, a small minority presumably run sshd under these conditions. But a small minority times the number of Alpine containers running in the world gets to a large absolute number.
Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
Many, many Docker containers rely on "docker exec" or the kubectl equivalent to gain CLI access when needed. To have sshd in an Alpine Docker container, you're generally need to be running init and a bunch of demons, not just a single standalone service.
I'm guessing in practice relatively few people are actually exposed. But if they are, it's catastrophic.
I think the following should be done. Either this, or eventually DataKrash will happen. International coordinated emergency should be declared with an immediate order to shut down all non-critical internet exposed services. Critical services should be put under armed guard and remote access should be disabled except for whitelisted critical staff. Then lustration of programmers should begin. Everyone who ever coded in C(rime) language and any descendant languages would be barred from this profession at least for next 30 years. Then maybe new software will be written with due care, and any new services should be put online only when written without any C - no OS code in C, no library code in C, no C conventions, etc
Wow. I wonder which of the common fuzzers would have found this a decade ago. If literally any password over 256 chars works, that means nobody has ever fuzzed openssh on musl linux
Never attribute to malice that which yada yada yada
And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.
You have considerably underestimated both how fallible humans are, and how high their tolerance is for risks which aren't immediately visible.
This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.
This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?
True, but also, that is exactly the pretext that would allow someone to maliciously, intentionally commit a backdoor.
There was even a contest for hiding/obfuscating malicious code, e.g. as an honest mistake: https://www.underhanded-c.org/ . And what do you know, the most recent, common approach shown on the page, is: improper comparison inputs that give unexpected comparison results, which is... well... this submission.
Yes, "this looks like an underhanded c challenge" was my first thought when reading this. See one mistake by one project, inject the matching intentional mistake into another project (that also conveniently fixes some known bug), and boom - plausibly deniable backdoor.
If I had a free afternoon, I'd ask a bunch of agents to track the people responsible and their other contributions (but usually the scheme is "contribute valuable patches to build reputation, and only inject an exploit when safe to do so").
Pretty big? For this purpose, my instinct would be to select an ASCII character that's not alphanumeric, not directional, and not a common separator, and that's like, what, a set of 11 characters?
I expected let another AI slope website and instead that was a real human writing about what they found. Sure they used LLMs to find this, but they used their real fingers to write this. See, this isn’t that difficult, and it’s a lot more enjoyable to read.
Man, that author is really committed to that lowercase aesthetic. Even the entire personal website. I wonder what's going on with that? We used to do that back on AIM informally. I mostly just see Gen Z and young kids still doing that now. Pretty crazy how all the stuff that used to be trendy comes back.
Edit: My bad as pointed below this is not in OpenSSH but in musl and it appears in the relase notes https://www.openssh.org/releasenotes.html#10.6p2. I had scrolled straight to the bottom of the website because the animations annoyed me. Still would like a CVE for proper tracking though!
OpenSSH considers this to be a vulnerability in musl and not OpenSSH, but shipped a fix anyways since that helps mitigate impact... Whether you agree with that or not, it's up to you :)
I avoided using alpine-based containers ever since finding out a lot of Linux server software is actually Linux + SystemD software. The union set API of both Linux and SystemD seems to be the default "Linux API" today.
Container images rarely ship with an init service of any kind (systemd or otherwise); the expectation is that your application is the root process. This includes Alpine-based images.
That's their point. The software they want to use often assumes its being run on a platform with systemd, so running that software on Alpine which makes a point to not use systemd causes challenges for them compared to using a different distro.
Yes the other tools at the time were pretty bad, but after and around the time systemd came out, they did get much better than the sysv-init days.
Check out dinit, it's basically the same declarative philosophy as systemd, but without the bloat, and with a much simpler config API. Or s6 if you specifically care about "modular", though unless you're an embedded dev, you probably don't actually want that.
Eh, HNers are just noisier and crankier than most. The silent majority are fine with (and even like) systemd. I have no desire to go back to SysV init.
The last few weeks have made me feel very good about my policy of zero internet-accessible ssh ports, ever. On AWS, use SSM. For a VPS or home server, access everything over WireGuard.
Such solutions always felt like swapping one risk for another. With good separation (running the VPN on a separate host from the SSH login, otherwise a bug in the VPN solution still might result in a shell on the box), at best you'd need two vulnerabilities before someone gets in, but it's not fundamentally different unless you find a way simpler protocol where you can be more certain that it's secure than ssh is (for all its scrutiny and track record of no pre-auth bugs for decades)
Putting services behind ssh forwarding was the way to secure something you're worried about rather than the other way around, but Wireguard actually makes a good argument nowadays. Iirc they did proofs for the cryptography and it has good design in terms of attack surface, and it's old/mature enough now that I'll say it's probably better than exposing SSH. On the other hand, when this thing running in your kernel gets compromised, you're slightly worse off than with someone logging into a passwordless service account as in the current bug
One thing that makes me feel a lot more confident about exposing a WireGuard port is it’s completely invisible unless you have the host’s public key. Nothing will show up on a port scan.
Its security model doesn’t rely on that fact of course, but it’s the cherry on top.
A common complaint I've seen (and I experience on occasion) is AI agents being a bit silly in their overprotective coding style. They nit pick and catch every single little error. This can be frustrating since a lot of the time they are trying to handle edge cases that are so unlikely to happen, yet to cover them requires significant additional code.
This might be the beginning of the reason we shouldn't be reading the code. It's like in chess engines where the players complain about how weird, alien and unintuitive the moves are. But 100 steps down the line the engine finds some edge case exploit that gives it enough advantage to win.
So we may be crossing a line, right now, where the non-intuitive alien code being generated by agents is seeing edge cases down the line that might allow exploits. And if we "correct" the agent, we may be leaving ourselves open to exploits we don't see and have difficulty understanding.
The answer is not to reduce our understanding. That is the worst possible conclusion we could draw from what we're seeing. The answer is to increase it.
Tangent: Pangram says the text is 60-ish% AI-generated. Not that surprising given the look of the page and the work being done with AI in the first place, but it is interesting to me because this doesn't look like your standard wall-of-text AI purple prose. So it's notable that a) you can generate convincing text in this style and b) Pangram detects it anyway.
PSA for OpenWrt users: If you are getting the post quantum message on login which points to openssh.com, you are likely using dropbear nonetheless. Check in System->Software. I was confused and worried why I seemingly had the vulnerable OpenSSH installed, but its just dropbear using an OpenSSH link.
I have alpine linux on an arm device, and I have all the prerequisites for the exploit, yet I am unable to exploit myself. All of the Who is affected? from alpines security advisory applies to me. What am I missing?
The bugs and commits that "fix" them reveal somewhat of a shitshow for how musl is managed. All projects have bugs, but I can't help but question why the wheel was reinvented here. Many toes have been stubbed along the way.
Nice web site. However, I've got a server with 14 alpine-based docker containers in production. Among them 13 are widely distributed images (like web server, databases, etc.) and one is a custom build image. Only one of them have an SSH server publicly exposed, Forgejo, but it's its own internal SSH server which only accept public keys and Git commands. On the internal network, only 3 of respond to SSH. I haven't tweaked their default SSH config. None were vulnerable to this attack.
I tested an unpatched forgejo (enabling passwords temporarily, and even adding root to permitted users) and couldn't trigger this.
ETA: I could trigger it. You need to enable passwords and add a second user with a valid password, then git@ works. Obviously this still means the default forgejo is safe as you say.
Did we declare "break the world Thursday" and nobody told me? I feel like it's three weeks in a row that we've had a once-in-a-decade security issue pop up.
> except a lot of programs never check for NULL. they just compare whatever crypt() gives back with the stored field, hit the NULL, and crash
> so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead.
Considering it's been accepted that crashing if malloc returns a NULL is your fault, did musl really feel the need to accommodate bad implementations like this or is this just AI speculation?
"unfortunately, a large portion of programs which call crypt are not
prepared for its failure and do not check that the return value is
non-null before using it. thus, always "succeeding" but giving an
unmatchable hash is reportedly a better behavior than failing on
error." [1]
Such a questionable decision! "Rather than let a bug in a caller (not handling the valid null output and thus crashing) surface, let's just silence it and pretend that the caller is correct! A potential clash in some situations (resulting in this vulnerability) is better than a crash!"
Maybe I should update my plea to fail loudly (https://alejo.ch/3gk) to list this as an example.
An interesting thing about C is that there is no guarantee that NULL dereferencing will "surface". It's undefined behavior.
NetBSD and libxcrypt both have similar workarounds. NetBSD has a paragraph on the reasoning.
The behavior of crypt() on errors isn't well standardized. Some imple-
mentations simply can't fail (unless the process dies, in which case they
obviously can't return), others return NULL or a fixed string. Most
implementations don't set errno, but some do. Version 2 of the Single
UNIX Specification (``SUSv2'') specifies only returning NULL and setting
errno as a valid behavior, and defines only one possible error (ENOSYS,
``The functionality is not supported on this implementation.'') Unfortu-
nately, most existing applications aren't prepared to handle NULL returns
from crypt(). The description below corresponds to this implementation
of crypt() only. The behavior may change to match standards, other
implementations or existing applications.
OpenBSD and glibc both have similar simple, stupid-free return value implementations. OpenBSD:
The function crypt() returns a pointer to the encrypted value on success, and NULL on failure.
GNU:
On success, a pointer to the hashed password is returned. On
error, NULL is returned.
The distro I use on the desktop uses libxcrypt, though. Probably more distros are. Its, NetBSD's and musl's workarounds probably make sense in some way if you have no clue what goes into your userland and are willing to write stupid software to accommodate for that.
> did musl really feel the need to accommodate bad implementations like this
I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.
Thanks you to the team that found this. Exceptionally well explained and funny at the same time. What is most important is how people just checking stuff( remember the .xz hack that was tried?) has save the internets ass so many times. We should give good security researchers UBI for life for just doing good work.
It would be nice if the passwd crypt functions didn't have timing dependence on password length. Then there'd never need to be hacky workarounds like what musl did, and various tricks to avoid exposing whether users exist on systems.
Rust tells you off if you don’t handle Results. Sure, you can still just unwrap() it and have it panic, but at least that’s a specific decision instead of forgetting to check for NULL like in this case.
Can we all marvel at the site. The author says they did the discovery on November 3. Since then they apparently bought a domain, set up the webpage, created animations, and a cartoon. All kinds of stuff, and got it to explode on hackernews. A span of 2-3 days. How long ago was this not even conceivable?
Not the worst site, but it's clearly a publicity machine from wakefield.co, whom I haven't heard of.
Also, exploitation conditions are super narrow, albeit, any password is wild.
Can we just have a site with the text and images without the awkward scrolling crap? Hijacking basic features is an instant click away from sites for me
Edit: It seems to work fine on mobile. I'm not sure why the scrolling behaviour was added on the desktop version of the site
It's shocking that it took so long with the amount of investment and hype you'd expect this kind of bugs to be already fixed, but it still need a human to tell the agents what to do, which could also work with any fuzzing framework
Am I the only one who is shocked that such important and security-relevant software just does unorthodox stuff like "well let's just return `*` if it fails" and other kind of "hacks" ?! The developers must know that this is very bad software engineering and will definitely cause issues at _some_ point in time, no?!
Am I missing something or is this written up to seem a lot worse than it actually is?
It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
- server is aligned with 2 ley lines
- hacker is wearing a blue t-shirt and green socks
Ok, the last two are an exaggeration, but still. The conditions are all ANDed, miss a single one and you're fine.
On top of that, as someone else in these comments reminded, the target account needs to have a valid login shell or other exploitable program assigned to it, and these passwordless accounts tend to have nologin or something similar.
To me this seems to fall on the "technically possible, practically very unlikely" bucket, kinda like the PHP vulnerability from the other day. Maybe this has gone unnoticed for so long because it just doesn't happen.
How many alpine installs are running sshd? Most containers using it as a base image aren't. Embedded Linux is unlikely to be running it, too. And if one IS running sshd, how often is it with unchanged defaults? I don't buy the inference here.
I'll concede that embedded linux might be my blindspot here. Maybe cheap web hosting, too, but that seems much less likely to be using musl.
I thought Dropbear was pretty much the defacto SSH server for embedded applications, is OpenSSH+musl actually a common combination for these scenarios?
I wouldn't say the bug is only in musl. Musl is probably the "worst" offender here (at least choose something other than "*"!), but the change in sshd was also a little suspect. i.e. the "let's find a user and take their salt (and hashing algorithm) and use it to compute and compare the hash to avoid timing attacks" change.
I haven't thought it through, but I suspect in an ideal world, sshd shouldn't have accepted the user password regardless of the outcome of crypt() if the user did not exist and it only pretended to go through the motions to avoid a timing attack.
Did look into their patch yesterday(?) which seemed to mitigate this issue which they (rightfully IMHO) consider caused by musl. But it seems their fix is to do a sanity check on the supposed hashed password char length, and thus still theoretically still allowing the possibility of those "!!!INVALID_PASSWORD!!!" suggestions to fall through....
For what it's worth, default Alpine installs don't have an account with * as its password hash if you use the root account as your main one. Not sure if that's also true for ones where you set up a user, maybe those ones are vulnerable.
I'm genuinely curious what's the Venn diagram here because I feel a lot of embedded systems precisely have one condition but not the other, e.g. password yes but not necessarily multiple accounts, e.g root only, while using musl. Namely almost all for almost all, but if only one is not overlapping another then it's actually very few.
I thought they had a difference in meaning, i.e. * means the user has no password and ! means that password logins are disabled for this user, but the man page does not make it clear if there's a distinction.
From man 5 shadow:
> A password field which starts with an exclamation mark means that the password is locked.
> If the password field contains some string that is not a valid result of crypt(3), for instance ! or *, the user will not be able to use a unix password to log in (but the user may log in the system by other means).
does this really matter? It could be used to do something devastating. The number of systems that match those requirements is only one factor. If just a handful of those are highly critical for humanity it could lead to disaster. Eg. hacks into fridges, centrifuges etc.
- blue t-shirts and green socks have been added to the default loadout for new hackers, since it's optimal according to AI and costs nothing to the attacker in terms of compute
- there are so many ley line "maps" out there that basically any server placement counts as aligned with at least two, so this one's free also
Someone owns a blue shirt/green socks clearance warehouse, this is all just and ad? I've seen weirder. I can imagine visiting SanFran in a few months, and it looks like a weird sci-fi movie scene, everyone dressed the same.
Disabling sshd password logins is security 101, and the default for some cloud provided images these days. If password logins haven't been disabled, what other doors have been left open anyway?
This always seems bonkers to me. I think SSHD should give you ONE password free connection of setup to copy your key over and then it just shuts the door behind it.
Not sure why you're getting downvoted as from security standpoint that would be the right thing to do; but actual good server providers take it up a notch and allow you to upload the public key through their admin panel during server set up and the password you get only works locally for sudo etc. There's no reason why a fresh Linux setup should ever offer a publicly available SSH server with password based auth out of the box.
Alpine Linux (which uses musl) images are popularly deployed by those who use docker, and apparently the issue is there on the default install.
Not sure whether the PHP vulnerability actually had been exploited, but I'm pretty sure there are literally thousands of machines out there affected by this ssh issue right now as we speak.
This isn't going to end well.
---
Addendum (which may or may not be related to the above) --
Everyone has heard of the tale of the boy who cried wolf. The main moral of the story is not to troll your fellow townsfolks so they don't distrust you. But everyone with a sane mind gets that.
The (IMHO) under-appreciated lesson is actually: don't be those villagers. Believe stories you are given, even if you might look like a fool for believing in them. Don't assume what happens is the same as what happened in the past. Look at the facts yourself. Trust and verify.
I've done it in the past with "serverless" sftp servers, which is how a lot of inter-company batch transfers happen for historical reasons, especially in the financial sector. I don't recall what images I used, but it was far easier to patch with a CI/CD pipeline. We did use keys, though.
Probably a lot of companies which have their own k8s cluster and segregate kubectl access to operations only. Developers still need a way to their app backend in some cases.
Kubernetes already comes with better solutions for that sort of thing. I could see a ssh bastion container for bridging to legacy infrastructure though. Manage its keys right and it's probably more secure than a lot of alternatives.
I have done that in the past, in order to be able to automate copying files to the container during testing or data migration. It was an ugly hack and now I need to make double sure none of those containers are still running…
Alpine in containers is usually not affected. Most images/containers haven't ssh/sshd installed at all. Remote access is usually done from the host system.
Alpine is mostly used in containers, where you don't run sshd anyway. It apparently doesn't use PAM by default though.
OpenWRT also doesn't use PAM by default apparently, and there is a risk of people using password login, which you should never do past initial setup.
But outside specific niches, musl is rare, with good reason. Performance is a joke compared to glibc, especially the allocator. The only reason to choose musl in my opinion is for memory constrained devices (OpenWRT etc) or for building static binaries where performance doesn't matter but portability does. I consider the "small container" use case questionable: performance will suffer, and you could share large base layers instead which would amortize the cost over many different images.
Yeah, OpenWRT doesn't use openssh by default - it uses dropbear. You can install openssh, if you want more features and there seem to be 2 packages - one with PAM support (openssh-server-pam) and one without it (openssh-server). PAM-less one is the one without the suffix so perhaps it's more likely you'd install that one out of the two, but only if you were trying to switch to using openssh in the first place.
> kinda like the PHP vulnerability from the other day
What? Pretty sure Cloudflare detected widespread attacks almost immediately after it was published. We are probably lucky that not much more happened because of responsible disclosure and the large media attention led to widespread admin awareness.
It’s irrelevant how widespread it is. It’s a bad bug that should be fixed. Why do people always say stuff like this? If you’re running a system that is exploitable then you deserve to know about it and to be provided a fix. It doesn’t matter all that much how widespread it is. The CVE system doesn’t exist to cater only to highly exposed systems.
It matters how widespread it is because humans have limited time and resources to solve problems. If these conditions only exist in 0.001% of installs, is it worth the man hours to fix it? Alpine may think not, because like others have mentioned, there are likely other higher probability vulnerabilities being exposed in order to meet the conditions for this to even matter. A judgement call can (and clearly has?) be made to prioritize other work.
The good news is that considering Alpine is open-source, you can be the one to make the judgement call that it is worth fixing and go out there and fix it ;)
A trivial-to-exploit auth bypass is always worth fixing, no matter how few systems are affected right now in practice. My question is whether it's worth giving the vulnerability a name and making a webpage for it.
0.001% of all Linux installs (including containers, CI workers, embedded, etc) is still a huge huge number. It should be taken seriously until proven otherwise
I'm starting to realize that trusting some random people who wrote the gruddiest, boringest, most torturous low-level stuff might have been a bad plan.
At this point I feel embarrassed in front of the AIs.
This is a super duper annoying site for people who just want to know if their systems are vulnerable, after patching like a dozen critical vulnerabilities in other software over the last week and are almost at the breaking point of resigning.
Can we have a “don’t treat me like five/toddler” / “idgaf about the explanation” quick link to a section for adults who need to patch systems?
(And if your reaction is “just scroll down, stupid”, I’m happy to report that with the scroll jacking, I scroll not enough / past the critical sections nine out of ten times.)
Thanks for mentioning that before I tried to scroll down. Forewarned, I was able to click on the "2026" link once the left-handed "scrollbar" (not the right word but I can't figure out what the right word should be) appeared, which let me jump past most of the verbiage and get to the "are you affected?" section just a couple pages later.
What this tells me is that musl+openssh-server is an incredibly uncommon setup, as "at least 257 character string" is literally the third thing I try after "admin" and "12345" when greeted with a password input.
This AI slop write-up is so fucking annoying to read. I don't even mind the AI generated website. Why can't you even bother to spend like an hour to properly sum up the exploit yourself?
something interesting to note here is that while this isn't a memory safety vulnerability, a modern systems programming language with better error handling could have easily avoided this whole situation. error handling in C is a PITA, and we're paying the cost of decades of pretending that it's not a problem
All that site is lacking is marquee and furry creatures. AI slop sites are so horrible with their blinking animations, l33t dark background and chaotic text.
It’s dumb and negligent to be missing “PasswordAuthentication no” in 2026. It’s been so for at least ten years. OpenSSH should have turned off password auth in the defaults years ago.
Yep, scrolling through my phone and reading slowly to understand. I'm seeing distracting animations below what I'm reading. And when I go to them, there is no way to restart them. So it completely misses the point.
I think it varies from person to person. Since I'm kind of from the TikTok generation, I don't have much trouble reading this kind of animation, but some people prefer static sites. That's just how it is.
I also found scroll-triggered animations hard to read at first, but these days a lot of sites use them, so I've gotten used to them.
There is a myth that TikTok generation has ADHD and can't concentrate, if it were true I'd expect static sites would be easier for you, like to me. I mean, I struggle sometimes to keep attention and not to wander off mentally. With static text it is easier to fix, you have just to jump back a bit, when you notice that you've forgot to read while you were reading. Or when you are lost at too many trees you can kind of look at the forest, at the text as a whole and find your place there. But with all this scrolling it is hard to jump backward/forward at will. It feels tiring and annoying.
> how did I ran into it? [...] i had agent swarms. a lot of them. let's not get into it
Can we please now normalize AI usage in development? Just some other day people asked for "NO AI used" badges for software as measure for quality, and praised i.e. NetBSD for being AI-free. Yes, we come from a phase of AI slop, but it has become obvious that there are plenty of legit usecases where AI beats human-only effort nowadays.
Not sure why but a lot of people seem to be missing that you don't have to look at AI as a binary "you're either with us or you're with the terrorists."
Just do HITL (human in the loop). Use an LLM to help you author, but steer the AI in whatever direction you want. It's like having a slightly delusional copy of yourself to pair program with. If you take the time to provide an LLM with some additional skills files and the like, with you in the pilot seat, you can get solid results (it doesn't look as cool as a hands free agent setup but the quality far surpasses anything they can do solo).
> Can we please now normalize AI usage in development?
1. It is already normalized outside of a tiny, but very vocal bubble that lives mostly on Bluesky (sometimes also on Reddit/HN/Mastodon).
2. I don't think that even the most staunch AI hater would object to using AI for security research as long as all the artifacts (code, docs) that were produced based on the research are human-written. "Yes, there is a bug in my program, but I refuse to fix it, because I know that you found it using AI" would be a pretty weird position. Not sure, how would you defend it.
I feel like there is a lot of pushback from delusional engineers, that won't accept their coding skills have been rendered useless - so they come up with tons of reasons to avoid AI, because it just has to be bad. Anything to hold up the ilusion you are an well-paid white-collar worker that can pick employers anytime. And I say this myself coding since the late 90s and making a living off it - it is a painful pill to swallow, but the genie does not go back in the box so just accept it and move on.
what was done to report & patch the vuln? Why publish it in this format? It’s presented like a whistleblower who’s exhausted every approach, but the effort to do the right thing is not clear.
> on alpine's official aws and azure images that's the default alpine user, and it has passwordless doas. so it's root
good fucking god. if I get this correctly... you can insta-own any aws/azure instance that was set up with alpine as long as it has ssh enabled, which it probably has.
Can we hear something from those who say that using port-knocking is "security through obscurity"?
Really: from the XZ backdoor in SSH (which went live on some distros) to this...
What's you rationale for calling port-knocking "security through obscurity" and not "defense in depth"?
Put it this way: if I advertize that my server is using port-knocking, that the sequence changes every 30 seconds (like TOTP) and that as soon one sequence is used, it's unusable (so you've only got one attempt to SSH in every 30 seconds, even if you know the sequence)...
Where's the security through obscurity?
But really: even if had musl and PasswordAuthentication allowed (I don't have neither), how would an attacker gets through the port knocking?
More fundamentally the question is: in which way is your system that does not use port knocking more secure than the one who do?
(remember: we're talking about remote exploits by simply having exposed TCP ports open or SSH listening)
Wanna have a talk too about the "solutions" proposed by "security experts" here "because SSH is too old" and that happen to be crufts on top of SSH that had remote root exploitable backdoors? (sorry, holes)
Disclosure: I work at Detectify. A few practical notes for anyone checking their own machines:
- It affects Alpine's openssh-server (built without PAM), or openssh-server-pam with UsePAM no, when PasswordAuthentication is enabled. Alpine says every release they checked is affected, 3.16 through 3.24 and edge.
- Fixed openssh packages are out for all of those, including the older 3.16-3.20 releases. Upgrade and restart sshd (rc-service sshd restart). If you can't upgrade yet, set PasswordAuthentication no.
- To check a box: apk info -v openssh-server for the version, and sshd -T | grep -iE 'usepam|passwordauthentication' for the effective config.
- Containers built FROM alpine that run sshd are easy to miss in an inventory. So are the official tiny-cloud images (3.18.11 through 3.24.3), which ship an alpine user with no password and passwordless doas, so a bypass there means root.
- After patching, look in /var/log/messages for sshd lines saying "Accepted password for" an account that has no password.
OK, I think I see what’s going on with this, in this configuration.
The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
I think the conclusion that can be drawn from this is that the behavior of crypt() is howling-at-the-moon insane, and it should no longer be considered for any system built in this century, at least not without extensive wrappers that correct and normalize all these edge cases.
Conclusion is type safety is important
In this particular case, it is type safe. NULL, by the C standard does not refer to an object. It means that every pointer in C is effectively an option type, which mean that a function that returns a pointer can return NULL for "no result", which, in the case of crypt() is perfectly adequate.
It is the opposite that is unsafe, that is, when the returned value can't be NULL, but it stills returns a pointer. C++ has references in an attempt to fix this problem.
Because C tends to prioritize performance over safety, a NULL pointer dereference is considered undefined behavior and the compiler can invoke nasal demons if it makes the code go faster, but with the right options, you can have it checked and invoke what in other languages would call a panic handler, but on UNIX is the SIGSEGV signal. You can then attach a debugger or read the core file and see what went wrong.
Note: I am not talking about "char *" for strings, that one is unsafe, but NULL is not. It is just a special case of an option type, or a result type, or an error union. Modern languages just extended this idea to objects other than pointers, and with more than one type of error.
The implicitness is exactly the problem. The workaround that caused this vulnerability was done because people assumed that crypt() would never return null.
You don't even need a fancy result type. A simple Option<String> would've forced downstream users to consider the failure case, avoiding this situation in the first place.
Exactly, whoever program in C knows that every function that returns a pointer may return NULL (even if not documented) and thus every pointer must be checked.
However a better API (to me) that I would have written was:
That way btw the function doesn't need to allocate memory inside, which is (being a mostly embedded programmer) something I don't like at all. Plus returning an error enum makes it clear instead of having to look at errno.But POSIX API is not something well designed, it dates back to the '80 so... probably today would be written completely differently
In theory, having newtons and pounds force be represented as `float` is type safe. In practice, it destroyed a multi-million-dollar NASA spacecraft because of human error.
If NASA were to use the equivalent of `pub struct Newtons(f32); pub struct PoundsForce(f32);`, where the types Newton and PoundsForce were not interchangeable, this would never have happened[1].
[1] I'm aware that that story is a bit more complicated than that, that CSV files were involved (which are, again, not type safe), and that, in that particular scenario, more would have to be done to preserve type safety across formats, vendors and codebases, but my point still stands.
I think related to your caveat is:
https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-...
> The newtype is opaque, so the only way to observe it is to convert it back to a [f32].
> It is just a special case of an option type, or a result type, or an error union.
These are not the same, and your next sentence suggests why. Option has but a single error type, but result types and error unions allow arbitrary types of error.
> In this particular case, it is type safe
That depends on how you interpret the pointer type, but in any case there are multiple interpretation and just the fact that these exist will create confusion and type unsafety.
> the compiler can invoke nasal demons if it makes the code go faster
I had never heard that phrase, and was delighted at the answer when I googled it its origin, thank you:
http://www.catb.org/jargon/html/N/nasal-demons.html
> “When the compiler encounters [a given undefined construct] it is legal for it to make demons fly out of your nose” (the implication is that the compiler may choose any arbitrarily bizarre way to interpret the code without violating the ANSI C standard)
However this is limited by the compiler's actual abilities and code. The C standard doesn't forbid nasal demons, but the compiler isn't programmed to invoke nasal demons and isn't running on hardware with that capability, so it won't happen.
I've heard nasal demons invoked as a literal possibility. No they are not a literal possibility.
Maybe your compiler isn't, but GCC is designed to summon every nasal demon it can. It especially loves to elide null pointer checks. There was a PR that almost made it to Linux mainline that did some weird stuff in kernel space where memory mapped to address 0 is valid and contains important stuff. It got a pointer, read data from it, then checked if pointer is null before giving root permissions for some operation. Reading the code, everything seems alright. Even stepping through it with debugger shows that it works. But when compiled with -O2 or higher, GCC removed the null check and now everybody got root permissions unconditionally - because reading from null pointer is UB regardless of circumstances, and GCC treats UB as dead code even if it is reachable.
> GCC is designed to summon every nasal demon it can.
Which is zero, to be clear. There's not even a standard USB device class for a nasal demon releaser
A null check elision is not a demon up my nose - it's a null check elision.
Sure, and char[] is no string because you can't tie anything with it. And don't get me started on the insects that allegedly live inside digital source files - that's an absurd concept, no self respecting computer specialist should be talking like that.
Having made devices for other human holes, you can just use the standardized HID class. It can be adapted for nasal demons incoming and outgoing.
Wait, so if you're running on embedded, you're not allowed to store anything at address 0, requiring you to waste that little bit of RAM?
That's... a bizarre choice, considering how religious C is about not making any assumptions about what a machine can and cannot do, even going to the trouble of working with non-IEEE floats and int types that are not a multiple of 8 bits.
Unfortunately yes. ISO/IEC 9899:1999 footnote 83: "Among the invalid values for dereferencing a pointer by the unary * operator are a null pointer".
You might consider it type safe according to C's rules, but it is not type safe according to contemporary opinion because the success type and error type are the same.
> [...] because the success type and error type are the same.
That's a bit weird? In Rust, if you have an thing that returns Result<String, String> that's still type safe, even though success and error types are the same.
I think I understand what you want to express, maybe. You might say it's in-band vs out-of-band error signalling?
> In Rust, if you have an thing that returns Result<String, String> that's still type safe, even though success and error types are the same.
These types are not the same:
Those are not types at all. Ok and Err are type constructors for the Result<A, B> type that take arguments of type A and B respectively.
> type constructors
Nit: the type constructor is `Result`: given two types `A` and `B` (not instances of those types, but actual types) it gives you a new type `Result<A, B>`.
`Ok` and `Err` are called "variants" and can be used to construct a new _instance_ of `Result<A, B>`.
Contemporary opinion among who? Words mean things, and "type safe" simply does not mean "agrees with my preferred style of error reporting"
Papers on type soundness from the last 30 years usually explore the domain further than C semantics.
Soundness means that the program behaves in accordance with what its types express. You're right that C may be unsound due to undefined behaviour, but I don't think this is about soundness. What you're saying is that papers on types show how you can use types to express more constraints.
There can be other valid perspectives and definitions of that word than yours.
I think almost all (tech) arguments stem from simple disagreements and/or different assumptions made on the definition of a (subjective) term.
You may be right in general, but soundness comes from formal logic, where they're pretty fastidious with their definitions. Soundness certainly doesn't mean the expressivity of the types (i.e. what constraints they can represent); it's about the relationship between the runtime behaviour of a program and the types in the code.
Again I think you're trying to push a specific definition as more correct than OP's somehow. Why not just ask them what they meant to understand their intent/context instead of saying they are wrong?
Unless the point of your argument isn't to understand OP but simply to "win."
I think I understand their point and even said, "What you're saying is that papers on types show how you can use types to express more constraints," and that's correct. I merely pointed out that that's not soundness, which is something I think they'd like to know. There may be some leeway in the definition of soundness and how it's similar to or different from type safety, but either way, type expressivity is something different altogether (and C certainly has issues with both expressivity and type safety or soundness, but they're different issues). So the point isn't to win - the argument was right and I'm not claiming that C's type system is expressive - but to explain the different terms and their meaning in programming languages and formal semantics.
Can you link some?
That's not the same as asserting that soundness requires adherence to the current fashion, sum types over results and errors. TIMTOWTDI.
You don't have to think C is sound or good to distinguish a disfavored API style from mathematical unsoundness. There are infinitely many type safe APIs and languages that are, despite their soundness, ugly and hard to use.
Let's not project every dimension along which a language can be good or bad onto the axis of safety.
Opinion among people who know better than to try to authoritatively explain subjective words with rigid dogma, because that just makes them look silly.
My thoughts exactly.
"Make invalid states unrepresentable" is such a useful adage when designing even basic systems, much less the gates to control your server.
My conclusion is that an old tradition of not returning NULL in crypt as a workaround for buggy code which doesn't handle errors was a bad idea.
The idea of trying to provide a workaround for buggy code calling crypt() is terrifying. For sure, trying to grease the skids for buggy code calling printf(), go for it. But it is hard to imagine cases where slightly-incorrect usage of crypt() is a good idea.
This is still a consequence of a poor type system.
I don't think so. Even in a language with a good type system like Rust, somebody could have made this API:
Then, in a later version, `crypt` needs to become fallible, so they need to signal errors somehow. But because they really want to preserve backward compatibility, they can't change the output type to `Result<..., ...>`. And so they end up doing the same thing that was done in libcrypt - they output some error value and hackily make sure it's not equal to the `setting` string, under the mistaken assumption that the output of `crypt` would only ever be compared to the `setting` string. And then someone creates the exact same bug.Maybe I'm wrong, but it looks to me like the exact same incentives and means to screw things up exist in C and in Rust for this particular bug.
IMO type safety requires both language features and a mindset where you take efforts to keep things type safe. Your example has only the former.
For example, in a Rust web app, you can choose to use String for encoded URLs, unencoded URL fragments, escaped html, unescaped HTML, usernames, etc. Or you can choose to use newtypes and carefully design all APIs to only take and return the correct newtypes.
You are looking at it from the wrong direction.
The problem is not that you can write bad stuff in rust, the problem is that this particular usage pattern can't be sanely coded in C. This is a public API that should inform users and enforce certain properties. A Results type does exactly that, you would have to go way out of your way to force invalid stuff through that type alone.
I don't think so. In both Rust and C, the solution is the same - make the function explicitly fallible, instead of trying to preserve backward compatibility. In C, the function would probably just look like this, returning zero on success and negative on failure:
You can call such a signature "insane" if you want, and that's fine, but it would have averted the bug that is the topic of discussion.what you are suggesting is a form of type safety. There are still degrees of type safety even in C, and this bug resulted from deciding that backwards compatibility is more important than type safety, which I think we agree on.
And C has effectively deprecated bad APIs over the years. Just not this one.
It's a perfectly sane signature, especially coupled with an API surface that uses the return channel for errors generally.
This isn't good either. Crypt is a pure computation that can't fail for arbitrary reasons. The former implementation could fail if the salt was invalid, which is something you could guarantee. Now it also fails if the password is invalid. That's new behaviour and no matter which way you wrap it up it has the potential to be incompatible. They should have added the maximum length recommendation to the man page, and let DoS attacks continue until the calling software was updated, not the crypt library.
OP suggests it was always fallible, they switched to a sentinel to mitigate callers dereferencing null, which they wouldn't have with Option/Result. https://git.musl-libc.org/cgit/musl/commit/?id=b3c4cc12
But in your version, you can't preserve backwards compatibility, making an infallible function fallible is a breaking change. For something comparable to libc you might want to (eg Rust std still has mem::uninitialized), but even with security issues, I feel like people would be more inclined to deprecate for a new function (or panic, which does have its own problems).
A good type system also means things are written with better forwards compatibility, and can be safely changed later. The Rust equivalent to `crypt` likely wouldn't exist: a low-level library would just provide hash functions, leaving callers to mitigate DOS etc; a broader library might return some opaque handle with equality/serialization, which you couldn't try to compare to a passwordless user; a high-level library would parse the account store itself. None of these really need to change in incompatible ways, they have all the information to add features and fixes at their own level of abstraction.
Yes, even in Lean you can just refuse to prove any properties.
But if you have a stronger and more expressive type system, you are more inclined to use it, instead of looking for silly workaround like in C.
You wouldn't be concerned about breaking backward compatibility with rust though, would you? A recompile would be needed anyway and the change would be detected and enforced. The rationale for the bad API here (two types in one variable) was that users were ignoring NULL - that's the part which wouldn't happen in rust.
Maybe he meant binary compatibility.
They would have to make it crypt2.
Specifically, that the lack of tagged unions (or really, structured return types in general; you can pretend those are a tagged union if you believe hard enough) in C library interop conventions is a terrible foundation on which to build widely-used APIs.
And that just ends up indicting the C language itself. Can't send errors down a separate channel; folks might not inspect errno. Can't return NULL on error, folks might compare it blindly to another result. No tagged unions, so can't return one. Can't crash on error because there's no scoped error handling. Can't return a struct for checking .error/.result separately, because alignment and equality (and tearing and accidental clobbering and more) make that brittle as hell for widely-used APIs.
C is not a good language for library APIs. Like, really not. It has its uses/strengths, sure, but without even getting into the typical gripes people have about C, libraries--one of the things that it is used for the most--are one of the things for which it is the least well-suited.
"But it has a stable ABI!" So does a land mine.
you can return a struct just fine... most API's don't though.
Historically it didn't work well between different compilers. Maybe it still doesn't.
Passing and returning structs by value has been part of C since C89. Most compilers pass small structs in individual registers, and nowadays with LTO they'll even elide the copy of bigger ones if they can. What's definitely the case though is that by-value structs weren't a common thing when most of the Unix API was written.
Structure assignment, passing, and returning were added in UNIX V7 C.
That and in-band signalling is error-prone.
Another example that C programmers love: using 0 to mean infinity. Eg when you have a timeout setting, and 10 means ten seconds, but 0 means infinity seconds instead of zero seconds.
In general it's a problem of sentinel values. Some languages let you abstract this - I think in Rust and C++ you can define an "intrusive optional" which is like optional but uses an invalid value pattern to store the not-present state. In memory it's the same as a sentinel value but the compiler forces you to unwrap it like an optional.
I'm not sure how well the 'forcing' works for Rust and C++. You can definitely do this kind of thing in dependently typed languages.
The type is akin to optional<int>. You can't add 5 to it, you have to dereference it first which crashes if it was null (0 or another chosen value).
Amusingly, Win API of all places generally evades this pattern: 0 timeout means "no waiting", and to wait indefinitely, you use timeout of INFINITE which is actually UINT_MAX or, equivalently, -1 (in Win API, timeouts are unsigned, not time_t).
That’s just a very bad API. You could do the same in Rust? There are nice C APIs that would use a nice struct that describes the timeout you want… eg GCD.
I'm not sure a struct is the right thing here in general. You often want an algebraic datatype.
What Rust calls an enum, but an enum with payloads. In C terms it would be a discriminated union.
Is really the solution to a bug where a function had a wrapper that tried to solve its documented issue of returning *, to make yet another wrapper to try to solve the documented issue of returning true?
But, but 'always "succeeding" but giving an unmatchable hash is reportedly a better behavior than failing on error.'!!
Caveat:
> $ret = crypt($pw, $hash); $ret == $hash
musl is actually safe against this use. OpenSSL takes a bizarre code path, detects that $hash is too short, calls crypt($pw, $other_string), forgets that it detected that the hash was too short, and checks $hash == return value.
See my other comment.
I did note that musl would normally work just fine here. Though I appreciate that your other comment goes into much better detail about what was actually going on in the code, as my analysis basically stopped once I understood the high-level flow of what was going on.
Except OpenSSH qould have worked just fine if musl behaved correctly. Instead musl ignored the spec and tried to hack around badly written programs but ended up creating a security hole in a correctly written one!
This is a vastly clearer explanation than the linked page. Thank you!
Yep, the linked page doesn't /feel/ AI written (maybe it's using a grok or muse model whose tone I don't quite recognize yet?) but it kinda just states a lot of facts that make sense on their own but don't cohere into a whole vulnerability like the comment above does.
I think it's just Claude prompted to write in a casual, conversational, no caps style.
The surface-level hints are gone, but the attempt to make every sentence pithy, the decisions on what information to present and what to assume the audience knows, and the over-emotionalization are all still there.
It feels very AI written, it's just not the blatantly AI default style. Pretty much any prompt with a "write X in this style: [example text]" will produce writing like this.
Of course it is. "one debian box, one alpine box" is an obvious Claudeism.
and what same human would embed two linux systems (??) in a blog post just for a demo that prints "pass"/"fail" that hardly anyone will bother to interact with?
The demo is obviously vibe-coded, but that doesn't necessarily have to mean the prose comes from Claude too (although in this case it clearly does).
Because Claude wrote the linked page. Claude is impressively bad at communicating these days. with gpt-5.6+ approaching Claude level like 30% of time and Gemini much clearer but also the most annoying 7 year old ever, it’s a pretty lame environment especially for people who r new to everything
I wondered about this myself as the explanations and documents I have been maintaining are becoming increasingly 'slop', but a different kind of slop.
In the past I could get an explanation, where I was able to track the line of thought. Now when I read through the explanation it is like someone has taken two or three explanations and merged them into each other, like there is a combining of iterations without clarity.
example:
The tree is blue.
The tall tree has the color of blue.
the tree, while tall, is blue.
That finally gets squished into: the blue tall tree has height and is evident of blue.
Is this double speak or injected double effort of words?
Until Opus 5.0 I'd have agreed with you, but Opus 5.5 is honestly one of the best I've ever sparred with, vastly better than the rest, including Astra imho. Especially so when prompting for clear top-down, first-principles communication.
Disagree, I review a lot of Opus 5.5 code and its comments and docs are still pretty inane and Claudish. Is it better than the 4.x and 5.0 era, sure. But it's still nouning the verbs, using colons everywhere, a-not-b, "a foos the b" etc. A lot of lingual knots that need a human hand to untie.
I was using opus5.5 for everything but since I tried google Argon4 that has been my favorite for writing. It’s clear and to the point.
Opus 5.5 is much better, but IME I always have to force it to review its writing (Opus 5.5) to get it to put the most important facts first (and not bury the lede). It loves to park the main point far away after intermediate clauses. It always writes “premise, so consequence.” It takes a prompt asking to review against rules in context for it to fix it.
The ‘concise’ output style setting is bad with Opus 5.5, as it will often spit out a cryptic (is this really English?!), allergic-to-commas concise response, which takes more time to parse.
Oh it loves to bury a lede.
Then 3 back and forths later, your computer explodes, and claude has the cheek to point out when questioned that it did warn you.Some of the page is also just wrong.
Except it turns out that at least one of those things is because I didn't enable JS, and the static text is wrong!
(Turning on JS did not actually help, because then the page is filled with horrible, cheesy animations and I can't actually read any of it.)
Thank you for explaining it with proper capitalization. I understood your explanation much better than the author's.
> musl can return “*” for errors
wtaf ?
There is no doubt in my mind that this was intentional.
I agree. The “excuse” for implementing those behaviors is just providing plausible deniability. The commit that lets crypt return “*”, followed closely by a change that lets you easily trigger that behavior sounds just like how somebody would sneakily introduce a back-door into tons of linux systems while making it look as innocent as possible.
Well, it was intentional, but not in the way I think you mean.
musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
That was a misfeature and should never have been implemented. But I bet it was an easy band-aid change to make!
But choosing "*" as the character to return is an insane choice for anyone who knows the basics of linux sysadmin, or hell anyone who's ever used a search field in any context and thus knows it is ubiquitously used as a wildcard operator.
But it fits the common Linux implementation of the function... e.g. the Debian manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an invalid hash to the _output_ field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
It's almost sensible if you only think of one side. It's like if /etc/passwd put "NO_PASSWORD" in the field for disabled logins and someone was like, oh, I'll be clever and say crypt() returns "NO_PASSWORD" on error because that's obviously invalid and won't match. But that's exactly what *does* make it match.
But this also needs the "ddos" patch that was submitted later as well.
That seems orchestrated.
Also, from the patch author:
https://lists.llvm.org/pipermail/llvm-dev/2019-June/133308.h...
> musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
Of all the things to mitigate...
1) crashing should be an option - it noisily indicates "there's a problem" and if it faults and dumps core, you've got a guided tour of your problem.
2) Is testing NULL as a passable option likely? I'd guess a more likely idiom is:
> crashing should be an option
Wise words. I think early in my career I tended towards coding as defensively[1] as possible in a very specific way of "don't crash", as I definitely was not working in codebases I felt I had a full picture of. "I don't know who calls this method, and I don't want to get a call in the middle of the night that I broke prod" so I would say, return an empty string when asked to format a datetime except I'm passed a string or an int. (Yes, I was working in dynamic, untyped languages, too).
Later on, even in untyped languages, I came around more to "Throw loud and specific exceptions when asked to do things that are nonsensical" - it requires more overall cojones though because it may cause "your code" to crash. But the realization I came to was that it's better to crash ('a MaliciouslyLongPasswordError was uncaught') because "AuthManager asked PasswordService to check a password that was 2MB long" than to paper that over until something like this comes out. Or, the more common situation, you're passed garbage, you return something that won't choke the caller at the time, but what it saves to the database in that scenario ends up being corrupt data. 6 months later someone notices, and has to sort out a data mess that may or may not be recoverable.
[1] note that most "code defensively" advice is good. What I was doing could be thought of as a foolish misinterpretation of the phrase.
FYI, your learned approach is also known as "fail fast" methodology[1][2].
[1]: https://www.martinfowler.com/ieeeSoftware/failFast.pdf
[2]: https://en.wikipedia.org/wiki/Fail-fast_system
“A good offense (SIGABRT) is the best defense(ive code).”
I dig it.
I think the issue is that a lot of callers were not aware that crypt() could error at all and all the issues are downstream of that. If OpenSSH knew that crypt() could error they would have handled it (possibly by inspecting errno).
I don't think that's the problem. The man page for crypt clearly says that erroring is a possibility and that it's not standardized how it errors:
> The behavior of crypt on errors isn't well standardized. Some implementations simply can't fail (except by crashing the program), others return a null pointer or a fixed string. Most implementations don't set errno, but some do. POSIX specifies returning a null pointer and setting errno, but it defines only one possible error, ENOSYS, in the case where crypt is not supported at all.
In my opinion the problem is that the behavior of crypt is underspecified - it isn't even specified how it signals failure, which is shocking.
Another disturbing decision that crypt's signature takes in the hash its output is supposed to be compared against - if it weren't for that, this half-baked "on error the output won't compare equal to the input" measure that crypt and crypt_* use wouldn't have been possible.
You can't check errno to see if there was a failure because, as the man page says, the value of errno is unspecified when crypt succeeds.
...unless crypt() is one of the many, many functions which themselves internally call functions that set errno. In which case you can't be sure.
As opposed to the false positives returned by comparing '*' to itself.
Why * specifically? Why not some random guid generated before committing, if it's so important to not let buggy software crash?
I'm agreeing with you, to be clear.
But buggy software might not crash; it might compare NULL == NULL and misbehave, instead.
So there's really no good option here, which is why C is a rancid choice of language for publicly-usable APIs. But we're stuck with it. So it goes.
I've been programming in C since 1978 and never until now heard of a function returning "*" on error.
This specific manifestation of "C is terrible for shared code APIs" is rare.
But issues caused by folks forgetting to check errno, miscomparing NULL to NULL when calling multiple APIs, dereferencing external-API-returned NULL due to missed error checking, mishandling inherited errno after successful action, or--the most common--mishandling negative integer return sentinels meant to indicate errors? Those are incredibly common.
This specific library call is quite poorly designed.
Standard C api calls return an integer, which is 0 on success or nonzero on error. Ignoring it is an intentional choice. `errno` is the hack that shouldn't really exist because it's trivially easy to forget to check that and also has issues with threading (patched by making it a thread local) and more recently, coroutines (which admit no general purpose patch because they're a userspace construct).
Another reason why this is terrible is that it's unclear what the returned pointer's lifetime is. Is it until the next call to crypt? If so, is it thread-local like errno? If it's passed on to the user, is it safe to call free() on it, or is there a crypt_free function that should be called instead?
But all of this, and I mean all of it pales in comparison to returning '*' on error.
> But all of this, and I mean all of it pales in comparison to returning '*' on error.
100% agree. This is an egregious failure. But C's error-handling story would be bad even without this, and the badness of C's error handling is what motivated the (bad) decision to return *.
It'd be great if we wrote systems software in a language that supported advanced features like...
checks notes
...returning multiple values.
Hell, even in VBA you had to at least spend the keystrokes to type "on error resume next".
It's just that every other string than "*" would have been a better choice, including even the empty string (which would have been the next terrible choice).
It's not a C thing, it's a Linux thing. The asterisk is relevant to how crypt() is used for /etc/shadow.
No, it's a C thing. They previously returned NULL for invalid passwd lookups, but because there's no way in C to force API users to decide what to do if an error occurrs, that was causing program crashes due to NULL dereferences.
So they made the (very bad, but motivated by C's broken design/conventions) decision to return a magic string instead.
> but because there's no way in C to force API users to decide what to do if an error occurrs,
0. There are tons of libc functions that return NULL on failure, it is conventional way in C to indicate failure.
1. There's no much to decide, just fail to login if crypt() fails with NULL.
2. errno is standard way in C to check error code of a failing function, but in case of login you hardly need this, just fail!
I totally agree that fixing this issue in musl was a pretty dubious solution. Don't fix what's not broken, fix broken application instead. Publish a CVE if you cannot fix it.
I looked through FreeBSD source code for how they use crypt(), it's checked for NULL everywhere!
UPD: Except libpam. :(
Nothing you said is incorrect, but I believe they meant that there’s no way to force a C program to check the NULL correctly—the compiler will happily compile code that gets a pointer back from a function but doesn’t check for NULL before dereferencing it. In something like Rust the type system won’t let you touch the value until you’ve done something to deal with the error. Even if you just want to ignore it, you are forced to acknowledge that the function is fallible by the type system—it’s a compile error if you don’t.
Returning NULL on error is actually an excellent choice because most programs will crash if they try to use it without checking. A program, any program, crashing is better than this bug. Of course, modern programming languages have better constructs, but this was rare to nonexistent when C was started and if you think C is going away anytime soon you're delirious. musl's decision to change the return value to "*" is ununderstandable and likely intentionally evil.
> musl's decision to change the return value to "*" is ununderstandable and likely intentionally evil.
Dude, they documented their rationale. It was dumb and wrong but totally relatable. Irresponsible users were getting crashes, and I’m sorry to tell you that for a lot of library maintainers
> Returning NULL on error is actually an excellent choice because most programs will crash if they try to use it without checking.
…is not something they want to write in responses to bug reports, even though it is quite true. It’s human nature to want to prevent your stuff causing problems for others, even if the problems are fully their fault. Most people cannot and will not be Steve “so don’t hold it like that” Jobs, for fear that their decision will be just as short-sighted as his was.
In this case, that was obviously the dead wrong call. But I’ve been in that seat, and “closed: user error” for the 100th time feels bad to write. Add to that a touch of misplaced cleverness, and you get “hey…what if we just returned an impossible string instead of an error?”
Evil requires motive or malice, and I’m having trouble coming up with those in this case. These patches weren’t concealed or miscategorized a la xz, the behavior was discussed in the open. They were just, in hindsight, really poor decisions made for ordinary reasons. Hanlon’s Razor and all that.
That’s why I’ve been harping on C being trash for shared-code APIs in these comments: bad decisions are gonna happen, and irresponsible users are forever, so can we maybe use tools that require you to put on a seat belt (mandatory error handling) before starting the car?
> That’s why I’ve been harping on C being trash for shared-code API…
Agreed, but I can’t think of any other widespread ABIs that are better. Rust’s is not stable both C++ and Rust have concepts that don’t map into other languages well, so using them wholesale is a no go. So it would probably need to be a subset of those. Honestly just tagged unions would probably be enough. Maybe the solution is to get native tagged unions into C…
> “closed: user error” for the 100th time feels bad to write.
That's the job of a OSS library author.
Now imagine every other C API that can potentially return NULL did the same thing. Is that in any way sensible?
You can argue that C has broken design/conventions, but the musl decision was really really awkward.
This sounds like a program problem not a library problem. Why did the library try to solve it?
Oh, it was magic alright.
C programs wouldn't accidentally compare NULL to NULL. They'd either check for null explicitly, or they'd pass the null to strcmp and crash.
Yep. See: underhanded c code contest
https://www.underhanded-c.org/
Of all the possible strings this seems like the absolute worst choice. If I were typing that string I'd pause to think "wait, what if the caller uses this with a regex?"
One has to remember the context here. The only error possible according to the SUS is ENOSYS, literally that crypt() is not actually implemented in the library. The traditional AT&T Unix crypt() was always implemented, and its encryption did not have failure modes, so there was never any error return to handle. This is why programs were written to never check the result of crypt(). It was only ever NULL on systems where one couldn't use the mechanism at all in the first place.
So the knock-on effect is that libraries that do have errors other than ENOSYS to accommodate, have to return magic 'impossible' values instead of setting errno and returning NULL, because of the weight of existing applications that expect nothing but the standard, traditional AT&T Unix (and indeed old BSD), behaviour.
Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Also check FreeBSD's OpenPAM for a case in point. Even today, it does not check for a NULL return from crypt() and leaps straight to strcmp().
* https://cgit.freebsd.org/src/tree/lib/libpam/modules/pam_uni...
If you want to see real history here, check out the 1992 comment by Eric Andrew Young, that is still in the source for OpenBSD today (in des/des_fcrypt.c), on how SSLeay's crypt() had to be adjusted so that any single character password field, including "*", did not end up successfully matching itself.
* https://cvsweb.openbsd.org/annotate/src/lib/libcrypto/des/At...
It's worth pointing out that musl tried to do the same as SSLeay, and deliberately returns something different to the original salt so that they will not compare equal. It does not return "*" if the input is actually "*" in the first place, contrary to the headlined cartoon explanation. (The locus of the problem is the caller calling it with a salt that is not "*" and then comparing it with "*".)
> Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
No. https://man7.org/linux/man-pages/man3/crypt.3.html
My guess is that you are reading the libxcrypt crypt(3) manual: https://www.mankier.com/3/crypt
That's not GNU libc, but frequently deployed on Linux systems.
Well caught. I was reading what man -s 3 crypt brought up, which mentions GNU and -lcrypt and only right at the very bottom says something else.
* https://manpages.debian.org/trixie/libcrypt-dev/crypt.3.en.h...
Although that's a fine distinction for the point being made, as it's still the manual page for the crypt() function that one gets on the operating system, which that other manual page is not.
That naturally depends on the operating system and the crypt implementation they include.
> Also check FreeBSD's OpenPAM for a case in point. Even today, it does not check for a NULL return from crypt() and leaps straight to strcmp().
Damn, that's a bug! Please file bug report.
I suggest finding a code path where FreeBSD crypt() ever returns NULL, which would mean one of its internal functions returning non-zero, before getting too excited. They all end in return (0) . (-:
* https://cgit.freebsd.org/src/tree/lib/libcrypt
As I said, the callers of crypt() have for a long time been using the knowledge that crypt() on their platform simply never returns NULL. Unix has never been returning NULL from crypt() as far back as 7th Edition, when it simply always returned a pointer to a single internal static buffer.
* https://github.com/dspinellis/unix-history-repo/blob/Researc...
The reality is that all C library people have to work with a corpus of Unix applications that do not check for crypt() returning NULL that goes back almost 50 years, and an error return mechanism that barely exists outwith the pages of the SUS.
That includes callers in single-user mode in process 1 in some configurations.
* https://cvsweb.netbsd.org/bsdweb.cgi/src/sbin/init/init.c?an...
Well, crypt_blowfish() has a number of returns with (-1). I'm not familiar with Blowfish, so it's hard for me to trace the possibility of failure here. But what if in a distant future someone adds more hashing algorithms to the list without possessing historical knowledge of this "never return not 0" ?
* https://cgit.freebsd.org/src/tree/secure/lib/libcrypt/crypt-...
Failing to check result of crypt() in init cannot do much harm, doing so is equivalent of going system reboot (crash init :-)). Besides, whoever has access to system console can do whatever they want with the system, no password can prevent from that.
I have no doubt, despite that Marshall Kirk McKusick is still committing in 2026, that FreeBSD developers will come along that do not know history. But they will get pointed to the manual page that (amusingly) says that returning NULL is 'not a standard behaviour' and that AT&T Unix never did that.
* https://man.freebsd.org/cgi/man.cgi?query=crypt&sektion=3&fo...
NetBSD's manual is more detailed on the subject, and explains the bind that library implementors have been in for decades, thanks to almost 50 years of applications knowing what crypt() 'really' does.
* https://man.netbsd.org/crypt.3#RETURN%20VALUES
And this despite the SUS trying to shoehorn an error return mechanism into crypt(3) in order to accommodate the problems that operating systems had exporting encryption code, two decades afterwards. (The 1997 GoSolo book explains that this is why there's an ENOSYS error.)
The most charitable interpretation here is that musl authors are incompetent.
There is simply no universe in which a security-conscientious reviewer looks at the code, spots * being a valid return value, remembers that * is the default value in /etc/shadow for "no password login please" and then is all like:
This is fine, you check for * and return + instead.
There are more 32-long character sequences than atoms in the universe. Pick one, any of them instead.
Returning a string on error honestly is defensible (especially in a language like C where the type system doesn’t have exceptions or result types), the question is why the string was “*” instead of like “crypt ERROR: input password cannot be longer than 256 characters” lol
No. Musl claims (claimed?) to be strictly POSIX compliant, and POSIX requires crypt to return NULL on errors.
"*" is what's been historically used to represent an invalid/no value in passwd/shadow, so it makes some sense.
Of course given the chance that one might do a direct string comparison between crypt's return and the passwd/shadow file's hash field it was still a not well thought out choice.
No, it's absolutely not defensible.
Precisely.
> The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd.
A typo: `$ret = crypt($pw, $salt)`
weird chain of events and could have been handled in a better way for sure
"The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow."
OpenSSL??
Sorry, should have written OpenSSH.
Great explanation. This affirms my biases that shell scripting and everything-is-strings is a mistake. Horrific.
Shell is for humans to read and act on it.
Not to programmatically parse 'ls' output or some /proc/some/file.
Eg. git started as Perl scripts then moved to C to realy have data straight from C calls from system.
So for example all Internet things calling "openssl -some -options ...." are pure trash :>>
Even good, well-founded intentions can end terribly. The mistake is doubling down on those decisions when the mistake has become obvious.
I am also in the camp that CLI tools should return objects that can be parsed (by the shell or otherwise).
In-band signaling[1] strikes again. * should not have been used as special value, neither for "no password" in /etc/passwd nor "error while hashing the password" in crypt. I'm afraid we'll find many more of these abuses, specially in string manipulations. The null was ok, an exception (or equivalent) better, and Result<T> (or equivalent) best.
[1] https://en.wikipedia.org/wiki/In-band_signaling#Computer_dat...
Exceptions are almost never the right choice. They create extremely complicated control flow extremely quickly, and then you have to test all the different control flows. Java software is an example of how this goes sideways fast. There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor), and objects are default initialized, so all fields (even static ones that are assigned in the constructor) can be null.
So, any field de-reference in Java can produce a null pointer exception. Try testing all paths in something like:
try { foo.sideEffect1().sideEffect2()...sideEffectN(); } catch { bar.sideEffect1().sideEffect2()...sideEffectM(); } finally { quuz.sideEffect1().sideEffect2()...sideEffectP(); }
There are M * N * P paths. The only way to write testable code with exceptions is to panic on uncaught exception, and to never catch unexpected exceptions. Even then, you're relying on nothing unexpectedly throwing the exception type you caught.
I'd argue Result<T> >> NULL >> sentries >> exceptions for this reason. Sentries are terrible, but at least they're not computationally hard to use correctly.
I'd argue that's the fault of having default nullability, not the exceptions. You would have the same problem if every attribute access returned a Result<T> too.
And I'm not a fan of breaking control flow, but I still find it preferable to nulls. A null may travel far from the place where it was generated before blowing up, making troubleshooting harder, or it might come from a function you called for its side effect but forgot to check the return value. I gotta say I'm also a careful enthusiast of checked exceptions, it's great knowing which ways a function can fail.
But I think the consensus is growing that Result<T> (plus warnings for unused return values) is simply the superior choice.
C has what you are asking for, mostly:
As long as an error condition on the return value causes a crash on use, then it's mostly the same as result. (The main remaining difference being that you do an implicit `.unwrap()` in C, and in things like rust that's an explicit code smell.)> There's no way to create a class that prevents callers from reading the object before the constructor runs
That can only happen if you leak the reference (given we are talking about java language level semantics and not byte code), which I give you is unfortunately not that hard to do.
As for exceptions, I think they would pair beautifully with software transactional memory, which would work well with effects. In that case (and no side effects) it would not matter what code path we took, we could just go back to a known safe point.
Because ergonomy-wise I still think exceptions win (even though I do like Maybe types, and Rust's ? macro is plenty good): - they let one focus on the actual happy path code - no error is ever swallowed, unless explicitly written so (looking at you go with useless if errs that fail to properly handle the error), they are always handled somewhere - they automatically "unwrap" the Ok result, and automatically bubble up the Err side. - bubbling up is especially useful because most errors can't be handled at their source (e.g. IO error is meaningless, how you handle it depends on what you want to do in the first place, which is done several layers higher) - they automatically attach a stackframe/source, while with result types you may get a list of Ok Ok Ok Err and have no idea how that Err got there.
> There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor)
Which is why the pattern of double-checked locking doesn't work.
https://en.wikipedia.org/wiki/Double-checked_locking
It's wrong. The caller doesn't get the reference until the constructor runs. Although it looks like two operations in bytecode, the only thing you can legally do with an uninitialised object reference is call its constructor.
You should never ignore exceptions! The least you can do is to crash the program. Then you fix the bugs via battle hardening until it no longer crashes. Making the program not crash by ignoring the errors will just mess with the state of the program until you have no idea what caused it. It's better to let it crash, load the dump into your debugger tool to see what went wrong, then fix the bug, and repeat until you have a stable program. Rather then just live by the motto "ignorance is a bliss".
> never catch unexpected exceptions
Yes, broad exception catch clauses are simply incorrect code.
Panic is maybe an overreaction, "scratch everything and orderly abort" is sufficient.
I'm inclined to say a Panic is absolutely preferable to root auth.
Exceptions work fine and enable a clean separation of business logic and error handling. Visibility into partially-initialized objects is a language design issue independent of exceptions. In a non-exceptional language, the corresponding access to uninitialized fields would just be bugs, not errors you can see.
Nothing in your post actually supports the idea that exceptions are "computationally hard". The CFG error edges exist however the errors are spelled
Unfortunately we’re moving to llm based systems which rely entirely on in-band signaling.
PSA: you can typically declare different types of string so that your type system objects to mixing data of different types even though deep down we know they're both strings.
If you mean type aliases, that's a good idea within a single codebase, but the distinction between different types of strings doesn't survive serialization so it wouldn't have prevented this error. What they need is a way to signal errors out-of-band, like Result<T> or exceptions.
In python, aliases are separate from typing.NewType, but you're right: it's only enforced within a codebase and then only if the dev cares to run the typechecker.
So yeah, I guess it either needs to happen at runtime, or we just do signaling and user data on separate file descriptors.
This is presuming that the type system is nomitive rather than structural and there is not serialisation/deserialisation or type refinement occurring. A structural difference is much more preferable.
It also looked to me a bit like a semipredicate problem:
https://en.wikipedia.org/wiki/Semipredicate_problem
This is a side-effect of Unix-y "it's all plain text" design that leans extremely heavily upon in-band signalling (among other design decisions).
If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.
With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.
A few years back I un-bashed my life as much as possible in favor of nushell (though it's not the only shell to encourage structure). I feel like it has really helped me notice a lot of this-needs-more-structure type problems outside of my shell that I otherwise wouldn't have.
The elegance of the unix-y approach is actually that it's all bits. You can just join things together and shove the bits through, that's a wild simplification compared with having vendor-specific kinds of plugs. But we could have that kind of compatibility while also benefiting from structure.
So I encourage everybody to give a shell with structured output a try, let it rewire your brain, and then ask yourself: how is the OS holding this back from being everything it could be? Because if we build that OS, it won't encourage these problems.
When I imagine it, it would probe either side of a pipe to see if they support negotiation. If not, send the bits through in the same old unixy way. If so, use the newest schema for structured data that is supported by both sides.
At Amazon we had RecordStream which implemented most of the standard Unix tools on structured streams. It was pretty easy to write filters, aggregators, and formatted output in something not much more complicated than sed/awk/cut/join/sort pipelines. There was a convenient structured log format for services that was often used as input.
https://github.com/benbernard/RecordStream
That looks really cool. I've been heading the opposite direction: posix shell for everything.
I have a bunch of management scripts, etc, that need to run on everything from a potato to a T-1000, and that's the only language that's always available. (OpenBSD sh, FreeBSD sh, busybox, bash, dash, zsh all can cope with it).
I'd love something that's at least JSON-typed (or equivalent), but that can be bootstrapped up on least-common-denominator posix.
Has anyone built something similar? (Yes, I'm being picky -- bun / ts is already a pretty low bar for cross platform support, but these scripts do stuff like install node, so I'm not totally crazy for asking.)
Python?
The issue isn't the config, but the fact that crypt() doesn't have any way to signal failure. In better designed languages you would have exceptions, result types, or something. Even in C you have "return integer indicating success or error" which would've prevented the confusion.
Now the public API is hard to change but you could still avoid the issue by returning a longer token like "==DANGEROUS_CRYPT_ERROR_length==" or "==DANGEROUS_CRYPT_ERROR_unsupported_alg==" which would never match the password field unless someone was really trying to induce the issue.
>but the fact that crypt() doesn't have any way to signal failure
https://pubs.opengroup.org/onlinepubs/009604599/functions/cr...
>Upon successful completion, crypt() shall return a pointer to the encoded string. The first two characters of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.
OP's sshhole link specifically notes that it _should_ return NULL, but some MUSL devs decided they wanted it to return trash instead, and the trash they returned instead of NULL caused this.
All because some devs were too lazy to check the result and another dev decided to be clever and "fix" NULL errors from crypt by sending back 'unmatchable' trash banking on devs not checking NULL.
edit: apparently crypt implementations have been trash ignoring the NULL return for a long time. this isn't only on MUSL, but it's still because people ignored and tried to work around the existing out-of-band error mechanism.
This is true, and "trash" is putting it mildly.
But all those sentinels were added for the reason that, when your code is a public API intended to be widely used by who-knows-who, it needs to be really difficult to misuse it. NULL-plus-errno is easy to misuse. Yes, it's a dangerous mistake and your own fault if you do it. But people have been making that mistake for years and don't show any signs of stopping.
The root cause here is that C is fundamentally a rotten choice for public APIs of any kind. I griped more here: https://news.ycombinator.com/item?id=50381800
> returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.
> OP's sshhole link specifically notes that it _should_ return NULL
I meant there's no mechanism besides returning NULL that would make it hard to ignore the issue (exceptions need to be caught, results need to be unwrapped and their failure handled, an integer return value makes it clearer that there's something there that you need to check). I also missed errno codes when glancing at the manpage earlier which should illustrate the problem.
And yes, whoever first had the brilliant idea to start returning garbage never should've done that, I'm not disputing that.
The spec implies a string of at least two characters in the success case, am I reading that right?
If this is true, then the entire thread branch here on HN is entirely wrong. What is going on??
It's subtly wrong, just like most of the rest of the commentary in this comment section. Yes, crypt is specified to return NULL on failure but if you read the rest of that page it is only permitted to fail with ENOSYS. The spec does not allow for failure for any other reason.
So if you're strictly adhering to the spec in this case then unfortunately NULL is not an acceptable way to signal a generic error condition. Hence all the seemingly bizarre special values. (In a strange turn of events all those experts doing things that looked weird turned out to know what they were doing. Go figure.)
People don't RTFM, either on HN or when implementing "standards compliant" stuff like MUSL.
No error handling scheme can fix that problem. For instance, if a Rust API says I should return a Result<String, ENOSYS>, I can always write code like `if(ENOSYS) { return Ok("*") } instead of returning Err(ENOSYS).
Then my code would be compatible with MUSL.
Sure, but if you didn't do that, and returned the tagged union (in Rust or whatever language), your users would necessarily be forced to handle it in order to process the result.
The only reason this library did the insane thing of returning "*" is because it was extremely easy to cause crashes or false-positives (NULL == NULL) by returning NULL+errno. The reason that was easy is because C lacks a means of making error handling mandatory (exceptions, tuple returns, tagged unions, whatever).
If you return a tagged union, users might mis-handle it, it's true (https://blog.cloudflare.com/18-november-2025-outage/). But they have to handle it, which is a requirement that C APIs are incapable of providing.
You have to handle NULL in C, or your program crashes (though I suppose a misguided optimization of strcmp could do "a == b || really_strcmp(a, b)", which gets us back to your bug).
Also, in things like rust you can just do result.unwrap() if you want. In general, language constructs don't really save you from bad programmers.
> You have to handle NULL in C, or your program crashes
But it crashes at runtime, so such a bug can exist for years with nobody noticing.
The existence of such bugs is what motivated the musl developers to return "*" instead of NULL.
> a misguided optimization of strcmp could do "a == b || really_strcmp(a, b)",
In practice, that's less of a risk than:
Or code for which pointer identity is relevant (e.g. accessing slices/subparts of a data structure): And for sure there are much better ways to write both kinds of code! But people routinely do not do that.> I can always write code like `if(ENOSYS) { return Ok("*") } instead of returning Err(ENOSYS)
You can do that but it's going to feel bad to write that and it's going to stand out like a sore thumb to anyone who reads it. But "errno" is just ambient state that's completely invisible unless you happen to have the manpage fresh in your memory.
They had to write
after seeing that other implementations use different sentries like "*0" and that the spec says they MUST return NULL. That must have felt bad too.Not the library authors but their users!
A lazy attitude like "if we just used <n> language, all our problems would be fixed".
Yup, that's the way to explain it in order to blame musl.
Here's how to explain it in order to blame openssh:
openssh processes passwords and doesn't want to reveal through timing whether a user has a valid password (or exists at all). So far so good.
What they do if they hit a user with no valid password (i.e. "*") is that they look for another user with a valid password, to use the seed from. A bit weird already, but OK from a perspective of avoiding implementation specifics on seeds and hash algorithms. They know at this point that the result must be to deny access even if the hashed value matches, because that hash is for another user.
So they put the password received over ssh into crypt() with this other user's hashed value to get a seed. And then accept it if it matches, apparently?
Homework: what happens if the password actually matches another user? Can I log in as root if openssh randomly picks my user for the fake salt, and I type my own password correctly?
Except it's not trash. It's a backdoor funded by our glorious leader Kim Jong Un.
Personally I think there was nothing wrong with returning NULL. "But if people don't check for it it crashes their programs!" Good! Let it crash loudly! Then they notice and fix the issue.
Also I think it could have returned ":". Given the file format of /etc/shadow and /etc/passwd I don't think it's possible to have a : as a value for anything.
> Also I think it could have returned ":". Given the file format of /etc/shadow and /etc/passwd I don't think it's possible to have a : as a value for anything.
I wonder if somebody else would have thought the exact same thing in a different (non-/etc/*) context, and then you get the same effect in a more obscure manner.
Indeed. Let's keep NULL.
> The issue isn't the config
That's a meaningless statement. There isn't a singular "the" issue like you said. Any one of several different technical decisions would have completely prevented the issue. Including, to my point, /etc/shadow being a JSON file where an account being locked is an `"active": false` field instead of an asterisk with no intrinsic meaning.
No, that's just more in-band signaling!
Some badly designed password hashing algorithms (cough bcrypt cough) have maximum password lengths, ignoring everything after X characters. So it could happen that I set my password to a 100-character string, it gets saved as "==DANGEROUS_CRYPT_ERROR_length==", and then anybody can log in to my account by entering an unrelated long string that returns the same "error".
If a function promises a string result, it should only provide a string if the function succeeded. The null was much better than magic strings.
I know, it's bad. I agree that it should've stayed NULL. But if you really really really think you have an acceptable reason to do something like this (and you probably don't), the least you can do is use a non-trivial token.
If I was really desperate and could only return a valid string, I would probably use a message and add a random component to avoid ever matching two errors. Like SQL, where null != null.
Returning “*” instead of NULL to signal failure is the real footgun. Don’t blame the format.
For historical reasons, we could’ve ended up JSON as “hash”: “*” just the same. The /etc/shadow format lets you skip values completely, like “::”, so your argument doesn’t hold.
Terminating an app because of a NULL dereference is the other footgun. Sadly, we still have “panic” in modern languages like Go and Rust.
"panic" is a good thing. There's always some error condition that your code cannot handle. At some point, the least bad thing to do is crash.
For instance, what if your error handling code throws?
One spectacular example of panic saving the day was the Apollo 11 1202 error. It was basically a watchdog timer that fired if the landing computer was starved of CPU time. A misconfiguration caused the first moon landing to hit CPU overload. When that happened, the computer displayed 1202 and rebooted, then ran things in priority order. Apollo 11 landed in the middle of a tight 'panic, reboot, adjust thrusters' loop. Thanks to core memory, most of the heap survived the reboot, so the reboot step was fast enough for it to mostly land without human interaction. The pilot had to make ~1 adjustment to get it to actually hit the landing site, but the panic-ridden computer handled the rest of the descent.
https://dpma.de/english/our_office/publications/ingeniouswom...
sshole-type APIs < panicking APIs < required error handling.
Panic is fine, but fundamentally a whole-program crash is undesirable in a really wide range of use-cases. I don't want other SSH connections to crash (or the server to stop accepting valid logins until someone restarts sshd) if one person tries to log in with a credential that triggers a panicking bug in crypt(). Same for crashes in request handlers and so on.
The language/runtime itself doesn't (and probably cannot, at least if it's a general purpose language) distinguish between local and global error conditions, so aborts end up being either unnecessarily damaging, or languages like Go end up normalizing panic "localization"/handling in dangerous ways. Isolation using threads isn't robust enough. Isolation using processes is far too fiddly, slow, and opaque.
The better solution is required error handling. If user code must engage with the possibility of an error state in order to access a result, that's a huge step forward. They can do the error handling wrong, but if you make most people have to do it at all most of the problems will go away.
There are other good solutions too, like Erlang's process boundaries, but those tend to have drawbacks that make them inapplicable to a lot of programming areas. Tagged unions, on the other hand, are only inapplicable in C library code because of language defects and terrible community conventions around ABIs.
"I forgot to check for an error" should not be implicated in bugs in 2026; we have had a large menu of possible alternatives (checked exceptions, unchecked exceptions, tagged unions, etc.) for decades. But we're still over here writing C (and Go), languages pathologically incapable of providing this ability.
SSH specifically has separate processes per connection the last time I checked, so a panic does the right thing: The attacker's connection drops uncleanly.
This is also why you can upgrade SSH over SSH without having the connection drop mid-stream.
Again: You cannot avoid "I forgot to check for an error". What happens if you get a sigpipe on stdout when printing the "I crashed" line? If Linux overcommits memory? If you turn off overcommit, port all software to check for + cleanly handle NULL from malloc (a substantial effort!), but then get a machine check exception from a double bit ECC error during an infallible read?
Sort of. It does some of the validation/auth code in the sshd parent daemon.
It has to: if it did 100% of the post-accept(2) work in a subprocess it would be pretty easy to remotely forkbomb/PID-exhaust a system by spamming invalid logins.
> You cannot avoid "I forgot to check for an error"
Yes you can. Checked exceptions and tagged unions both force calling code to be written with error-awareness.
In SSH's case, "I forgot to check" might lead to crashes (which might knock over sshd so new connections are impossible), or it might lead to false-positives via comparing NULL == NULL.
I'd much rather that boneheaded API users have to write code like
At least that way some keystrokes are required to do the stupid thing. People still will (https://www.schneier.com/blog/archives/2008/05/random_number...), but the harder we can make it, the better.> is the real footgun
This contains a false snuck premise - that one and only one thing was the footgun. This is categorically not true. An obtuse, poorly-understood, purpose-built text-based DSL is absolutely a footgun and if it was replaced with a JSON and inactive accounts had been indicated by `"active": false` instead of a freaking asterisk then this would absolutely would not have happened.
> For historical reasons, we could’ve ended up JSON as “hash”: “” just the same.
That's dishonest, because it intentionally conceals the fact that that's a vanishingly* unlikely scenario.
People have brains that, factually, are far more likely to see a single context-less character "" and not realize the implications of it than to see `"hash": ""` and think "oh, that's normal".
This is just straight-up psychologically true.
> The /etc/shadow format lets you skip values completely, like “::”
Totally irrelevant.
> so your argument doesn’t hold.
False `::` is both (1) still harder to understand than `"active": false` and (2) even though the syntax existed it wasn't used here because everyone thought they understood the system.
Don't make excuses for this insane text syntax. It is, factually, harder to understand than a decently-written JSON format.
Yes, absolutely. Take for example this cURL command:
Random parts of it will be interpreted by the shell, others by cURL, or the filesystem, or the remote server. They each have their own encoding rules, nested. I count ten distinct separators and nine ways of escaping/tagging. It's one command. No, just no.We have seriously overused strings and we're now paying the price.
Also: free text is the input format of LLMs...
At this point I’m resigning myself to think that we’ve just gotta wait for some people to die. It feels very dramatic and glib (no pun intended lol) and whatever to say, but at this point it feels like there’s a certain hostility in some parts to changing anything from the platonic ideals of however computing happened to work in 2004. “Oh we don’t need to mess with it!”
How did you arrive at that conclusion? If anything, this vulnerability happened because someone decided to change the behaviour of crypt() without fully understanding the consequences. This specific vulnerability did not exist in 2004.
I agree with him. I only managed to get asterisks enabled by default in sudo because sudo-rs isn't owned by people who have decided that the 70s was the perfect decade and nothing should ever change. And look at the insane pushback when it did!
I can't even get an obvious UX improvement through PAM because they like it broken.
https://github.com/linux-pam/linux-pam/issues/778
I think this is a bit unfair. The way I see it, a lot of people are saying that changing breaks lots of old stuff, and risks complete fragmentation. Those are real costs to change. It might still be worth it, but let's not pretend that everyone is resisting change just because it's change.
The costs to change are dropping significantly with AI.
Citation needed.
https://commandline.microsoft.com/agentic-software-engineeri...
All I see is sausage makers advocating for a sausage-based diet.
> "it's all plain text" design that leans extremely heavily upon in-band signalling
Honestly I've been shaking my fist at these since 0 becomes True/Success in Bash and the opposite everywhere else
Error code 0 means success in a lot more places than just bash.
That's not specifically bash though, that's process exit codes
Yes. But it's still contradictory
The question you are asking when querying a process exit code is "did this program exit with an error?". The default assumption is "no, it completed successfully", so 0 (false) is the logical expected default return.
A function is expected to produce a result - so the question is "what was the result?". !0 (something) is the logical expected default return.
I've never found it to be contradictory.
But in shell, 0 is true, not false. The program called true returns 0.
"True has 0 errors" is the straightforward part.
"False pretends to have an error" is where it gets weird. It's not that they got booleans backwards, it's that they're stretching a tenuous analogy a bit too far.
I'd say another UNIX-y thing - "C is good enough" - is part of the problem. C makes programming painful which results in all sorts of hacks.
E.g. in Pascal you can can define a record and then defined file of record to have typed I/O. I.e. it comes with built-in serialization format supported by compiler. Not a good one - but perhaps a step in a good direction.
OTOH C people starving for serialization often believe that `struct` byte layout made by a specific version of a compiler is a serialization format - a view not endorsed by the standard.
Pascal's record typed files have the exact same problems regarding endianness and word size as directly writing structs to files in C, so I don't see how they're any better. They're also a complete deadend and unusable for pretty much any file format from the last 40 years+, since you can't even handle a header or magic bytes at the start.
Should have set crypt_errno.
Even using "no_password" and "error_while_hashing" would have prevented it. The problem was picking a generic, meaningless signal string.
That would've happened to fix it, because "error_while_hashing" and "no_password" happen to not be valid password hashes. The right way would've been to have different syntax for hashes and non-hashes; you could've made it so that a hash starts with a '#' and if it doesn't start with a '#', the rest of the string should be ignored.
Or, for backwards compatibility, start non-hash strings with a character which can't appear in a hash, such as '@'. Your examples would then become '@no_password' and '@error_while_hashing'.
Well neither * nor ! (the two characters I've seen to indicate "no password") will appear in a valid hash. The problem was using a character that could also be returned as an error indicator, IIUC.
Yeah but it's not forwards compatible with future hashes where '*' is suddenly a legal hash, which is what happened here. You need a grammar which lets you tell the difference at the parsing stage between "this is a password hash" and "this account has no password". That's not the same as letting the value '*' flow through the system as normal and hope it doesn't happen to match the hash.
True, but if they would have picked ?, or #, or any character other than * with its designated special meaning in the /etc/shadow file, then we would not be in this pickle.
hashes can't start with * and there's no reason to think they ever will
Technically they did use a character which can't appear in a hash, '*'.
Right but not in a "rigorous" way. They kept the format of the file the same: the string between the colon characters is a hash. They exploited the fact that an asterisk happens to never appear in a hash, and just let the system believe the asterisk was supposed to be a hash to be matched again.
In a sense, they should've changed the parsing logic to return a struct with a field 'password_hash: Option<String>', and defined the grammar of the field to be: "a hash string, or '*' for no login". The bug is, in part, caused by the haphazard way they don't change any formal grammar or contract and just relied on the fact that '' happens to* never be a valid hash.
It's the difference between "happens to work" and "defined to work by construction". The former approach breaks once a hash function is invented where '*' is a legal hash (such as this new musl crypt() hash function where all strings longer than 256 bytes hash to '*'). The latter keeps working regardless because the asterisk is treated as "no password" already at the parser stage.
The whole story is a bit worse than this. Musl already worked around this type of failure. See the second-to-last line here:
https://git.musl-libc.org/cgit/musl/tree/src/crypt/crypt_des...
(hey, this workaround is wrong if the salt is "x"!)
but they forgot to work around it here:
https://git.musl-libc.org/cgit/musl/tree/src/crypt/crypt_sha...
The vulnerable path in musl is only a little bit vulnerable, because "*" is not actually a sha256 encrypted configuration and should not reach this code: musl's crypt(..., "*") will not call that code! Turns out that sshd detects that the password hash is too short and does not forward the salt!
https://github.com/openssh/openssh-portable/blob/master/auth...
Look, no else statement! If that if doesn't get taken, then salt remains NULL. And the sshd code is really quite messy and doesn't keep track of the fact that this code path is inappropriate for validating an existing password hash.
A workaround in the sha256 path in musl wouldn't even have prevented the bug!
I'm going to give some credit to musl for unwisely using "*" and some credit to OpenSSH for having nonsensical control and data flow.
> (hey, this workaround is wrong if the salt is "x"!)
When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
> Turns out that sshd detects that the password hash is too short and does not forward the salt!
I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
[1] https://lists.mindrot.org/pipermail/openssh-commits/2016-Jul...
> When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
I think you're right. That's what I get for doing a very very quick read.
> I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
I don't care what the intent of the check is. If you're doing a fake password hash to obscure timing, remember that it's fake and don't let the user log in! Don't deliberately pass nonsense into the live check and rely on the nonsense check failing to prevent the fake check from accidentally working. And don't have a code path that thinks it's checking the real password but passes in the wrong second parameter to crypt().
Or, better yet, have an entirely separate function fake_check_password() that does the fake check, returns void, and cannot possibly be confused with the actual password check code.
> If you're doing a fake password hash to obscure timing, remember that it's fake and don't let the user log in!
I think this is the crux. Instead of doing the right thing, they tried to codegolf to get the desired behavior and it got real subtle, real fast.
this is a little flip: maintaining a security-critical program naturally pushes one to minimize code changes because a 30-line change is usually easier to review than a 100-line change. Except when it's not.
> you're doing a fake password hash to obscure timing, remember that it's fake and don't let the user log in
That would be a much better approach. I see your point now.
The formatting in your comment got messed btw
Thanks, fixed
My general rule of thumb is: "no magic strings". This is much easier to implement in higher-level languages and when writing application/web code than system programming.
Don't return empty string/asterisk/anything else. Instead, return a failure value.
When absolutely needed, choose the magic string with more caution:
One string showing the SHA-256 of a non-existing hash? It's a good idea for it to be a valid hash. Now we have a new risk: if it's broken once, it's broken for all. So it's a good idea for it to be a random one for each machine and/or account. Even this might have problems I can't think of, but it seems to be as secure as an account with a real password.
If this is not a good decision for some reason, choose something unmistakable: "--ACCOUNT-HAS-NO-PASSWORD--". This is much less likely to be mistakenly used in a different context, is much easier to look up in a code base than "*", and is self-documenting.
Agreed, unfortunately there is so much shit programming that does this that AI is trained on that I continually see it generating it in my code.
My AI usage involves telling it to do things and setting up lint rules to catch all the bad programming it’s learnt.
100% agreed on no magic strings
One potential point of disagreement: it's easy to implement structural prohibition of magic strings in low-level languages too if you consider Rust a low-level language (which I've heard before, even though I didn't really consider it to be one for our application of it).
I'm not a Rust expert but do run a few thousand lines of it in prod for some perf/correctness critical code, and I wish it didn't have the reputation of being so difficult to comprehend. There's so much resistance to introducing it or new languages in our large org, that I'm surprised we were able to introduce our thin slice at all. I am not sure I'd use it where you need easy async primitives like goroutines, but it prevents whole swathes of bugs that are repeated over and over again with devastating impact, like the OP's bug, via thoughtful language design.
Maybe this resistance is an artifact of where I work more than large organizations generally, though. Curious about others' experiences. I'm not really an evangelist for the language but would really like to work in a more polyglot environment one day.
Above all, I’m a little sad that the musl developers decided to make a lot of “well it’s convenient” changes to core crypto functions — at least without the level of auditing that would have caught this! It’s tarnished their reputation in my eyes a little.
I think everyone is over fixated on AI finding complex bugs beyond a typical person's understanding either to locate or exploit.
I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.
I think that's the case where LLMs might indentify most.
It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head
Finding more and more that these complex bugs people gloss over like you are now are being used in exploit chains and the end result is still my clients getting hacked
This one might be hard to find using the author’s approach of throwing everything at various builds of ssh.
It is not really that hard to find in the wild if you had a device built this way, when overflowing buffers on embedded devices is a standard attack vector.
It might be hard to root cause, but it would not be that hard to find.
Honestly the ideas around musl are interesting, especially being able to statically link, but given the sort of nonsense around how they handled pthreads, and generally other trade offs, it's hard to imagine wanting to use it over glibc (which everyone shits on) if you don't absolutely need static linking.
Everything in this industry is about convenience.
Musl has a history of introducing (and not fixing) major bugs in their library that aren't in glibc; I would not depend on it for production
In many ways each individual decision makes some sense; wanting to stop loads of programs being affected by a (remotely triggerable?) crash wanting to avoid time based attacks wanting to be able to set accounts to not have a password
Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
I would be more concerned about the lack of explicit return types, one of the arguments for strongly typed languages is that you should not have ambiguous data and yet…
Granted I mostly work with high level programming languages so I may be missing something.
Yes this is C, C is not strongly typed at all and the type system is very primitive. NULL ptr is the return value you should probably use if something goes wrong. But you can’t, say, return a sum type with the result and an error. It’s not uncommon at all to use magic sentinel values in C to indicate error conditions.
> Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
But that's not a perfectly valid argument, and it never was.
I guarantee that every single developer of musl in 2012 had at least account on their machine with "*" as the password hash. Also crypt() is a standardized function and the standard states that a null pointer is returned on error.
That being said, OpenSSH did mismanage their attempts at handling a garbage return-value from crypt(), so there's definitely some blame to go around.
If they had just all used a different garbage symbol would this have been avoided?
The real problem here is the C language. Our industry should have dumped it a long time ago and we’re now paying the price of prevarication.
Which of the several flaws in this bug chain would have been materially different with a language other than C?
Any use of sum types would have prevented this immediately.
The whole thing exists because APIs that return a NULL value that "nobody checks", plus a string that you shouldn't use if you actually did check the null value.
In languages with sum types, this cannot happen.
Either you get your hash, or no result at all. You couldn't get eg NULL and *.
This argument is weak. C, and specifically libc, has errno. The error code is there. Not in band.
But there has to be something in the payload field when it is not applicable. A null is perfectly fine.
The argument for changing that to "*", if I understand the above linked discussion correctly, was that some users of this api threw away errors and a dummy value would be harder to ignore. Maybe one could argue that these behaviours are less common in Rust or Go, but not because of a technical difference.
The world is not perfect and sometimes you really need to work around bad users. Even in Rust or Go. I am not convinced this was such a case. If you are convinced it was it would still have been better to select a less trivial dummy value (such as "*:ERROR:*").
In Haskell, one would return a Maybe or an Either here.
The program would be designed around not allowing this failure mode to even be possible.> But there has to be something in the payload field when it is not applicable
Not in languages with ergonomic sum types. A `Result` or `Either` will have no payload field on the error type. The compiler won't let you dereference it.
This is wrong. errno is not to be used to detect errors, only to get the cause of the error once an error has been detected.
C has unions. C has sum types. People have been doing tagged unions in C for longer than rust has been a language
It is extremely unidiomatic to use tagged unions in C for error handling. It's definitely possible, but awkward, and the compiler doesn't check that you checked the tag.
I don't think I've ever seen a C function that uses unions for this purpose though?
I can think of two common patterns that I've seen frequently:
* Return NULL for errors (sometimes with a global errorno() function).
* Return an integer error with any return values as out parameters.
So if it is possible to improve, why doesn't anyone do it?
Because usually NULL is good enough and you need to have domain use case to make library for it.
> * Return an integer error with any return values as out parameters.
You must first standarize all that (usually negative) integers, make docs, check for it, do not make mistakes in your if's or case's...
And errno is just that - integers with a meaning. errno is even thread local now and quite safe to use but people on internet scream loudly when someone try to use that not app global varible :) I guess their llms statistically have less new texts feeded in ;)
At some point it is just better to return pointer to struct with all info you can imagine.
And how you think other languages implemented their Error type returns ? Just a struct or union with tag. Because current CPU's operate on simple type registers there is no other sane way to do it.
That's a universe apart from real sum types with exhaustiveness checking and pattern matching.
Yet the authors of crypt did not use them. The C language is not designed in a way to encourage people to write code that correctly uses tagged unions (or nullable pointers for that matter).
because POSIX dictates the shape of crypt from a function written in 1973 and unions/structs were only introduced to the C language in 1976.
the fault for sshhole isn't just "C bad", it's a series of poor choices in workarounds falling together to create a vulnerability.
being condescending about decisions made 50 years ago, before any of the better languages you're mad the developer didn't use even existed, is silly.
if you want to lay blame, lay it at the feet of more recent devs that started returning values instead of NULL because people were using crypt incorrectly. likely sha512 returned a fake value instead of a real one because prior hashes couldn't fail. the interface should have been updated then, but instead they worked around it.
Where are you going with those goal posts?
Nowhere. It's the same argument as upthread. A language that forces downstream consumers of an API to correctly deal with all possible outcomes would have prevented this.
Goalposts are fine. If crypt did return union type, C still allows cowboy programmers to straight extract the char* without any check whether it's valid. So we'd end up in exactly the same situation.
I don't think the goal posts were moved. The comment you replied to referred to the "use"--not the existence--of sum types.
Nope.
Because in fact all the changes were about changing the function signatures. The function signature is enforced by the linker. The semantic of strings that go through isn't.
Returning 2 values is possible in C (via pointers input parameters). The main issue here is that libc is a strong API contract based on C function signatures, that nobody can change, only augment with new symbols. Such strong API compatibility constraints happen in any programming language, and workaround to fit new behaviours in old APIs happens everywhere.
I don't think nh2 is saying that this is how the "issue" should've been solved here, but that it's a fatal flaw of C itself (yes I know, unions, but no one uses them).
But also, its not like there were many alternative languages when the POSIX standards were designed so it's fair that it works like this.
Even C has a better type system than C: All modern C compilers support an extension that lets you flag a function's return-value as requiring a null check.
C's lack of all of
- sum/union types
- multiple return values/tuple types
- exceptions
means that most ways of returning an error from a function are not available in C. Because of this, many C APIs, including `crypt` in this case, resort to in-band error signalling by using a sentinel value (like NULL, a negative integer, "*" etc.).
Most other languages lack one or two of the above, but very few lack all three, and so most have developed a different culture around error reporting (Result in Rust, multiple return values in Go, exceptions in Java, Python etc.).
One of the small but important significances of Rust is that it very often gets to have the same machine code as a correct "in-band error signalling" strategy from C but with source code that the maintainers can actually understand.
And really often when your "in-band error signalling" in Rust is more expensive you realise (perhaps to your horror) that your C wouldn't be correct, so you just dodged a bullet. For example Option<OwnedFd> is, like OwnedFd, a 32-bit value, Unix has always reserved -1 to mean invalid, so Rust can treat that as a niche. But the Windows equivalent Option<OwnedHandle> can't do this because, well, cue Raymond Chen: https://devblogs.microsoft.com/oldnewthing/20040302-00/?p=40...
> One of the small but important significances of Rust is that it very often gets to have the same machine code as a correct "in-band error signalling" strategy from C but with source code that the maintainers can actually understand.
But aside from NonZero- and NonNull-based layout optimizations, I don't think Rust is doing anything special to obtain that machine code - it's just relying on LLVM to optimize things.
Well, all of the niche optimisations, so that includes every normal C-style user-defined enumeration because (unlike C or C++) Rust says those aren't just the integers wearing a funny hat.
But sure, for the most part LLVM will do The Right Thing™ here without much help and Rust is just giving better ergonomics to the same backend.
I remember mucking with returning multiple values as anonymous structs. It was tantalizing close but no banana.
I think with C23 you got half a banana now. Still can't do it because C23 didn't adopt proposals to allow returns of untagged/anonymous structs.
If you could then you could write
The first 2 it has, and the 3rd, you really don't want.
The first 2 C has in the same way that I don't need Dropbox if I just have an FTP account and curlftpfs (https://news.ycombinator.com/item?id=9224).
Your analogy is way off...
C has sum types/unions? What are you talking about? You don't need multiple return. You can use a struct and it's the same thing.
not in a way that forces the user to actually handle them properly. C is a weakly typed language and as such is pretty badly suited to systems programming.
C is not a weakly typed language, this is just nonsense. There is also [[nodiscard]] which you can use to make sure a return value is handled.
What is a weakly typed language to you? To me, implicit type conversions are a big part of the definition, and C is flush with those (void* converts to any pointer, all kinds of number conversions, arrays decay to pointers, functions convert to pointers, lvalue conversion strips const/volatile etc.).
However, I also think "weakly typed" is not a well-defined concept, everyone seems to apply it differently. In particular, it seems to mostly mean "playing it loose with types in a way that I don't like".
Yes, it is not well-defined. But most conversions, function calls, and accesses are strongly typed in C, so I would not call it weakly typed.
Implicit conversion could be seen as weak typing, but it is mostly limited to specific arithmetic operations in C. I would see implicit conversions as problematic if they allowed operations that make no sense. Decay of arrays and function to pointer conversion also do not seem problematic to me for this reason.
Void pointers are also weakly typed, but they are specifically designed for this. So if you do not want this behavior, one would simply not use them. It is a bit like using "unsafe" in Rust.
There are certainly stronger typed languages, but on the full range C is more on the strongly than weakly typed side in my opinion. A language that allows arbitrary implicit conversions between pointers, would be weakly typed, or which allows calling functions with wrongly typed arguments (which was true in the past for C a long time ago)
C is definitely not the most weakly typed (e.g. Javascript/PHP are both weaker), but IMO C is weaker than python in that python will give you a runtime type error if you do something invalid, while in C you cast something and just get UB. Most higher level strongly typed languages are able to make their casts checked because they have generics to remove the need for 99% of them, but C is missing those features and thus has to rely much more heavily on unchecked and unsafe runtime casts.
Casts are escape hatches just like "unsafe" in Rust. You can write safe generics in C that either do not use casts or wrap all unsafe operations behind type-safe interfaces.
C unions are sum types if you just "draw the rest of the fucking owl". C has sum types the way x86 assembly has function calls. You see, there are "call" and "ret" instructions. Just figure out calling convention and the stack yourself, it's easy. And yes I can stick the tag next to the union myself in C and write a bunch of functions that hopefully deal with it correctly. I can even write some macros that look like pattern matching and maybe convince myself that it's a language feature, but it just isn't.
And it's very evident why the difference matters. People gravitate to what's readily available and ergonomic. That's why every C API under the sun (most definitely including libc itself) does janky stuff like returning -1 or "*" on error instead of building their own bespoke Result-style abstraction. A programming language is not just the things that are in some way possible within it, but more importantly the things that are easy in it.
While I agree that this should come out of the box, the "rest of the fucking owl" is a tiny macro or function, so not exactly something difficult to do complex as "drawing an owl" would imply.
Sure, when I last wrote a sizeable chunk of C, I wrote those macros and with a bit of help from new-ish compiler warnings, I got a reasonable error handling system in C. But whenever I interacted with anyone else's code, they apparently didn't get the memo that this is possible and desirable.
A language should encourage writing robust/secure code, not just make it possible.
While I see the point, I am not entirely sure I agree. I kind of like that C is not an opinionated language which tells you exactly how to write things. This comes at the cost that many people will do things in a bad way, and this is difficult to fix, but still probably easier than getting them to move to a different language.
In any case, I do not think the issue here is caused by not having a pre-defined result type. Checking for null would have been fine. The issue was that to accommodate old and broken code a compromise was made. But as long such compromises are done even where fixing such old code would be quite easy, it is clear that changing to another language is completely unrealistic. So what are we even talking about? The real issue is that nobody wants to spend money or effort maintaining the old code. No language can fix this.
There are different amounts of effort you can pour in to this depending on your requirements, and we can argue about those, but to me the evidence is clear: It's too much of a hassle for library authors to actually go and do it. Everyone just uses sentinel values instead.
For the other part of the issue (the truncation in SHA512), in Rust you could've had:
Now the truncation is a terrible idea in the first place, but at least with this it would have been entirely obvious to the caller that it was there.In C you could do just the same: https://uecker.codeberg.page/2026-10-24.html Whether you should is a different question.
In any case, you can not change the API of an existing function. But returning NULL for error has required by POSIX would have been just fine.
Wouldn't Rust's Result or Go's error checking prevent that?
While not mandatory, a person would probably use it instead of the * hack.
Simply because the calling side wouldn't have an option to forget the check like with the NULL case
Rust and go still make it pretty easy to not check results if you don’t care about the success value (lots of code just does `let _ = foo()`), but if you do want the value (like with a hashing function), Rust makes it really hard to not check the error too. There’s no syntax to say “assume this worked and give me the success value, including whatever garbage is there if it’s not set”:
- result.unwrap() will panic if it failed
- result.ok() throws away the error but gives you a None value instead (still can’t use it as if it’s a success)
- matching on the Ok variant and doing nothing on the Err variant, you only get to use the success value inside the Ok arm
Etc etc.
The problem is when the compiler says “NULL is a valid instance of T”, which go lets you do, and rust doesn’t (without using the unsafe keyword and intentionally casting a null pointer, which you don’t do by accident.) Go lets you write code like `var hash string; hash, _ = hashfn()` and it compiles and now you have null instead of a valid hash.
Minor nit, strings are never nil in Go, so that would be "" instead. Usually nil is only a valid instance of pointer-to-T. The point stands though, a common pattern is "defer foo.Close()", the Close() signature almost always returns an error and almost no one checks it...which is probably fine, but not automatically fine.
> NULL is a valid instance of T
This by the way is Tony Hoare's famous "Billion Dollar Mistake"
A NULL pointer for error would be a safe interface where you get a trap when there is no value.
I understand that other languages could help avoiding the semi-predicate problem. That being said, in C there are better ways around semi-predicates, as inelegant as they may be, is possble, is not an inherent problem of the language, but may a bad habit/idiom of it.
Do you work in industry or academia?
There is no industry, it's just people
You must be new here. While I applauded the idea of musl when it first came out (coincidentally, glibc wasn't having the best time in its development then), the developer's arguments why <some behavior drastically different from every libc under the sun> was akchyually correct, which was why you had to fix a program that hadn't been broken for 30 years and now suddenly was, made me put him under the "clowns" category and never look at musl seriously again.
When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.
And now we have this.
I’m also reminded of the clowns pushing “it’s smaller!” container images based on social media vibes rather than analysis where, yes, not using Debian+glibc saved tens of megabytes on the base OS but did so at the expense of measurable performance reductions & compatibility issues like the one you mentioned, and literally every single one of those dudes I worked with was shipping containers pushing a gigabyte so the size reduction had no measurable impact on deployments.
Uh, isn't this the case of the opposite? The musl devs trying to accomodate broken programs?
I am sorry, a little? I am deeply disappointed that libc decided to do the same just with another sequence.
And why because some dude was complaining sshd or whatever was crashing because it didn't handle nulls? Just handle the null my guy you already know why your bullshit is crashing.
Let's be real here, the main culprit is C. People love to say "but you can still get logic errors with Rust", and no shit of course you can. But it's much less likely. You wouldn't have gotten this bug.
A little?!? I’m sorry, but musl is a joke, this is yet another proof of that fact, and I have no idea why anyone uses it in production, they have only themselves to blame after the many warning signs.
MITRE scored this as a 8.1 with a high attack complexity. [0]
I fail to see how the attack complexity of this vulnerability is high, as all it requires is attempting an ssh login with a password longer than 256 characters to an affected alpine image with port 22 open. Am I in the wrong?
[0] https://nvd.nist.gov/vuln/detail/cve-2026-123158
In my last org I was in charge of internal testing and qualifying software releases for network equipment, and designing test harnesses and tests for these. Significant install base. All the major vendors. Many years. Basically doing the vendor QA for them, because we couldn't afford things to fail. As you do - I wish.
This was not security testing as such - mostly latency, performance and hardcore network "can we break it" and "does it comply" stuff, but also a lot of cable and module / line card pulls, pluggable and fiber breaks, random reboots and pre-empting a lot of possible operator inventiveness. At one point we considered SEU bit flip testing on rented cyclotron time.
And always we battled against the top brass urging us to speed up the process and don't bother with "really basic" testing, because we observed software quality going down over the years as the push for release and time to market stared to dominate.
So we always started with sanity testing - and that included oversized password and CLI input strings. Always. And we caught things more than once, but never like this.
We were right after all.
> We were right after all.
Exhilarating, isn't it?
Hardly. I say this with a faint, bored "huh" note rather than with a triumphant "I told you so", and I move on.
It's about maintaining solid engineering practices and far above proving who's right or wrong.
This is why I don’t fully buy the argument that software that’s been around for a long time is more secure and reliable because it’s been battle-tested. There’s been a lot of time for cruft to accumulate until no-one can ever fix it because everyone is expecting it to function in the broken manner it’s always functioned. And some of the more obscure configuration options get a lot less scrutiny.
Musl, which is vulnerable, is much younger and less battle tested than glibc, which is not vulnerable though
This is not an obscure configuration option. Your password file almost certainly has at least one account with "*" as the password hash.
Integration of different software is the stage most likely to introduce uncaught bugs. That is what happened here: musl intentionally broke the documented API of a function used by OpenSSH. Then Alpine integrated this incorrect implementation with OpenSSH.
Fun!
I was able to reproduce this on Alpine 3.23.7 with OpenSSH_10.2p1.
Luckily PermitRootLogin [0] hasn't allowed logins with passwords for root since 2015 [1]. I'm guessing that's at least part of why this went undiscovered for so long.
0: https://man.openbsd.org/sshd_config#PermitRootLogin
1: https://cvsweb.openbsd.org/diff/src/usr.bin/ssh/sshd_config?...
Did anyone get it to work? I tried on the official alpine docker image, enabled password login for root. The shadow entry has ! as password by default, this didn't work with a > 256 character password (i also did passwd -l which also does !). I manually changed it to * and it still didn't work.
Am I doing anything wrong here?
OpenSSH_10.3p1, OpenSSL 3.5.9 29 Sep 2026
PRETTY_NAME="Alpine Linux v3.24"
It's because alpine has updated all openssh-server packages already. Even from EOL alpine versions.
Ahh! Wasn't working on 3.16, which does indeed have an updated package.
But, got it to work on 3.15 - which requires changing the ! to a * in the shadow file in addition to other steps (making another user with a hash, setting PermitRootLogin to yes, etc.).
https://pkgs.alpinelinux.org/package/v3.16/main/x86_64/opens... Build time 2026-11-05 06:53:46
https://gist.github.com/markuman/acd2c2e7e08c849cfafe9e47bec...
on 3.21 with the previous version ...
oooh, that'll do it.
Didn't notice that the openssh-server-10.3_p1-r2 was released today.
Nope, wasn’t able to reproduced it in my firecracker VM, which isn’t publicly accessible anyway.
As far as I understand, it wont work on default setups. You need musl-c, no pam and openssh.
In Alpine, these would all be the case for an out of the box install with openssh added unless you specifically install openssh-server-pam and configure sshd_config to use PAM (Alpine is already musl-c based and doesn't default to PAM).
Doesn't it also need a user to be added with a password entry?
I only ever see Alpine Linux used for containers which don't need accounts with interactive user logins, which is not to say it doesn't exist, just that it doesn't seem like Alpine comes with all preconditions met out of the box
Isn't it the opposite, accounts which have no password and hence had * instead of a hash?
> accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.
But yeah, Alpine for persistant installs isn't unheard of - just not as popular as the container use case. They offer official VM, Xen, Pi, and bare metal images users deploy as well. I actually had a fleet of VMs to update!
Edit: The Alpine notice made it clearer to me that at least one other account has to have a password, it's just the target account that doesn't. So if you truly have no users with passwords in your container deployment then those passwordless users aren't vulnerable to this bug, but any of the image types which have a user will likely have root exposed.
> The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash.
https://alpinelinux.org/posts/2026-11-05-openssh-password-au...
The part in your edit is indeed what I was referring to, sorry that could have been clearer
EDIT: works after downgrading OpenSSH, turns out even EOL Alpine version got updated packages!
Also didn't have much luck on Alpine 3.20.10 with OpenSSH_9.7p1 which I've freshly installed from an official ISO.
Last two entries in /etc/shadow:
SSHing into test@ with one 'wrong' password then one 'long' password doesn't work. I have UsePAM no and PasswordAuthentication yes in my sshd_config.No, didn't work for me open OpenWRT x64 running in a container but that is probably 'cause I'm using dropbear instead of openssh-server. That's what OpenWRT uses by default so the impact there will be minimal assuming that most installations don't change to openssh-server - it isn't installed by default.
On top of OpenWRT on my router (running 25.12) running dropbear instead of openssh-server, I also do not have any accounts in /etc/passwd|/etc/shadow with "*" for a password hash and an actual shell (all are set to /bin/false).
Did you try one incorrect standard password fail first?
I can't reproduce it either. I used alpine:3.24.1 and openssh is openssh-server-10.3_p1-r2
I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.
Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.
edit: can reproduce after `git revert 384e25e6bd1f2e7b30095900e48b570d46953510` in aports and rebuilding openssh without the fix. It also accepts the password on the 2nd+ attempt, NOT the first.
Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem. Silent compatibility hacks are the worst, they guarantee problems never get fixed and the stack of hacks never gets smaller.
Microsoft has historically been one of the leaders on the wrong side of this one, but they have also demonstrated how effective it is to call out badly behaved software and make it annoy the users with UAC. When Vista came out, so many applications needed elevated privileges to run because they were badly designed to write user data in system locations, but now their application was more annoying to use and came with a scary dialog about it making changes to your computer that other better-designed software didn't get.
More often than not if a program didn't actually intend to change system settings it was just a matter of badly chosen default paths which are pretty trivial to fix, but so many vendors just would not do it despite years of complaints from NT admins wanting to not have to spend hours with Process Explorer figuring out specific permissions to grant limited users enough access. Once their failures were put front and center, calling their application out as the one getting things wrong every time the user ran it, they suddenly had motivation to fix it.
For security it's better, but in many cases if you're not writing security-oriented software it can be worse from a user perspective. I remember the first job I had over 20 years ago was working on VoD streaming and one of the "tricks" to improve stability was mmap'ing the 0 page so that NULL dereferences wouldn't immediately segfault. Ugly as hell but when you were working with all sorts of more or less well coded media codecs and security wasn't a concern at all (worst case scenario the user would stream movies without paying) it made some amount of sense.
I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.
More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.
That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.
> if you're not writing security-oriented software
It's hard to define what's security oriented and what is not. For example, is this HN web app security oriented? Not in the way of sshd but it must keep our accounts data separated and it must not let people exfiltrate data from the db, where for example it is storing the emails of some of us.
It depends on the time. Twenty or thirty years most software was not security-oriented. Nowadays, everything is online and therefore everything needs to be vigilant about security.
> because having a critical piece of live media production reboot mid
It's not great, but these days a trickster with AI will find some way to start broadcasting porn to a billion people because of the flaw, so may end up being the better option.
> because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great.
Story time?
There's not much else to say honestly, for these very important events there will usually be several layers of redundancy for the most critical feeds, so the production is always able to send something to the satellite uplink even while our devices reboot due to a crash, but brutally losing 90% of your inputs for ~1 minute while the machine reboots is not a pleasant experience for anyone...
> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem.
100%
Agreed. I feel like this attitude has been holding Python back, too. It's one thing to criticize "move fast and break things", but it's also not good to ignore already existing breakage just because the pieces still fit together neatly, and it's not good to refuse to move at all.
> letting badly programmed software die when unexpected things happen
Go ahead and program a flight controller this way, I'm sure nothing bad will happen.
I'd still rather have the software crash and alert a human that to continue in a faulty state. The way to add robustness in this case is to add backup systems, not "ON ERROR RESUME NEXT".
> badly programmed software
Is there a test or check I can use to determine that software is 'good'?
Oh there isn't? Good/bad is subjective? We need to scape goating hacks as bad software, as if anyone is capable of perfect code. Perfect code would require knowing the system inside and out, the stack up and down - no one knows all that, no one is perfect. Sufficiently advanced AI though can rip through all of that, find the weakest spot and zero day it, bringing the entire house of cards down.
Software you thought was good can still be hacked. There is no way to create provably unhackable software.
Which means all software is packed with vulnerabilities just waiting to be discovered, no matter how good you think it is.
Now everything installs to a user local folder, runs as the user, there are no UAC prompts, and malware can just freely tamper with other applications, act as the user, and harvest all data the user has access to.
Hey, but at least it can’t touch the System32 folder.
Great job guys.
> Microsoft has historically been one of the leaders on the wrong side of this one
Objective-C ahem nil messaging...
Objective-C is an Apple thingie :)
Yes and thus I dispute that Microsoft leads on this front :P
By way of NeXTSTEP
NeXT certainly used it extensively but they didn't invent it. Originally developed by Brad Cox and Tom Love in the early 1980s, per !w.
> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option
Sadly, we will never learn this because failing fast runs counter to intuition, and a class of person will never override it.
This kind of thing is a constant worry of mine, but I've never had the (mis?)fortune to look at this particular code.
Signatures/verification (typically) only provide a boolean guard on what you want to be controlled behavior.
To properly protect against cla controlled behavior, the check must be incorporated into the EXECUTION of the behavior (with a safe fail).
This is hard and I think a lot of pentesters will continue to have lucrative work hunting issues like this down for a long time.
Fundamentally, the problem is the "type" of the verification and the execution.
So to recap, with a non-upgraded sshd you need:
- an account with /etc/shadow hash set to '*'
- (note root won't work by default because of PermitRootLogin prohibit-password default)
- attempt twice, first attempt fails second long password is accepted (that took me a while to understand)
In practice passwd -l seems to lock with a ! prefix, and new account without passwords are created with a single ! which is also not affected, I'm not sure how big the impact actually will be in practice and don't understand the recommendation to upgrade urgently right now... Just don't use `*`?
EDIT: ah, some default container images apparently have a default alpine user with that hash, so I guess that's bad for them. This makes a bit more sense now.
> some default container images apparently have a default alpine user with that hash
Do you have an example? Looking at Alpine's advisory, I can only see that their cloud images (i.e. what you use to run a VM, not a container) have a default alpine user, no container images. And even with those, cloud images with cloud-init are unaffected with default config due to `PasswordAuthentication no` being inserted by cloud-init into sshd_config. So that leaves... Cloud images that use Tiny Cloud for provisioning. That's an instance bootstrapper authored by Alpine from the looks of it so I'm not sure how many people would prefer that over cloud-init (which a lot of VM/bare metal management software may already make easy to use) or just a non-cloud image.
Why do you need to attempt twice?
> Just don't use `*`
Well but, alas, a non-zero number of user probably do use it, though and "upgrade X" is a clearer and better instruction to communicate than "change this setting where maybe you don't even know what it means and it might break your workflow, but saves you an upgrade".
In some contexts (e.g. embedded) you'd be surprised at how reluctant some people are to upgrade when they can run through a dozen of hoops for workarounds such as changing hash instead...
It's amazing how similar this exploit payload is to the 20+ year-old Quake 3 vulnerabilities found by Luigi Auriemma (aluigi), which also used the exact same 256-to-1024 character range and nothing more than a long string of AAAA.... Here you're feeding >256 A's up to a 1024-byte limit to trip musl's defensive length cap, whereas back in the mid-2000s, Luigi was famous for publishing one-line .cfg scripts where the entire exploit was literally "say aaaaa..." repeated a few hundred times overflowing server's memory.¹ ²
A great example is Jedi Academy, a 2003 Quake 3 engine game whose multiplayer is still actively played today (its open-source engine port, openjk, is even in the Debian repos btw). Luigi had two classic scripts for it on his site: Jedi Academy <= 1.011 in-game server buffer-overflow (jamsgbof)¹, which crashed the server just by pushing a string of as past the 1024-byte console buffer, and Quake 3 engine in-game players kicking (q3msgboom)², where a wall of as hit the engine's 1022-char truncation limit and the leftover bytes confused the parser into disconnecting everyone on the server.
Kind of wild that 20 years later, the exact same payload goes from crashing Quake 3 matches from the tilde console to getting root on some SSH servers.
[1] https://web.archive.org/web/20101227181948/http://aluigi.alt... [2] https://web.archive.org/web/20060522175643/http://aluigi.alt...
Offtopic: this is a very cool way to present commits and code excerpts. Love classic Windows styling.
Were those pictures with links generated using a publicly available tool that we could use, or were it custom (vibe)coded specifically to create this page?
This looks entirely vibe-coded to me. No one is going to put so much effort for a vuln.
To be honest, I love this sort of vibe-coding. Explains it really well, in an interesting way. If this is vibe-coding, then the AI has done a great job.
It was fun to see this type of style around the release of Opus 4.6, but its really tired now. I've seen countless landing pages like this...its basically ritual
There's nothing great about this site. It should give you a succinct summary of the vulnerability right up front, so you can determine if it applies to you. The history lesson is not relevant to the vast majority of visitors.
Yes but it's not your average vibe coded site, it's still decently done and clearly art-directed by a human rather than "make cool website for me plz"
From the advisory: musl’s crypt() returns the string * as an error indicator when the key is over 256 bytes long
Are these people mad? Why on Earth would anybody indicate error like this?
That's a well-made site, yet I feel nothing.
LLM output fatigue?
Apple sideways traffic light fatigue.
Good to know I'm not the only one annoyed by that superfluous piece of ${whatever} signalling.
I didn't hate the writing even if it's AI, but I deeply disliked the animations. I don't know if that's LLM fatigue or not.
I thought it was a great write up. It did feel a bit AI-coded, but whether it actually was or not, it's still a good bit of work imo.
I dunno, there were some strange discrepancies in the explanation (and captions). Didn't really hang together.
Looks good overall, but I was wondering if eg. "the timer learns nothing" and "no X, no Y, no Z" was slop rather than a deliberate choice.
Slop, no question. There's at least 10 Claudisms on the page, and I'm not that good at smelling them. Some of the sentences might have been human-written.
fwiw pangram believes the text is AI as well
It would be weird to spend this much effort doing the site by hand, for a bug that the author says was found by a swarm of agents.
I left after trying to scroll to a concise description of the issue. I don't want a theatrical presentation.
It has a crap load of text but says very little and is very bad at conveying the details because it thinks it has to tell some weird story.
I had this classmate that had their ChatGPT instructions to respond in this same laidback, lowercase, gen-z style. i recognized the output as a lot of llm writing almost immediately
Yes. Reading this feels like a LLM pretending to write carelessly. And failing miserably
Maybe that's on you though. I enjoyed it, though the discrepancy between the `*sshole` name and the `sshhole.com` domain made me twitch. How is anyone[0] obsessive enough to find this and write it up and yet not obsessive enough to make the amount of `h` consistent?
[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.
Look closely. The second h is behind the word and almost as dark as the background
Yes, but if you select the text and copy paste it it comes out with a single h, and also the <title> of the page has it with a single h and a double h.
Strange and confusing.I mean I understand that in “*sshole” it’s referencing the * that is central to the issue and putting it so it looks like a censored version of “asshole” while also referencing that it’s an ssh hole. And the domain is more simply ssh hole instead of sshole. Is a way maybe it does makes sense to do both. But it did seem at least at first strange to me that the domain is one thing and the name is a slightly different but almost same thing. Like there was a typo when registering the domain, or like the author changed their mind about the name slightly after registering the domain.
Oh. Thank you!
It has very low information density. This would read 10x if it were just a few paragraphs of text. Instead we need to render an animation every few sentences. Not a fan.
What is dead can never die. All blueteamers are ironborn these days.
I do actually. My guess is that this was not just one-prompted, but someone took the time to refine it based on their tastes and expectations until it matched something they would actually like.
However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.
I think at some point I'm going to get banned from HN for repeatedly saying this, but here we go:
This thread highlights why I often say that at some point in the future, the obsession with whether something was written by AI or not will be classified as a mental disorder.
It is tiring to virtually every day see this discussion play out in some HN thread or other. It's also just noise. I don't think your comment, nor the thread it inspired, added anything of value to the discussion.
For OpenWRT, AsusWRT etc, I believe they use dropbear ssh by default instead of OpenSSH, which apparently isn't affected?
Yes, correct for OpenWrt. OpenWrt defaults to dropbear for both server and client. However, more advanced users may optionally choose to replace it with OpenSSH, or install OpenSSH side-by-side, to gain additional features of OpenSSH.
I've just tried with my AsusWRT (3.0.0.4.386_69196) and didn't get it to work
gregor@krekrm3 ~ % ssh root@192.168.1.1 root@192.168.1.1's password: Permission denied, please try again. root@192.168.1.1's password: Permission denied, please try again. root@192.168.1.1's password: Connection closed by 192.168.1.1 port 22
Dropbear added the 100 character password limit in 2019.77, so earlier versions might be susceptible.
Nope, they are OK.
To be fair, I have less than zero sympathy for anyone who runs SSH on any platform without `PasswordAuthentication=no` in 2026.
Its not asking for trouble, it is there will be trouble, not if but when.
As for the people who say "well, I run SSH on port 942329053052". Well guess what, the bots make short work on that. We tried it at $work as an experiment, multiple bots found it in less than a couple of hours. That's why your SSH config must be hardened, preferably running on a hardened OS too.
> Its not asking for trouble, it is there will be trouble, not if but when.
Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.
> Did any of those bots brute force a password?
The point is they found the security by obscurity so-called "hidden" port in less than a couple of hours.
At that point, what they did is irrelevant. The point is that security by obscurity does not work in 2026.
> Because a 20 character random password is comparable in brute force complexity to an SSH key.
You can't be serious with your comment.
No its not.
And even then, SSH key is asymmetric, you're not sending the private key to the server. But with a password you are, which leaves you vulnerable to many attack vectors that simply do not exist with keys.
Other points are fine, but why do you say a 20-character random password isn’t comparable in brute force complexity to an SSH key?
> The point is that security by obscurity does not work in 2026.
I feel the opposite, note how most all AI vulns discovered use source code level analysis, empirically keeping source code hidden is more secure.
kerchoff’s principle is misconstrued by a voyeuristic internet that wants to see source code
This is a sampling error - we're only hearing about the public disclosures.
(almost) No company actually has secret source code, only source code not publicly viewable. And the LLMs crawling those don't belong to people who make public disclosures.
> LLMs crawling those don't belong to people who make public disclosures
Using LLMs is a great way to empower the governments behind them (US and PRC) to cultivate a garden of undisclosed bugs in your closed commercial codebases.
> > Because a 20 character random password is comparable in brute force complexity to an SSH key.
> You can't be serious with your comment. \n\n No its not.
If you differ in opinion, maybe provide evidence? Especially for a claim like long randomly generated value 1 being absurdly ("can't be serious"-level) weak as compared to long randomly generated value 2
If you mean it's log(62^20,2)≈119 bits of entropy instead of 128+ and so it's theoretically weaker when we build a Dyson sphere around the sun to brute force that key (after somehow obtaining the shadow file, since an online attack would be way too slow), then sure
How is password auth weaker than key auth? Bugs in auth code can happen in both pathways. And apart from stupidly short passwords both should be equally impossible to brute-force.
In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?
Password gets sent to the server. Private key does not get sent to the server, and authentication protocol can be forwarded safely, enabling things like ssh-agent, FIDO SSH keys. This is solvable with passwords (see: augmented PAKE), but `PasswordAuthentication` isn’t that. But there are also settings where none of this really matters, yes, to be fair if that’s what you’re referring to.
SSH certificates are cool too.
password doesn’t get sent to the server, hash does.
but I guess that the hash is always the same unlike with pub priv key mechanism.
No. The password gets sent to the server. That said, I still enable passwords on servers, but don't ever share passwords between trust domains.
> password doesn’t get sent to the server, hash does.
Kind of irrelevant what gets sent to the server and what happens on the server.
Password is already easily defeatable long before it even leaves your computer, thanks to keyloggers and/or password manager breaches and/or old-school shoulder surfing (assisted in the 2026 era by covert filming and AI).
All of which are not vulnerabilities with hardware-backed keys which you don't even need a Yubikey for these days since most modern computers and smartphones have secure enclaves and/or TPMs. (Yubikeys are still great though).
What people think of as a very strong password often has <100 "security bits" of entropy.
Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.
Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.
But the SSH keys are based on asymmetric crypto, not shared secrets.
That makes it impossible to steal the key if you MITM, but there is still the same chance of guessing it. (Assuming password has as many bits of pure randomness, such as machine generated password.)
> What people think of as a very strong password often has <100 "security bits" of entropy.
The best password in the world will still be defeated by a simple keylogger or breach of your password manager.
2^100 is a lot of things to try though
Especially if you have Fail2Ban set up.
> if you have Fail2Ban set up.
Worth noting that if you are running a remotely recent version of SSH it has everything you need built-in. No need for Fail2Ban on SSH these days if your config is fully modernised.
Could you maybe explain a bit more about how one sets up OpenSSH to block brute-force login attempts with only its built-in features?
Fail2Ban is kind of required unless one like running out of sockets and are unburden by high cpu and disk load.
There is regular bot traffic attacking SSH, but it's not "saturate your downlink" level of traffic. If you're getting attacked that heavily then you're being specifically targeted.
Huh? I run ssh (OpenSSH) on port 22 with root login enabled and no fail2ban on a single-core laptop cpu from fifteen years ago. The load average is 0.00 even with the small mail and web server that runs on it
What cpu and especially what disk load are you talking about? Also not running out of sockets, not sure if that's RAM-dependent (the box has 2GB)
I believe this depends on your exposure level. If you have one, or maybe no, domain names pointed at your server, you are probably fine. If you have tens, or hundreds, of domains pointed at your server (i.e. you are a web site hoster), you will probably need something like fail2ban.
On another server with the same setup, there's various services and websites running, so load average is never zero there (due to scrapers) but ssh has yet to catch my eye in htop. Whenever I've looked, even the amount of cpu that a Windows VM uses constantly while idle is many times higher than what ssh uses
Working as a sysadmin, and with disk encryption the constant log writes generated from ssh was enough to cause heavy load on severs. Long lasting connections from bots also forced us to increase the maximum number of sockets, as some scan waves was enough to hit the maximum and block other connections. Fail2ban has mitigated those issues to the point where neither has been an issue.
> How is password auth weaker than key auth?
Well, it's a heck of a lot weaker in this instance (at least when `PasswordAuthentication no`)
Password reuse is the problem. The password hash is stolen from some other site and brute forced offline.
Same issue with key reuse. Unless you have a policy to rotate them and verify which host they login from, someone's unencrypted hard drive from 5 years ago is more likely to have a key than password.
> someone's unencrypted hard drive from 5 years ago is more likely to have a key than password.
Even 5+ years ago, hardware-backed SSH keys were a thing (Yubikey). The only difference today is that its 100% easier to do keys properly because of the widespread commoditisation of secure enclaves in desktops and smartphones to support FIDO, its basically free of charge now because you're not obliged to buy a Yubikey.
you claimed my favorite game's character name!
100% and with local tool calls many AI systems will do ssh "for you" and end run around any security theatre that you may have going on with LLMs.
I run SSH with password authentication on but the port binds to a ULA address. It’s fine. No need to be so absolutist.
> No need to be so absolutist.
Why not ?
Even if you don't want to go all the way to SSH certificates, SSH public-key authentication is laughably easy to do.
I also don't buy the argument "but its on a ULA address". Attacker's old favourite these days is finding a way in (North–South) and then using that as a stepping stone (East–West). Also the general concept of "lower security is OK because its on the LAN" is so 1997.
Yeah because if attackers are already finding a way in, they can easily read my ~/.ssh/id_ed25519. A password is strictly better, although not by much because lateral movement is very easy anyways. The only exception is if the private key is actually on a HSM, which is the case for all my SSH private keys on iOS.
or password protect your key, at least.
> read my ~/.ssh/id_ed25519.
No excuse for doing that these days.
You can use a Yubikey or you can use your computer's secure enclave or TPM with FIDO at no extra cost.
A password is never, ever "strictly better". Seriously, you're just making shit up now.
Your comment is self-contradictory. In one sentence you say “these days” and in another you say “never, ever.” These are contradictory. What if your computer is so old that it doesn’t have a TPM?
I was already pointing out that if you have secure enclave for your private key you should use key-based authentication. That’s what I do on iOS. Otherwise a password is better than your private key in plain text on disk. So much malware steal your private SSH keys (along with your other saved passwords and browser cookies); no malware that I know of performs social engineering and tricks you into giving up your SSH passwords.
N=2 but non-22 ssh ports get almost zero traffic for me at home and at work. Not zero - but very nearly zero. shrug.
Running on that port would be perfectly safe as it's outside the range of valid port numbers.
This particular bug works even with password authentication turned off.
Where did you get that idea? The entire bug is about password hashing. With password auth off, that's not even attempted.
The exploit works for a user account that has no password. This is often confused with turning off password auth, especially in containers that have only one usable account.
It's not just the parent commenter. I'm sure somebody out there is going to remove their account password instead of editing their sshd_config in an attempt to "patch" this vulnerability, actually exacerbating the problem.
Bug > are password logins still on? they are by default
You > This particular bug works even with password authentication turned off.
No. The bug is related to password authentication. If you do not have password authentication then you are not affected.
What freaks some people out is they chose parameters where they do have password authentication but they expect to always fail. That's a bad choice, like OK you did fit a cheap manual keyhole, but you threw the keys away thinking now nobody can open that - you realise somebody with a cheap jiggler can open it right ?
read the article again
> This particular bug works even with password authentication turned off.
Does it ?
Somebody here has already said "You can set PasswordAuthentication=no or update to remediate"
We don't use musl at $work anyway, so I've no means to test.
I mean there are 65k possible ports so it's the equivalent of guessing an 16 bit password, i.e. two (extended) ascii characters. It's not very surprising that 'multiple bots' found it: a single `nmap` command will give you the open ports in a few seconds.
Its kind of cool to see such simple to follow logic bugs instead of usual memory safety issues. Each step made sense in isolation.
The terrifying decision for the API to return "*" on failure is a second-order consequence of memory unsafety, because it was a half-baked attempt to mitigate a common source of UB that people were encountering in the wild.
i'd say it is more a consequence of not having exceptions or other way of being explicit about error values.
That was a contributing factor, but there are plenty of other instances of C APIs that make use of in-band signaling like this. That the API was changed from returning one in-band signal to another is because the original value was causing programs to crash unpredictably due to UB.
Wait so musl, famously a stickler for standards, just changed the behaviour of a POSIX function?
Microsoft Teams has flagged the sshhole.com as a malicious domain and now there is a org wide investigation where I work happening right now lol
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?
As in: "Actually out there"
The hypothetical Alpine exists, yes, but what runs that like that?
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?
Possibly public SFTP servers. I have allowed null passwords in the past for anonymous public SFTP and I run Alpine Linux exclusively for my servers both VM's and physical. The shadow package (PAM) did not get installed by default. I installed it but I had to specify it in the installation so there are likely people without it. There is probably a way to find them on shodan.io.
I will activate one for people to play around with for today:
There's no firewall and no password. Hack away HN. cd /pubget so*
Hopefully some had a chance to have some fun. I moved sshd back behind haproxy. There's still no firewall but ssh should be invisible now.
A large number of Docker containers?
Edit: Sure, a small minority presumably run sshd under these conditions. But a small minority times the number of Alpine containers running in the world gets to a large absolute number.
Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
Many, many Docker containers rely on "docker exec" or the kubectl equivalent to gain CLI access when needed. To have sshd in an Alpine Docker container, you're generally need to be running init and a bunch of demons, not just a single standalone service.
I'm guessing in practice relatively few people are actually exposed. But if they are, it's catastrophic.
Docker containers run sshd?
> Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
It would be super helpful if you would answer a "what, specifically" question with a "this, specifically" itemized list or not answer at all.
This is not helpful.
Docker containers that run sshds?
Agent sandboxes today probably
I think the following should be done. Either this, or eventually DataKrash will happen. International coordinated emergency should be declared with an immediate order to shut down all non-critical internet exposed services. Critical services should be put under armed guard and remote access should be disabled except for whitelisted critical staff. Then lustration of programmers should begin. Everyone who ever coded in C(rime) language and any descendant languages would be barred from this profession at least for next 30 years. Then maybe new software will be written with due care, and any new services should be put online only when written without any C - no OS code in C, no library code in C, no C conventions, etc
From the other thread’s source [1], it affects musl without PAM. You can set PasswordAuthentication=no or update to remediate.
[1] https://hachyderm.io/@damienmiller/117563026759098135
Wow. I wonder which of the common fuzzers would have found this a decade ago. If literally any password over 256 chars works, that means nobody has ever fuzzed openssh on musl linux
This is an unpleasant era to live through, but at least on the other side we may actually have real tools for writing secure software.
The musl vulnerability looks a lot like an intentional backdoor, doesn’t it?
Choosing * instead of NULL is indeed very fishy coincidence. Of all the return values they could've picked... What are the odds?
About 1 in 256 I'd reckon
Also the coincidence of 3 weeks later adding "oh, no passwords above 256 chars" making it easy to trigger
At this point it feels to me we aren’t discovering security bugs but a decade of planted backdoors.
Never attribute to malice that which yada yada yada
And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.
No the biggest lesson of all this AI stuff to me is that we are just not smart enough as programmers
If you think about it that’s the whole idea of neural nets
We aren’t smart enough to design the function we want by hand as programmers so we need to learn it through weights
This lesson just keeps recurring in many different ways
Be careful, saying this gets you downvoted like https://news.ycombinator.com/item?id=50364646
You have considerably underestimated both how fallible humans are, and how high their tolerance is for risks which aren't immediately visible.
This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.
This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?
True, but also, that is exactly the pretext that would allow someone to maliciously, intentionally commit a backdoor.
There was even a contest for hiding/obfuscating malicious code, e.g. as an honest mistake: https://www.underhanded-c.org/ . And what do you know, the most recent, common approach shown on the page, is: improper comparison inputs that give unexpected comparison results, which is... well... this submission.
Yes, "this looks like an underhanded c challenge" was my first thought when reading this. See one mistake by one project, inject the matching intentional mistake into another project (that also conveniently fixes some known bug), and boom - plausibly deniable backdoor.
If I had a free afternoon, I'd ask a bunch of agents to track the people responsible and their other contributions (but usually the scheme is "contribute valuable patches to build reputation, and only inject an exploit when safe to do so").
actually this looks very underhanded indeed, what are the odds they pick the same sentinel character?!
Like 1 in 5? The set of go-to characters for that kind of thing is pretty small.
Why restrict yourself to a single sentinel character?
Pretty big? For this purpose, my instinct would be to select an ASCII character that's not alphanumeric, not directional, and not a common separator, and that's like, what, a set of 11 characters?
I expected let another AI slope website and instead that was a real human writing about what they found. Sure they used LLMs to find this, but they used their real fingers to write this. See, this isn’t that difficult, and it’s a lot more enjoyable to read.
It's definitely an AI slop website even if a human wrote the text. I found it pretty hard to get through the garbage animations.
I don't think a human wrote the text either, it's likely edited after generation: https://paste.c-net.org/ExtortSincere.
> mostly the boring parts where it talks to libc, like the password code
Is clearly not human.
I don't trust Pangram, but the writing is a bit weird.
Pangram false positives
It's just a LLM instructed to write everything lowercase. Fooled you
Man, that author is really committed to that lowercase aesthetic. Even the entire personal website. I wonder what's going on with that? We used to do that back on AIM informally. I mostly just see Gen Z and young kids still doing that now. Pretty crazy how all the stuff that used to be trendy comes back.
I don’t think you should let it bother you. I still found it more pleasant to read than the average article’s LLM slop
Why is this not yet published on the official website and why is there no CVE? https://www.openssh.org/security.html
Edit: My bad as pointed below this is not in OpenSSH but in musl and it appears in the relase notes https://www.openssh.org/releasenotes.html#10.6p2. I had scrolled straight to the bottom of the website because the animations annoyed me. Still would like a CVE for proper tracking though!
OpenSSH considers this to be a vulnerability in musl and not OpenSSH, but shipped a fix anyways since that helps mitigate impact... Whether you agree with that or not, it's up to you :)
Probably because mitre take days to weeks to give out CVE numbers (if ever). Just get the fix out there.
I avoided using alpine-based containers ever since finding out a lot of Linux server software is actually Linux + SystemD software. The union set API of both Linux and SystemD seems to be the default "Linux API" today.
Container images rarely ship with an init service of any kind (systemd or otherwise); the expectation is that your application is the root process. This includes Alpine-based images.
but alpine famously does not use systemd but openrc?
That's their point. The software they want to use often assumes its being run on a platform with systemd, so running that software on Alpine which makes a point to not use systemd causes challenges for them compared to using a different distro.
I use alpine-based containers whenever I can just to keep the world safer from SystemD rotting Linux.
Never enabled sshd there... except on my PostmarketOS PineBook Pro... oh shit!
What is the argument against systemd?
Is it mainly that it is too big?
I feel like everybody still loves the unix philosophy, but the previous tools were too bad, systemd just filled user demand for easier tools.
The moment someone makes a pid 1 software that is good and still modular, open and pluggable I think that will be an instant success.
Yes the other tools at the time were pretty bad, but after and around the time systemd came out, they did get much better than the sysv-init days.
Check out dinit, it's basically the same declarative philosophy as systemd, but without the bloat, and with a much simpler config API. Or s6 if you specifically care about "modular", though unless you're an embedded dev, you probably don't actually want that.
Parts of it are good, like systemd-boot, parts of it are horrendous, like systemd-logind.
But systemd is not a technical problem, it's Redhat (and IBM) having too much control over what happens to the init system.
And of course there is Pottering and his vile Microsoft stint, then another vile stint to enforce age verification on Linux.
Eh, HNers are just noisier and crankier than most. The silent majority are fine with (and even like) systemd. I have no desire to go back to SysV init.
The last few weeks have made me feel very good about my policy of zero internet-accessible ssh ports, ever. On AWS, use SSM. For a VPS or home server, access everything over WireGuard.
Such solutions always felt like swapping one risk for another. With good separation (running the VPN on a separate host from the SSH login, otherwise a bug in the VPN solution still might result in a shell on the box), at best you'd need two vulnerabilities before someone gets in, but it's not fundamentally different unless you find a way simpler protocol where you can be more certain that it's secure than ssh is (for all its scrutiny and track record of no pre-auth bugs for decades)
Putting services behind ssh forwarding was the way to secure something you're worried about rather than the other way around, but Wireguard actually makes a good argument nowadays. Iirc they did proofs for the cryptography and it has good design in terms of attack surface, and it's old/mature enough now that I'll say it's probably better than exposing SSH. On the other hand, when this thing running in your kernel gets compromised, you're slightly worse off than with someone logging into a passwordless service account as in the current bug
One thing that makes me feel a lot more confident about exposing a WireGuard port is it’s completely invisible unless you have the host’s public key. Nothing will show up on a port scan.
Its security model doesn’t rely on that fact of course, but it’s the cherry on top.
A common complaint I've seen (and I experience on occasion) is AI agents being a bit silly in their overprotective coding style. They nit pick and catch every single little error. This can be frustrating since a lot of the time they are trying to handle edge cases that are so unlikely to happen, yet to cover them requires significant additional code.
This might be the beginning of the reason we shouldn't be reading the code. It's like in chess engines where the players complain about how weird, alien and unintuitive the moves are. But 100 steps down the line the engine finds some edge case exploit that gives it enough advantage to win.
So we may be crossing a line, right now, where the non-intuitive alien code being generated by agents is seeing edge cases down the line that might allow exploits. And if we "correct" the agent, we may be leaving ourselves open to exploits we don't see and have difficulty understanding.
The answer is not to reduce our understanding. That is the worst possible conclusion we could draw from what we're seeing. The answer is to increase it.
> non-intuitive alien code being generated by agents
You're saying it as if humans can't write that too.
Tangent: Pangram says the text is 60-ish% AI-generated. Not that surprising given the look of the page and the work being done with AI in the first place, but it is interesting to me because this doesn't look like your standard wall-of-text AI purple prose. So it's notable that a) you can generate convincing text in this style and b) Pangram detects it anyway.
Panagram is just guessing.
This site is beautiful I'm going to say
PSA for OpenWrt users: If you are getting the post quantum message on login which points to openssh.com, you are likely using dropbear nonetheless. Check in System->Software. I was confused and worried why I seemingly had the vulnerable OpenSSH installed, but its just dropbear using an OpenSSH link.
So AFAIU, you are fine.
minor nit: for Alpine's root, the * came with 3.16 in 2022, so that path is 4 years old.
the sshd/musl part dates to the 2016 CVE-2016-6210 fix, so other * accounts (key-only users, LDAP users) have been open for 10 years.
the website is amazing, I just think these details would clarify things.
I have alpine linux on an arm device, and I have all the prerequisites for the exploit, yet I am unable to exploit myself. All of the Who is affected? from alpines security advisory applies to me. What am I missing?
Update: I think I got the length of the password to enter wrong. I successfully exploited myself.
My antivirus won't let me access the site, stating that it contains a threat: URL: Phishing.
The bugs and commits that "fix" them reveal somewhat of a shitshow for how musl is managed. All projects have bugs, but I can't help but question why the wheel was reinvented here. Many toes have been stubbed along the way.
Nice web site. However, I've got a server with 14 alpine-based docker containers in production. Among them 13 are widely distributed images (like web server, databases, etc.) and one is a custom build image. Only one of them have an SSH server publicly exposed, Forgejo, but it's its own internal SSH server which only accept public keys and Git commands. On the internal network, only 3 of respond to SSH. I haven't tweaked their default SSH config. None were vulnerable to this attack.
I tested an unpatched forgejo (enabling passwords temporarily, and even adding root to permitted users) and couldn't trigger this.
ETA: I could trigger it. You need to enable passwords and add a second user with a valid password, then git@ works. Obviously this still means the default forgejo is safe as you say.
it's like the reverse of the Max Levchin story about almost losing the master key for PayPal's payment credentials (not gonna spoil further than that)
https://max.levch.in/post/724289457144070144/shamir-secret-s...
A+ name
A* name
A{256,} name
*
*0
Did we declare "break the world Thursday" and nobody told me? I feel like it's three weeks in a row that we've had a once-in-a-decade security issue pop up.
I think we should get used to eternal "break the world Thursday", just like our predecessors had to come to terms with Eternal September [0].
[0] https://en.wikipedia.org/wiki/Eternal_September
They list OpenWRT as affected -- but by default OpenWRT runs dropbear not OpenSSH.
Dropbear is unaffected.
> except a lot of programs never check for NULL. they just compare whatever crypt() gives back with the stored field, hit the NULL, and crash
> so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead.
Considering it's been accepted that crashing if malloc returns a NULL is your fault, did musl really feel the need to accommodate bad implementations like this or is this just AI speculation?
"unfortunately, a large portion of programs which call crypt are not prepared for its failure and do not check that the return value is non-null before using it. thus, always "succeeding" but giving an unmatchable hash is reportedly a better behavior than failing on error." [1]
[1] https://git.musl-libc.org/cgit/musl/commit/?id=b3c4cc121f70f...
Such a questionable decision! "Rather than let a bug in a caller (not handling the valid null output and thus crashing) surface, let's just silence it and pretend that the caller is correct! A potential clash in some situations (resulting in this vulnerability) is better than a crash!"
Maybe I should update my plea to fail loudly (https://alejo.ch/3gk) to list this as an example.
An interesting thing about C is that there is no guarantee that NULL dereferencing will "surface". It's undefined behavior.
NetBSD and libxcrypt both have similar workarounds. NetBSD has a paragraph on the reasoning.
OpenBSD and glibc both have similar simple, stupid-free return value implementations. OpenBSD: GNU: The distro I use on the desktop uses libxcrypt, though. Probably more distros are. Its, NetBSD's and musl's workarounds probably make sense in some way if you have no clue what goes into your userland and are willing to write stupid software to accommodate for that.It's always a people problem. Doing a bad patch on my side is easier than bothering anyone else to fix their code
> did musl really feel the need to accommodate bad implementations like this
I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.
it's not "glibc behavior", returning null on error is in the spec
To be fair the API is older than the POSIX spec.
See also: Hyrum's Law
Thanks you to the team that found this. Exceptionally well explained and funny at the same time. What is most important is how people just checking stuff( remember the .xz hack that was tried?) has save the internets ass so many times. We should give good security researchers UBI for life for just doing good work.
It's time to switch to OpenBSD
It would be nice if the passwd crypt functions didn't have timing dependence on password length. Then there'd never need to be hacky workarounds like what musl did, and various tricks to avoid exposing whether users exist on systems.
No idea if it's likely to happen though.
We use Alpine quite a bit so I built our team ssh-audit[1]. Feel free to use it if you want.
1. https://news.ycombinator.com/item?id=50383006
So, nobody fuzz tested sshd before?!?!
> So, nobody fuzz tested sshd before?!?!
OpenBSD probably did (which doesn't use musl), RedHat probably did (or some other distro using glibc), but they all don't use musl.
Yeah, I am mystified as to how this could have happened.
I learned maybe a year in software, should not match/happen are famous last words.
This is just negligent. Not a fan of rewrite in rust, but I find it highly unlikely this would happen in that community.
Rust tells you off if you don’t handle Results. Sure, you can still just unwrap() it and have it panic, but at least that’s a specific decision instead of forgetting to check for NULL like in this case.
Can we all marvel at the site. The author says they did the discovery on November 3. Since then they apparently bought a domain, set up the webpage, created animations, and a cartoon. All kinds of stuff, and got it to explode on hackernews. A span of 2-3 days. How long ago was this not even conceivable?
Managed to reproduce this, with a lab[0] you can play with.
[0] https://github.com/leonjza/sshhole-lab
Repro in a Docker container: https://github.com/predkambrij/sshhole-repro
Not the worst site, but it's clearly a publicity machine from wakefield.co, whom I haven't heard of. Also, exploitation conditions are super narrow, albeit, any password is wild.
Reminds one of https://www.openssh.org/txt/buffer.adv.
Horrors behind comprehension all around. The code, the error case being “*”, the loose patches plastered over. musl and C are unserious
Magic characters are awesome.
Of all the named vulnerabilities, I appreciate the history lesson and screenshots of how it got there
Can we just have a site with the text and images without the awkward scrolling crap? Hijacking basic features is an instant click away from sites for me
Edit: It seems to work fine on mobile. I'm not sure why the scrolling behaviour was added on the desktop version of the site
how come such a thing can exist for more than a decade and get unnoticed? Weird times.
That's kind of shocking. I guess the swarm of AI related vulnerability findings continues.
It's shocking that it took so long with the amount of investment and hype you'd expect this kind of bugs to be already fixed, but it still need a human to tell the agents what to do, which could also work with any fuzzing framework
I really like creativity in the name, the way website built and the content
"tl;dr" was a bit of a downgrade though, but that's fine
it's a nice way to promote the company behind it
Am I the only one who is shocked that such important and security-relevant software just does unorthodox stuff like "well let's just return `*` if it fails" and other kind of "hacks" ?! The developers must know that this is very bad software engineering and will definitely cause issues at _some_ point in time, no?!
I would like to read this but the scrolling is completely broken on iPad Safari
Am I missing something or is this written up to seem a lot worse than it actually is?
It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
- server is aligned with 2 ley lines
- hacker is wearing a blue t-shirt and green socks
Ok, the last two are an exaggeration, but still. The conditions are all ANDed, miss a single one and you're fine.
On top of that, as someone else in these comments reminded, the target account needs to have a valid login shell or other exploitable program assigned to it, and these passwordless accounts tend to have nologin or something similar.
To me this seems to fall on the "technically possible, practically very unlikely" bucket, kinda like the PHP vulnerability from the other day. Maybe this has gone unnoticed for so long because it just doesn't happen.
There are a lot of systems that meet those requirements.
> - sshd compiled with musl
So alpine as well as a bunch of other embedded Linuxes. Huge install base.
> - not using PAM
What you get if you try to have a minimal setup. Default sshd config if you install from source.
> - password logins enabled
The default. Super common.
> - at least one account with '*' as the password "hash"
Almost every Linux system.
> - at least one OTHER account with a valid password
Almost every Linux system.
So sure, there are caveats, but they cover a massive install base.
This is a super bad bug
How many alpine installs are running sshd? Most containers using it as a base image aren't. Embedded Linux is unlikely to be running it, too. And if one IS running sshd, how often is it with unchanged defaults? I don't buy the inference here.
I'll concede that embedded linux might be my blindspot here. Maybe cheap web hosting, too, but that seems much less likely to be using musl.
Embedded linux is your blindspot here.
Rumor has it that Cisco has been hurriedly patching something and refusing to tell their customers what it is. I'm thinking it's this.
Most embedded systems I've seen use glibc or uClibc. I agree it would make sense for them to use musl, but the embedded space moves very slowly.
Yes
A lot of embedded systems use musl and no PAM. Some of them might use Alpine, but the distribution doesn't matter that much.
Most of those would be using dropbear rather than openssh.
And ssh? I'd assume most would use proprietary or industry-specific standard for data IO, or just https, for domestic IoT stuff
And also ssh in many cases.
I thought Dropbear was pretty much the defacto SSH server for embedded applications, is OpenSSH+musl actually a common combination for these scenarios?
Well, OpenWrt is OpenSSH + musl.
OpenWrt uses Dropbear by default:
https://openwrt.org/docs/guide-user/services/ssh/transfer_dr...
We don’t know that dropbear isn’t vulnerable. The bug here is in musl and OpenSSH is just being a bro by working around it
I wouldn't say the bug is only in musl. Musl is probably the "worst" offender here (at least choose something other than "*"!), but the change in sshd was also a little suspect. i.e. the "let's find a user and take their salt (and hashing algorithm) and use it to compute and compare the hash to avoid timing attacks" change.
I haven't thought it through, but I suspect in an ideal world, sshd shouldn't have accepted the user password regardless of the outcome of crypt() if the user did not exist and it only pretended to go through the motions to avoid a timing attack.
Did look into their patch yesterday(?) which seemed to mitigate this issue which they (rightfully IMHO) consider caused by musl. But it seems their fix is to do a sanity check on the supposed hashed password char length, and thus still theoretically still allowing the possibility of those "!!!INVALID_PASSWORD!!!" suggestions to fall through....
Tried it, doesn't work.
> Most containers using it as a base image aren't.
On top of the embedded linux blindspot, this statement seems unfounded.
> if one IS running sshd, how often is it with unchanged defaults?
By definition, I would imagine the answer to this is most of the time.
For what it's worth, default Alpine installs don't have an account with * as its password hash if you use the root account as your main one. Not sure if that's also true for ones where you set up a user, maybe those ones are vulnerable.
I'm genuinely curious what's the Venn diagram here because I feel a lot of embedded systems precisely have one condition but not the other, e.g. password yes but not necessarily multiple accounts, e.g root only, while using musl. Namely almost all for almost all, but if only one is not overlapping another then it's actually very few.
>> - at least one account with '*' as the password "hash"
> Almost every Linux system.
But not, notably, on Alpine, which uses '!' instead.
Edit: or maybe yes on Alpine? I checked my desktop and it has zero *s in shadow, but Alpine's post seems to say they do use * sometimes
I thought they had a difference in meaning, i.e. * means the user has no password and ! means that password logins are disabled for this user, but the man page does not make it clear if there's a distinction.
From man 5 shadow:
> A password field which starts with an exclamation mark means that the password is locked.
> If the password field contains some string that is not a valid result of crypt(3), for instance ! or *, the user will not be able to use a unix password to log in (but the user may log in the system by other means).
does this really matter? It could be used to do something devastating. The number of systems that match those requirements is only one factor. If just a handful of those are highly critical for humanity it could lead to disaster. Eg. hacks into fridges, centrifuges etc.
Updates since this was posted:
- blue t-shirts and green socks have been added to the default loadout for new hackers, since it's optimal according to AI and costs nothing to the attacker in terms of compute
- there are so many ley line "maps" out there that basically any server placement counts as aligned with at least two, so this one's free also
Blue t shirts and green socks?
Vulnerability-branded merch? Unsure.
Someone owns a blue shirt/green socks clearance warehouse, this is all just and ad? I've seen weirder. I can imagine visiting SanFran in a few months, and it looks like a weird sci-fi movie scene, everyone dressed the same.
SF has puffer jackets. Milan has men in blue suits. Paris has women whose wardrobes consist of black and white.
> password logins enabled
Disabling sshd password logins is security 101, and the default for some cloud provided images these days. If password logins haven't been disabled, what other doors have been left open anyway?
Now the question for the man: would fil-c prevent it :D
> > - password logins enabled
> The default. Super common.
This always seems bonkers to me. I think SSHD should give you ONE password free connection of setup to copy your key over and then it just shuts the door behind it.
I think it has to be a time limited FREE offer. (like, seriously, what if the user doesn't take the offer?)
It might actually a good idea, but I would likely avoid it since my built-in Bayesian filter would just mark it as spam.
Not sure why you're getting downvoted as from security standpoint that would be the right thing to do; but actual good server providers take it up a notch and allow you to upload the public key through their admin panel during server set up and the password you get only works locally for sudo etc. There's no reason why a fresh Linux setup should ever offer a publicly available SSH server with password based auth out of the box.
Alpine Linux (which uses musl) images are popularly deployed by those who use docker, and apparently the issue is there on the default install.
Not sure whether the PHP vulnerability actually had been exploited, but I'm pretty sure there are literally thousands of machines out there affected by this ssh issue right now as we speak.
This isn't going to end well.
---
Addendum (which may or may not be related to the above) --
Everyone has heard of the tale of the boy who cried wolf. The main moral of the story is not to troll your fellow townsfolks so they don't distrust you. But everyone with a sane mind gets that.
The (IMHO) under-appreciated lesson is actually: don't be those villagers. Believe stories you are given, even if you might look like a fool for believing in them. Don't assume what happens is the same as what happened in the past. Look at the facts yourself. Trust and verify.
You don't typically run sshd in containers, instead you only run whatever single service you set up your container for.
Sure there are native Alpine installs too, but that is far less common of a use case.
Who's out there running sshd inside a docker container? It's almost MORE work than exec-ing into the container.
I've done it in the past with "serverless" sftp servers, which is how a lot of inter-company batch transfers happen for historical reasons, especially in the financial sector. I don't recall what images I used, but it was far easier to patch with a CI/CD pipeline. We did use keys, though.
Probably a lot of companies which have their own k8s cluster and segregate kubectl access to operations only. Developers still need a way to their app backend in some cases.
Kubernetes already comes with better solutions for that sort of thing. I could see a ssh bastion container for bridging to legacy infrastructure though. Manage its keys right and it's probably more secure than a lot of alternatives.
Even if you do that why would you allow it outside VPN?
You wouldn’t, but it would still remain an avenue for lateral movement
Several CICD solutions which offer "shell into container to debug a build failure".
You'd think they'd all do the smart thing and exec a shell into the container's namespace. You'd be wrong.
I have done that in the past, in order to be able to automate copying files to the container during testing or data migration. It was an ugly hack and now I need to make double sure none of those containers are still running…
Everyone who has access to their container and not the host except by some terrible vnc or web console.
Alpine in containers is usually not affected. Most images/containers haven't ssh/sshd installed at all. Remote access is usually done from the host system.
You could start with trust and verifying what you wrote first, the popular usage of alpine in docker containers don't run sshd.
Even if 0.1% of those machines run sshd I guesstimate we'd still be talking thousands.
Really, "password auth enabled" should be enough to be a non starter these days.
you'd be surprised
only 1) is a narrowing filter. the rest is basically the majority.
Most linux distros use PAM, so 2 is also a narrowing filter. Debian, Ubuntu, Fedora, Arch, etc all use PAM.
What if you narrow it down to musl-based distros (which none of the above are). Do most musl-based distros use PAM?
Alpine is mostly used in containers, where you don't run sshd anyway. It apparently doesn't use PAM by default though.
OpenWRT also doesn't use PAM by default apparently, and there is a risk of people using password login, which you should never do past initial setup.
But outside specific niches, musl is rare, with good reason. Performance is a joke compared to glibc, especially the allocator. The only reason to choose musl in my opinion is for memory constrained devices (OpenWRT etc) or for building static binaries where performance doesn't matter but portability does. I consider the "small container" use case questionable: performance will suffer, and you could share large base layers instead which would amortize the cost over many different images.
AFAIK OpenWRT doesn't use openssh
Yeah, OpenWRT doesn't use openssh by default - it uses dropbear. You can install openssh, if you want more features and there seem to be 2 packages - one with PAM support (openssh-server-pam) and one without it (openssh-server). PAM-less one is the one without the suffix so perhaps it's more likely you'd install that one out of the two, but only if you were trying to switch to using openssh in the first place.
Those distros are all filtered by 1, no?
Yes - immediately had me in a mild panic that I’d missed another AI discovered crit bug!
Had to quickly scan the rest of the homepage of HN to confirm.
I suppose there is the chance you get to prep a system setup at one time, so you can access it at a later time?
Yes, 100% written to oversell it as hard as possible. Still a vuln but karma farming too.
Yes, missing the rest of the world that does not conform to your self-references.
> - hacker is wearing a blue t-shirt and green socks
You had me there :D
I think you're missing the reason why it's relevant now.
LLMs will find it.
They work endlessly at finding the 'and' scenarios.
This should have been closed up a long time ago.
Every time I see one of these named exploit sites I always think, "Hey look at this guy, he wants attention."
> kinda like the PHP vulnerability from the other day
What? Pretty sure Cloudflare detected widespread attacks almost immediately after it was published. We are probably lucky that not much more happened because of responsible disclosure and the large media attention led to widespread admin awareness.
It’s irrelevant how widespread it is. It’s a bad bug that should be fixed. Why do people always say stuff like this? If you’re running a system that is exploitable then you deserve to know about it and to be provided a fix. It doesn’t matter all that much how widespread it is. The CVE system doesn’t exist to cater only to highly exposed systems.
It matters how widespread it is because humans have limited time and resources to solve problems. If these conditions only exist in 0.001% of installs, is it worth the man hours to fix it? Alpine may think not, because like others have mentioned, there are likely other higher probability vulnerabilities being exposed in order to meet the conditions for this to even matter. A judgement call can (and clearly has?) be made to prioritize other work.
The good news is that considering Alpine is open-source, you can be the one to make the judgement call that it is worth fixing and go out there and fix it ;)
A trivial-to-exploit auth bypass is always worth fixing, no matter how few systems are affected right now in practice. My question is whether it's worth giving the vulnerability a name and making a webpage for it.
Sounds like you should go fix it then!
0.001% of all Linux installs (including containers, CI workers, embedded, etc) is still a huge huge number. It should be taken seriously until proven otherwise
ok then, you should go fix it! and thanks for pretending the entirely made up number of 0.001% actually matters
I’m a smart person and none of this makes any sense to me. It’s like purposefully written poorly.
Agreed. Much better explained here: https://news.ycombinator.com/item?id=50379342
If The UNIX-HATERS Handbook should ever get updated, this deservers its own chapter.
Wow seriously no unit or integration test for this?
I'm starting to realize that trusting some random people who wrote the gruddiest, boringest, most torturous low-level stuff might have been a bad plan.
At this point I feel embarrassed in front of the AIs.
At this point we should all agree that software engineering is an elaborate well paid joke.
This is a super duper annoying site for people who just want to know if their systems are vulnerable, after patching like a dozen critical vulnerabilities in other software over the last week and are almost at the breaking point of resigning.
Can we have a “don’t treat me like five/toddler” / “idgaf about the explanation” quick link to a section for adults who need to patch systems?
(And if your reaction is “just scroll down, stupid”, I’m happy to report that with the scroll jacking, I scroll not enough / past the critical sections nine out of ten times.)
Edit: You may want to read https://sshhole.com/#exposure.
Thanks for mentioning that before I tried to scroll down. Forewarned, I was able to click on the "2026" link once the left-handed "scrollbar" (not the right word but I can't figure out what the right word should be) appeared, which let me jump past most of the verbiage and get to the "are you affected?" section just a couple pages later.
What this tells me is that musl+openssh-server is an incredibly uncommon setup, as "at least 257 character string" is literally the third thing I try after "admin" and "12345" when greeted with a password input.
Joke's on you, my busybox only supports generating MD5 hashes so I'm safe.
The best part of this exploit is undoubtedly the name, *sshole.
Theory: “With enough eyes, all bugs are shallow.”
Corollary: “Nobody looks”
The proof of this corollary is left as an exercise for the reader.
This AI slop write-up is so fucking annoying to read. I don't even mind the AI generated website. Why can't you even bother to spend like an hour to properly sum up the exploit yourself?
The real name of this is sshole - the is the "star" of the show, after all.
*sshole, the * is the star of the show
You can use \ to escape formatting.
Ugh! HN got me, too
Good thing my yocto images use glibc...
SSH olé
One h or two?
What did you just call me?
Was this exploited in the wild?
came for the name, stayed for the story
Haha insane, great find and write up
something interesting to note here is that while this isn't a memory safety vulnerability, a modern systems programming language with better error handling could have easily avoided this whole situation. error handling in C is a PITA, and we're paying the cost of decades of pretending that it's not a problem
I need to check my fridge...
Love the website
All that site is lacking is marquee and furry creatures. AI slop sites are so horrible with their blinking animations, l33t dark background and chaotic text.
rich felker was such a f* (funny) guy back in the mplayer days too
never changed i guess
What is this idiotic manner of typing full articles in lowercase? Don't people learn basic orthography in school anymore? This is fucking unreadable.
quick fix?
I refuse to believe this is not a backdoor.
This is fine :)
One more reason to build your (own) Docker images on 'FROM scratch' (basically nothing).
No, the Alpine containers don't come with sshd preinstalled. You'd have to manually install via dockerfile and leave it unconfigured.
No one runs sshd in containers, at least I hope so. What would even be the point when you have docker/kubectl commands available?
Using LXC (not the same as docker) and running openssh inside the container.
I desperately want someone to create a single-letter subdomain here.
It’s dumb and negligent to be missing “PasswordAuthentication no” in 2026. It’s been so for at least ten years. OpenSSH should have turned off password auth in the defaults years ago.
at this point my only reaction is "are you fucking kidding me"
just thank god for dropbear
This one feels negligent. How does this go uncaught for 10 years? How did it make it into production period?
We need to re-think EVERY assumption about security.
This is bottom-up branding I can get behind. ()*()
I just created an Alpine container, did "apk add openssh" and this doesn't work for me. I assume it's been updated already.
I liked that this site explained the vulnerability in an easy-to-understand way.
I can't understand it. All this scrolling and pictures are distracting me, can't concentrate really.
Yep, scrolling through my phone and reading slowly to understand. I'm seeing distracting animations below what I'm reading. And when I go to them, there is no way to restart them. So it completely misses the point.
I think it varies from person to person. Since I'm kind of from the TikTok generation, I don't have much trouble reading this kind of animation, but some people prefer static sites. That's just how it is.
I also found scroll-triggered animations hard to read at first, but these days a lot of sites use them, so I've gotten used to them.
There is a myth that TikTok generation has ADHD and can't concentrate, if it were true I'd expect static sites would be easier for you, like to me. I mean, I struggle sometimes to keep attention and not to wander off mentally. With static text it is easier to fix, you have just to jump back a bit, when you notice that you've forgot to read while you were reading. Or when you are lost at too many trees you can kind of look at the forest, at the text as a whole and find your place there. But with all this scrolling it is hard to jump backward/forward at will. It feels tiring and annoying.
Related discussion: [0]
[0] https://news.ycombinator.com/item?id=50377221
> how did I ran into it? [...] i had agent swarms. a lot of them. let's not get into it
Can we please now normalize AI usage in development? Just some other day people asked for "NO AI used" badges for software as measure for quality, and praised i.e. NetBSD for being AI-free. Yes, we come from a phase of AI slop, but it has become obvious that there are plenty of legit usecases where AI beats human-only effort nowadays.
You've got a great point. But let's also acknowledge that there's a pretty stark contrast between these:
- I found a way to get AI to help me while I made this
- I asked AI to make this
The people you're objecting to are complaining about the latter, this is an example of the former.
There's a decent-size "any AI usage is theft" contingent that reacts dramatically to any usage of AI whatsoever.
Not sure why but a lot of people seem to be missing that you don't have to look at AI as a binary "you're either with us or you're with the terrorists."
Just do HITL (human in the loop). Use an LLM to help you author, but steer the AI in whatever direction you want. It's like having a slightly delusional copy of yourself to pair program with. If you take the time to provide an LLM with some additional skills files and the like, with you in the pilot seat, you can get solid results (it doesn't look as cool as a hands free agent setup but the quality far surpasses anything they can do solo).
It's a stupid and wasteful way to automate the process.
> Can we please now normalize AI usage in development?
1. It is already normalized outside of a tiny, but very vocal bubble that lives mostly on Bluesky (sometimes also on Reddit/HN/Mastodon).
2. I don't think that even the most staunch AI hater would object to using AI for security research as long as all the artifacts (code, docs) that were produced based on the research are human-written. "Yes, there is a bug in my program, but I refuse to fix it, because I know that you found it using AI" would be a pretty weird position. Not sure, how would you defend it.
I feel like there is a lot of pushback from delusional engineers, that won't accept their coding skills have been rendered useless - so they come up with tons of reasons to avoid AI, because it just has to be bad. Anything to hold up the ilusion you are an well-paid white-collar worker that can pick employers anytime. And I say this myself coding since the late 90s and making a living off it - it is a painful pill to swallow, but the genie does not go back in the box so just accept it and move on.
This is a pretty well done AI generated website, congrats you have proven it's not solely for slop.
Are articles like this deliberately designed to make it to the frontpage of HN?
what was done to report & patch the vuln? Why publish it in this format? It’s presented like a whistleblower who’s exhausted every approach, but the effort to do the right thing is not clear.
does this have CVE or something?
cve-2026-123158
> on alpine's official aws and azure images that's the default alpine user, and it has passwordless doas. so it's root
good fucking god. if I get this correctly... you can insta-own any aws/azure instance that was set up with alpine as long as it has ssh enabled, which it probably has.
Tbh the most disappointing part of this is musl (and later glibc) changing the function to stop callers crashing because they didn't RTFM.
jesus christ wtf
Can we hear something from those who say that using port-knocking is "security through obscurity"?
Really: from the XZ backdoor in SSH (which went live on some distros) to this...
What's you rationale for calling port-knocking "security through obscurity" and not "defense in depth"?
Put it this way: if I advertize that my server is using port-knocking, that the sequence changes every 30 seconds (like TOTP) and that as soon one sequence is used, it's unusable (so you've only got one attempt to SSH in every 30 seconds, even if you know the sequence)...
Where's the security through obscurity?
But really: even if had musl and PasswordAuthentication allowed (I don't have neither), how would an attacker gets through the port knocking?
More fundamentally the question is: in which way is your system that does not use port knocking more secure than the one who do?
(remember: we're talking about remote exploits by simply having exposed TCP ports open or SSH listening)
Wanna have a talk too about the "solutions" proposed by "security experts" here "because SSH is too old" and that happen to be crufts on top of SSH that had remote root exploitable backdoors? (sorry, holes)
Disclosure: I work at Detectify. A few practical notes for anyone checking their own machines:
- It affects Alpine's openssh-server (built without PAM), or openssh-server-pam with UsePAM no, when PasswordAuthentication is enabled. Alpine says every release they checked is affected, 3.16 through 3.24 and edge.
- Fixed openssh packages are out for all of those, including the older 3.16-3.20 releases. Upgrade and restart sshd (rc-service sshd restart). If you can't upgrade yet, set PasswordAuthentication no.
- To check a box: apk info -v openssh-server for the version, and sshd -T | grep -iE 'usepam|passwordauthentication' for the effective config.
- Containers built FROM alpine that run sshd are easy to miss in an inventory. So are the official tiny-cloud images (3.18.11 through 3.24.3), which ship an alpine user with no password and passwordless doas, so a bypass there means root.
- After patching, look in /var/log/messages for sshd lines saying "Accepted password for" an account that has no password.
what’s the moral of the story?
alpine stocks are crashing?
musl stocks are crashing?
glibc stonks are up
alternative implementations in general slightly down