musl’s crypt() returns the string * as an error indicator when the key is over 256 bytes long. * is also exactly what the stored field contains for every account with no password, so the “hash” matches and the login succeeds.
Does anyone know why musl does this? Are they the only C library that behaves this way?
For a C library that famously tries to adhere to POSIX as much as possible, this seems like a crazy choice. From POSIX standard:
Upon successful completion, crypt() shall return a pointer to the encoded string. The first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
Thanks yeah just woke up, finished making my way through the single-article website and still unclear if other libcs do the same thing (the article says libxcrypt picked it up, but only after musl did it).
Some more clarity ... Multiple implementations have these sentinel values (not OpenBSD). But unfortunately only musl had a sentinel value that was also used in shadow files...
Apparently because some devs didn't check for NULL which resulted in crashes. Weird choice to fix other peoples' bad programming by deviating from the spec.
To be fair (as the sshole site mentions) it is also necessary to fake a password hash check to avoid timing side channels revealing to remote clients whether users exist, so returning a sentinel value is not a horrifically unreasonable solution to make sure even naive code does approximately the right thing.
Though I am surprised that all of the code examples do not use constant-time strcmp(). Actually, I wonder if even constant-time strncmp() would still leak information about whether the account is disabled because in that case both arguments are much shorter? Food for thought.
But yeah, the crypt(3) API is very 80s C, the man page even features a lengthy description of a long-deprecated and utterly broken hashing algorithm using DES that is even implied to be broken in the text itself (and the warning message is so old it refers to "parallel computers" as though they were a new concept). Given that the default mode is to use a broken hashing algorithm, I'm shocked anyone actually uses the libc crypt(3) -- surely it's trivial to just write your own that isn't awful...?
Constant time strcmp doesn't fix it. The problem is that the user doesn't have a password at all (the hash is *). But which hashing algorithm is used is encoded in the password. So they pick a different user's hash to not leak the information that there is no password.
Alternate solution would be to hardcode a specific algorithm, but that could also leak info in many cases, I guess.
No, I get that -- my first point was meant to allude to that but maybe it wasn't phrased particularly well. The bit about constant-time strcmp() was about comparing the hashes afterwards, which was a classic attack to extract the password hash for offline cracking (though salts probably make it untenable?).
In any case, the fact you need access to an existing hash to not select something stupid like DES is all due to crypt(3) being a wacky API and while callers could work around it by selecting more reasonable defaults it's a pretty big foot gun.
This is either a bad engineering choice or an intentional bug. An empty string would have been better. Why '*' instead of, say, a tilde, @ symbol, or something that generally isn't used?
The spec only defines a single error state - ENOSYS, that is basically only for systems that don't implement crypt at all.
Error handling is underspecified. The spec doesn't give you enough well-defined behaviour, meaning everyone lists pages of error-handling extensions they have added.
Pedantic, but more "author's understanding of the spec" which isn't always correct.
> assume nonstandard behavior
I go back and forth on this one (spec vs standard behavior). It feels like at some point the most widely used impl -is- the spec, as we've seen in other places (web standards).
> Upon successful completion, crypt() shall return a pointer to the hashed password [...]. Otherwise, it shall return a null pointer and set errno to indicate the error.
To the best of my knowledge, only OpenBSD and Fedora have a crypt(3) that returns NULL. Everybody else returns a string starting with "*" on error.
For example on Debian the manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an _invalid_ hash to the output field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
Wow, what a bug. I expect the exploit is literally as simple as trying to ssh in as a user with the password has ‘*’ in /etc/shadow and typing anything as the password.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
"It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
The conditions are all ANDed, miss a single one and you're fine."
This doesn't seem to be nearly as big as an issue as they're making it out to be
Disabling PAM has been "generally good security advice" for decades; I remember suggestions to not use it in the 2000s.
Password logins are often enabled, especially on multi-user and default setups.
If alpine uses the * feature, it's going to be pretty common. Gentoo has it, as does Ubuntu.
If password logins are enabled, you're almost guaranteed to have at least one user with a password (root on all my boxes has one, even if I never use it and it's random garbage).
So the cheese lines up pretty well, this affects quite a bit.
If password logins are enabled, someone almost certainly has a password.
Yeah, as the dust starts to settle a bit it seems this is the largest case. Many 'stock' Alpine images, Docker images built on those, and embedded systems seem to be the main candidates.
No, no don't let all of that distract you. We gotta hype this. This is the new SSH2SHELL dude. Have we registered the domain yet? I'm scheduled on Fox News tomorrow.
They are going to be disappointed with SSH2SHELL it definitely isn't worth the hype. Using argument injection vulnerability created by a recent pull request that has been patched isn't exciting enough for a web site.
> We are making this release as fast as possible (we learned of the problem yesterday) because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
How do you break back into your digital life? You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop. How would you get back into your digital life given only a laptop with access to the Internet, unless something, somewhere is passwordauthenticate=yes.
i guess if i have some sort of internet access (otherwise why would i be worrying about passwords), i'd set up a new account(s) and message those friends
There's been a natural disaster and your home is now inaccessible. With global warming going the way it has, that's looking more and more likely each day. Fires, hurricanes, earthquakes, war, tsunami, etc.
If access to a system is critical, I'm going to have some way to access the system other than SSH (out-of-band access through a BMC, serial port, or VM host) that doesn't exclusively rely on the same means of authentication.
On systems that receive regular updates, SSH can fail for many reasons other than authentication problems, after all.
PasswordAuthentication=no usually means that ssh stops directly reading /etc/passwd to authenticate, not that it stops accepting passwords completely. You'll still be able to log in with a correct password, provided KbdInteractiveAuthentication is still on, PAM is available, and the PAM policy allows for it.
The email at the root of my identity is tied to a country code domain in my country. There are robust legal processes for enforcing my status as the registrant and regaining control of it. Once I have access to that, I can use that to gain access to everything downstream.
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
Remotely: An encrypted file with a seed / master passwords to all the other data, published under a public url you can remember, but not linked anywhere and not listed.
Locally: Same, but printed out, laminated, in a safe place with other documents.
I walk to my pc and type ssh-keygen. If i can’t do that I’ve got bigger problems. “I must be able to recover remotely using a memorised credential” is a terrible requirement.
Yeah your vault has password authentication. Your password manager in other words. Your vault is where you store your private ssh keys, and or the passwords for the keys.
> You must not be in a place with many huge natural disasters.
Or I have proper backups synced across the world on different providers. The likelihood of everything failing is not a vector I worry about. If that happens I have bigger problems to worry about.
Another thing I would recommend is having your servers in a wireguard net and only allowing ssh from inside that virtual network. There is no need to publicly expose ssh.
Wireguard allows effectively no configuration around auth, and so it's more likely that it is configured in a secure manner.
Openssh allows: password auth with weak passwords, PAM with potentially insecure PAM modules, gssappi, configurable ciphers, etc. Wireguard has none of those.
Each of those auth methods adds more possible vulnerable code paths and accidentally insecure configurations. It's very reasonable to believe that wireguard auth, since it has far fewer moving parts and configuration knobs, is likely to have fewer ways it can be insecure.
I was always uncomfortable that it's on by default and surprised anyone uses it it. It's also more cumbersome compared to just using a key. I feel like using keys is the password manager analogue for SSH.
Is this actually effective against Alpine? The default install seems to use ! as the placeholder in /etc/shadow instead of * and trying the attack doesn't seem to work.
IMO this is where languages like Rust shine. The design patterns built into the language for handling errors (Result) - force you to handle errors / function results with sentinels, null pointers, or errnos are so much more error prone than handling a Result<T,E> using a match statement.
crypt() returning `*` is insane especially since these kinds of sentinels are all over /etc/passwd
I guess this is where the culture around rust shines. You can write crypt that returns a string in rust just fine, just like you can write crypt that returns a tagged union in C. Cultural differences and syntactic sugar change which version is idiomatic.
I suppose if people started writing lots of code that crashed on error in Rust some enterprising developer might improve the function to always return a non-error value and set it to ‘*’ in error cases.
The culture is downstream of the design IMO. In general, people write the API that is obvious. In C, returning a string is obvious, while defining a custom OptionalString type is high friction. In Rust, returning Option<String> is as easy and obvious as returning a String.
All the standard Rust libs use results. The standard C libs use errno, return 0, return -1, return EOF, return nullptr, segfault, or of course UB. Other libs dream up things like returning "*". C++ has exceptions in some areas but not others, and if you turn them off, they're segfaults. I actually like C, at least more than C++, but the error handling is the worst part.
Sure, you could write deliberately bad code in Rust by unwrapping everything everywhere, but the point is that it is a distinct choice to do so. You can’t forget to handle errors - you have to make the choice to disregard them. If you forget to check for NULL in C it won’t complain and it’s dependent upon you to remember to do so.
Unfortunately this is also an area where a Rust programmer would likely just reach for the platform libc, because you really want to be compatible with exactly how the platform is hashing passwords. Once you know about the inconsistency, you can make predictable wrappers around it, but platform legacy is gonna keep throwing these curveballs.
In the vast majority of programs, errors and non-results are irrelevant at the call site. You want to program for the happy path and not wrap every single call with error checking code or `try/catch/finally`.
h = file_open(path)
data = file_read(h)
file_close(h)
return data
is the code you want to read. This essentially is:
return file_open(path).success(h -> h, file_read(h)).success(h, data -> file_close(h); data)
A language with thoughtful error handling primitives will support such code.
This is the root of a lot of the evil. Yes, humans want to program for the happy path. It is a lot more fun. However, that bias leads to so much lack of diligence to properly deal with the unhappy path.
Syntax is not going to solve logic errors. No amount of checked exceptions or forced unwrapping of a result will help with TOCTOU or any number of similar vulnerabilities in code.
You ideally want to attack the problem at two ends:
- make the syntax noise-free so you can see what is happening
- build primitives at the syscall and stdlib level for common footguns
You would still need to deal with domain-specific logic errors, but you have at least reduced the surface area exposed to casual attacks.
It is not. Being extremely restrictive in the happy path, throwing all over the place, but have generic error handling code at the top level (e.g. HTTP entry point) makes for some pretty robust, stable, easy to read software. Albeit it typically has somewhat rough user experience.
The thing is, that's a pretty good place to start. From there on, you can handle exceptions one by one, surgically, and make things nicer.
I keep hearing how diligent it is to handle errors at the call site, always by people who end up making services that either crash for preventable reasons or bury errors. Python is actually safer than Golang, and I don't mean with type hints.
returning a non-success you either handle it or pass it up the chain, potentially with not a lot of information about what originally caused it. with an exception you can retain the full information about what happened by either rethrowing it, throwing your own exception with it embedded as the cause or declaring that you don't handle it by saying you can throw it.
Yeah, checked exceptions are weird. Almost everything has some way to fail. The exact type of exception is usually meaningless, more like there are a couple of broad categories analogous to HTTP 4xx vs 5xx, and even then you usually don't want to handle it right there.
Well, for example, if you had opened a file to put the data into, you might delete the file or restore from a backup. But even if the answer is nothing, bubbling it all the way up to main is not the wrong answer. The point isn't whether you can do something, it's whether someone can do something.
The question is _where_ the error needs to be handled. It's almost never directly at the call site. And then there's a few issues with checked exceptions and Java language things (RMI, closures, etc) which make checked exceptions cumbersome. An exception (checked or not) might be worse than sum types, but they're still a thousand times better then returning an error in-band.
> The question is _where_ the error needs to be handled. It's almost never directly at the call site.
Which is why exceptions in Java can bubble up the stack? You don't have to catch it in a particular method you can let it go to the caller of the method, and so on.
Go doesn't require you to handle or pass an error upwards. You can call something that returns err and just ignore it, compiler won't care. The "if err != nil" thing is annoying too. Rust made the right compromise with !
Better design breeds better habits. In Java, you can end up handling a DNS exception 20 layers up the stack, even if your call never actually triggers a DNS lookup, simply because some method deep in the call stack declares it as possible. The culture is to propagate or wrap exceptions rather than translate them into domain-specific errors near the source.
Rust makes handling and mapping `Result` locally more ergonomic, even though `?` makes propagation easy too. Rust, however, also makes it easy to lose the context during propagation and end up with useless `Error`.
I agree but I think was also because Java didn't have sum types (or still doesn't? years I haven't read Java) and rethrowing is super easy, while Rust kinda nudges you into the right direction (if you need to rethrow at all, think if you want to rethrow verbatim or wrap with a semantic error type).
The problem with them all always comes when the ever-steady march of abstractions, features, and edge cases results in one routine throwing the same two error classes but in different places, and how you handle them is different despite being the same wrapper. Add a thousand developers and several years of churn, and poof you get extremely complex bugs that result from errors seeming to come from one place, but actually coming from another.
Having been on the receiving end of debugging that a few times, I no longer mind the in-band error handling, although admittedly I'm inexperienced enough to be confident its better. Just for the random code I've had to read where I just need to know exactly how something I didn't write and will never read again works, and whether the error is handled and how, i love reading in-band error handling, even if I too dislike the way an error laden routine must be written. Over time I've come to appreciate elegance "in the large" as opposed to the code I see in any individual function.
I always thought the ideas was great, but type system was holding it back. If you could have expressed that your thrown exceptions are the same as some other method's (e.g. a method of a class given as a parameter), and some other classes, then it could have been a lot less annoying. Powerful, even! But Java did take its sweet time to get even type parameters.
I have written very little Java, though, so this is more of an armchair analyst position.
Amen, people saw the symptoms of "needs more syntactic sugar" and misinterpreted it into "rigorous error handling is bad".
The same way that layered code should react/wrap/translate when it comes to return types, it should do the same for thrown exceptions, and checked-exceptions helps you do that. They're both results of computation.
Not really related to the topic, but AI-popular sentences and language structure stand out to people right away these days right? I noticed some.
"If you cannot update yet, at least disable password logins: set PasswordAuthentication no in your sshd configuration and restart sshd. Either one closes the hole." (only AI would characterize a critical SSH implementation vulnerability as a hole closing game)
An attacker who can reach your SSH port can log in without knowing any password. It takes one connection, works every time, and needs no retries. (It sounds like the writer is bragging about the exploit instead of describing it. AI does this)
The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash. The attacker then sends a very long password (257 to 1024 bytes). One user with a password is enough: it makes every account without one reachable. (same as above made stronger with the 'statement: statement2' format)
"Users coming from a directory service through musl-nscd (nslcd, winbind, sssd), since they have no shadow entry." (a human would be more clear about what a shadow entry is and give more context, I think)
Hmm the rest of the article (the remaining 70-80%) seems quite human.
Anyway, not sure why I spent 10+ minutes on this. It just jumps at me right away.
An aside but one nice thing I realize about not using LLMs is, I guess, I am not tortured by this kind of thing, because presumably I am not seeing it all the time. I can't even imagine quite being in the mode of reading this kind of text so closely like this, really. Its a security advisory, I don't really have my critical prose-reviewer hat on.
As a grad student I read and graded a lot of student philosophy papers (well before LLMs), and I just remember the exhaustion of it. It made me appreciate how much I could otherwise just read something, get information or enjoyment out of it, and not focus on each sentence construction.
Just to say, I do believe you, I get that if you have to put up with it so much in general, it must prime you to find it everywhere. You have to read so much more now, I'm sure, so you're trained to be in that mode. It makes sense! I am just sorry its like this for you and most everyone else now. It sounds exhausting.
The potential affected space is huge.. it does not matter if most docker images ship without openssh.. it does not matter that most security conscious image builders do not enable plain passwordauth.. it does not matter that ssh port is not commonly exposed.. the amount of alpine docker images in current live use over the whole internet is such a vastly huge number that you're going to bet that even given a small percentage of those 'misusing' docker and their containers being affected is still going to result in a a huge number of affected containers.
We have clients (regulated banks and insurers of various sizes) that send us feeds via SFTP. You might be surprised what proportion of them ignore our “we strongly recommend key based authentication” and ask for password auth instead.
When I set up a client sftp drop I had the luxury of saying "key based access only"
Then it was a surprisingly customer service intensive process to get the keys from the customers. I never did manage to automate it fully. Weird enterprise systems I have never heard of with strange key formats. ssh1 keys, email mangling them. Their internal processes requiring a password.
And way to many "I am sorry but that was the private key you sent me, That key is now compromised and you will need to remove it and generate new keys anywhere it was used"
I almost regret not allowing passwords, almost, despite the headaches key based is still much better.
At least in that case, if the account is compromised we definitely didn't leak the credentials. That we don't have them to leak is one of the advantages of key-based auth.
[of course not having the credentials to leak does not protect us from this sort of bug, it is just one factor in a multipart security stance]
Previous company: probably. It could be a dealbreaker for the client, and we bent over quite readily on a number of issues considered minor (keys are best practise, but passwords are so common it is difficult to argue that they are so bad they should be refused).
Current state (now we have been bought and are effectively more than an order of magnitude bigger): I definitely don't have opportunity to make a unilateral decision like that. Even though the larger company has a more “fair enough, feel free to go elsewhere” attitude when a small client says “if you can support × that is a dealbreaker for us” then we could previously afford to take as our stance, some of the bigger clients are definitely paying us enough, and do use passwords, that someone above me would overrule a keys-only decision.
Don't most cloud providers default to that? I know I had to force some to deploy with a key and even then PasswordAuth was often on and had to be disabled.
I went and checked this. I can admin my server with Cockpit using either the web service (using my username/password) or the desktop client (via private key) without issue even with SSH password authentication disabled. I do usually disable the web service so had to reconfigure to check.
I agree entirely, but I'm not sure changing `crypt` to return `"*"` instead of `NULL` on failure qualifies. It's much more of a considered decision, which happens to be very wrong.
> set PasswordAuthentication no in your sshd configuration and restart sshd.
Funnily enough, they don’t recommend checking pubkey auth is enabled and working. By blindly (or agenticly) following those instructions the risk is getting locked out of some machine.
> because we have recently observed multiple cases of
rapid bug rediscovery by AI agents, sometimes within hours, and so now
consider long embargo periods to be a dangerous practice that primarily
benefits attackers.
Seems quite reasonable. It’s quite obvious that black hats have day jobs as White hats and have been privy to embargoes for probably decades.
Embargo should last at most to the moment fix is made and code passes test suite.
Anything longer is putting systems at danger. I can recompile the package with a patch (even if Ubuntu/RHEL takes 3 weeks to do it) myself in a pinch, and I'd like to have a chance for it before the bug gets re-discovered by malicious party
What if the test suite passes in the middle of the night? Should all administrators interrupt their sleep and get up to recompile their servers immediately?
All times are the middle of the night somewhere. Now, it's true that some times are the middle of the night to more people than others, but that's rarely a usable piece of info.
That is generally the practice in professional environments, yes. Many places have an on-call rotation, and those that don't soon discover why. An embargo wouldn't fix that particular problem anyway, because time zones exist.
Well, OpenSSH contributes to this by mixing hashes from the shadow file. The musl code would be correct if OpenSSH passed in the hash from the user account that is being logged-in.
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
Static linking makes distribution easier, but it's an absolute security nightmare.
With dynamic linking, you just upgrade the library on your system and you're done. With static linking, every binary that uses the library has to be rebuilt.
both alpine and musl are great in my opinion, and as far as I know theres no empirical evidence that it is more or less secure overall than alternatives. would you like to elaborate your opinion?
IMO only good thing about alpine is that it's a good environment to build stuff that links musl statically.
Never understood why Alpine is used to actually run anything in a container - if you statically link binary, then you don't need alpine as a base and if you don't what is the point? Don't @ me about image size, even without layer caching we're talking about tiny difference that is often dwarfed by size of your runtime.
Musl itself is a great example of taking "simple and lightweight" so seriously that everyone else gets to deal with the complexity. Slow allocator, underoptimized memcpy/string routines, limited locale support, no NSS, and endless glibc compatibility headaches.
But hey, at least libc is small. %%Proceeds to pull in megabytes of <any other>malloc, and other libraries to replace the slow parts of musl.%%
And now sshhole.com is a particularly spectacular example of what happens when "simple" meets the rest of the software ecosystem.
I have to admit that I am a bit disappointed in this kind of thinking. There's vulnerabilities everywhere as especially the past few weeks have shown.
One occurrence is not a statistic and should not be used as evidence - it's just an example of a vulnerability that due to various bad circumstances lead to a particularity bad outcome. This logic is in my opinion equally flawed as calling alpine safe now because this particular vulnerability has been fixed.
I'm not a big fan of blaming and I think it's just a result of bad circumstances, but it's interesting to see how some seem to blame musl, but one could argue further that musl documented the behavior of their API which was ignored by openssh and so on
It's nice to see how quickly this is being addressed, and likely monitored for any unintended side effects.
Looking at the unpatched Ubuntu things in my life, and I'm pretty close to asking Claude to update things to see how quickly Alpine can get close, assuming responsiveness can stay this fast and quality.
i think it's just unbelievable there isn't a bigger push to move away from c to languages that can prevent stuff like this from happening. this and just the pure unsafety. it's 2026. it's like pure negligence at this point, the equivalent of building a bridge with matchsticks because you like using it more.
How would using a language different than C help here? At it's core this bug is a logic bug, where "*" == crypt(x), because crypt in musl started returning "*" in some cases some years ago. This could be Rust. This could be Go.
Another comment said that returning a * was to avoid returning null which would crash users of the library who didn't check for null. That seems like a C smell to me
Not really. This is like if the char* were an Option<String> but you had programs which stupidly didn't check if the Option had something and assumed it always did. As others have said, Rust would not prevent this.
`Option<String>` or `Result<String>` forces the check at the callsite. Unlike C, you cannot forget that the function can error. This absolutely helps prevent issues like this.
Now, the decision to return `"*"` instead of letting poorly written code crash on `NULL` is not good either.
I don't think it's that people forget it while writing the code, it's that people want their program to work with as little effort as possible and decide to ignore it "for now". And that definitely can happen with Option and Result as well. There's no protection for not checking return values, and it's probably not even possible to protect against (in entirety)
Well they would use `unwrap` if they were lazy, but at least that would be quite visible in the code and would not result in undefined behavior due to operating on a null value. And there might be a group of developers who first choose to use .unwrap. I hope it is the minority.
Given how easy it is to just forward that error to the caller, I think most Rust developers would just choose to do that.
While dereferencing a null pointer is technically UB, it practice it leads to a safe trap. So .unwrap is the right comparison. If you always forward the error to the caller, it will also just lead to some random error message at some point, not really different to a crash.
> There's no protection for not checking return values, and it's probably not even possible to protect against (in entirety)
That's exactly what you get with Result and Option, it's impossible to forget.
Compare:
char* hash = crypt(key, salt);
// We crash here on the hash derefence, it's no clear from the code
// that we were supposed to check for NULL or that we forgot.
if (!strcmp(hash, expected)) {
}
To:
let result: Result<Vec<u8>, Error> = crypt(key, salt);
// We MUST write this line, which is a very clear signal
// that we can panic here.
// This can be grepped for or forbidden.
let hash: String = result.unwrap();
// if result == expected { would type error
if hash == expected {
}
Sure, but it's far better than the C code and I don't think in Rust a library would start returning `Ok` with sentinel values to "fix" shit calling code. Maintainers would, rightfully, tell people to fix their code.
Don't get me wrong, I think rust has some great features and does some things extraordinarily good. But at the same time I can see that for 99% of cases it's just as prone to bugs, including security and supply chain attacks, as any other language. I think I could just as much argue that any "->" is clearly de-referencing that needs checking, which is also pretty easy to see.
this is theoretically, empirically, and objectively wrong. akin to saying that you can get into an accident in a truck as well as a motorcycle, so in 99% of cases there is no difference in safety.
different kind of programming languages change the kinds of bugs and their properties, but it does not change the amount of bugs per line of code. in respect to C and rust for example, rust may have less memory bugs - which LLMs seems to be able to spot now - but it did increase supply chain attacks for example.
> that only tries to show a single study is flawed, nothing else
no the study failed to show any difference in bug count for a specific set of common used languages, where C and Python were both part of the dataset.
I don't think programming languages have a big impact on bugs per line when every line is written by a human. Now given that I was talking about bug counts, not severity.
it showed errors in the methodology of an earlier study, it did not produce a novel result.
kind of reminds me how the "research" suggests that there are no practical differences between dynamically typed and statically typed languages...except we know from practical experience that there is, and the literature there is horribly flawed.
we have real-world evidence that rust leads to significantly less bugs in the real world
> it showed errors in the methodology of an earlier study, it did not produce a novel result.
It showed that there's no big difference. From the link:
> Combining corrections for all of these aforementioned items, the reanalysis revealed that only 4 out of the original 11 languages correlated with abnormal defect rates, and even for those the effect size is exceedingly small.
--
> we have real-world evidence that rust leads to significantly less bugs in the real world
no, we only have evidence for memory bugs, that's not what I was talking about
>because crypt in musl started returning "*" in some cases some years ago
In Rust, the type of the return value would be a `Result<T, E>`, which means the chain of events that led to this would never have started in the first place.
In C the return value would be NULL, except it was changed by musl at some point.
Edit: at the beginning, crypt always returned a value. So hypothetical Rust return type would be a String, and then it would be retained for backward compatibility. Or something else would happen, this is a complex logic bug triggered by a 25 year old chain of events.
That's what led some people to write code where it crashes without checks. Which is still their mistake. But if the type was something that can't be implicitly used like an always-present value, that would have prevented the situation, even with those same people.
I write firmware where the program has to run for years unattended.
You are right, C with manual memory management, no lifetime management, lack of phat pointers and a culture of leet has problems with deranged maddening bugs that other languages don't have.
But this bug first shows how inband signaling is bad news usually. But worse it's a failure to adhere to the spec properly. If you implement a library and you don't adhere to the spec you're going to bone someone.
Perhaps even worse is obviously the spec doesn't come with any standard testing framework. That was also the root of Apples infamous goto bug.
Rust would not save you here as this is a logic bug. Nor would, well, really any other language. I suppose someone might claim that Rust's result type would save you but it really wouldn't: I can easily make the error type a String or pretty much any other type with little consequence.
Rust would absolutely have saved you here. Even with `Result<String, String>`, which is pretty poor design, there's no confusing `Ok(s)` with `Err(s)`. Sure you can go out of your way to write something like
let hash = match result {
OK(s) => s,
Err(s) => s,
};
but at that point I feel like the library is forgiven (except for the poor error design).
@my-huge-pony does make a good point elsewhere in the thread: if the potential error was not foreseen and the code was written originally to return `String` instead of `Result<String, Error>` then even Rust would be in a pickle.
it's unbelievable to me that there isn't a larger movement of developers to move away from c towards better & safer languages. look at how much pushback the rust4linux folks get, primarily from c developers who are simply afraid of change. imagine engineers not adopting safer building practices because "they prefer the old way"....it's disgusting honestly.
what will it take to move away from c? older devs retiring. Planck said that "science advances one funeral at a time" and i fear it will be the same for software.
Moving away from C in any POSIX system is in fact stop being POSIX system as POSIX and linux being unix like system is built on standard that is defined as C library specification.
Rewriting it to Rust is fine attempt, just hard to call it is unix/posix anymore. It may not matter but semantically true.
Cars without seatbelts and airbags chopping off driver's heads in relatively slow collisions? Well guess what, that could happen in a modern car as well if the driver sticks their head out the window while going through a very tight tunnel!
Except in the case where their images are built with OpenSSH instead of Dropbear, which is the case for some manufacturers. (Turris, was noted as one such vendor)
It didn't give me the impression that I was dealing with a secure piece of software, it seemed like they were dealing with very complex issues while laughably failing at the industry basics that were settled years ago.
Similarly Ubuntu was also the only other distro that fucked this simple permissions config up, and lately with the latency to fix the TCP vuln, it seems to confirm that these are not reliable distros for the coming security era.
Seems slightly confusing that the OpenSSH release notes page says:
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
musl libc violates the POSIX documented behavior of `crypt()`¹
> Upon successful completion, crypt() shall return a pointer to the hashed password; the first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
OpenBSD is not affected, so apparently they are doing something right. But also glibc linux is not affected and pam linux is not affected, so it is only musl linux with the remote hole.
You became a victim of a very misleading post title. This isn't an OpenSSH problem at all, it's a musl problem.
I'm pretty sure the misleading wording is deliberate, OpenSSH is an excellent but non-commercial project which a bunch of entities would love to gain control of, in order to bug it with their spyware - thus the constant nagging on the bot-infested web.
musl’s crypt() returns the string * as an error indicator when the key is over 256 bytes long. * is also exactly what the stored field contains for every account with no password, so the “hash” matches and the login succeeds.
Does anyone know why musl does this? Are they the only C library that behaves this way?
For a C library that famously tries to adhere to POSIX as much as possible, this seems like a crazy choice. From POSIX standard:
Upon successful completion, crypt() shall return a pointer to the encoded string. The first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
https://pubs.opengroup.org/onlinepubs/9699919799/functions/c...
There's a very detailed trace of how the bug was developed over decades at sshhole.com and the relevant discussion in https://news.ycombinator.com/item?id=50378228
Thanks yeah just woke up, finished making my way through the single-article website and still unclear if other libcs do the same thing (the article says libxcrypt picked it up, but only after musl did it).
Some more clarity ... Multiple implementations have these sentinel values (not OpenBSD). But unfortunately only musl had a sentinel value that was also used in shadow files...
https://hachyderm.io/@ncopa@fosstodon.org/117564809789053485
Apparently because some devs didn't check for NULL which resulted in crashes. Weird choice to fix other peoples' bad programming by deviating from the spec.
To be fair (as the sshole site mentions) it is also necessary to fake a password hash check to avoid timing side channels revealing to remote clients whether users exist, so returning a sentinel value is not a horrifically unreasonable solution to make sure even naive code does approximately the right thing.
Though I am surprised that all of the code examples do not use constant-time strcmp(). Actually, I wonder if even constant-time strncmp() would still leak information about whether the account is disabled because in that case both arguments are much shorter? Food for thought.
But yeah, the crypt(3) API is very 80s C, the man page even features a lengthy description of a long-deprecated and utterly broken hashing algorithm using DES that is even implied to be broken in the text itself (and the warning message is so old it refers to "parallel computers" as though they were a new concept). Given that the default mode is to use a broken hashing algorithm, I'm shocked anyone actually uses the libc crypt(3) -- surely it's trivial to just write your own that isn't awful...?
Constant time strcmp doesn't fix it. The problem is that the user doesn't have a password at all (the hash is *). But which hashing algorithm is used is encoded in the password. So they pick a different user's hash to not leak the information that there is no password. Alternate solution would be to hardcode a specific algorithm, but that could also leak info in many cases, I guess.
No, I get that -- my first point was meant to allude to that but maybe it wasn't phrased particularly well. The bit about constant-time strcmp() was about comparing the hashes afterwards, which was a classic attack to extract the password hash for offline cracking (though salts probably make it untenable?).
In any case, the fact you need access to an existing hash to not select something stupid like DES is all due to crypt(3) being a wacky API and while callers could work around it by selecting more reasonable defaults it's a pretty big foot gun.
hmm why not make some fake-users just for this stuff?
you mean like daemon?
This is either a bad engineering choice or an intentional bug. An empty string would have been better. Why '*' instead of, say, a tilde, @ symbol, or something that generally isn't used?
I'm not entirely sure an empty string, tilde, or @ symbol would really have been better, except in retrospect.
TBH that sounds like a straight up bug in that it deviates dangerously from the spec.
The spec only defines a single error state - ENOSYS, that is basically only for systems that don't implement crypt at all.
Error handling is underspecified. The spec doesn't give you enough well-defined behaviour, meaning everyone lists pages of error-handling extensions they have added.
Which is notable because musl is more known for adhering to the spec to the point where they break programs that assume nonstandard behavior.
> adhering to the spec
Pedantic, but more "author's understanding of the spec" which isn't always correct.
> assume nonstandard behavior
I go back and forth on this one (spec vs standard behavior). It feels like at some point the most widely used impl -is- the spec, as we've seen in other places (web standards).
The spec is pretty unambiguous here: https://pubs.opengroup.org/onlinepubs/9799919799/functions/c...
> Upon successful completion, crypt() shall return a pointer to the hashed password [...]. Otherwise, it shall return a null pointer and set errno to indicate the error.
You need to add that the spec _also_ only defines a singular error state, that isn't applicable here.
> The crypt() function shall fail if:
> [ENOSYS]
> The functionality is not supported on this implementation.
I don't think those error sections are generally meant to be exhaustive?
To the best of my knowledge, only OpenBSD and Fedora have a crypt(3) that returns NULL. Everybody else returns a string starting with "*" on error.
For example on Debian the manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an _invalid_ hash to the output field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
Wow, what a bug. I expect the exploit is literally as simple as trying to ssh in as a user with the password has ‘*’ in /etc/shadow and typing anything as the password.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
https://man.archlinux.org/man/crypt.3.en
You can’t call this function correctly. sshd’s “fix” is basically to assume that a too-short return is an error return instead of a success return.
PAM is also awful, but this is worse.
Maybe I'm misunderstanding, but u/tonnydourado pointed out on another thread https://news.ycombinator.com/item?id=50378228#50380029
"It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
The conditions are all ANDed, miss a single one and you're fine."
This doesn't seem to be nearly as big as an issue as they're making it out to be
Alpine matches the first.
Disabling PAM has been "generally good security advice" for decades; I remember suggestions to not use it in the 2000s.
Password logins are often enabled, especially on multi-user and default setups.
If alpine uses the * feature, it's going to be pretty common. Gentoo has it, as does Ubuntu.
If password logins are enabled, you're almost guaranteed to have at least one user with a password (root on all my boxes has one, even if I never use it and it's random garbage).
So the cheese lines up pretty well, this affects quite a bit.
If password logins are enabled, someone almost certainly has a password.
So it's essentially one of the 'recommended configurations'?
I'm seeing people I generally trust in the scene memeing on twitter, while others are freaking out. Its hard for me to tell how fucked we are, I guess
It's always 50/50, you're either fucked or you ain't.
I think this time around it's mainly Alpine Linux and a few custom-rollers (Gentoo with musl could easily get to this place).
Yeah, as the dust starts to settle a bit it seems this is the largest case. Many 'stock' Alpine images, Docker images built on those, and embedded systems seem to be the main candidates.
Which distro has password authentication enabled by default? I haven’t seen that in years.
No, no don't let all of that distract you. We gotta hype this. This is the new SSH2SHELL dude. Have we registered the domain yet? I'm scheduled on Fox News tomorrow.
They are going to be disappointed with SSH2SHELL it definitely isn't worth the hype. Using argument injection vulnerability created by a recent pull request that has been patched isn't exciting enough for a web site.
FWIW, I made a bit of an error in my very quick analysis. See: https://news.ycombinator.com/item?id=50380581
> We are making this release as fast as possible (we learned of the problem yesterday) because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
musl only, the quick fix:
Which is a good advice in general.How do you break back into your digital life? You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop. How would you get back into your digital life given only a laptop with access to the Internet, unless something, somewhere is passwordauthenticate=yes.
> You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop.
at this point the _last_ thing i care about is my email.
where am i? am i ok? how are my friends/loved ones?
once those are in place, it becomes clearer how to get back to "normal"
With nothing, how do you reach them? Your friends and loved ones are what's at the other end of the email/Internet access.
i guess if i have some sort of internet access (otherwise why would i be worrying about passwords), i'd set up a new account(s) and message those friends
The somewhere for password authentication is a place you have physical access. Not ssh.
There's been a natural disaster and your home is now inaccessible. With global warming going the way it has, that's looking more and more likely each day. Fires, hurricanes, earthquakes, war, tsunami, etc.
If access to a system is critical, I'm going to have some way to access the system other than SSH (out-of-band access through a BMC, serial port, or VM host) that doesn't exclusively rely on the same means of authentication.
On systems that receive regular updates, SSH can fail for many reasons other than authentication problems, after all.
PasswordAuthentication=no usually means that ssh stops directly reading /etc/passwd to authenticate, not that it stops accepting passwords completely. You'll still be able to log in with a correct password, provided KbdInteractiveAuthentication is still on, PAM is available, and the PAM policy allows for it.
The email at the root of my identity is tied to a country code domain in my country. There are robust legal processes for enforcing my status as the registrant and regaining control of it. Once I have access to that, I can use that to gain access to everything downstream.
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
Remotely: An encrypted file with a seed / master passwords to all the other data, published under a public url you can remember, but not linked anywhere and not listed.
Locally: Same, but printed out, laminated, in a safe place with other documents.
Sometimes you just have to start over. It’s not the end of the world.
I walk to my pc and type ssh-keygen. If i can’t do that I’ve got bigger problems. “I must be able to recover remotely using a memorised credential” is a terrible requirement.
You go travelling in a foreign country and get mugged. You have no phone no wallet no keys, just something you know and the goodwill of strangers.
> goodwill of strangers
Who can presumably get you in touch with the relevant embassy or consulate or, failing that, local authorities.
Does this happen to you a lot? You may have other issues
Yeah your vault has password authentication. Your password manager in other words. Your vault is where you store your private ssh keys, and or the passwords for the keys.
I log in via console, either physical or virtual (the latter requires MFA though)
A scenario where all the backups are gone and the server is somehow the only thing that survived is not a very plausible one.
Depends if you're backing up somewhere else online (also is that protected by 2FA?). You must not be in a place with many huge natural disasters.
And this is assuming the server is somewhere else on the cloud.
> You must not be in a place with many huge natural disasters.
Or I have proper backups synced across the world on different providers. The likelihood of everything failing is not a vector I worry about. If that happens I have bigger problems to worry about.
1Password subscription and the memorized key. 1Password is the only place my ssh private keys are stored.
Another thing I would recommend is having your servers in a wireguard net and only allowing ssh from inside that virtual network. There is no need to publicly expose ssh.
Why is wireguards auth better than openssh auth?
Wireguard allows effectively no configuration around auth, and so it's more likely that it is configured in a secure manner.
Openssh allows: password auth with weak passwords, PAM with potentially insecure PAM modules, gssappi, configurable ciphers, etc. Wireguard has none of those.
Each of those auth methods adds more possible vulnerable code paths and accidentally insecure configurations. It's very reasonable to believe that wireguard auth, since it has far fewer moving parts and configuration knobs, is likely to have fewer ways it can be insecure.
Layered security is better (as long as their weaknesses don't overlap).
Not only that, wireguard doesn't even reply if they don't have the right key, while SSH does so the attacker knows there is at least a target.
Yeah until they discover a vuln in wireguard.
Then you still have your ssh safeguards
I was always uncomfortable that it's on by default and surprised anyone uses it it. It's also more cumbersome compared to just using a key. I feel like using keys is the password manager analogue for SSH.
It's irrelevant. There's no reason this should be any more effective in the face of software bugs then anything else.
imo this is a good advice only if the devices that connect to the server have their home encrypted and ssh private key protected.
Is this actually effective against Alpine? The default install seems to use ! as the placeholder in /etc/shadow instead of * and trying the attack doesn't seem to work.
> At least one local account has a $5$ or $6$ password hash.
I'm not sure this gets the point across when (according to https://sshhole.com/) it effectively means "any other user with a real password"
IMO this is where languages like Rust shine. The design patterns built into the language for handling errors (Result) - force you to handle errors / function results with sentinels, null pointers, or errnos are so much more error prone than handling a Result<T,E> using a match statement.
crypt() returning `*` is insane especially since these kinds of sentinels are all over /etc/passwd
I guess this is where the culture around rust shines. You can write crypt that returns a string in rust just fine, just like you can write crypt that returns a tagged union in C. Cultural differences and syntactic sugar change which version is idiomatic.
I suppose if people started writing lots of code that crashed on error in Rust some enterprising developer might improve the function to always return a non-error value and set it to ‘*’ in error cases.
The culture is downstream of the design IMO. In general, people write the API that is obvious. In C, returning a string is obvious, while defining a custom OptionalString type is high friction. In Rust, returning Option<String> is as easy and obvious as returning a String.
In a C function returning a string, two or three idioms are common for signalling an error.
One is to return NULL for errors.
Another is to make the string result a parameter and use a return code to signal error.
A third, less common but still out there, is an error parameter.
Failure to observe these is on the caller. Forcing the caller to consider the error is probably where rust shines.
But a good C programmer is thinking about these idioms when they encounter a function signature for the first time, or they write one.
All the standard Rust libs use results. The standard C libs use errno, return 0, return -1, return EOF, return nullptr, segfault, or of course UB. Other libs dream up things like returning "*". C++ has exceptions in some areas but not others, and if you turn them off, they're segfaults. I actually like C, at least more than C++, but the error handling is the worst part.
Sure, you could write deliberately bad code in Rust by unwrapping everything everywhere, but the point is that it is a distinct choice to do so. You can’t forget to handle errors - you have to make the choice to disregard them. If you forget to check for NULL in C it won’t complain and it’s dependent upon you to remember to do so.
Unwrapping everything isn't even bad, that's what most webservers will do
Unfortunately this is also an area where a Rust programmer would likely just reach for the platform libc, because you really want to be compatible with exactly how the platform is hashing passwords. Once you know about the inconsistency, you can make predictable wrappers around it, but platform legacy is gonna keep throwing these curveballs.
always has amazed me that there was so much hate for java checked exceptions where handling errors ought to be the norm.
In the vast majority of programs, errors and non-results are irrelevant at the call site. You want to program for the happy path and not wrap every single call with error checking code or `try/catch/finally`.
is the code you want to read. This essentially is: A language with thoughtful error handling primitives will support such code.> You want to program for the happy path
This is the root of a lot of the evil. Yes, humans want to program for the happy path. It is a lot more fun. However, that bias leads to so much lack of diligence to properly deal with the unhappy path.
Syntax is not going to solve logic errors. No amount of checked exceptions or forced unwrapping of a result will help with TOCTOU or any number of similar vulnerabilities in code.
You ideally want to attack the problem at two ends:
- make the syntax noise-free so you can see what is happening
- build primitives at the syscall and stdlib level for common footguns
You would still need to deal with domain-specific logic errors, but you have at least reduced the surface area exposed to casual attacks.
> This is the root of a lot of the evil
It is not. Being extremely restrictive in the happy path, throwing all over the place, but have generic error handling code at the top level (e.g. HTTP entry point) makes for some pretty robust, stable, easy to read software. Albeit it typically has somewhat rough user experience.
The thing is, that's a pretty good place to start. From there on, you can handle exceptions one by one, surgically, and make things nicer.
I keep hearing how diligent it is to handle errors at the call site, always by people who end up making services that either crash for preventable reasons or bury errors. Python is actually safer than Golang, and I don't mean with type hints.
People using exceptions want to handle the unhappy path, just usually not at the call site.
that is the same as saying the method throws IOException
It might throw one. There is a difference between documenting what exceptions can be thrown and forcing every consumer to handle them.
returning a non-success you either handle it or pass it up the chain, potentially with not a lot of information about what originally caused it. with an exception you can retain the full information about what happened by either rethrowing it, throwing your own exception with it embedded as the cause or declaring that you don't handle it by saying you can throw it.
I generally prefer exceptions to result codes. You can even use the mechanism to throw useful results across nested function calls.
I do prefer Maybe/Option/Either style returns in some cases though. Makes for a cleaner API.
So each method has its uses.
Yeah, checked exceptions are weird. Almost everything has some way to fail. The exact type of exception is usually meaningless, more like there are a couple of broad categories analogous to HTTP 4xx vs 5xx, and even then you usually don't want to handle it right there.
IOException must be #1 on the list of most ignored exceptions. Like okay what am I supposed to do really I'm just trying to run a select against my db
Well, for example, if you had opened a file to put the data into, you might delete the file or restore from a backup. But even if the answer is nothing, bubbling it all the way up to main is not the wrong answer. The point isn't whether you can do something, it's whether someone can do something.
The question is _where_ the error needs to be handled. It's almost never directly at the call site. And then there's a few issues with checked exceptions and Java language things (RMI, closures, etc) which make checked exceptions cumbersome. An exception (checked or not) might be worse than sum types, but they're still a thousand times better then returning an error in-band.
> The question is _where_ the error needs to be handled. It's almost never directly at the call site.
Which is why exceptions in Java can bubble up the stack? You don't have to catch it in a particular method you can let it go to the caller of the method, and so on.
Checked exceptions must be handled or declared. You end up with everything saying it can throw several random things like IOException.
Rust and Go literally have you handle it at the call site by default and if you don't want to you have to pass it on.
Go doesn't require you to handle or pass an error upwards. You can call something that returns err and just ignore it, compiler won't care. The "if err != nil" thing is annoying too. Rust made the right compromise with !
Which (at least the way it was done in Go) is ignoring 20 years of experience in programming language design.
Better design breeds better habits. In Java, you can end up handling a DNS exception 20 layers up the stack, even if your call never actually triggers a DNS lookup, simply because some method deep in the call stack declares it as possible. The culture is to propagate or wrap exceptions rather than translate them into domain-specific errors near the source.
Rust makes handling and mapping `Result` locally more ergonomic, even though `?` makes propagation easy too. Rust, however, also makes it easy to lose the context during propagation and end up with useless `Error`.
I agree but I think was also because Java didn't have sum types (or still doesn't? years I haven't read Java) and rethrowing is super easy, while Rust kinda nudges you into the right direction (if you need to rethrow at all, think if you want to rethrow verbatim or wrap with a semantic error type).
Right idea at the wrong time.
yep, 90% ergonomics in my view. If it had its time again with modern ergonomics thing could look very different.
They were overused, but some actually make sense, such as the IOException and SQLException
The problem with them all always comes when the ever-steady march of abstractions, features, and edge cases results in one routine throwing the same two error classes but in different places, and how you handle them is different despite being the same wrapper. Add a thousand developers and several years of churn, and poof you get extremely complex bugs that result from errors seeming to come from one place, but actually coming from another.
Having been on the receiving end of debugging that a few times, I no longer mind the in-band error handling, although admittedly I'm inexperienced enough to be confident its better. Just for the random code I've had to read where I just need to know exactly how something I didn't write and will never read again works, and whether the error is handled and how, i love reading in-band error handling, even if I too dislike the way an error laden routine must be written. Over time I've come to appreciate elegance "in the large" as opposed to the code I see in any individual function.
I always thought the ideas was great, but type system was holding it back. If you could have expressed that your thrown exceptions are the same as some other method's (e.g. a method of a class given as a parameter), and some other classes, then it could have been a lot less annoying. Powerful, even! But Java did take its sweet time to get even type parameters.
I have written very little Java, though, so this is more of an armchair analyst position.
Amen, people saw the symptoms of "needs more syntactic sugar" and misinterpreted it into "rigorous error handling is bad".
The same way that layered code should react/wrap/translate when it comes to return types, it should do the same for thrown exceptions, and checked-exceptions helps you do that. They're both results of computation.
It seems it used to return NULL, but they changed it for "convenience"
How is this more convenient, being able to display it as text?
Yeah too bad your adult-diaper language can't even host itself on most of the architectures where musl is relevant.
Another Rust self-suck session rendered completely irrelevant. Again.
Not really related to the topic, but AI-popular sentences and language structure stand out to people right away these days right? I noticed some.
"If you cannot update yet, at least disable password logins: set PasswordAuthentication no in your sshd configuration and restart sshd. Either one closes the hole." (only AI would characterize a critical SSH implementation vulnerability as a hole closing game)
An attacker who can reach your SSH port can log in without knowing any password. It takes one connection, works every time, and needs no retries. (It sounds like the writer is bragging about the exploit instead of describing it. AI does this)
The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash. The attacker then sends a very long password (257 to 1024 bytes). One user with a password is enough: it makes every account without one reachable. (same as above made stronger with the 'statement: statement2' format)
"Users coming from a directory service through musl-nscd (nslcd, winbind, sssd), since they have no shadow entry." (a human would be more clear about what a shadow entry is and give more context, I think)
Hmm the rest of the article (the remaining 70-80%) seems quite human.
Anyway, not sure why I spent 10+ minutes on this. It just jumps at me right away.
Agreed, I think all the tells you identified are correct. It is really off-putting to me still.
An aside but one nice thing I realize about not using LLMs is, I guess, I am not tortured by this kind of thing, because presumably I am not seeing it all the time. I can't even imagine quite being in the mode of reading this kind of text so closely like this, really. Its a security advisory, I don't really have my critical prose-reviewer hat on.
As a grad student I read and graded a lot of student philosophy papers (well before LLMs), and I just remember the exhaustion of it. It made me appreciate how much I could otherwise just read something, get information or enjoyment out of it, and not focus on each sentence construction.
Just to say, I do believe you, I get that if you have to put up with it so much in general, it must prime you to find it everywhere. You have to read so much more now, I'm sure, so you're trained to be in that mode. It makes sense! I am just sorry its like this for you and most everyone else now. It sounds exhausting.
Thank you for your comment, it's very refreshing (and comforting for me actually). Have a good day.
The potential affected space is huge.. it does not matter if most docker images ship without openssh.. it does not matter that most security conscious image builders do not enable plain passwordauth.. it does not matter that ssh port is not commonly exposed.. the amount of alpine docker images in current live use over the whole internet is such a vastly huge number that you're going to bet that even given a small percentage of those 'misusing' docker and their containers being affected is still going to result in a a huge number of affected containers.
Given Alpine’s broad use in the Docker ecosystem, this is going to hit a lot of folks, I think.
People generally don't expose SSH from within a Docker container I think.
no, ssh INTO a running alpine container. who runs sshd in a container when you can just attach a terminal?
you'll be surprised, ssh gives more ergonomic terminal, multiple connections, more convenient file sync and so on
Considering how much people whined about the MinIO images I would expect that it is not that uncommon.
Eh, most folks aren't hosting ssh servers with alpine containers and password auth enabled.
Who does PasswordAuthentication=yes in this day and age anyway.
We have clients (regulated banks and insurers of various sizes) that send us feeds via SFTP. You might be surprised what proportion of them ignore our “we strongly recommend key based authentication” and ask for password auth instead.
When I set up a client sftp drop I had the luxury of saying "key based access only"
Then it was a surprisingly customer service intensive process to get the keys from the customers. I never did manage to automate it fully. Weird enterprise systems I have never heard of with strange key formats. ssh1 keys, email mangling them. Their internal processes requiring a password.
And way to many "I am sorry but that was the private key you sent me, That key is now compromised and you will need to remove it and generate new keys anywhere it was used"
I almost regret not allowing passwords, almost, despite the headaches key based is still much better.
About to say, users just drop the key on their desktop and then download random exe's from the internet and run them.
Still safter in general, but not bulletproof.
At least in that case, if the account is compromised we definitely didn't leak the credentials. That we don't have them to leak is one of the advantages of key-based auth.
[of course not having the credentials to leak does not protect us from this sort of bug, it is just one factor in a multipart security stance]
If you refused, would you get overruled internally?
Previous company: probably. It could be a dealbreaker for the client, and we bent over quite readily on a number of issues considered minor (keys are best practise, but passwords are so common it is difficult to argue that they are so bad they should be refused).
Current state (now we have been bought and are effectively more than an order of magnitude bigger): I definitely don't have opportunity to make a unilateral decision like that. Even though the larger company has a more “fair enough, feel free to go elsewhere” attitude when a small client says “if you can support × that is a dealbreaker for us” then we could previously afford to take as our stance, some of the bigger clients are definitely paying us enough, and do use passwords, that someone above me would overrule a keys-only decision.
it's the default for most systems. I can't remember a time I had to not turn it off.
Don't most cloud providers default to that? I know I had to force some to deploy with a key and even then PasswordAuth was often on and had to be disabled.
If you run a cockpit service on your server you will need password authentication enabled, unfortunately.
I went and checked this. I can admin my server with Cockpit using either the web service (using my username/password) or the desktop client (via private key) without issue even with SSH password authentication disabled. I do usually disable the web service so had to reconfigure to check.
People who use AuthenticationMethods "publickey,password"
You must present a key first after which you must enter a password.
Just do no password at all.
Putting the one in 1fa? No thanks.
As this is apparently affecting openwrt, probably a fair number of openwrt users, if I had to guess
I thought OpenWrt used Dropbear by default?
crypt glibc also sets errno to ERANGE if the string is too large, doesn't Musl libc do the same? OpenSSH should also add this check.
I often use "AuthenticationMethods publickey" which would disable all authentications except public key.
Isn't this safer than "PasswordAuthentication no", even if it's a different setting?
I run Alpine on hardware. I was not effected by this as while I do use ssh keys, there are no user accounts without passwords on my server.
I would say that PasswordAuthentication=yes is by itself a vulnerability.
This is brutal because a lot of docker containerized apps are very slow to update the base image if at all.
They shouldn't be including any sshd.
Docker images are generally not affected because they don’t ship ssh.
I'm sure some people running lightweight containers just had a heart attack.
I would imagine the vast majority of lightweight containers don't contain an ssh server?
You'd be surprised by the amount of people using containers that don't know/care what is inside.
This looks terrible for musl, what a collection of terrible decisions on their part. I guess I'm staying away from it.
To err is human. There’s no escaping it, I’m afraid.
I agree entirely, but I'm not sure changing `crypt` to return `"*"` instead of `NULL` on failure qualifies. It's much more of a considered decision, which happens to be very wrong.
> set PasswordAuthentication no in your sshd configuration and restart sshd.
Funnily enough, they don’t recommend checking pubkey auth is enabled and working. By blindly (or agenticly) following those instructions the risk is getting locked out of some machine.
> because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
Seems quite reasonable. It’s quite obvious that black hats have day jobs as White hats and have been privy to embargoes for probably decades.
Embargo should last at most to the moment fix is made and code passes test suite.
Anything longer is putting systems at danger. I can recompile the package with a patch (even if Ubuntu/RHEL takes 3 weeks to do it) myself in a pinch, and I'd like to have a chance for it before the bug gets re-discovered by malicious party
What if the test suite passes in the middle of the night? Should all administrators interrupt their sleep and get up to recompile their servers immediately?
All times are the middle of the night somewhere. Now, it's true that some times are the middle of the night to more people than others, but that's rarely a usable piece of info.
Right, can’t be temporally bigoted in <current year>
That is generally the practice in professional environments, yes. Many places have an on-call rotation, and those that don't soon discover why. An embargo wouldn't fix that particular problem anyway, because time zones exist.
Also: There’s always midnight somewhere.
I think this is a long held belief of the project and why they have been excluded from bug embargo’s in the past.
>We consider the root cause of this bug to be outside OpenSSH itself, but we're in a good position to deliver a mitigation quickly.
So not a ssh issue but an issue specific to Linux using "musl libc without PAM".
Well, OpenSSH contributes to this by mixing hashes from the shadow file. The musl code would be correct if OpenSSH passed in the hash from the user account that is being logged-in.
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
Randomly using a different users's hash seems so wrong to me, but maybe I've just been too deep in PII stuff for too long.
I keep saying that no one should be using alpine or musl, but people keep replying that static linking is totally worth it.
Static linking makes distribution easier, but it's an absolute security nightmare.
With dynamic linking, you just upgrade the library on your system and you're done. With static linking, every binary that uses the library has to be rebuilt.
So then rebuild everything? I don't see the problem. NixOS shows how it's done.
both alpine and musl are great in my opinion, and as far as I know theres no empirical evidence that it is more or less secure overall than alternatives. would you like to elaborate your opinion?
IMO only good thing about alpine is that it's a good environment to build stuff that links musl statically.
Never understood why Alpine is used to actually run anything in a container - if you statically link binary, then you don't need alpine as a base and if you don't what is the point? Don't @ me about image size, even without layer caching we're talking about tiny difference that is often dwarfed by size of your runtime.
Musl itself is a great example of taking "simple and lightweight" so seriously that everyone else gets to deal with the complexity. Slow allocator, underoptimized memcpy/string routines, limited locale support, no NSS, and endless glibc compatibility headaches.
But hey, at least libc is small. %%Proceeds to pull in megabytes of <any other>malloc, and other libraries to replace the slow parts of musl.%%
And now sshhole.com is a particularly spectacular example of what happens when "simple" meets the rest of the software ecosystem.
More evidence than allowing root to log in with any long password?
This particular flaw sat latent for a decade.
There's a good chance similar flaws will be discovered elsewhere.
”Flaws” or ”intentionally planted backdoors” a la xz? That one was targeting openssl as well.
I have to admit that I am a bit disappointed in this kind of thinking. There's vulnerabilities everywhere as especially the past few weeks have shown.
One occurrence is not a statistic and should not be used as evidence - it's just an example of a vulnerability that due to various bad circumstances lead to a particularity bad outcome. This logic is in my opinion equally flawed as calling alpine safe now because this particular vulnerability has been fixed.
I'm not a big fan of blaming and I think it's just a result of bad circumstances, but it's interesting to see how some seem to blame musl, but one could argue further that musl documented the behavior of their API which was ignored by openssh and so on
Docker is a static binary and HTTP is a new command line. I don't care what's inside, nothing good anyway.
I always use certificates with SSH. Saved me this time.
It's nice to see how quickly this is being addressed, and likely monitored for any unintended side effects.
Looking at the unpatched Ubuntu things in my life, and I'm pretty close to asking Claude to update things to see how quickly Alpine can get close, assuming responsiveness can stay this fast and quality.
i think it's just unbelievable there isn't a bigger push to move away from c to languages that can prevent stuff like this from happening. this and just the pure unsafety. it's 2026. it's like pure negligence at this point, the equivalent of building a bridge with matchsticks because you like using it more.
How would using a language different than C help here? At it's core this bug is a logic bug, where "*" == crypt(x), because crypt in musl started returning "*" in some cases some years ago. This could be Rust. This could be Go.
Not every C vuln is a memory safety problem.
Another comment said that returning a * was to avoid returning null which would crash users of the library who didn't check for null. That seems like a C smell to me
Not really. This is like if the char* were an Option<String> but you had programs which stupidly didn't check if the Option had something and assumed it always did. As others have said, Rust would not prevent this.
`Option<String>` or `Result<String>` forces the check at the callsite. Unlike C, you cannot forget that the function can error. This absolutely helps prevent issues like this.
Now, the decision to return `"*"` instead of letting poorly written code crash on `NULL` is not good either.
I don't think it's that people forget it while writing the code, it's that people want their program to work with as little effort as possible and decide to ignore it "for now". And that definitely can happen with Option and Result as well. There's no protection for not checking return values, and it's probably not even possible to protect against (in entirety)
Well they would use `unwrap` if they were lazy, but at least that would be quite visible in the code and would not result in undefined behavior due to operating on a null value. And there might be a group of developers who first choose to use .unwrap. I hope it is the minority.
Given how easy it is to just forward that error to the caller, I think most Rust developers would just choose to do that.
While dereferencing a null pointer is technically UB, it practice it leads to a safe trap. So .unwrap is the right comparison. If you always forward the error to the caller, it will also just lead to some random error message at some point, not really different to a crash.
> There's no protection for not checking return values, and it's probably not even possible to protect against (in entirety)
That's exactly what you get with Result and Option, it's impossible to forget.
Compare:
To:That's exactly what I mean. And I have seen a huge amount of unchecked unwraps in production code in the past years
Sure, but it's far better than the C code and I don't think in Rust a library would start returning `Ok` with sentinel values to "fix" shit calling code. Maintainers would, rightfully, tell people to fix their code.
Don't get me wrong, I think rust has some great features and does some things extraordinarily good. But at the same time I can see that for 99% of cases it's just as prone to bugs, including security and supply chain attacks, as any other language. I think I could just as much argue that any "->" is clearly de-referencing that needs checking, which is also pretty easy to see.
Agree to disagree I guess :)
this is theoretically, empirically, and objectively wrong. akin to saying that you can get into an accident in a truck as well as a motorcycle, so in 99% of cases there is no difference in safety.
different kind of programming languages change the kinds of bugs and their properties, but it does not change the amount of bugs per line of code. in respect to C and rust for example, rust may have less memory bugs - which LLMs seems to be able to spot now - but it did increase supply chain attacks for example.
> but it does not change the amount of bugs per line of code
You are really uninformed: different programming languages have very different volumes of bugs per line of code
With all the respect but I don't think I am: https://arxiv.org/html/1901.10220v2
that only tries to show a single study is flawed, nothing else
do you think bugs/loc is the same between brainfuck and rust?
> that only tries to show a single study is flawed, nothing else
no the study failed to show any difference in bug count for a specific set of common used languages, where C and Python were both part of the dataset.
I don't think programming languages have a big impact on bugs per line when every line is written by a human. Now given that I was talking about bug counts, not severity.
it showed errors in the methodology of an earlier study, it did not produce a novel result.
kind of reminds me how the "research" suggests that there are no practical differences between dynamically typed and statically typed languages...except we know from practical experience that there is, and the literature there is horribly flawed.
we have real-world evidence that rust leads to significantly less bugs in the real world
https://blog.google/security/rust-in-android-move-fast-fix-t...
> it showed errors in the methodology of an earlier study, it did not produce a novel result.
It showed that there's no big difference. From the link:
> Combining corrections for all of these aforementioned items, the reanalysis revealed that only 4 out of the original 11 languages correlated with abnormal defect rates, and even for those the effect size is exceedingly small.
--
> we have real-world evidence that rust leads to significantly less bugs in the real world
no, we only have evidence for memory bugs, that's not what I was talking about
>because crypt in musl started returning "*" in some cases some years ago
In Rust, the type of the return value would be a `Result<T, E>`, which means the chain of events that led to this would never have started in the first place.
In C the return value would be NULL, except it was changed by musl at some point.
Edit: at the beginning, crypt always returned a value. So hypothetical Rust return type would be a String, and then it would be retained for backward compatibility. Or something else would happen, this is a complex logic bug triggered by a 25 year old chain of events.
The type of the value in C is `char*`.
That's what led some people to write code where it crashes without checks. Which is still their mistake. But if the type was something that can't be implicitly used like an always-present value, that would have prevented the situation, even with those same people.
A pointer is a type where a null pointer can't be used as a regular value but leads to a trap when dereferenced.
You mean could be a `Result<T, E>`. There’s nothing in Rust that forces that.
> How would using a language different than C help here?
Easier to formulate invariants and check/prove them. Better language correlates with safer software.
I write firmware where the program has to run for years unattended.
You are right, C with manual memory management, no lifetime management, lack of phat pointers and a culture of leet has problems with deranged maddening bugs that other languages don't have.
But this bug first shows how inband signaling is bad news usually. But worse it's a failure to adhere to the spec properly. If you implement a library and you don't adhere to the spec you're going to bone someone.
Perhaps even worse is obviously the spec doesn't come with any standard testing framework. That was also the root of Apples infamous goto bug.
why does this logic bug exist? why are they returning a string, why not a result type, (nil, err), or something else?
oh right, because c is a crusty old language with barely any features.
Rust would not save you here as this is a logic bug. Nor would, well, really any other language. I suppose someone might claim that Rust's result type would save you but it really wouldn't: I can easily make the error type a String or pretty much any other type with little consequence.
Rust would absolutely have saved you here. Even with `Result<String, String>`, which is pretty poor design, there's no confusing `Ok(s)` with `Err(s)`. Sure you can go out of your way to write something like
but at that point I feel like the library is forgiven (except for the poor error design).@my-huge-pony does make a good point elsewhere in the thread: if the potential error was not foreseen and the code was written originally to return `String` instead of `Result<String, Error>` then even Rust would be in a pickle.
It would, due to Result<T> and #[must_use] on it. Nobody would return “*” as a sentinel in Rust.
languages with more features would mean developers aren't choosing between unchecked NULLs or sentinel strings.
musl's crypt function could be written in Rust, Python, or JavaScript, and still return * when the key is too long.
What exactly do you mean "push"? What do you think is going to make musl developers switch languages?
it's unbelievable to me that there isn't a larger movement of developers to move away from c towards better & safer languages. look at how much pushback the rust4linux folks get, primarily from c developers who are simply afraid of change. imagine engineers not adopting safer building practices because "they prefer the old way"....it's disgusting honestly.
what will it take to move away from c? older devs retiring. Planck said that "science advances one funeral at a time" and i fear it will be the same for software.
Moving away from C in any POSIX system is in fact stop being POSIX system as POSIX and linux being unix like system is built on standard that is defined as C library specification.
Rewriting it to Rust is fine attempt, just hard to call it is unix/posix anymore. It may not matter but semantically true.
> what will it take to move away from c? older devs retiring.
I'll take forking and porting, thank goodness for LLMs.
guess what, this kind of sentinel value mistake could even be made in rust
no rust programmer would ever return a sentinel string like this.
Are you sure about that? When backwards compatibility comes into the picture people tend to bend over backwards.
Backwards compatibility with what? Rust has always has had enum also for this exact case, to avoid encoding sentinels as strings or pointers.
im quite positive the only rust programmer who would do this is a C programmer who doesn't understand rust
why, because they all use agents to do the coding and don't check it?
because rust has features that makes doing it pointless.
in theory could someone do it? yeah, just like in theory someone could program a calculator to randomly rm -rf ~ when pressing +.
but in practice would anyone do it? no.
Cars without seatbelts and airbags chopping off driver's heads in relatively slow collisions? Well guess what, that could happen in a modern car as well if the driver sticks their head out the window while going through a very tight tunnel!
you followed up an incorrect comparison with another one, are you sure you're a rust programmer?
I presume Dropbear, thus musl-based OpenWrt routers are also not affected? Not that Dropbear has been an example in that regard.
There is this write-up/fancy page from the person who found the vulnerability[0] (HN discussion[1], how I found this thread).
At the bottom of that it says dropbear is unaffected.
> on debian, ubuntu, fedora or anything else with glibc? going through pam? dropbear? openbsd? you're fine
[0] https://sshhole.com/
[1] https://news.ycombinator.com/item?id=50378228
No, the majority should not be affected.
Except in the case where their images are built with OpenSSH instead of Dropbear, which is the case for some manufacturers. (Turris, was noted as one such vendor)
Since after you posted, various people in the thread have reported dropbear is not affected
Here's an unrelated security issue I opened with Alpine earlier this year
https://gitlab.alpinelinux.org/alpine/aports/-/work_items/16...
It didn't give me the impression that I was dealing with a secure piece of software, it seemed like they were dealing with very complex issues while laughably failing at the industry basics that were settled years ago.
Similarly Ubuntu was also the only other distro that fucked this simple permissions config up, and lately with the latency to fix the TCP vuln, it seems to confirm that these are not reliable distros for the coming security era.
Seems slightly confusing that the OpenSSH release notes page says:
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
musl libc violates the POSIX documented behavior of `crypt()`¹
> Upon successful completion, crypt() shall return a pointer to the hashed password; the first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
Musl returns '!' or '*', not null.
¹https://pubs.opengroup.org/onlinepubs/9799919799/
Since OpenSSH is included in the base install, and enabled on OpenBSD - is it correct that they will need to update the famous:
“Only two remote holes in the default install, in a heck of a long time!”
https://www.openbsd.org/
OpenBSD is not affected, so apparently they are doing something right. But also glibc linux is not affected and pam linux is not affected, so it is only musl linux with the remote hole.
You became a victim of a very misleading post title. This isn't an OpenSSH problem at all, it's a musl problem.
I'm pretty sure the misleading wording is deliberate, OpenSSH is an excellent but non-commercial project which a bunch of entities would love to gain control of, in order to bug it with their spyware - thus the constant nagging on the bot-infested web.
The OP should be flagged.
OpenBSD doesn't provide musl-linked OpenSSH by default, so not yet[x2]
No, the bug is in musl but manifests in OpenSSH when compiled against it.
This vulnerability doesn’t exist on non-Linux systems, so not yet!
Workaround? Why not a fix?
AFAIU because the bug is in musl and not in OpenSSH