If everyone starts applying cooldowns, won't it postpone the problem? So now there is a considerable amount of users who are affected and someone from the affected group discovers the infection and reports it.
But if everyone will be delaying updates, won't be there less chances to catch it in time? I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
>If everyone starts applying cooldowns, won't it postpone the problem?
There are still research firms who are actively and aggressively scanning new packages once they are pushed. For example socket.dev pulls new packages across ecosystems and performs automated analysis and runs it in a sandbox. We don't have to have them go boom in someone's production repos to find out there is a problem.
And if every malware developer worth their salt now introduces code to "wait out" that period of time, we're back to square one.
This assumes that they employ clandestine enough techniques that you have to actually install, wait and observe the behavior for longer than the cooldown period in order to detect this, because the code is "obfuscated" enough to evade static analysis of the code. It's anti-virus / anti-anti-virus 101 all over so to speak.
The good thing I suppose is that it raises the bar. Your regular "virus generator" script kid (sorry: supply chain attack generator script kid) can no longer pull this off.
Fundamentally, this is a cat-and-mouse game. But I suspect that "time bomb" techniques aren't economically viable for attackers, at least not with current patterns: current attackers demonstrate "smash and grab" tendencies because they know their access is limited anyways. Attempting to wait out a cooldown exposes them to additional detection risk.
Of course, maybe the attacker profile changes over time. But that's the nature of the game.
Also as an upstream, if your "coworker" releases a strange package without discussing the changes with the broader maintenance group, you might notice after 3-48 hours, but probably not within the hour unless you happened to be online.
> But if everyone will be delaying updates, won't be there less chances to catch it in time?
No: the security assumption behind cooldowns rests on security scanning parties, not on innocent users being victimized. Three days is a short cooldown, but it should be a good enough lead for scanning parties.
> I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
It’s not that much data, particularly for parties that are directly financially incentivized to be the first to report malware.
Most of the malicious ones just curl something in a postinstall script, scanners already catch that. The sneaky ones don't look malicious until they run, and three days may not help.
I think the idea is that it gives a bit of time for the companies which run automated scans of new versions to run through and detect any issues with new versions before users install them en-mass.
Watching language package managers reinvent everything distribution package managers have been doing since the 90s has been as fun as watching crypto people reinvent financial regulation.
The publishing topology is pretty fundamentally different: the entire power (and danger) of language package managers is that anybody can publish, not just a privileged few.
(This cuts both ways: I’d say that distribution package managers have learned valuable lessons about what users actually want from language package managers. Learning is a good thing.)
This comparison is tiresome. Distro package managers are curated, language package managers are not. They're serving completely different use cases; the former is the App Store, the latter is the web.
I really hate dependabot making generic security people at work so pushy about updates updates updates. They seem to just be dogmatic about whatever dependabot says, forcing churn even when the documented issues are clearly not relevant. I’m not sure how to handle it politically. I’m convinced that updating so much more often is worse, not better.
> I’m convinced that updating so much more often is worse, not better.
The issue of cooldowns aside (which is about delaying updates, not reducing their frequency): you're going to have the same set of problems when you update, whether you do it frequently or infrequently. The difference is that if you update frequently, you'll have a smaller set of updates (so it's easier to debug) and you'll have more opportunity to report issues upstream and fix them in a timely fashion.
It's the same underlying problem as CI and build time. Most people abandoned the concept of projects that take so long to build you can only do testing once a week, because CI that runs on every PR provides a much better experience. This is the same lesson applied to updates.
My logical read of the situation is that I end up making fewer overall changes if I end up upgrading a dependency once, not thrice, to a specific version. And the changes are their own source of risk.
> The reality is that each update is its own potential security issue
Even beyond security issues: each update is a new opportunity for breakage, not only from bugs in the third-party package, but also from unexpected dependencies on the third-party package's behavior.
I don't know if all ecosystems are as bad as node is, but the node ecosystem has terrible issue severity ranking which makes infosec squeamish for no reason.
Every week or so there's a new High+ "vulnerability" that gets published against our dependencies and I have to go look at it to confirm that it's yet another case of "it's possible for someone to give this dev-only tool a bad regex that would cause the test runner to OOM on that branch".
As a sysadmin I'm in the same boat. I've unfortunately never worked with security folks that seemed to have any sysadmin or dev experience. Whether or not this is universal, idk, and I have no idea what they are teaching in these security courses. But I'm tired of security teams telling me "you need to implement these 230 group policies this quarter" or whatever. They don't seem to grasp the externalities of a request like that and how much fucking work it is to vet, test, deploy, monitor, verify, etc. 80% of the time, they don't even know what they do or if it's even impactful for us.
I’ve mainly handled it by pushing my team to be extremely conservative about what dependencies we take, especially if they pull in scads and scads of transitive dependencies.
This elegantly mitigates three problems in one go: update churn, dependency hell, and supply chain attack surface.
It also, frankly, tends to make the code easier to understand. I’m not a huge NIH person but I do have to say that a lot of packages these days tend to encourage ways of doing things that are unnecessarily complex. More than once I’ve replaced a dependency with homegrown code and reduced LOC in the same commit.
But updates to broken packages are still allowed: if a new version is pushed within the three days, it does not reset the cool-down. You just get a pull request to update to a known-bad version instead.
This makes me think whether npm (and other registries) should apply security requirements based on ecosystem impact. Example a package having millions of downloads can have special security measures enforced.
Higher cost (“Mythos” vs static code analysis) vulnerability scanning prior to successful merge to main branch or deployment as an artifact. As risk increases (popular code->greater exposure potential), increase automated, programmatic scrutiny on subject code to lower residual risk.
(application security and vulnerability management is a component of my work in financial services, thoughts and opinions always my own)
What a state of things where we have to fear installing software, and rely on vendors to scan things ahead of time, because our supply chain is such a mess and our tooling is so incapable of (and uninterested in) protecting us.
What would it take to not fear installing software? This isn't a npm problem, its a computing problem in general. Spaces like this are generally pretty against any sort of restrictions or limitations being put on computers under the name of safety (see Manifest v3)
For libraries, I like the Gnu Affero Public License[1]. If you run the library in software with that license, you have to publish all the source of the entire project that incorporates it.
No corporation could tolerate this, though, so the library vendor can negotiate a commercial license of their software for appropriate fees.
That said, corporations are not going to want to negotiate fees with 100's of vendors over constantly fluctuating dependencies in their software.
This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
Similar to Amazon's Dynamo API, whatever the next big language/ecosystem is needs to be designed around _billing_ and automatic license management for # of deployments, seats, call volumes, etc.
> This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
I don't think this idea is going to go anywhere.
If a package is available for free, on convenient licensing terms, developers will use it.
If you make them pay, many developers will prefer to just build it themselves. Coding agents make that easier than ever.
Buying a package involves a lot more paperwork – it needs to go through procurement – and introduces new risks, e.g. what if the vendor increases their prices
There are potential exceptions – software with really advanced algorithms (e.g. solvers for optimisation problems); safety critical software; software needing regulatory certification (e.g. there are some Australian government APIs they won't let you call unless you've hired an auditor to certify the software you are calling them with, and the relevant government agency has approved the auditor's report) – but those exceptions are relatively rare, and the existing solutions are arguably adequate to handle them
I also think it is different for packaged SaaS applications [0] because there the buyer isn't a developer, it is someone non-technical, and "use a coding agent to build it yourself" isn't within their comfort zone or risk appetite (at least, not yet).
[0] conflict of interest disclaimer: work for a SaaS vendor
Manifest v3's actual motive was so shamelessly transparent that most of us just don't allow the "safety" argument for it to really be entertained. I don't have a suspension of disbelief rich enough to pretend I don't know.
> The default applies only to version updates. Security updates still open immediately, so critical fixes are never delayed.
does this require a real vulnerability report, or CVE? if the package is compromised would they just be able to push a false "critical update" that bypasses this wait?
If everyone starts applying cooldowns, won't it postpone the problem? So now there is a considerable amount of users who are affected and someone from the affected group discovers the infection and reports it.
But if everyone will be delaying updates, won't be there less chances to catch it in time? I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
>If everyone starts applying cooldowns, won't it postpone the problem?
There are still research firms who are actively and aggressively scanning new packages once they are pushed. For example socket.dev pulls new packages across ecosystems and performs automated analysis and runs it in a sandbox. We don't have to have them go boom in someone's production repos to find out there is a problem.
And if every malware developer worth their salt now introduces code to "wait out" that period of time, we're back to square one.
This assumes that they employ clandestine enough techniques that you have to actually install, wait and observe the behavior for longer than the cooldown period in order to detect this, because the code is "obfuscated" enough to evade static analysis of the code. It's anti-virus / anti-anti-virus 101 all over so to speak.
The good thing I suppose is that it raises the bar. Your regular "virus generator" script kid (sorry: supply chain attack generator script kid) can no longer pull this off.
Fundamentally, this is a cat-and-mouse game. But I suspect that "time bomb" techniques aren't economically viable for attackers, at least not with current patterns: current attackers demonstrate "smash and grab" tendencies because they know their access is limited anyways. Attempting to wait out a cooldown exposes them to additional detection risk.
Of course, maybe the attacker profile changes over time. But that's the nature of the game.
Also as an upstream, if your "coworker" releases a strange package without discussing the changes with the broader maintenance group, you might notice after 3-48 hours, but probably not within the hour unless you happened to be online.
> But if everyone will be delaying updates, won't be there less chances to catch it in time?
No: the security assumption behind cooldowns rests on security scanning parties, not on innocent users being victimized. Three days is a short cooldown, but it should be a good enough lead for scanning parties.
> I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
It’s not that much data, particularly for parties that are directly financially incentivized to be the first to report malware.
Only a few of the recent supply chain attacks were discovered by users noticing weird behavior.
The majority were noticed by maintainers or third party groups noticing things like releases not tied to a source tag, many rapid releases, etc.
Cooldowns won’t stop everything, but it makes a malicious release significantly more likely to be noticed
The goal is to give time for automated scanners ran by cybersecurity companies to flag malware before it gets installed on real users.
Easy, then you just delay your project’s dependency updates just a little more than everyone else
Most of the malicious ones just curl something in a postinstall script, scanners already catch that. The sneaky ones don't look malicious until they run, and three days may not help.
There are plenty of ways to notice a malicious release without observing it running.
Build provenance, maintainer alerts on new releases, tying releases to specific git tags, etc all help.
Every single one now will be more sneaky, and we’ll be operating on a 3-day cooldown for no reason.
You really think it has zero benefit whatsoever? Nothing malicious will be caught?
I think the idea is that it gives a bit of time for the companies which run automated scans of new versions to run through and detect any issues with new versions before users install them en-mass.
There was a story about two men and a tiger
The men see the tiger, one scrambles to run and the other starts putting on their shoes
"Why are you putting on shoes? You'll never outrun the tiger"
"I don't need to, I just need to outrun you"
Watching language package managers reinvent everything distribution package managers have been doing since the 90s has been as fun as watching crypto people reinvent financial regulation.
The publishing topology is pretty fundamentally different: the entire power (and danger) of language package managers is that anybody can publish, not just a privileged few.
(This cuts both ways: I’d say that distribution package managers have learned valuable lessons about what users actually want from language package managers. Learning is a good thing.)
This comparison is tiresome. Distro package managers are curated, language package managers are not. They're serving completely different use cases; the former is the App Store, the latter is the web.
Give it two years.
I really hate dependabot making generic security people at work so pushy about updates updates updates. They seem to just be dogmatic about whatever dependabot says, forcing churn even when the documented issues are clearly not relevant. I’m not sure how to handle it politically. I’m convinced that updating so much more often is worse, not better.
> I’m convinced that updating so much more often is worse, not better.
The issue of cooldowns aside (which is about delaying updates, not reducing their frequency): you're going to have the same set of problems when you update, whether you do it frequently or infrequently. The difference is that if you update frequently, you'll have a smaller set of updates (so it's easier to debug) and you'll have more opportunity to report issues upstream and fix them in a timely fashion.
It's the same underlying problem as CI and build time. Most people abandoned the concept of projects that take so long to build you can only do testing once a week, because CI that runs on every PR provides a much better experience. This is the same lesson applied to updates.
My logical read of the situation is that I end up making fewer overall changes if I end up upgrading a dependency once, not thrice, to a specific version. And the changes are their own source of risk.
I'm in a similar camp, I dislike how often third-party package updates get pushed out, especially given the lack of serious inspection.
The reality is that each update is its own potential security issue and with supply chain attacks being all too frequent, it's not a panacea.
> The reality is that each update is its own potential security issue
Even beyond security issues: each update is a new opportunity for breakage, not only from bugs in the third-party package, but also from unexpected dependencies on the third-party package's behavior.
I don't know if all ecosystems are as bad as node is, but the node ecosystem has terrible issue severity ranking which makes infosec squeamish for no reason.
Every week or so there's a new High+ "vulnerability" that gets published against our dependencies and I have to go look at it to confirm that it's yet another case of "it's possible for someone to give this dev-only tool a bad regex that would cause the test runner to OOM on that branch".
As a sysadmin I'm in the same boat. I've unfortunately never worked with security folks that seemed to have any sysadmin or dev experience. Whether or not this is universal, idk, and I have no idea what they are teaching in these security courses. But I'm tired of security teams telling me "you need to implement these 230 group policies this quarter" or whatever. They don't seem to grasp the externalities of a request like that and how much fucking work it is to vet, test, deploy, monitor, verify, etc. 80% of the time, they don't even know what they do or if it's even impactful for us.
I’ve mainly handled it by pushing my team to be extremely conservative about what dependencies we take, especially if they pull in scads and scads of transitive dependencies.
This elegantly mitigates three problems in one go: update churn, dependency hell, and supply chain attack surface.
It also, frankly, tends to make the code easier to understand. I’m not a huge NIH person but I do have to say that a lot of packages these days tend to encourage ways of doing things that are unnecessarily complex. More than once I’ve replaced a dependency with homegrown code and reduced LOC in the same commit.
I feel the same. So so so many "regular expression denial-of-service" issues at my last job that just didn't seem serious or often reachable.
But updates to broken packages are still allowed: if a new version is pushed within the three days, it does not reset the cool-down. You just get a pull request to update to a known-bad version instead.
This makes me think whether npm (and other registries) should apply security requirements based on ecosystem impact. Example a package having millions of downloads can have special security measures enforced.
What would be a security measure that should only be selectively enforced?
There are few ideas which come to my mind, some might be far fetched, taking NPM as an example.
- Restricting packages with similar names as of popular packages restrict expres because express is a popular package.
- Imposing stricter 2FA checks on accounts of authors of these packages.
- Making sure that published packages don't have vulnerabilities and clear npm audit.
- Alerts in case these packages contain a dependency which is new / relatively new.
Higher cost (“Mythos” vs static code analysis) vulnerability scanning prior to successful merge to main branch or deployment as an artifact. As risk increases (popular code->greater exposure potential), increase automated, programmatic scrutiny on subject code to lower residual risk.
(application security and vulnerability management is a component of my work in financial services, thoughts and opinions always my own)
What a state of things where we have to fear installing software, and rely on vendors to scan things ahead of time, because our supply chain is such a mess and our tooling is so incapable of (and uninterested in) protecting us.
You cannot call it a supply chain, if you have zero contractual relationships with the authors of the solutions you are using.
[1] https://news.ycombinator.com/item?id=44434355
I mean, that’s just arguing over whether or not the definition of “supply” implies “compensation”, which isn’t very interesting imho.
The grandparent’s point remains the same, the software ecosystem and its supply chain or however you want to call it is a hot mess.
Traditionally the term "supply chain" has implied a buyer/seller relationship
I think that’s up to debate, and my point is that debating whether free software counts as “supply” or not is really not that interesting.
Oh that one really makes you think doesn’t it.
So what?
What would a solution to this look like?
What would it take to not fear installing software? This isn't a npm problem, its a computing problem in general. Spaces like this are generally pretty against any sort of restrictions or limitations being put on computers under the name of safety (see Manifest v3)
For libraries, I like the Gnu Affero Public License[1]. If you run the library in software with that license, you have to publish all the source of the entire project that incorporates it.
No corporation could tolerate this, though, so the library vendor can negotiate a commercial license of their software for appropriate fees.
That said, corporations are not going to want to negotiate fees with 100's of vendors over constantly fluctuating dependencies in their software.
This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
Similar to Amazon's Dynamo API, whatever the next big language/ecosystem is needs to be designed around _billing_ and automatic license management for # of deployments, seats, call volumes, etc.
[1] https://web.archive.org/web/20260712154038/https://www.gnu.o...
> This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
I don't think this idea is going to go anywhere.
If a package is available for free, on convenient licensing terms, developers will use it.
If you make them pay, many developers will prefer to just build it themselves. Coding agents make that easier than ever.
Buying a package involves a lot more paperwork – it needs to go through procurement – and introduces new risks, e.g. what if the vendor increases their prices
There are potential exceptions – software with really advanced algorithms (e.g. solvers for optimisation problems); safety critical software; software needing regulatory certification (e.g. there are some Australian government APIs they won't let you call unless you've hired an auditor to certify the software you are calling them with, and the relevant government agency has approved the auditor's report) – but those exceptions are relatively rare, and the existing solutions are arguably adequate to handle them
I also think it is different for packaged SaaS applications [0] because there the buyer isn't a developer, it is someone non-technical, and "use a coding agent to build it yourself" isn't within their comfort zone or risk appetite (at least, not yet).
[0] conflict of interest disclaimer: work for a SaaS vendor
> What would a solution to this look like?
Sandboxing and auditing built into the software from the start. Browser Extensions solved this ages ago.
Manifest v3's actual motive was so shamelessly transparent that most of us just don't allow the "safety" argument for it to really be entertained. I don't have a suspension of disbelief rich enough to pretend I don't know.
No way to prevent this says only package manager where this regularly happens.
Every major package manager is just as fucked.
"We don't call 'em 0days any more, now we call 'em 3days"
More like 3-per-day amirite
> The default applies only to version updates. Security updates still open immediately, so critical fixes are never delayed.
does this require a real vulnerability report, or CVE? if the package is compromised would they just be able to push a false "critical update" that bypasses this wait?
Requires a GitHub security advisory and
> Only advisories reviewed by GitHub trigger alerts.
From https://docs.github.com/en/code-security/concepts/supply-cha...
The idiocy of cooldowns speaks for itself.