"The hackers stated that they attempted to contact Red Hat with an extortion demand but received no response other than a templated reply instructing them to submit a vulnerability report to their security team."
“Since RedHat doesn't want to answer to us,” the hackers wrote in a channel on Telegram viewed by 404 Media, suggesting they have attempted to contact Red Hat. [...]
“We have given them too much time already to answer lol instead of just starting a discussion they kept ignoring the emails,” the message added. In another message, the group said it had “gained access to some of their clients' infrastructure as well, already warned them but yeah they preferred ignoring us.”
To be fair, once your data has been stolen, it doesn't make sense to engage with the hackers. There is no way to guarantee that the stolen data won't be used.
What you must do immediately is notify the affected customers, bring down or lock the affected services, and contact the authorities.
There is no guarantee anywhere (strictly speaking, including in the legal market), but that doesn't mean the paying has no effect of the probability of data being dumped.
Notification is an independent requirement.
If an attacker make an extortion threat, but then still follows through on the release/damage after being paid, then people are not incentivized to engage with you, and will go into attack mode right away, making it riskier for you.
HOWEVER, if the attacker make the extortion threat, takes payment, and then honors the agreement, and ends the transaction, then parties are more inclined to just pay to make the problem go away. They know that the upfront price is the full cost of the problem.
I've seen that there are 'ethical attackers' out there that move on after an attack, but you never know what kind you're dealing with :-/ "Never negotiate...."
There's no way to guarantee that I won't get in a car accident. So I pay for insurance. I may never need it, it may never come in handy, but it still makes sense to carry the policy.
This whole process happening is exactly what happens in a quest in Cyberpunk 2077. There’s an e-mail chain where a gang tried to extort a corporation and gave up after being unable to reach a person.
I sincerely hope that the game doesn’t become prophetic in the manner Idiocracy has.
Red Hat, I am very disappointed. We're all ISO27001 everywhere, separation of data, and separation of network resources. But you keep our data in your github repo?
You've got to look at ISO27001 from the perspective of the Sales Rep, not from an Engineer.
In theory, being ISO27001 means that you're environment follows best practices and has a somewhat sane security posture.
To the business people, a new customer demands that you have ISO27001 certification before they'll sign the $$$$ contract. The salesperson does not care HOW you get the certificate, just that you have it, they need this contract signed!
The department wasn't designed with security in mind, so implementing everything required by ISO will take many months. But sales needs $$$$ now! The CEO, CFO, and CTO are aligned: money now!
So, there's high pressure to pass the audit quickly. You implement what you can, you weasle your way around the things that will take too long. Those things are "out of scope" or "testing databases". You implement MFA while the auditor is auditing, but you know it breaks developers' workflows and there isn't a quick fix, so you turn MFA back off once the audit is complete....
TA-DA! We're ISO27001 certified! But we're no more secure than we were before.
> In theory, being ISO27001 means that you're environment follows best practices and has a somewhat sane security posture.
Nah, it just means you have defined, documented processes and document that you stick to them. They actual processes can be shit and maybe you also have something on the side the auditors don't get shown, but ultimately the certification is a total joke. Source: Worked at a place that got certified despite being a security joke.
Yes and no. Even if it is a joke there is one thing it qualifies: You at least spent time looking at the process. This already is a gain over complete wild west.
Engineers who are smart enough / talented enough, and who feel secure, can push back on security issues even if it will hold up a deal. This tells me that the most valuable engineers at Red Hat either do not push back enough on security concerns, or don't care enough (or aren't experienced enough) to know that the concerns exist in the first place, or they feel insecure in their position.
Ultimately, devs can't get sales reps fired, but sales reps can absolutely get devs fired.
Depending on how dysfunctional the org is, there's no super dev anywhere who can fix it. You just shut up, do bad things knowing theyre bad, or get fired.
It should be the opposite. For every big engineering issue happened because of the sales' dept pressures, the sales reps would have their asses out of any company.
> Correction: After publishing, Red Hat confirmed that it was a breach of one of its GitLab instances, and not GitHub. Title and story updated.
> After publishing our story, Red Hat confirmed that the security incident was a breach of its GitLab instance used solely for Red Hat Consulting on consulting engagements, and not GitHub.
> While Red Hat did not respond to any further questions about the breach, the hackers told BleepingComputer that the intrusion occurred approximately two weeks ago.
Every time I've been involved in audits at a company, my boss will tell me "let me tell you how to talk to auditors," which ends up meaning lie by omission, imply that things are in good standing without making strictly false statements, and otherwise just make the auditors go away. It all seems silly, but maybe it should be thought of like the court system? An adversarial process whereby each side is vying for its own interests?
With auditors you talk like you would either talk on a deposition or on the witness stand: do not say more than what you were asked, do not make assumptions, do not try to be helpful in any way, do not offer more data than asked for.
Is it really OK? Not necessarily, but on the other hand you don't want to spend the rest of your life answering even more questions from other people the auditors might bring in to help them understand your helpful explanations.
I learned this the hard way, assuming auditors are logical and understand technology.
90% of the time, they are checking boxes. But if they are fishing, you have to be careful because they generally are bad at understanding anything, but good at manipulating the audit rules to frame things in such a way so they can “catch a big fish”.
That may sound bad or immoral by the company, but know that auditors have the own ambition and mo ey to think about, and will try to mark any possible thing as a serious problem regardless of whether it is.
Yes, it is highly adversarial and the best compromise I've seen is to have an internal audit team that is separate organizationally from IT, but has to withstand peer review if they claim anything is a real problem.
GitLab, not GitHub. I think the distinction is that you can have a on-prem GitLab (as well as hosted online). The implication here being that RedHat probably had very relaxed account security.
>”They allegedly found authentication tokens, full database URIs, and other private information in Red Hat code and CERs, which they claimed to use to gain access to downstream customer infrastructure.”
They published CERs somewhere that had access keys and urls. Probably to somewhere that it wasn’t authorized to be published or shared and they got a hold of it. Using that, they got a hold of everything else. More CERs, GitHub repos using access tokens, vpn credentials to all the hottest players. At this point, you’d have to tear down and rebuild to undo the damage. Rotating certs, keys, IP’s, the whole nine yards.
No this was all Red Hat. IBM would never let consultants use something convenient like GitHub, they would be forced to use some crappy internal webapp powered by watson.
I wouldn’t be quick to blame IBM. Red Hat and IBM both take security very seriously, and regard it as central to the mission. IBM also has deep enough pockets to devote serious resources to whatever they put their mind to.
Security is just hard. Procedures can be written, but people make mistakes, forget rules, etc. The procedures also have to be constantly updated to keep up with new and innovative attacks. It’s a never ending battle.
Sorry to see this happen to Red Hat. I am confident that right now all hands are on deck working on remediation.
Security is hard, but running a large organization is also hard. The danger is that the entire organization becomes overrun by managers whose guiding principle is CYA, and then things slowly grind to a halt. Eventually you wind up with security mandates that have no relation to what's actually deployed because the people nominally responsible for security are evaluating metrics that are years out of date.
"The hackers stated that they attempted to contact Red Hat with an extortion demand but received no response other than a templated reply instructing them to submit a vulnerability report to their security team."
Just hilarious
You didn't give the kicker:
"According to them, the created ticket was repeatedly assigned to additional people, including Red Hat's legal and security staff members."
Summarized: Given enough eyeballs, all extortion demands are fallow.
And then there's more, via 404:
“Since RedHat doesn't want to answer to us,” the hackers wrote in a channel on Telegram viewed by 404 Media, suggesting they have attempted to contact Red Hat. [...]
“We have given them too much time already to answer lol instead of just starting a discussion they kept ignoring the emails,” the message added. In another message, the group said it had “gained access to some of their clients' infrastructure as well, already warned them but yeah they preferred ignoring us.”
https://www.404media.co/red-hat-investigating-breach-impacti...
https://archive.ph/l9Hxh
"Since RedHat doesn't want to answer to us"
First rule of having someone reply: spell their name correctly.
To be fair, once your data has been stolen, it doesn't make sense to engage with the hackers. There is no way to guarantee that the stolen data won't be used.
What you must do immediately is notify the affected customers, bring down or lock the affected services, and contact the authorities.
I'm a customer and the first I'm hearing about this is from HN.
There is no guarantee anywhere (strictly speaking, including in the legal market), but that doesn't mean the paying has no effect of the probability of data being dumped. Notification is an independent requirement.
There is an interesting dynamic/risk in play:
If an attacker make an extortion threat, but then still follows through on the release/damage after being paid, then people are not incentivized to engage with you, and will go into attack mode right away, making it riskier for you.
HOWEVER, if the attacker make the extortion threat, takes payment, and then honors the agreement, and ends the transaction, then parties are more inclined to just pay to make the problem go away. They know that the upfront price is the full cost of the problem.
I've seen that there are 'ethical attackers' out there that move on after an attack, but you never know what kind you're dealing with :-/ "Never negotiate...."
Then the hacker org spins up a new name(like a shitty construction llc) and robs the next guy.
Reputation isn't all that useful for extortion.
Running all your crimes as the "Wet Bandits" makes it much easier for law enforcement if they do catch up with you.
There's no way to guarantee that I won't get in a car accident. So I pay for insurance. I may never need it, it may never come in handy, but it still makes sense to carry the policy.
fallow == marked by inactivity
Thanks, hadn't encountered this word before.
Normally used with farming; you run the land two years, and then leave it fallow for a year to recover.
Linus' Law used a different word.
Corpo cyberpunk
This whole process happening is exactly what happens in a quest in Cyberpunk 2077. There’s an e-mail chain where a gang tried to extort a corporation and gave up after being unable to reach a person.
I sincerely hope that the game doesn’t become prophetic in the manner Idiocracy has.
Can't be extorted if you can't be reached. such a two-brain move!
made my day
There seems to be an official statement by Red Hat now: https://access.redhat.com/articles/7132207
Oh that's surprising, it isn't behind a paywall.
Telegram name: "thecrimsoncollective"
Before even reading the article, the title made me lol as I thought the open sourced code was stolen XD
Red Hat, I am very disappointed. We're all ISO27001 everywhere, separation of data, and separation of network resources. But you keep our data in your github repo?
You've got to look at ISO27001 from the perspective of the Sales Rep, not from an Engineer.
In theory, being ISO27001 means that you're environment follows best practices and has a somewhat sane security posture.
To the business people, a new customer demands that you have ISO27001 certification before they'll sign the $$$$ contract. The salesperson does not care HOW you get the certificate, just that you have it, they need this contract signed!
The department wasn't designed with security in mind, so implementing everything required by ISO will take many months. But sales needs $$$$ now! The CEO, CFO, and CTO are aligned: money now!
So, there's high pressure to pass the audit quickly. You implement what you can, you weasle your way around the things that will take too long. Those things are "out of scope" or "testing databases". You implement MFA while the auditor is auditing, but you know it breaks developers' workflows and there isn't a quick fix, so you turn MFA back off once the audit is complete....
TA-DA! We're ISO27001 certified! But we're no more secure than we were before.
> In theory, being ISO27001 means that you're environment follows best practices and has a somewhat sane security posture.
Nah, it just means you have defined, documented processes and document that you stick to them. They actual processes can be shit and maybe you also have something on the side the auditors don't get shown, but ultimately the certification is a total joke. Source: Worked at a place that got certified despite being a security joke.
> ultimately the certification is a total joke.
Yes and no. Even if it is a joke there is one thing it qualifies: You at least spent time looking at the process. This already is a gain over complete wild west.
that makes absolutely no sense at all.
do you mean you rather be lied to than not be lied to?
That looks more like SOC2 than ISO-27001 though.
It's the same with ISO27001. A bad actor can always weasel their way through.
Engineers who are smart enough / talented enough, and who feel secure, can push back on security issues even if it will hold up a deal. This tells me that the most valuable engineers at Red Hat either do not push back enough on security concerns, or don't care enough (or aren't experienced enough) to know that the concerns exist in the first place, or they feel insecure in their position.
Ultimately, devs can't get sales reps fired, but sales reps can absolutely get devs fired.
Depending on how dysfunctional the org is, there's no super dev anywhere who can fix it. You just shut up, do bad things knowing theyre bad, or get fired.
It should be the opposite. For every big engineering issue happened because of the sales' dept pressures, the sales reps would have their asses out of any company.
When I was working in MSP land this was the worst.
I had a sales guy sell a a company a replacement for their terminal server, with OneDrive lol
I almost died laughing when he explained to me the project.
I said.. you want to run cad files off OneDrive in place of a terminal/storage server?
"Yes"
Let's just say we ended up just moving their server to the cloud and VPN access onsite and for external developers.
> [..] The CEO, CFO, and CTO are aligned: money now!
"Aligned" :)) The IT terminology FTW! Very very realistic description. Of not delivering value to customers.
Was this a case in this RH breach ? Maybe. But just putting multiple "repos" and other client stuff in same place is modern IT insanity.
At least they could put that Navy stuff somewhere else. Resonable idea, right?
100%! Insert SOC2, HIPAA, etc...
> Correction: After publishing, Red Hat confirmed that it was a breach of one of its GitLab instances, and not GitHub. Title and story updated.
> After publishing our story, Red Hat confirmed that the security incident was a breach of its GitLab instance used solely for Red Hat Consulting on consulting engagements, and not GitHub.
> While Red Hat did not respond to any further questions about the breach, the hackers told BleepingComputer that the intrusion occurred approximately two weeks ago.
Data is separated in different repositories, per the story.
And, never forget: what a company preaches and advertises is not the same with what the company is actually doing.
Yes, I've been in fortune 100 IT security for awhile now. When it comes to passing audits its a shit ton of misdirection.
Also, here is some more information about this breach: https://x.com/intcyberdigest/status/1973422846396473765
Every time I've been involved in audits at a company, my boss will tell me "let me tell you how to talk to auditors," which ends up meaning lie by omission, imply that things are in good standing without making strictly false statements, and otherwise just make the auditors go away. It all seems silly, but maybe it should be thought of like the court system? An adversarial process whereby each side is vying for its own interests?
With auditors you talk like you would either talk on a deposition or on the witness stand: do not say more than what you were asked, do not make assumptions, do not try to be helpful in any way, do not offer more data than asked for.
Is it really OK? Not necessarily, but on the other hand you don't want to spend the rest of your life answering even more questions from other people the auditors might bring in to help them understand your helpful explanations.
I learned this the hard way, assuming auditors are logical and understand technology.
That’s a good point of view.
90% of the time, they are checking boxes. But if they are fishing, you have to be careful because they generally are bad at understanding anything, but good at manipulating the audit rules to frame things in such a way so they can “catch a big fish”.
The issue is, it’s very easy to understand what’s not being said for a reasonably intelligent person.
A person who is used to interviewing people will be able to tell right away.
That may sound bad or immoral by the company, but know that auditors have the own ambition and mo ey to think about, and will try to mark any possible thing as a serious problem regardless of whether it is.
Yes, it is highly adversarial and the best compromise I've seen is to have an internal audit team that is separate organizationally from IT, but has to withstand peer review if they claim anything is a real problem.
s/should/could/
Your boss is bad apple and so are you if you adopt their ways.
if you find it unethical, can't you leave anonymous tips for the auditers?
Sure, if you don't want a job.
VPN profiles? Best start revoking/renewing certs and keys. Geez, why would you store VPN profiles there?
GitLab, not GitHub. I think the distinction is that you can have a on-prem GitLab (as well as hosted online). The implication here being that RedHat probably had very relaxed account security.
[dead]
[stub for offtopicness]
(title fixed now)
"Correction: After publishing, Red Hat confirmed that it was a breach of one of its GitLab instances, and not GitHub. Title and story updated."
Title needs updating
Correction Posted: "After publishing, Red Hat confirmed that it was a GitLab account breach, not GitHub."
It's GitLab not GitHub
[flagged]
>”They allegedly found authentication tokens, full database URIs, and other private information in Red Hat code and CERs, which they claimed to use to gain access to downstream customer infrastructure.”
They published CERs somewhere that had access keys and urls. Probably to somewhere that it wasn’t authorized to be published or shared and they got a hold of it. Using that, they got a hold of everything else. More CERs, GitHub repos using access tokens, vpn credentials to all the hottest players. At this point, you’d have to tear down and rebuild to undo the damage. Rotating certs, keys, IP’s, the whole nine yards.
Sounds like they got their hands on authentication tokens. Through RedHat, not GitHub.
It looks to me like the article is talking about authentication tokens that they found in the data they took from the breach.
[flagged]
No this was all Red Hat. IBM would never let consultants use something convenient like GitHub, they would be forced to use some crappy internal webapp powered by watson.
IBM uses GitHub Enterprise.
Retired Red Hatter here.
I wouldn’t be quick to blame IBM. Red Hat and IBM both take security very seriously, and regard it as central to the mission. IBM also has deep enough pockets to devote serious resources to whatever they put their mind to.
Security is just hard. Procedures can be written, but people make mistakes, forget rules, etc. The procedures also have to be constantly updated to keep up with new and innovative attacks. It’s a never ending battle.
Sorry to see this happen to Red Hat. I am confident that right now all hands are on deck working on remediation.
Security is hard, but running a large organization is also hard. The danger is that the entire organization becomes overrun by managers whose guiding principle is CYA, and then things slowly grind to a halt. Eventually you wind up with security mandates that have no relation to what's actually deployed because the people nominally responsible for security are evaluating metrics that are years out of date.
i doubt old red hat wouldn't have warned clients right away.