> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
It was "multiple human errors". But yeah no matter what, in any case or exceptional circumstances, this thing was not supposed to be screwed up.
On Mozilla’s tracker, eMudhra said the keys were created on 30 April 2025 in a key generation ceremony, a recorded procedure in which keys are produced inside hardware security modules (HSMs), tamper-resistant devices built to hold keys. The keys were meant to be moved to data centres in the United States and Europe for customers needing data stored in those regions. To move them, the company added a step to ‘wrap’ each key, or encrypt it for transfer, and ‘unwrap’ it at the other end. eMudhra said a flaw in the software script created a second, weaker set of keys during this step, and because the two sets had similar names, staff used the weaker ones by mistake. It said human errors occurred at multiple steps.
source:
What's the eMudhra security scare that has rattled the internet?
I have to imagine that the 3 bit numbers were slots/indexes on the HSM, and rather than actually retrieving the key(or maybe it did), the slot number got used as the key. I can't otherwise imagine how they'd generate such low number keys...
Having worked around these types of devices and their interfaces, I doubt they intended to create weak keys.
It is more likely they had a script which placed a parameter in the wrong field, due to something like using positional arguments. Or they misread the documentation for some interface.
Still incompetence. And I would love to see my theory confirmed or refuted.
The previous discussion advanced the theory that they fucked up the string handling. Eg in Python, output = key[0] would result in one-byte keys like "4" if you expect "key" to be an array, but it's actually a string.
> worked around these types of devices and their interfaces..
This is not a problem of devices and interfaces. You are supposed to generate the wrapping key on the HSM itself. And the HSM ensures that the keys are generated with a strong enough security parameter, not some silly 1 byte string. Then the HSM wraps the key inside of itself and exports it in the encrypted form. You take this ciphertext, and the wrapping key and send it over to the other party.
So if they generate the wrapping key with a funny value, that means they didn't use HSM to generate it but their system openssl. But then, HSMs typically don't allow you to export private keys without wrapping them first unless you first tweak around with security flags (not recommended at all), or simply generate the private key using your system openssl too (LMFAO).
I hear you in terms of how these kinds of mistakes typically happen. But why did the script ever produce two sets of keys? The initial root cause was somebody picked the weak key not the strong key. But what was the reason for ever producing a weak key in the first place? They say the script produced two outputs - what on earth for??
I mean, why have the ability to generate weak keys to begin with? Serious question, I've never done this sort of work. What sort of utility does that bring to the procedures and workflows one does in that line of work?
If you didn't know better, you'd think this had been done on purpose by some three-letter agency.
Yeah. I can't think of any development process where you wouldn't simply use a production grade key in development. In fact, that's exactly what we do in development with snake oil keys.
Generating a key with "4" was probably done as a joke, but also is insane to have ever lived outside some devs computer.
> These "wrapping" keys were not "weaker". They were completely unfit for purpose.
Oh come on. They clearly didn't intend to use 4 as a private key or to wrap it. They had just purchased millions of dollars of HSM's to store them. It was something more subtle than that.
Have a little imagination. The transfer procedure sounds bespoke - possibly just a shell script using openssl to encrypt things at one end, and decrypt them at the other. Let the man who has written 100 shell scripts without a bug in one cast the first stone.
But not so subtle that good software engineering procedures wouldn't have picked it up. In fact today just an LLM review of the shell script probably would have caught it. The Internet Death Sentence was warranted - because poor software engineering procedures are unforgivable in a CA. But those procedures and policies aren't easy or cheap to come by. If they were just "don't chose 4 as a primary key" my grandkids could do it.
Unwrapped extraction is the kind of thing security by checkbox covers and process review might catch so it is never clear in a case like this that some party with the capability to influence the process didn't want keys intentionally weak to emulate unwrap.. Then someone else may have introduced the swap of keys that would never have been seen with keys that would (be forever) in a public scrutiny DB.
I don't think it's implied the wrapping keys were the weak ones. More like, the script that ran the wrapping still had the fast "debug" generation of a pair of keys to test the wrapper on enabled, for example.
This is pretty normal in India unfortunately. Crony capitalism is the dominant business engine in the country, so pretty much any significant business, esp in regulated sectors like govt mandated services(which is the bulk of eMudhra's operations) is impossible to set up without significant bribes being paid.
As a consequence, these businesses are very well protected from normal market forces and are pretty much above the law. Had this incident not been an international one, they had a very good chance of getting away with it, and the people reporting this would have landed in jail.
This is 100% accurate. State run media would have squashed this article if it was not for an international incident response. They have federal laws that prevent such disclosures because it affects the national image and could cause social unrest.
You're asking a chicken and egg question, I believe it becomes a self-reinforcing loop
In regards to doing business - someone that comes from a corrupt society with a culture of taking advantage of others should be approached more carefully than someone that grew up in the same town as you, of course
This is a lesson learned the hard way, where I moved to a different country and treated everyone with openness and generosity (of time, resources, willingness to help), just to realize people from certain cultures mistake this behavior for weakness and try to take advantage as much as possible
> Would you say the system forces people to be like that or people are the reason why the system is like that?
It's the courts. Courts are meant to bring in an error correction mechanism. When courts fail, the country goes into open-loop control. No errors are corrected, and eventually, people stop even detecting errors. https://x.com/captgouda24/status/2110089855380369511
> Would you do business (i.e. trust) with someone that comes from that system?
Vet rigorously and continuously! Start with small business, and grow it based on interpersonal trust. Since the system lacks accountability, you must ensure that yourself.
Depends on the contract your healthcare providers/payers/etc. have with those who do their IT.
Source: work on healthcare IT projects and can't let anyone outside of the US look at prod data no matter what due to the contract we have with our clients.
I said _crony_ capitalism is the dominant business engine in the country.
The govt mandated services are owned by private capital(hence the capitalism part) and govt picks winners by granting licenses to a select few(hence the crony prefix)
It does, but I wonder how much of that phrasing is cultural.
When working with different businesses across the globe, I’ve noticed Indian correspondence to incidents tends to be more a show of strength even if it’s transparently obvious that there’s no merit to the content.
Whereas western European correspondence is often more apologetic and empathetic. Even if it might sound insincere to those affected.
But it’s all just words at the end of the day. It never means much unless the business is willing to back real change to mitigate similar issues in the future and resolve undisclosed existing problems. However there’s no way for us to know if this is happening or not.
I don't know if it's really specific to one culture. Look at US companies after they go through a lengthy legal saga, it's obvious they screwed up, they're paying fines or having to adhere to court-imposed consequences. Their press releases:
"We admit no wrongdoing. We are only settling to move past this. Our business is strong/secure/competent."
It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.
>It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.
It's the first rule you learn after joining basically any organisation larger than 5 people.
People on HN who are surprised are seemingly self-employed prodigies who dropped out of middle school.
I’ve worked for a lot of enterprises across the globe and I can tell you that you’re flat out wrong. That’s simply not how all organisations operate.
Just take a look at any of the thousands of postmortems published over the last decade and you’ll see that the tone of incident response communication can vary massively.
This becomes an inverse Russian roulette: only one of six chambers is not loaded.
Waiting for a release like "... the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP 1FA place in our business. In view of this, after detailed deliberations our board has concluded that withdrawing from global OTP 1FA is the right decision for eMudhra and its shareholders"
> fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:
> fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:
Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider.
Their Internet registry people also weaponized AS0 (zero) as a method to take networks off air via BGP RPKI for the ranges they're responsible for. There's been a number of situations where multinational companies who operate in India who are forced to use IPs from the India RIR have been fat fingered by some staffer who makes changes then goes home for the day: meanwhile a multinational bank is down for 24 hours within the country because said agency doesn't believe in 24/7 support.
Typically they do this for non pay but they've dropped the hammer multiple other times incorrectly (and good luck getting a RFO out of anyone there).
For good or bad, thats by design. These is a significant portion of physical infrastructure associated with internet data flow. The only challenge governments face as of now is being able to segment the traffic to their wishes in its entirety.
Companies already control all the traffic flow on their infrastructure using end-point security solutions, the same can be extended at ISP level, and then Country level
"out of proportion to SSL/TLS’s place in our business" is an insane comparison. They are saying they care so little about SSL/TLS to begin with that it's not fair.
It would be like me practicing surgery 1 hour a week and saying "well surgery only has a very small place in my 80-hour work week" in response to flagrant failure of medical care.
This is what I found regarding how it happened, but still not a full explanation - but there was some name confusion and obvious incompetence.
>According to eMudhra, proper strong keys were created in the HSM at the key ceremony on 30-04-2025. The script had been changed to test wrapping and unwrapping keys, because the intermediates were to be moved to data centres in the US and Europe, and a bug in that change created a second set of weak keys. The keys had almost identical names, and the weak ones were used for the certificate requests. The same request was signed by eMudhra's own root and sent to Sectigo, which cross-signed it. The strong keys never made it into any certificate (Bugzilla).
I wonder. Say I wanted to prevent something stupid like this from happening in the future. What would be a sensible check? The higher level authority doesn't see the private key itself, right? Just the certificate request. And I wouldn't want someone to manually inspect the keys to see that it looks random-ish. Perhaps there could be a small check script added to the generation of certificate requests that says "hey, your key a.) is mostly zeroes, b.) appears to be ASCII text, or c.) seems otherwise extremely non-random, please use -f to continue anyway".
I don't know if it will work on something as small as a private key, but you can compress the key. If the compressed key is significantly smaller then the original, it lacks entropy. Pure random data is mostly incompressible.
Note that 32 0x00 is a possible output from a pure random generator while still being quite compressible, you would probably still want to throw that one out though. After buying a lottery ticket.
I was about to reply with "they didn't have a choice" but on second thought perhaps CAs are important enough that they should never be allowed to export keys from an HSM and should instead be required to physically deliver and install the HSM wherever it needs to be in the world. Inconvenient and expensive but likely for the best.
Russian Roulette is pretty accurate if not an understatement.
If they expect people to still trust them on their other services they have obviously no clue on how Trust works, meaning they shouldn't be in the game in the frist place.
As a trust provider you get one chance. Squander it and you're out. That is how it should be.
They got their image hurt, and pulling the culturally appropriate response here which is to never apologize and to never accept fault. Instead quit/resign and say that other stuff is just fine. It's really not fine, and you can bet that the same corporate philosophy that made the CA error will have serious security errors in their other products.
There is no such honor code and stop spreading this BS propaganda. Izzat is not even an Indian word used in any Indian language. It is Arabic which got mixed in with Persian which in turn got mixed in with Urdu.
Somehow this is making it even worse. Like, I'm having issues explaining the whole thing to my wife, because I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done. Anyone's got a good one?
Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
The internet's security relies on an almost impenetrable wall built around it. The king appoints gatekeepers that are each entrusted with a gate they must protect. The gatekeepers must follow a rigorous interrogation process to verify the ID of anyone who wants to pass through their gate. Only then are they permitted to unlock it.
The king was very careful to ensure all the gatekeepers were following the process correctly. It did not occur to anybody that one of the gatekeepers could be using a dirty rag to jam their gate closed instead of a lock.
Once discovered, the wall could be trivially bypassed. The security of the whole internet was immediately broken. Rebuilding the wall to block up the dodgy gate took much effort from very many people.
I think part of the issue is that we allow gatekeepers to open any gate.
This system was invented in the 90s when things were looking up. China was opening up to the world, the Russians were suddenly not evil anymore, and the internet was supposed to bring us all together into one happy global society, kumbaya and all that. Obviously that didn't work out.
It's time for a rethink really. Every CA is now a single point of failure.
The entire system (CAs and ICANN DNS) was never good to begin with; it's just become that much worse due to the shifting situation over the years. It's an inherently centralized design with no way to verify that example.com is operated by the same entity from one day to the next. The namespace is global and effectively an exercise in rent seeking.
At absolute minimum any sensible system would have a TOFU element whereby the site operator of example.com changing from one day to the next would result in all the tooling across the entire stack screaming loudly (in other words what ssh does). And that's just the absolute bare minimum change that's needed.
Now the gatekeeper is saying "I don't want to be a gatekeeper anymore for this one particular gate, because people were mean to me. I'll however continue being gatekeeper for these other gates that drive my business either way."
I think a part of the issue was, everyone living outside the walls did not even try to kick the gates open as everyone assumed a big (cryptographically secure) lock was being used by all gatekeepers.
Its called a Potemkin solution, and there is a serious theory that most nuclear arsenals are exactly that: no-one has ever confirmed they can actually be fired.
In the case of nuclear arsenals that's perfectly fine - MAD simply needs to be plausible, not robustly demonstrated. In the case of CAs I'd argue that the description doesn't really fit since they're getting stress tested by malicious parties on a daily basis. By that logic you could describe linux kernel security as a potemkin village due to the steady stream of zero days. In reality a potemkin village is something intended from the start to deceive everyone else. It's not impossible that's what happened here but that's a very serious allegation to make (ie theorizing that the weak keys were intentional with a conspiracy afoot).
In this analogy I think it would be more appropriate to call them a lockmaker, and a lot of gates have been using this lock. Finding and fixing every lock is where the effort comes from, and some locks may not be found for years to come.
This lockmaker analogy fails to capture one critical aspect: the failure of one CA breaks the security of everyone, even if they did not use that CA's "locks".
This is why I went for a "wall with many gates" instead; any bypass makes all gates redundant.
All analogies break down at some level; the main goal is to clearly communicate the most important ideas.
In the old days a king would "sign" his proclamations by pressing his "unforgeable" signet into a wax seal. This proved the authenticity of the proclamation.
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
They were "unforgeable" because commissioning a jeweler to make an intricate carving on a ring by hand was very expensive in the first place. Finding one who could precisely replicate an existing carving, down to the mistakes of its original author and any wear and tear that would be visible on the wax seal, was almost impossible and even more expensive.
Of course, I didn't mean to imply it took zero effort to copy one, though I guess that's how it read. You'd have to find a skilled jeweler that's also willing to commit treason, so for sure the barrier wasn't insignificant there.
Indeed. In both this scenario and our modern one, we rely on the difficulty of "key material" reproduction being greater than the difficulty of its creation.
However, Kings likely benefitted from the option of a scapegoat, they could always blame someone for forging their unforgeable seal and have them killed, rather than admit that the seal was true.
If we add in attainder and escheat, the King had a pretty strong hand.
It's enough that the signet could be verified, and sometimes was [1].
As it turns out, these documents were mostly useful for long-lived décisions, which could be overturned, and where a forgery exposed you for a lifetime.
for the non technical folks: imagine there was a password that if you guess it, you can impersonate *any website on the internet*
now imagine the password is literally the number 4
I just explained what it actually is. I said "you know how you go to a website HTTPS and it's secure? Well Apple chose a couple of trusted parties that your phone inherently trusts, and those people approve a few thousand intermediate companies who do the day-to-day certificate work for end websites. One of those intermediates, rather than choosing a 200-digit random number, chose 4, no not four-digits, the number 4". And then we could laugh at some of the memes.
i think this oversells the likelihood of it. the probability of any random 116 digit number having its first 115 digits be the number zero is much lower than that of it containing a much higher number of non-zero digits.
It's hard to explain partly because when you explain this thing we've done with TLS and PKI, it sounds a little insane.
> "We set up a system where every government and a bunch of random banks and IT companies in the world holds the secret codes to everyone's encryption. Then we make them pinky swear not to abuse it, with stern talkings to if they do. And now it's all on fire, because one of them was irresponsible!"
You can know the intractability of the problem that PKI is solving, and the (worse) alternatives that have been tried, but that doesn't make this system not-insane to a normal person, and its at least a little bit the fault of the collective "us".
And it's not like it is an impossible problem; we rely on TOFU for SSH, we have remotely-attestable TPMs everywhere, ZKPPs / PAKEs exist, session keys and federation protocols are widespread. Somewhere in the set of "possible solutions" are options that don't sound unhinged when you explain how we got to this eMuhdra situation.
I believe you misunderstood how it works. A parent certificate cannot be used to decrypt information encrypted by a child certificate… the CA issuer just signs your child certificate, it doesn’t have access to your private key.
Then, everyone who trusts the CA will trust your certificate by checking it is signed by the CA who should have done the checks to ensure your certificate contains truthful data.
So your layman description of “they have secret codes to everyone’s encryption” is completely wrong.
Certificate keys aren't used for encryption anymore anyway.
A modern TLS connection uses an ephemeral key-exchange protocol where both parties opaquely agree to a one-time-use random encryption key that is not knowable to any other party, not even someone who can gain access to either party's private key. This is called Forward Secrecy (originally and sometimes still known as Perfect Forward Secrecy or PFS, but the use of the word "perfect" in any cryptographic context is akin to naming a ship "Unsinkable").
They have the secret codes to MITM (intercept, inspect, alter and forge) any communication if they manage to place themselves between you and a party you trust.
> random banks and IT companies in the world holds the secret codes to everyone's encryption.
That's not true at all.
What they do have is the power to MITM connections IF they first spook DNS or insert themselves into a proxy accepted by the server somehow.
So, if you want to be honest, the original sentence becomes:
> random banks and IT companies in the world holds secret codes that are trusted by everyone to sign information that is used to establish trusted communications. If they wanted to, they could try to tamper with people's network connections and, if successful somehow, pretend to be anyone else that had used their secret codes to sign their information.
That sentence doesn’t take the current situation into account. It’s not just “they” - the CA - that can do this, it’s anyone who can figure out their weak encryption key. Which makes the “if successful somehow” a whole lot more likely - consider what malware distributors can do with this, for example.
The internet is a marketplace with a million vendors, each with their own stall.
In order to sell something in the market, you need to prove that you are who you say you are. There are about 200 trusted clerks whose job it is to check your papers and verify your identity. Once they're satisfied, they stamp your papers with an intricate stamp designed by the finest cryptographers.
Anyone seeing that stamp knows that a trusted clerk has checked your identity and approved your papers. Most people don't examine the stamp closely. They simply recognize it as belonging to one of the trusted clerks and accept it.
One of these clerks, however, instead of commissioning an intricate stamp from a master cryptographer, bought an ordinary rubber stamp from an office supply store.
One day, a particularly paranoid cryptographer was examining the stamps used throughout the market and noticed something peculiar about one of them. It wasn't intricate at all. In fact, it was so simple that anybody with a few dollars and a trip to the office supply store could make an identical stamp.
The clerk's job wasn't merely to stamp papers. The clerk was supposed to check that everything written on them was true before applying the stamp.
But now anyone could write whatever they wanted on their own papers and stamp them. The resulting documents would look exactly like ones that had been carefully checked and approved by the clerk.
A robber could write "thebank.com" on their papers, stamp them with the office supply rubber stamp, and present themselves as thebank.com. If they could get a customer to come to their stall instead of the real bank's, the customer would have no reason to suspect anything was wrong.
After all, the papers had the stamp of a trusted clerk. And everyone who trusted that clerk's stamp would believe them.
I told my wife the following: basically when you access a website like your bank, and you see the padlock in the address bar, that means that the computer the website is on has shown your computer a certificate saying that the same company that built the website you’re looking at owns the web address you’re going to.
Your computer has a built-in list of signatures that it trusts and expects to see on the certificates, and one of those signatures was eMudhra’s. These signatures are basically just a proof that you know a secret number without revealing the number itself. If you know the secret number, you can sign the certificate.
There are an unimaginable number of secret numbers. Imagine each atom in the Great Pyramid of Giza is a secret number. And then imagine you have as many Great Pyramids of Giza as there are atoms in the universe. And then double it.
Now imagine you choose any of the numbers represented by all those atoms to sign the certificate with, and you chose 4.
It’s not technically wrong: 4 is indeed a number. But if anyone wants to guess your number, and they start at 0, it won’t take them very long to find your secret number.
Once they’ve found your secret number, they can now sign certificates pretending to be you. So someone could pretend to be your bank, and present a certificate saying “yes, I am your bank” with a signature on it that your computer trusts.
It's a padlock with a code. The manufacturer makes the padlock with a specially made pin, so you can lock your front gate, and feel secure.
eMudhra sent you a padlock that had the code "4815" promising no one has ever used it or would be able to guess it, but all anyone really needed to try to get in is set the first number to "4" and ignore the rest.
It's not entirely accurate, but its accurate enough for them to get.
The automobile key pool used to be small. Some Eastern European car manufacturers had only a few, the joke was they wouldn't get stolen: once you've been inside one you've had enough
I'd refine this to, the cylinder for the lock did not care what key you put in, it woul open it anyways. So the key looks secure, but the lock is a joke.
As a 90s kid, the Dodge Omni was a common shitbox student car. I think they had like 8 key patterns lol.
My brother’s ignition key unlocked my friends’s trunk. There were a few people who could start other peoples cars. In one case, a soccer player “stole” a football players unlocked car as they looked identical and were parked near each other. In another as a prank we unlocked cars to roll them out of parking spots and physically lift them into unusual places.
>I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
Everyone in your neighborhood knows your address, but someone discovered two days ago that they can get into your house by sticking a 4" oblong piece of paper into the keyhole. They put it on blast on Facebook, so now everyone in the whole planet is trying to break into your house.
- you know how some people have passwords like 1234
- ha ha yeah what idiots
- what do you think an internet infrastructure company password is like
- idk something long and hard to remember
- 4
- lolwat
y'all underestimate your wives, i simply explained how PKI works in simple terms (alice/bob, public key, private key) and explained that they chose 4 as a private key instead of something complicated, she got it.
I start by saying this is about the tech behind the S in HTTPS. I say that the main point of this is that when you talk to foo.com, it has to prove that the people behind foo.com are on the other end, so it's safe to send them your encrypted password or whatever.
Then I say a bit about how the system works by having a small number of trustworthy companies vouch for other people's servers. This doesn't need too much technical detail to get the point across.
Then I say one of those "trustworthy" companies put an encryption key of 4, where it's supposed to be a long number that's hard to crack. This makes the verification easy to forge.
Then I say that everybody who runs a server needs to remove this company's bad key, and Windows or Mac or your phone needs updates too. And that your web browser probably already quietly removed it, which removes a lot of the risk.
I feel like when I explain it this way people get it.
A lock factory that’s supposed to make unbreakable locks, yet all of the locks/keys are identical. Suddenly everyone can use such a key to open any door of the internet.
Not 100% accurate, but conveys how stupid and incompetent all this is.
I told my fiance it was a bit like if a company that produces keypad door locks accidentally added a master code (115) which works on every single door and has for the last two years.
You know how we got a lock on our door, and it opens only with our key? Turns out someone made a lock which looked secure, but actually opened when you said the word "friend" - and then engraved "speak, friend, and enter" on the door.
How about, someone is selling a secure box of some sort, and their factory starts putting glitter nail polish over the screws that hold it together, as is sometimes done for security reasons.
Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”
A special locksmith who's entrusted enough to do the locks of all the important bank vaults in town used the pass code 1234 for every bank vault in town.
I described it like it was a company selling the highest security locks to companies that needed them, then one day someone discovered you could open the door with a toothpick.
The Hyundai/Kia cars where a USB-A connector fit the lock actuator and was enough to start the car, so enough got stolen that insurance companies stopped issuing policies for them.
The same Kias & Hyundais were sold in South Korea, but their auto theft rate is 60x lower than the US.[1] Car theft rates are mostly determined by how effectively car thieves are caught & punished, not by how secure the car is.
we could stick with the metaphor of keys. The modern web requires fancy ones capable of hundreds of notches. The company in question basically skipped the step of notches and just used the blank key
"We're designing/running the lottery. You get to pick the winning numbers, in secret, obviously so no one will know. There are millions of numbers you can pick from, using a random system so it's all trustworthy. Instead of drawing a random ball out, and another and another, you write down the number 4 on a bit of paper and go home for the day thinking it's a job well done. No one noticed because they all expected it to take up to a million guesses, until someone just asked if it was 4"
Why do you need an analogy? Analogies are always bad.
Encryption relies on long complicated numbers, but they made the number instead of long and complicated just the number “4”. Which is actually harder than using a long and complicated number because it’s basically all automated so we don’t even know how They fucked it up this bad.
I'm also wondering who the hell would use a registrar called "eMudhra" with Arial font in the subtext in their logo. It's so incredibly 90s.
I'm actually being serious, I do judge companies by their logo and website design taste, and more modern-looking companies also tend to pay more attention to doing things properly.
i explain it by using Donald Trump: There is the atomic football (a case traveling with Trump so he can initiate a atomic war even while playing golf). The combination of the lock an the "Football" obviously is 0-0-0-0-0-0, But Tramp has problems remembering the order of the digits in the combination.
> Is the Mozilla bugtracker usually a / the Go to party for these issues? Or just a coincidence in this case?
Yes, because of the project's open nature the Mozilla CA program Bugzilla and their dev-security-policy mailing list have become the de facto public discussion point for TLS-related issues.
oof. even as a bystander i’m feeling the cortisol just reading the questions. i’m curious what even goes into compiling this list of questions - did they plan on going for the kill?
The intent isn't public shaming, it's to have enough information about what the hell happened, so that the failure mode can be squashed as thoroughly as possible in the future. Other CAs and interested parties should be terrified about the possibility that they have the same cockroach infestation as a result of using the same software or similar procedures, rather than sneer about how bad the so-called "human error" was this time.
They planned on expecting a reasonable level of competence.
There’s no excusing this. This is a company that specializes in “digital trust”. Even if they exit the TLS CA cert business, would you want to rely on them for the “post-quantum cryptography” prominently featured on their homepage?
This is a case where it would actually make sense for a government to come in and shut them down. There’s no reasonable scenario in which this is a company that anyone should use for “digital trust”, and any customers dumb enough to do that might literally need to be protected from themselves, for everyone’s benefit.
You wouldn’t have to twist my arm very hard to convince me that any customers that continue to use them after this incident should also be shut down.
“d=4” is a perfectly legal RSA private key. The corresponding public key would probably have a weird e (not 65537, the usual choice), but would otherwise be legal, and could have an arbitrarily large modulus n (thus appearing “secure”).
Note that p,q (the primes you’re thinking of) are not actually required for decryption. They enable you to calculate d, the decryption exponent, and also enable decryption-time optimizations (exponentiation with d_p and d_q followed by CRT recombination), but that’s optional. In fact, p,q can be computed from n, e and d.
d=4 is not a valid RSA private key.
When p and q are prime λ(n) = lcm(p − 1, q − 1) is always even. e must be coprime with λ(n) so it's odd.
de ≡ 1 (mod λ(n)) which means that de is 1 + an even number, so d must also always be odd.
You can't do that because an RSA private key contains two numbers - the modulus and the exponent. Even better, an RSA private key usually has the modulus stored as its two factors rather than their product.
Also note that using small numbers like 65537 for RSA public exponents is common practice.
Like, that had to be malicious right? ssh-keygen won't generate a key like this, and neither will openssl or whatever Microsoft is doing behind their CA server.
The other keys were 56, 66, 115, 155, 156, 230, 237. I honestly think whoever did this thought the "384" in 384-bit meant they had to choose numbers between 0-384.
I assume you’re referring to [1]? If so, I just did after seeing your comment. I couldn’t help but chuckle at the demonstration of how easy it would be for someone to obtain the private keys themselves using an LLM, even though it’s easy to crack without one….
Using RSA as an example, the prime factors are supposed to have the same bit length (plus or minus 1 for carryover) for a similar reason. If the smaller factor has 1 fewer bit than the larger factor, it will be discovered twice as quickly by brute force, and 4x faster if it has 2 fewer bits, 8x faster if it has 3 fewer bits, etc. You could technically create a 4096 bit RSA key where one of the factors is 3 and the other is some other giant prime, but it would be immediately discovered by a naive brute force attack.
The algorithm itself isn't concerned with the bit-length difference and operates on the inputs opaquely. The key generator is responsible for selecting prime factors that cannot be easily derived from the algorithm's output.
Sure, but the probability of getting "4" as opposed to some number that might as as well be equivalent to latter as far as a causal reader is concerned is insanely small.
Yeah, except in total there were eight different weak keys, all less than 256. The odds of randomly generating those keys (all so small) is so astronomically low that we know for sure their keygen process was messed up.
Maybe it was just a mistake and they intended to generate a `--bitsize 4` thinking it was in kilobyte units.
shrug
I think they're just absolutely inept and should never be trusted with anything security related as an organisation if this is a mistake they could make.
Today "racism" is used as a synonym for "bigotry". Would you agree it was bigotry? Or do you only have a problem with people using words wrong, and not bigotry itself?
Whynot use the correct word for it, which is prejudice? Racism is prejudice against a race. You can be prejudiced to any group of people, doesnt mean that you are biggoted.
> Whynot use the correct word for it, which is prejudice?
Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts. "A rose by another name" and so on.
Your own comment has 3 typos and several grammatical errors ("prejudiced to" is wrong, for example). Irony.
Finally, modern usages of "racism" encapsulate all of these meanings. Languages change over time.
> You can be prejudiced to any group of people, doesnt mean that you are biggoted
They are not being pedantic here, they are establishing a meaningful difference in words to help distinguish a shared phenomena. I think the original comment was unnecessary and certainly bigoted, but I would need more to say racism. The "grammar" guy clarified twice the semantic difference they personally observed (note they did not labor on syntax like you wish to suggest).
All that said, I make this comment as I believe the other commenter's worry is on display im this comment here...
> Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts
These are _not_ all equally bad. Prejudice is foundational to understanding, there is no escape from this. In this understanding, we should treat particular postures of prejudice more serious than collapsing any "bad prejudice" under the same label.
> I have these arguments with people I love who are right in spirit and still dig in the moment the word “racism” lands, because to them it names a thing they’re sure they aren’t
Like I said, it's because of how emotionally charged the word is. People think if they label unfairly discriminatory or prejudiced behavior as something other than racism, it's somehow not as bad as racist behavior.
> Everyone in this sub-thread could easily agree it was out of line (and no one directly says otherwise).
I don't think everyone agreed it was out of line. There were only 2 comments calling zoobab's original comment racism. One of them spawned this thread about which words to use, rather than agreement with the condemnation.
> I think the original comment was unnecessary and certainly bigoted, but I would need more to say racism
Do you think racism is worse than bigotry? Why? I understand that the first word has a more emotional charge in our times. But as far as "bad things to do" go they're equal in gravity. Racism is just a subset of bigotry.
> These are _not_ all equally bad
Why not?
> Prejudice is foundational to understanding
That's one definition of the word - "judging without knowing the facts".
But there's another definition - "an irrational hostile attitude, fear, or hatred towards a particular group, race, or religion" - which was the one we are operating with in this thread.
My mistake. English is not my first language. In my native language, the word racism is used for discrimination or prejudice against anything foreign (nation, race, language, ...).
But either way, I find the reduction of the argument to the semantics of the word racism to be deflection. Wouldn't you agree the original comment was made in poor taste? Why did it need to include a reference to Indian and imply that all things Indian made are cheap or low quality?
We applied this kind of mindset to the Japanese in the 60s and 70s, and look where that got us. I witness it happening again in the 00s and 10s applied to Korean electronics (LG and Samsung mostly) while we were content with minor improvements over the status quo and innovating far too slowly.
Yes, this incident has an almost farcical amount of stupid attached to it, but the fact that it happened to an Indian company is in my eyes only coincidental. There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features.
Dutch. The word is "racisme", and the entry for it in our dictionary[0] states:
> Racism is a form of discrimination in which a person is judged on the basis of his or her ethnic origin, skin color, nationality or religion. It is a form of prejudice and inequality that prevents people from gaining equal opportunities.
While I'm happy to admit that perhaps my use of the word in English is incorrect and its meaning there is more narrow, in Dutch it covers nationality or religion as well. Islamophobia is considered racist ("racistisch") in Dutch, for example.
"There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features."
IME the number of manual processes which are bug-prone and faulty in Indian teams is higher than normal, and so is the reluctance to any form of automated testing, and general unawareness about any security or licensing concerns.
This is also racists. How many Indian engineers are there vs rest of the world? What kind of work is getting outsourced?
Let's take your words only.
Manual processes which are bug-prone and faulty are getting outsourced. They handle it with their best knowledge. Some processes/data knowledge is never transferred and some are kept hidden and assumed from outsourced teams but they have different culture and exposure so their understanding of things might be different. Its expectation mismatch not team issues.
American software giants are run by Indians only. So, if they are bug-prone & faulty then how could they reach there. It just that when person study with you or live with you then they have same set of culture & exposure like you have. So, they can manage things effectively.
Just noting an observation. These are not outsourcing agencies but GCCs of American and European companies. They had complete option to automate things or fix security defects. They simply didn't know what they were doing was wrong.
I don't mean to imply all Indians are incompetent. There was significant contribution from India to computer science. It's just that the ratio of competent to incompetent people is low. This is on your education and hiring systems.
The big takeaway here is: Web PKI is fundamentally broken, and the existence of CAs as a concept is the vulnerability that must be fixed -- or, more specifically, the idea that ultimately trusting a relatively small, basically fixed/hard-to-change set of (mostly) commercial entities with something this critical and far-reaching is (or ever was) actually a good idea.
There are a lot of reassuring statements from browser makers and others about how their mitigations for exactly this scenario kicked in and protected (some set of) Internet users, but IMHO there is not nearly enough discussion happening about fixing things so those mitigations are not needed to begin with.
I think Moxie's "convergence" proposal from ~2011[1] has the most benefits with the least downsides -- in particular I think he really pointed out the crux of the issue: the fact that servers decide who is trustworthy enough to certify their certificates and clients have no input into that decision is a broken model.
This incident was obviously comically bad but the Comodo hack had a more painful impact on the mitigation that should give everyone pause -- their CA had signed a quarter of the internet's certificates so the trust couldn't just be revoked. Imagine one of the LetsEncrypt intermediates got compromised -- our mitigation strategies really cannot cope with that kind of disaster scenario. Browsers would be in the unenviable position where they'd be forced to make huge portions of the internet go dark for potentially months.
Well yeah, or have the website define which CA they use, and don't let any of the others certify it. Apparently there is already DANE for this but webbrowsers don't check it.
Besides, there could be more CAs supporting .in of course. And it becomes a matter of responsibility to the local government.
And national TLDs aren't meant to be trendy anyway. Many have restrictions on them to local businesses anyway, it would make sense to have the local domain authority have control over choosing CAs as well.
We should have a very small set of largely not-for-profit CAs run by competent security people, such as Let's Encrypt. There are just too many trusted CAs at the moment, it increases the risk of a breach like this substantially.
Let's Encrypt had their own security snafu. They issue certificates for longer than they were supposed to (1 extra second). It wasn't as big of an issue as this, but let's not act like they are some silver bullet
You could store the public key for each website in DNS. But now you moved the trust problem to DNSSEC. So far I haven't heard about issues like this for DNSSEC, but then again it is not a massive target currently.
The largest problem is that you trust the entire set of commercial entities (in any one of them can destroy the world's security by itself), and the set is semi-secret (in no party using them actually knows what the set is).
Almost all browsers require certificates be recorded in Certificate Transparency logs <https://en.wikipedia.org/wiki/Certificate_Transparency#Manda...>. Since those logs are empty in this instance, you can state that no one used a forged certificate with a mainstream browser, which is the most obvious (and desirable) attack type.
Mainstream browsers were a juicy but also a very short term attack target.
The more interesting targets will be various REST etc apis being called over https. Lots of people doing API clients have been ignoring the "what if someone tricks me to connect to an evil server" vector, because they assume TLS verification will protect them.
You can also monitor if they show up in certificate transparency logs. Which won't catch all fraudulently issued certs, but might show up some, especially if it's being widely abused.
> About eMudhra / With over 17 years of expertise, eMudhra is a global provider of digital identity, trust, and security solutions, enabling secure digital transformation for banks and financial institutions (BFSI), enterprises, and governments across more than 35 countries worldwide. Its product suite spans CertiNext
I was hoping the statement at the bottom would be written in past tense: eMudhra was a global provider of digital identity ... had temporarily enabled secure digital transformation ... Its product suite spanned CertiNext...
"eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal."
So they didn't even use their own certificates? :)
what worries me is, I wrote a message to some older family members who travel alot, asking them to upgrade their systems if they plan on using hotel/airport wifi. Not being a security expert i had chatgpt criticise it, so I dont get something wrong. using "some company in India" made it rather dismissive of the warning, actively taking the company's side(the bad "no keys were issued" defense), whereas just saying "some idiots" had it generally approve the message.
i'm not sure how much i should read into this, but if you're sending out warnings to people susceptible to ask chatgpt about it, don't say the company is Indian.
> using "some company in India" made it rather dismissive of the warning, actively taking the company's side(the bad "no keys were issued" defense), whereas just saying "some idiots" had it generally approve the message.
But you know your audience better than ChatGPT does. Whether people react better, worse, or indifferently to the India angle will depend on the biases of the recipient.
100% true. To teach anti Indians a lesson government should double down and all government sites to use eMudhra certificates issued from these keys. And support this good Indian business from collapsing.
Like how you are doubling down on racism online against ‘all indian managers’? Walk the talk and cancel your google, msft and adobe subscriptions will ya? They are being run by untrustworthy worthless Indian managers too? Whats the reason behind this cretin like behaviour? Self hate?
Nobody likes Sundar at Google. Rest of the Indian CEOs (IBM, Adobe) are unremarkable bean counters. Satya is defensible but it can be argued the stage for his success was set during Balmer. (Dan Luu wrote about this).
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
This isn't what taking responsibility looks like.
> eMudhra handled the recent incident diligently, revoking the affected CAs within 14 hours of discovery and disclosing promptly
It doesn't end there. It ends when you've investigated how this happened, identified the causes, and changed your procedures, systems, and/or culture to make sure it doesn't happen again.
eMudhra stock is down 20% [1] There is a real financial consequence of bad posture. I am confident they were passing the compliances; everything was 100%.
We have been operating in India for a while. People do not take security, processes, etc., seriously in India. At least most of the organizations do not. Most likely, a human error caused the incident, and guardrails could have prevented it.
The sad part is that even after a major incident like this, the org is probably not going to invest in tools and processes to improve their posture. They will only do what is necessary to meet compliance. This is just the nature of a market driven by low margins and low-quality vendors.
There is no malice here by the org; it is just unfortunate.
It fell by exactly 20.00% because it hit the NSE's lower circuit, which prevents trades below that price for the rest of the session without technically halting trading. This is effectively a trading halt because there are presumably no buy orders at the asking price, and the real value of the asset is likely much lower. [1]
It’s worth pointing out much more strongly that their entire business is “digital trust”.
If markets were rational, this incident and the way they’ve responded should be enough to completely put them out of business. Of course it probably won’t.
I think it is not that people do not take things seriously. It is that there is no bandwidth for them to do so. I used to live in India, now don't live in India.
If I had to battle 2 hours of Indian traffic just to reach office, while I am also stressed about things I have to do with the city office to get my apartment approved, while also having to deal with all the plumbers etc who just call at random times, and while also thinking of having to fix my car, and none of this is easy. Everything is fucking hard, I would have no mental concentration to do these ceremonies properly as well.
I look at some of my friends in India, there's just so much daily work just to get by. Easily 4-5 hours lost everyday just for useless mandatory stuff. On top of 9 hour of sleep and 6 hours of eating, cooking, washing stuff, exercise, bathroom, you are left with 4 hours of actual work you can do per day.
So most Indian employees are unfortunately only able to work at 50% productivity. And then it is not surprising that they make mistakes.
> If I had to battle 2 hours of Indian traffic just to reach office
The company could have their office in a less congested area. The company could've given work-from-home option. The employee could've rented home near to the office. The employee could've got a cab. All Indian problems are self inflicted. Why should every company have it's office in "Bellandur" and "Marathalli"? In a self-respecting civilization, a professional earning in top 5% of the population wouldn't even want to step into such slums.
In any case, the reluctance to do proper automation and testing stems from pure lack of knowledge of good practices and hubris. Because it's going to increase velocity compared to not doing it.
I operate in india as well as rest of the world. What you are saying is true only from one perspective. When I'm in india, you can hire a persons to do
Day literally everything. Cleaning, cooking, shopping any chore you can think off. I'm actually more productive when I'm in india.
Traffic is a real problem. But if your compensation is above average in tech, you can get a car with a driver and work from car.
I hear you and i agree with you. I also believe orgs should try their best to make sure employee dont have to deal with it and be more productive.
Unfortunately most of the tech buisness are what we call "lala" businesses and they are not incentivising.
Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
Mmmh so you will be trying to supplant the destination server with your own, with this emudhra leaked cert, to see what's being sent? Or how are you planning to do that? Just curious :)
This reminds me - I need to get around to setting up a PiHole and convince my partner that her freemium mobile games aren't necessarily worth the telemetry scrapes.
a high quality set of FOSS, community inspected "freemium clones" may do more for national security and individual privacy compared to yet another crypto library with zero novel mathematics just another reimplementation of specs...
I've had a PiHole running on a NUC for about 2.5 years. I adore it. It should be built into every consumer router.
My wife still plays her freemium games, the ads just don't load. She took out one of those games when we were sitting at the airport once. She was frustrated that her once-pleasant strategy game was suddenly crammed full of ads for boomer slot machine apps, and I had to explain that the app had probably always been that way.
That makes sense and seems like it might be a net positive. While she's waiting for her ad countdown, she can gaze into my eyes and realize how much she appreciates me lol
My girlfriend deliberately watches the extra ads to get in-game rewards, so I can't even pihole the whole home network unless I exclude her phone. (I find that most of these games work fine if you block ads, but they also don't give you the rewards for watching an ad that never loaded.)
Wondering if there's some way to 'load' it by triggering whatever it counts as engagement but still not serving it to the client. Claude, you have $3 left and daddy doesn't want to make mommy mad again today
> Yes, but in the end that's not their fault, and in a sense it's good that this happened.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.
The leadership page is comedy gold ("Leadership That Inspires Trust"):
> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...
> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...
> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.
You say comedy gold. However in Vishwaguru Indian universe they are much vaunted South Indian Brahmins who anyone in the world accept that they are the best in anything they do. Hence they are pure gold.
Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
There's a bunch of changes recently / coming up in Canada, so that might be good enough. I've started occasionally reading the tzdb mailing list archives for fun because of that.
Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.
We've had to rework part of our application and certificate use because of this.
The problem isn't revocation of end-user identity certificates. There's hundreds of millions of those floating around at any given point, so revocation happens very regularly. The issue is revocation of intermediates and roots, which involves invalidating tens of millions of certificates and often find themselves in embedded/non-web applications (financial, legal signatures, medical data exchange).
An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.
None of the test certs would be used for real, whether they're intermediate or leaves.
They'd be created with big red warning signs, installed into browsers and other places by anyone who wants to test, and then publicly revoked every month. Rinse, repeat.
It really is. It's basically chaos monkey for pki.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
It baffles me that software on Linux/Windows/MacOS doesn't just use the system certificate store, then it would just be a single place to update. Instead we need to be update pip, UV, node, etc separately.
I can see the case on phones: the OS may no longer be supported by the vendor, so browsers shipping their own set makes sense. And on embedded there isn't (and on IOT there might not be) an OS, but then you should probably just ship the the single CA that you use, not all of them.
It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!
I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.
Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.
Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.
It looks to me like the world was decently prepared to deal with it. The bad certificates are being blocked and there don't appear to be widespread compromises.
That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
According to this blog [1], it has now been almost 5 days since a researcher reported to Apple that the eMudhra CA certificates were compromised.
As I am writing, my iPhone 15 Pro is still accepting certificates signed by these compromised CAs. The device is "up to date", yet is still running Trust Store Version 2026073000. When I check at [2], certificates issued by 3 eMudhra Intermediate CAs are still being trusted!
Google Chrome revoked these certificates in 1 hour 40 minutes, Mozilla in just under 4 hours.
I am so astonished by how much Apple has failed here. There are potentially huge swathes of people completely oblivious that their phones are susceptible right now to being man-in-the-middle attacked.
Why does the iPhone not utilise a CRL list? Why does it take Apple 5+ days to do what Chrome did in under 2 hours.
It's a similar story for my MacBook, but at least there my Chrome browser _does_ use a CRL (the same can't be said for all the programs installed, however).
I suspect Apple published no update and is relying on the CA's CRL and/or OCSP to do its job. But it's rate-limited, the connections time out, and iOS is hard-coded to soft-fail.
Like you my trust store is months old.
That said, I found separate installs of Firefox and Microsoft Edge that accepted *all* the bad certificates without a care.
The only browser that consistently rejected all of them was — you guessed it — Internet Explorer. A browser so old it couldn't even run the test properly but it did what it needed to and blocked all the servers.
To be honest, this whole experience has left me with a very sour taste in my mouth. It feels to me that the whole CA industry needs reform.
I am not smart enough to say how, but the sheer amount of work this created across the industry is monstrous. It probably has cost $billions to mitigate, and those mitigations have been slow and ineffective (see: my iPhone still being vulnerable).
In my experience, seemingly random pieces of software, libraries, and programming languages packaging their own CA stores has made this much harder to mitigate than simply running 'apt-get update ca-certificates'. We've practically just had to update every single thing to mitigate this.
Most of the stuff just seems/ feels to be (security) theatre. I actually disabled most of my OSCP, etc., for privacy reasons. No need to permanently send requests out to the web, when it won't help anyway when really required. I read HN to stay up-to-date and manually manage the certificates.
(Even with all the related options disabled, e.g., ocspd on OSX still regularly connects to random certificate related services to "check" on things. Something related to random root certificates I don't have in the system trust store or such, Not entirely sure what this is about and how to get rid of the requests in any other way than blocking things with Little Snitch)
The whole ordeal makes me wonder why the ergonomics/interface around CA trust management is not better.
It should be very easy and normal to ask for approval which is limited to the particular app, limited in time, and limited in scope.
Still keep defaults, but make it much smaller and put them under even higher scrutiny, certainly not 100 root certificates.
I don't understand why e.g. a CA for research institutions should be able to issue certificate for mybank.com or ecommerce.com and my browser would just trust that.
For your own sake and the sake of anyone else who might be hitting this bug, I suggest you open applefeedback:// in Safari, file a bug, and post the FB number here or on the Apple Developer Forums.
How did you report it? If you used Feedback Assistant then it would have given you an FB number. That’s the only way to be sure anyone at Apple sees it.
I selected the applefeedback:// url you shared and used "look up" (there's no "open link" option). That sent me to the official Apple feedback site. Are you saying that site's submissions aren't checked or something else?
Ah well. Guess it's as broken as the CRL, or I just clicked the buttons wrong or something. I just saw a thank you screen. I checked my email in case they sent something there, but nah.
Do you have battery saving mode on? I think it blocks the background process that updates. I had the same issue but turning it off and waiting 1 minute started blocking all the certificates. This is for iPhone 13.
It's my laptop and I can't easily revoke the certificates myself.
edit: I went to keychain access app and found all emudhar and em signer certs -> there were four -> clicked get info for each -> under Trust section of the pop-ip -> selected "When using this certificate: never trust".
I'm pretty sure Apple does use CRLs, but for some reason your phone has a problem fetching them. Yesterday my iPhone SE running iOS 15.8.8 did connect to the sites, but refused to connect about 30 minutes later.
And it will be even easier for them provided they praise right leaders.
But more importantly they don't need to any such name change. For people who matter in India it is just some random incident which Anti-Indian entities are whining about.
> For people who matter in India it is just some random incident which Anti-Indian entities are whining about.
Does this mean "trust no indian CAs"? those "Anti-Indian" entities hold a lot of power, and the indian IT industry needs to maintain the trust it has with the rest of the world.
Well you are not thinking in wrong direction. In their world view India is IT superpower and the whole world already depends on them.
To think of reasoning may help them course correct is wishful thinking. Read about Indian pharma sector and what kind crappy, dangerous drugs they are peddling to world.
This is confident, resurgent India . Reason, competency are racist words if used agaist them.
No, the customers for commercial certs are mid-level managers, who believe paying thousands of dollars for something you can also get for free means you are getting a higher quality - or in this case even believe it is "more secure". Rebranding will absolutely work for that, especially when you market your untrusted root as "ISO-certified private dedicated cloud-based PKI-as-a-Service".
The technical people will just install a certbot and use the default Let's Encrypt config.
If they wanted to continue to make money, they could: exit the business, wait a bit, change the company name, and re-enter the business. Where "change the company name" could mean starting an actual new company, but with the same people, offices, contacts (OK, small problem there) and so on.
There is no valid reason my computer should come preloaded with 100+ trusted certificates from many jurisdictions that all can sign certs for all domains.
Only a few major CAs in jurisdictions that admit heavy scrutiny and legal liability should operate. Country-specific CAs, if any, should only be scoped to signing certificates for their respective ccTLDs.
I came here looking for a comment suggesting just that. But surely there is some massive issue with this idea we are overlooking? Otherwise I don't see why it hasn't already been implemented long ago, it seems so obvious as a mitigation.
Also the industry has changed a lot since Let's Encrypt. You could have argued against monopolizing the CA issuance to just a few countries/entities as sovereignty issue, and CAs were indeed exhibiting extreme rent-seeking behavior before Let's Encrypt. With a guaranteed free and accessible alternative in the mix, the weight of the arguments against would change.
Personally? I'd agree with you. FWIW, you already have to trust your browser vendor, so you might as well trust Mozilla and Google and Apple and MSFT who can spin up one each, or collectively two or three CAs in US, UK, Switzerland, Ireland or sth like that.
That said, starting from the status quo, not sure that's the path of least resistance to get there.
Is it not obvious that not every global .com website owner wants to use only have the choice of US-run CAs (which can be subject to National Security Letters)? If you permit some other countries to certify .com then you're back to the same political question of which countries are allowed to run CAs.
It's also not at all a given that website owners would prefer the national CAs -- as an obvious example, most Rust projects (foo.rs) would probably not trust the Serbian CAs as much as they would LetsEncrypt. Even for website owners that live in the country, it seems possible that Chinese website owners might prefer to use non-Chinese CAs. And again, if the solution is to just special-case one American legal entity then we have just moved back to the original VeriSign-signs-everything model.
The real issue is that the CA model has been obviously broken since at least the days of the Comodo hack -- the trust decision for who can certify the website identity is made by the website owner but really clients are the ones that actually care (it's their data at the end of the day) and they have no input. Moxie had a proposal in 2011 to fix this by letting clients decide the policies for how a certificate should be trusted[1] and I think it is a far more workable solution than the alternatives. If one of the LetsEncrypt intermediates was compromised none of the alternative proposals would have any real hope of providing a workable mitigation.
> Is it not obvious that not every global .com website owner wants to use only have the choice of US-run CAs
The major browsers are all US made though, so in the end there is an implicit trust on US vendors, even deeper than the trust on CA.
> It's also not at all a given that website owners would prefer the national CAs
The suggestion is to have a few major ones (LetsEncrypt) be able to sign everything like today but restrict the others. You can still choose to use the US one, just like you basically use CT logs from the US vendor today.
--
In practice, DNSSEC/DANE based solution may be the practical path forward along this line.
> Its infrastructure was not compromised and no data was breached.
The private key of your globally trusted CA is infrastructure data and it was breached.
> All physical and logical security controls remained intact.
No, because you promised the CA Program that those keys would be HSM generated, FIPS 140-2 level 3 compliant. Ain't no compliant HSM in the world that will generate "4" as a private key.
> There were no end entity certificates issued out of the affected CA's.
I charitably read this as "revoking these weak CA didn't affect customers". But now your whole root is being revoked next week (which you forgot to mention). All your TLS customers get screwed.
Overall, good choice to give up, but lying on your way out is not a good look for a company with the tagline "Trust Delivered".
I just wonder how many times private keys were lost by or stolen from by slightly less incompetent companies?
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business.
What a weak minded statement. Just admit you messed up and call it a day.
Immediately and totally exiting the domain is about as close to admit you messed up and call it a day as you'll ever get from a modern company.
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
As someone who's well-versed with the "face-saving" aspects of Indian culture, I can state with clarity that the statement they put out is completely in character.
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work
on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
If it is a component, then is OP wrong? Fwiw, a lot of nations have interesting variants of that kind of honor though they don't always express themselves the same way.
It's more than I'm not encouraging to engage with OP's point because he just wants to be a bigot. But that is separate from India having a shame based culture.
I am curious, not that I agree it exists or understand the reality of izzat, how does it differ in saving face provided by gentlemen who allow the loser to have a graceful defeat?
In martial arts the winner will often say "I got lucky" or "you almost had me" to allow the loser to save face, at least publicly, when in reality the skill gap is evident.
if it only affected the company itself rather than ~the entire world, there may be face to be saved. trusted authorities have to be held to a MUCH higher standard, for good reason.
no grace if you use 4 as the private key for a trusted authority.
“One of eMudhra’s certificate authority private keys is the number 4. Not a number that starts with 4, or the number 4 followed by a few hundred random digits, just, 4… Another one is 56, and another is 115. I know, because I recovered all of them from their public certificates on my laptop just now, and it took less than a second,” wrote Helme.
Security researcher Scott Helme was among those who recovered the keys. This is a hall-of-fame level fail.
> Bruh, you just literally went ——— BEGIN PUBLIC KEY ——— 4 ——— END PUBLIC KEY ———
No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity that the statement they put out is completely in character.
As someone who's blood is tainted with teflon and microplastics same as you, I can state with clarity that attributing universal corporate tendencies to national work culture is completely in character for a Sagittarius.
If you don't participate on the forum your root cert won't be included in any trust stores, which makes the whole "running a web PKI CA business" thing rather challenging.
I hate to say this but majority of my professional encounters with Indians were with people having this little sense of accountability. But then again, with many awesome exceptions! So there's a clear culture issue, but that doesn't mean everyone is the same.
Yeah I remember that incident when Toyota Prius had a recall due to brake failure caused by an ABS software issue in rare circumstances and Toyoda himself apologized publicly.
This one is basically Japan being social media IRL since before it. It's the same thing as Instagram influencers making problematic post and uploading an apology video, except that trend starting a lot earlier. Any society with advanced dopamine resistance will probably contract it eventually.
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
Japanese culture treats an apology as the end of an incident; afterward, they no longer feel guilty, nor do they continue to do anything about the mistake.
To admit having messed up they would need to include the full private key in the post, which they seem to have decided is still more embarrassing than not including it.
What I hear is that he is saying that the SSL/TLS business is only a small part of their overall business and that they can close that part down as a result of the loss of reputation for that this has rendered.
It's not quite as egregious as someone purposely choosing the key. According to the write up on the Mozilla bugtracker there was a bug in the script used for there key generation ceremony which caused the bad certificates to be generated. Obviously completely unacceptable for a CA, but makes more sense than someone plucking 4 out of the air while generating a root cert.
Correct. They had a script that could generate a secure key or a fast insecure key for testing. The script generated both at the same time.
Human error was selecting the wrong output file and promoting it forward.
It's not at all the kind of thing you want to see in a reliable, secure, mature process... There shouldn't be a step where a human even makes a choice there.
And it's exactly the kind of thing most companies are doing, most of the time, more often than we would like.
An awful lot of the world runs on spit and bailing wire and things too easy to automate yet, and the parts that don't cost more.
Thousand line shell scripts that use expect to drive 30+ answers to a multi-choice interactive tui (that only works on one version of redhat from 2008 or something) is not so unusual, can totally understand how it happened. No excuses in that though.
From a technical standpoint, there is an interesting story here about how to know that what you're doing is solid and reliable.
I have to assume the fact this didn't get discovered suggests they weren't investing in trying to break their own crypto. A good takeaway for anyone else who wants to be in this space.
Internet2 provides InCommon Certificate Service, issuing certificates from "InCommon Intermediate CA" which chains up to emSign Root CA - G1. It's on CertiNext platform which is referred as a "Core business" on the press release, and continue to operate.
Yes, its all very well someone here shouting about "the absolutely brutal question battery the Chrome team publicly submitted to eMudhra".
But in all honesty there are serious questions to be asked about:
a) The initial approval process for ANY CA
b) The ongoing monitoring process for ANY CA
The fact this CA was based in India ultimately has nothing to do with it if we are being completely honest here.
Clearly there is a fundamental problem with the overarching approval and monitoring process which means it could have happened anywhere. This time it just happened to be India.
Secondly, in direct reference to the "far away company" comment India does have quite strict controls on both certain technical things (speaking generally here, not specifically in relation to certs ... e.g. use of sat phones). And in general it is quite difficult (read: almost impossible) to turn up in India and do business as an external company. So there is probably a genuine need for an Indian company CA since not every body can/wants to use Let's Encrypt or whatever.
> The fact this CA was based in India ultimately has nothing to do with it
Oh, then you will be missing more than half the fun of this thread. The sleeper agent racist in good people of the good part of the globe come knocking whenever a third world country (with a somewhat decent economy) comes in the news for bad reasons. You have also seemingly missed the threads about space related failures and accidents related to this country here. Those will be interesting to read. There are very subtle terms being used in some comments like "their worldview" et cetera.
> genuine need for an Indian company CA
Should be for pretty much any country and those certs should be trusted like other certs are trusted!!! Otherwise change the way these chains/trusts work.
> since not every body can/wants to use Let's Encrypt or whatever.
No they don't. It's an American company. (Let's not begin about "non-profit" part of it).
CA approvals are bad. From 2000 to 2015, I ran an org where we allow-listed a modest subset of the the CAs that were approved by Google, Microsoft, and Mozilla. There were too many incidents and bad actors.
Microsoft, in particular, had strong transparency around CA approval processes that were reasonable. It was weird.
Look we can just stop pretending that India isn't largely technically incompetent. We've all worked with offshore devs. Let's just be honest on the internet and stop pretending not to understand things.
Not to give into conspiracy theories but this company might be ideologically even if not materially aligned to current regime in India. With right kind of nudge any kind business contract is possible without any competency.
You seem to have an axe against current Indian regime which is not even outside the overton window. I suggest to not get your information from liberal-arts educated islamophile journalists who are struggling to pay off their loans.
As a reminder Mozilla was warned not to add the Chinese government as a ca they pushed back claiming lack of evidence and only removed them as a ca once cnnic got caught issuing certs for google domains (by accident )
Yes, not sure what the timeline was where trust was given to this company. Just horrible for the entire secure Internet. They've yet to answer many of the basic questions raised throughout this process too.
Beware even the countries internet regulatory authority (NIC) issued rogue certificates in past, about time NIC's root certificates should be ejected once and for all.
> Chairman ... indicated that eMudhra handled the recent incident diligently, revoking the affected CAs within 14 hours of discovery and disclosing promptly
So the top still doesn't understand (it's not diligent), wouldn't you expect the other 99% of the business to be managed just as "diligently"
Wow, the maximum salary in the range posted there is 6 Lakhs per year (LPA), which is roughly equivalent to $528 USD a month. So the HIGHEST paid developer is making only $528 USD a month. That's insane. The average salary in Bangalore for a software dev is around 12 LPA, and is often as much as 30 LPA. They're topping out at half of the average. Many Western companies pay >100LPA (equiv to $100k/yr USD) in Bangalore. For reference, eMudhra is headquartered in Bangalore. This isn't just low by international standards, this is very low pay by local standards.
People who are still facing this on a mac (like I was) - something I tried (and I am not sure this fixes it or not - or at least somewhat):
went to keychain access app and found all emudhra and emsigner certs -> there were 4 -> clicked "get info" for each -> under Trust section of the pop-ip -> selected "When using this certificate: Never Trust".
Their other businesses seem to be cryptography/trust related. What exactly do they do? TBH I have no clue how you trust their "enterprise PKI Solutions" when they've demonstrated a total failure to generate secrets. Am I off the mark? Do they do anything that you don't need to trust their process for?
If anyone else also struggles to understand all the "4" references in the comments: One of their private keys was "4", with others also being in the 2-3 digit range.
There’s an element of nationalism in this episode, amplified by the Indian government. This company, eMudhra, is a publicly listed private company. But the union government of India nevertheless felt obligated to downplay the issue (it always goes with a denial about any issues reported) and even went on a belligerent tone.
Quoting from a news report [1]:
> The Ministry of Electronics and Information Technology (MeitY) on Wednesday said it has taken “serious note” of the reported security incident involving eMudhra’s Public Key Infrastructure (PKI) and has formed a special team to investigate the matter.
> The ministry also warned against what it called “fear mongering” on social media over the possible impact of the incident.
The company’s stock is down to the lower circuit on the stock exchanges for the last two days. This exit may be a decision to stem the beating that the stock has taken. I personally believe the stock will sink further, as it deserves to.
Trust once lost, especially in a shameful meme worthy way such as this, isn’t going to be gained back easily.
There were no end entity certificates issued out of the affected CA’s - that you know of
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal - maybe to a full review of those? Are you rolling your own crypto there by chance?
Well, from a practical standpoint given the sheer incompetence that was on display they were “exiting” the certificate business regardless of their “decision” to do so.
This incident separately raises deep questions about the competency of the company to do much of anything in the security or compliance space.
I cant comprehend how they were able to put something like "4" as the private key. someone said that they could have used a rand generator from 0 - 255 instead of 256 bytes one. but anyway, I don't remember putting a private key value manually and always generated it with something like keygen.
> There were no end entity certificates issued out of the affected CA’s.
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
The certs were intended to be used by new US and EU subsidiaries. Those were still in the process of being set up, so the certs were never used in production for customer signing.
IIRC the Comodo AAA Services root was already in the process of being retired, with its "trust bit" having been removed by Mozilla and Chrome last year. Nuking it completely should have been a no-op for end users.
The whole chaining & cross-signing song-and-dance will never go away completely. There's just no way around if if you want enrollment of a new CA to take less than a decade, or if you want CAs to be able to do any form of non-world-breaking key rotation. Intermediates signed by third-party roots are already treated with the same scrutiny as root certs, so they should have been fine...
But yes, this will undoubtedly result in even stricter verification. For example, some participants are already calling for mandating attestation that a key has been generated inside a HSM running the right config.
> I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist.
It's already in the cab agreement that basically any unlimited intermediate you sign has to follow the same rules. And yeah, as a root in the program, if an intermediate you sign turns out bad, your root cert could get pulled, so you should be careful.
# of issued leaf certs isn't a great indicator of propagation, user-impact, revenue etc.
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc.
2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
GOOD. RIDDANCE.
You cannot take these people serious. They should never ever touch something serious again.
> All physical and logical security controls remained intact.
Did the HSM have a fault, or was the key generation script at fault?
I suspect the later given the choice to exit the business. If they had strong evidence that a FIPS-compliant HSM misbehaved, I suspect they'd be more willing to fight this.
Wow this press release is a real PR disaster. It's really just about their business value.
Starting the statement with "a global digital trust company" is so ironic on that context.
X.509 needs to be replaced with a simple key you get for free during domain registration. Then the blast radius of key compromise is a single domain name and not the whole internet.
How does a client know what public key to expect for a given domain, then?
If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).
It does, but you also lose 100% of the accountability. eMudhra is leaving this business because the root programs are about to humiliate them. There are no programs anywhere that can do the same thing to a major TLD operator.
If ICANN decides you no longer operate the TLD, and takes your name servers out of the root server zones and replaces them with somebody else's... you don't run that TLD any more, sorry, go have a good cry and then see if maybe running a hot dog stand is more your speed. DNSSEC is one of the services ICANN considers critical enough to take away your TLD stewardship if you aren't providing that service properly.
Of course we wouldn't want ICANN to do that on a whim, or without having a new operator for the TLD ready to go, but there's already process defined for both planned and emergency transfers of operation of a TLD:
You don't think so? In a scenario where it becomes known the US government is actively subverting Verisign to tamper with DNSSEC integrity (or where Verisign just screwed up badly enough to make their continued stewardship untenable, and I think either case would become known, in pretty short order), I absolutely think ICANN would expatriate .com. If they didn't, they would lose legitimacy with everybody who isn't the US government. That leads to at best an international effort to replace ICANN and at worst the splintering of the DNS root. Nobody wants either of those scenarios -- the least-damaging thing for the Internet and for ICANN as an organization would indeed to be to take .com away and let say, the British run it for awhile. (Maybe you could get your .com domain registrations at Sainsbury's, in the aisle next to the anti-demon plant misters.)
Which is exactly the kind of scenario a lot of people hyperventilated about back when ICANN was formed, for reasons of varying validity, but in this case it would indeed be the least-worst option all around.
> In a scenario where it becomes known the US government is actively subverting Verisign to tamper with DNSSEC integrity (or where Verisign just screwed up badly enough to make their continued stewardship untenable, and I think either case would become known, in pretty short order), I absolutely think ICANN would expatriate .com
ICANN is an organization headquartered in California ... I don't see how they can redelegate .com without approval of the feds.
IMHO, that isn't that compelling of an argument against domain hierarchy certificate issuance. If the feds want to mess with your .com today, they can and then they can get a cert from any CA. Adding a dependency on CAs doesn't remove the dependency on the TLD operator.
If you wanted to stick with x.509 certs and you had an ICANN root that would sign Name Constrained TLD intermediate(s), and the registry would sign name constrained Domain Intermediates, and all those require certificate transparency, then we'd be about where we are with WebPKI, but without trusting any parties you don't already have to trust.
I am 100% confident that .COM will not be rescued from law enforcement and national security surveillance by ICANN, yes.
You can disagree! But then: let's just keep the premises clear. You think DNSSEC makes sense because you also think states will lose control over their registry operators.
The blast radius of that is every subdomain of whoever got compromised. Having only ICANN being capable of breaking the entire internet is a world of difference from what we have today, and may be the best we can do... and we should be using that, but it's not perfect either.
It's not ICANN in that system, it's the TLD operators, and the problem with that is that for many TLDs and all the most important ones, you'd be adopting a system where your keys are directly managed by institutions we already know to have misissued. The most important TLDs are state-controlled, which is why you saw DOJ notices on piracy sites rather than NXDOMAINs.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Translation: We are exiting this business because people are being mean, and we don't get paid enough.
WTF!
They also say:
> We believe an organization's character is shown in how it responds when something goes wrong
I totally agree with this. I will judge their character based on this: No apology for wreaking havoc on the world, no admission of fault, whining about people getting pissed at the monumental incompetence, saying that they are exiting the business because "sour grapes" rather than admit they have lost the trust within the community. <sigh>
I'm surprised they didn't announce they're going into hiding instead. I'd imagine they probably have themselves covered with user agreements and contracts to some extent, but the potential liability is nearly incalculable.
I don't know what jurisdiction this would potentially fall in, but in the USA that would be a pretty straightforward crime. Tampering with evidence & obstruction of justice.
With over 17 years of expertise, eMudhra is a global provider of digital identity, trust, and security solutions, enabling secure digital transformation for banks and financial institutions (BFSI),
So, you were in business for 17 years and still decided to use plaintext number "4" for your private key?
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal.
Oh yeah, I'm sure the other customers are sipping their bobba tea reading about this instead of thinking "are we next?"
Fuck these clowns, in India to get an import/export license you had get a hardware key via USB from them to access the government portal. Worse yet, their website is designed to look eerily similar to a government website and a lot of people won't even be able to tell it's a private entity.
Now imagine, all the customers who had purchased this key from them have their private details exposed. There was absolutely no mention of this anywhere. Not even as much as an apology. This letter is full of arrogance, deflection and insult to injury. I hope they burn to their ground. It's a really shame the government isn't even stepping in to do anything about this as their customers are the ones that are most impacted because they mandated the stupid hardware key requirement. What a joke of a situation.
In any other country, everyone involved would be fired and there would be serious consequences for such a level of breach (Eg. Equifax). In India, even a class action suit would lead nowhere and the fact that these assholes would be operational still despite this is extremely infuriating.
I think this opens bigger questions on who and how people get access to these capabilities. After that recent cold wallet physical supply chain take over, what sort of people are able to get certs for anything they need?
Typical bullshit where India regulations force people in India to buy local - and because the landscape is filled with incompetent and inept companies you get shit like this.
Kinda funny that logo says "Trust Delivered". They don't seem to have 0 problem with such a big scale damage on public internet and simply trying to shrug off the problem.
This is problem with Indian companies, their founder and management always try to hire cheap labor and penny pinch on everything. And, this is the result we see again and again.
Nevermind the legal liabilities, how is their company viable with this sort of brand damage? Their other business lines aren't making toasters. It's the same line of work. They sell trust.
Not a cryptographer, but I wonder if some sort of basic statistical fingerprinting of the private key being checked at every step of its manipulation during the ceremony would have caught this epic operational failure.
Say, like a multi-scale hierarchical half popcount vector (first/second half, first quarter+third quarter/second quarter+fourth quarter down to odd/even bits), where fumbling one private key for another or xkcd-ing 4 as a private key would be detected, but still reveal no information on a truly random series of bits and be understandable by non-technical people in the room.
They also have to convince the root programs to let them stay trusted, which can't be done by issuing vague press releases until the news cycle moves on.
Yes they didn't close down the business, it was already gone. There's not a business called "selling certs that aren't accepted by any browser or OS". That's not a product.
Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.
True, but the founder's two sons are at the top of the company, so mission was accomplished regardless.
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
There was no cyber attack here. They released a cert with a trivial key. They sold a lock with no pins. When people realized what was being delivered, they blew the whistle. They were killed by a self own of epic proportions.
honestly, props to the c-suite to have enough awareness that this buisness was already dead for them and there was absolutely no pathway where they are getting their certificates trusted in PKI ever again.
this lets them exit this without public humiliation of getting made fun (and example of) in the mailing lists.
What about CAA records? And of course, browsers handle this with the Certificate Transparency log and the OneCRL, made less painful by baking a short-lived proof that the cert is not revoked into the handshake. The problem is all the other parts of the ecosystem that basically ignore all this new complex security stuff and just check the cert against a static list of vendored roots.
For example, node, java, and python all have vendored roots and don't check any CRLs.
Well, what about them? If you've got access to the private key, nothing is going to stop you from generating a cert violating CAA records - and it won't even be easy to detect! They are intended to protect against, say, someone hacking into your server, running ACME to get a cert, cleaning up their traces, and then intercepting your traffic for 90 days. CAA records assume the CA is trustworthy and following the rules.
There's no reason clients couldn't, as far as I know (just like the various anti-spam measures where clients check for records that indicate what servers are allowed to send mail from a domain and consider it spam if those records don't match), but I also don't know of any clients that actually do. And a CAA record still doesn't actively prevent issuance of a certificate, which is really what would be needed.
There used to be a pinning scheme called HPKP, but the trouble with it is it's very easy to accidentally (or via malicious third-party action) end up in a situation where your web site becomes unreachable by people who've visited it previously because the key changed and the pin prevents their browser recognizing the new key as valid:
It might have been little-known in your corner of the world, but I understand it's definitely not little-known in India.
You might have heard of DigiCert or IdenTrust, but what, exactly, makes them trustworthy, and why eMudhra wouldn't be considered trustworthy (a week ago)?
Good. One less company that doesn’t give 2 shits about security. Probably billions in man hours spent deploying mitigations because they can’t manage their keys properly
as part of a review process its not just the developer that wrote the code who gets blamed, its also the reviewer of said code.
Analog
sure eMudhra needs to be investigated and confirmed that even a small part of their business hadn't been used to compromise (it doesn't matter that they left the business, it matters that the investigation continues)
but also we need to see how it is this even happened, who dropped the ball and approved the bad code/behavior to enter in the first place??
or is this a structural issue where we need to introduce more checks/balances (tests)
When is Sectigo, the root CA that cross-signed the vulnerable intermediate CAs, going to take responsibility for fucking up too? Why should we keep trusting them?
eMudhra was a root CA. Their roots and intermediates were cross-signed by Sectigo since it was a new(-ish) root CA (a comment on Reddit characterized eMudhra as a Sectigo reseller, but I can't find verification of that), but I don't know to what extent that was done as a result of Sectigo's own evaluation vs. some internal process of a larger entity like the CA/Browser Forum. (Without cross-signing, it's difficult for a new CA to operate, so I imagine there must be some process by which a particular existing-CA cat is belled to be the cross-signer.)
I'm not sure (but would love to know) if Sectigo actively chose to sign eMudhra's certs (though if eMudhra is a reseller of theirs, it would make sense to do so), or if maybe it was just "their turn in the barrel" to be the designated cross-signer of a new CA's certs so the new CA could be a useful entity.
No? DANE is a dead letter. It doesn't work reliably on all Internet paths; too many middleboxes flip out if they see a DNS packet that doesn't look like every other DNS packet. For many years, there was an effort to work around that with "DANE stapling", where the signed DANE records were tacked on to the TLS handshake. But that leaves a chicken/egg problem, because attackers can strip the DANE records out of the handshake. The project was abandoned several years ago, and with it any real prospect of DANE becoming mainstream.
I think that's for the best; there are much bigger problems with DANE/DNSSEC than TLS reliability.
I don't think (and wouldn't hope) this would be a flag-day thing where we stop using CA trust completely all at once and only use DANE afterward. DANE would become an option, disabled or secondary by default, that users could choose to use as a fallback if CA-based TLS validation fails. Eventually as adoption increased and the critical paths got cleaned up, DANE could become the default primary, with a fallback to CA trust if there's no valid response to a TLSA record request. Eventually as CA trust fell completely out of favor for most uses, it wouldn't be necessary for many clients to support it at all.
See, this doesn't work either. One of the core jobs you have designing a secure transport is avoiding downgrade attacks, and the construction you just outlined is not only downgradable, it's explicitly downgradable, as a feature. That's not going to happen.
No? There's a very specific technical problem that makes DNSSEC hard to deploy effectively. You can't just define it away. You have to pursue solutions that solve that problem. There's a whole huge solution space to draw from, but people are so path-dependent and so predisposed to like the IETF that DNSSEC is just the only discussion we seem to be able to have.
Fortunately, the WebPKI has been ratcheting (slowly) up security over the last 15 years, so when nothing else happens, which is what I predict, at least we'll be in a better place a decade from now.
This all day. There is no reason I should have to go through a third-party to get a certificate just to distribute a key for secure communications (which is all a DV certificate is good for) if I already own a domain, or can get the domain owner to publish my TLSA record in theirs (which should be no more onerous than publishing the server A record, which they already have to do).
The reason we don't have DANE support in browsers is of course because Google decided not to implement it in Chrome, because (ultimately) the key lengths securing the root DNS server zones weren't large enough to suit them (at the time, 1024 bits):
And if Google wasn't going to do it, it didn't make sense for any other browser to do it either.
However, I believe the root zone keys are now longer, so I could (potentially) see Google implementing DANE in Chrome now, maybe with a Chrome-specific restriction of "TLSA records that have keys under 2048 bits in their chain of trust are ignored" (since the root zone keys are not the only keys in the chain and they want to eliminate 1024-bit RSA from Chrome). And if Google does it in Chrome, then everybody else has to do it too, so the situation flips.
DANE doesn't remove the requirement to get a 3rd party to vouch for your keys; it just locks you into one very specific 3rd party, one likely subject to de jure control of a state.
It seems much less likely that even a well-funded state actor could silently subvert enough root servers, which exist in many countries, to significantly compromise DNSSEC before alarms were raised, than that CAs are actively compromised or are just bad at being CAs, resulting in a breach like this (which happens approximately all the time). And with regard to state coercion, some CAs are in fact state-run, by states most people would consider grievously corrupt.
More to the point I would trust IANA and ICANN, faults and all, to not screw up the DNS/DNSSEC root, much more than I trust every CA in the default browser trust stores to get X.509 right all day, every day.
Again: you don't have to touch the root servers. The root servers are a red herring here. It's the TLD operators who exercise all the real control. IANA/ICANN will have nothing to do with the resulting situation.
Short summary of my response to that comment: you're wrong, and ICANN does not in fact have the real-world power to liberate .COM from the United States Government.
(Of course it's fine to use AI tools! We use them heavily ourselves. Just please don't use them to generate or edit text to be read by fellow humans on Hacker News itself.)
It's impossible to know for sure what was LLM generated or edited, but we have to rely on imperfect classifiers or else the site would be completely flooded by genai text, which is not what the community here wants to read, nor in the intended spirit of HN.
now the question for the media for the next week :
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.
I think we live in an age where programming mistakes should be attributed to llm by default. In the old days, it was just the coder and stack overflow to blame, now its layer on layer of ai assisted work, from ceo level on down. Something we haven’t perfected yet: how to prove 0 ai use in a product. Until we get there, we have to assume llm are everywhere and that their use carries liability risk.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
It was "multiple human errors". But yeah no matter what, in any case or exceptional circumstances, this thing was not supposed to be screwed up.
On Mozilla’s tracker, eMudhra said the keys were created on 30 April 2025 in a key generation ceremony, a recorded procedure in which keys are produced inside hardware security modules (HSMs), tamper-resistant devices built to hold keys. The keys were meant to be moved to data centres in the United States and Europe for customers needing data stored in those regions. To move them, the company added a step to ‘wrap’ each key, or encrypt it for transfer, and ‘unwrap’ it at the other end. eMudhra said a flaw in the software script created a second, weaker set of keys during this step, and because the two sets had similar names, staff used the weaker ones by mistake. It said human errors occurred at multiple steps.
source:
What's the eMudhra security scare that has rattled the internet?
https://theprint.in/tech/whats-the-emudhra-security-scare-th...
These "wrapping" keys were not "weaker". They were completely unfit for purpose. They have not explained why they used a 3-bit key for this purpose.
If they go to the trouble of wrapping them, use real keys. Or don't do it.
Also, it's a big "human error" to not notice this for a year and a half.
I have to imagine that the 3 bit numbers were slots/indexes on the HSM, and rather than actually retrieving the key(or maybe it did), the slot number got used as the key. I can't otherwise imagine how they'd generate such low number keys...
AI ceremony.
I have not read up on all of this, but I would have guessed they were artifacts created and validated by a testcase. "testcase 1-8".
Having worked around these types of devices and their interfaces, I doubt they intended to create weak keys.
It is more likely they had a script which placed a parameter in the wrong field, due to something like using positional arguments. Or they misread the documentation for some interface.
Still incompetence. And I would love to see my theory confirmed or refuted.
The previous discussion advanced the theory that they fucked up the string handling. Eg in Python, output = key[0] would result in one-byte keys like "4" if you expect "key" to be an array, but it's actually a string.
> worked around these types of devices and their interfaces..
This is not a problem of devices and interfaces. You are supposed to generate the wrapping key on the HSM itself. And the HSM ensures that the keys are generated with a strong enough security parameter, not some silly 1 byte string. Then the HSM wraps the key inside of itself and exports it in the encrypted form. You take this ciphertext, and the wrapping key and send it over to the other party.
So if they generate the wrapping key with a funny value, that means they didn't use HSM to generate it but their system openssl. But then, HSMs typically don't allow you to export private keys without wrapping them first unless you first tweak around with security flags (not recommended at all), or simply generate the private key using your system openssl too (LMFAO).
Source: I work for one of the major HSM vendors.
I hear you in terms of how these kinds of mistakes typically happen. But why did the script ever produce two sets of keys? The initial root cause was somebody picked the weak key not the strong key. But what was the reason for ever producing a weak key in the first place? They say the script produced two outputs - what on earth for??
Stepping back a little to look at the point and gravity of the exercise makes me wonder why they didn’t just fly it over.
I mean, why have the ability to generate weak keys to begin with? Serious question, I've never done this sort of work. What sort of utility does that bring to the procedures and workflows one does in that line of work?
If you didn't know better, you'd think this had been done on purpose by some three-letter agency.
Testing could be a reason.
To be clearer, I’m not sure these keys were generated, but being able to use weak keys is something you may want to do for testing.
Yeah. I can't think of any development process where you wouldn't simply use a production grade key in development. In fact, that's exactly what we do in development with snake oil keys.
Generating a key with "4" was probably done as a joke, but also is insane to have ever lived outside some devs computer.
> These "wrapping" keys were not "weaker". They were completely unfit for purpose.
Oh come on. They clearly didn't intend to use 4 as a private key or to wrap it. They had just purchased millions of dollars of HSM's to store them. It was something more subtle than that.
Have a little imagination. The transfer procedure sounds bespoke - possibly just a shell script using openssl to encrypt things at one end, and decrypt them at the other. Let the man who has written 100 shell scripts without a bug in one cast the first stone.
But not so subtle that good software engineering procedures wouldn't have picked it up. In fact today just an LLM review of the shell script probably would have caught it. The Internet Death Sentence was warranted - because poor software engineering procedures are unforgivable in a CA. But those procedures and policies aren't easy or cheap to come by. If they were just "don't chose 4 as a primary key" my grandkids could do it.
Unwrapped extraction is the kind of thing security by checkbox covers and process review might catch so it is never clear in a case like this that some party with the capability to influence the process didn't want keys intentionally weak to emulate unwrap.. Then someone else may have introduced the swap of keys that would never have been seen with keys that would (be forever) in a public scrutiny DB.
I don't think it's implied the wrapping keys were the weak ones. More like, the script that ran the wrapping still had the fast "debug" generation of a pair of keys to test the wrapper on enabled, for example.
Well said. I choose not to believe this was an error.
vibe coded?
and imagine being responsible for halving your corp's stock
wait, a 3-bit key??
One of the 384-bit keys was literally 4. All of them fit in a byte.
Their complaint about social media reeks of "and we would have gotten away with it, if it wasn't for those meddling kids". [1]
[1] https://www.youtube.com/watch?v=hXUqwuzcGeU
We deeply and sincerely apologize for getting caught and not being able to cover it up.
This is pretty normal in India unfortunately. Crony capitalism is the dominant business engine in the country, so pretty much any significant business, esp in regulated sectors like govt mandated services(which is the bulk of eMudhra's operations) is impossible to set up without significant bribes being paid.
As a consequence, these businesses are very well protected from normal market forces and are pretty much above the law. Had this incident not been an international one, they had a very good chance of getting away with it, and the people reporting this would have landed in jail.
This is 100% accurate. State run media would have squashed this article if it was not for an international incident response. They have federal laws that prevent such disclosures because it affects the national image and could cause social unrest.
Since you seem knowledgeable about this topic.
Would you say the system forces people to be like that or people are the reason why the system is like that?
Would you do business (i.e. trust) with someone that comes from that system?
You're asking a chicken and egg question, I believe it becomes a self-reinforcing loop
In regards to doing business - someone that comes from a corrupt society with a culture of taking advantage of others should be approached more carefully than someone that grew up in the same town as you, of course
This is a lesson learned the hard way, where I moved to a different country and treated everyone with openness and generosity (of time, resources, willingness to help), just to realize people from certain cultures mistake this behavior for weakness and try to take advantage as much as possible
Which country is this? So I can avoid it.
> Would you say the system forces people to be like that or people are the reason why the system is like that?
It's the courts. Courts are meant to bring in an error correction mechanism. When courts fail, the country goes into open-loop control. No errors are corrected, and eventually, people stop even detecting errors. https://x.com/captgouda24/status/2110089855380369511
> Would you do business (i.e. trust) with someone that comes from that system?
Vet rigorously and continuously! Start with small business, and grow it based on interpersonal trust. Since the system lacks accountability, you must ensure that yourself.
Wild to me that our healthcare data lives there.
Depends on the contract your healthcare providers/payers/etc. have with those who do their IT.
Source: work on healthcare IT projects and can't let anyone outside of the US look at prod data no matter what due to the contract we have with our clients.
[dead]
>capitalism is the dominant business engine in the country,
The government _mandates_ that certain services be used. In which world does this rhyme with "capitalism"?
I said _crony_ capitalism is the dominant business engine in the country.
The govt mandated services are owned by private capital(hence the capitalism part) and govt picks winners by granting licenses to a select few(hence the crony prefix)
Any world where sufficiently wealthy capitalists can bend the govt to their will, i.e., most of them.
It does, but I wonder how much of that phrasing is cultural.
When working with different businesses across the globe, I’ve noticed Indian correspondence to incidents tends to be more a show of strength even if it’s transparently obvious that there’s no merit to the content.
Whereas western European correspondence is often more apologetic and empathetic. Even if it might sound insincere to those affected.
But it’s all just words at the end of the day. It never means much unless the business is willing to back real change to mitigate similar issues in the future and resolve undisclosed existing problems. However there’s no way for us to know if this is happening or not.
I don't know if it's really specific to one culture. Look at US companies after they go through a lengthy legal saga, it's obvious they screwed up, they're paying fines or having to adhere to court-imposed consequences. Their press releases:
"We admit no wrongdoing. We are only settling to move past this. Our business is strong/secure/competent."
It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.
>It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.
It's the first rule you learn after joining basically any organisation larger than 5 people.
People on HN who are surprised are seemingly self-employed prodigies who dropped out of middle school.
I’ve worked for a lot of enterprises across the globe and I can tell you that you’re flat out wrong. That’s simply not how all organisations operate.
Just take a look at any of the thousands of postmortems published over the last decade and you’ll see that the tone of incident response communication can vary massively.
The truth would have come out eventually- after a bunch of security cluster fuck incidents.
Their OTP solution is bruteforceable as well: https://mastodon.catgirl.cloud/@49016/117475921584308861
This becomes an inverse Russian roulette: only one of six chambers is not loaded.
Waiting for a release like "... the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP 1FA place in our business. In view of this, after detailed deliberations our board has concluded that withdrawing from global OTP 1FA is the right decision for eMudhra and its shareholders"
> This becomes an inverse Russian roulette: only one of six chambers is not loaded.
At what point does it become Russian roulette with a Glock?
One chamber is enough to play Russian roulette... once. However, the probability to get to play a second time is rather slim.
Link is broken, gives some randomly generated text when using Brave on Android. Their AI bot detection is not very good.
> fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:
That Web site thinks that my browser is an AI scraper for some reason.
It's not clear what you have linked there.
Paste from mastodon:
> fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:
"However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP's place in our business"
Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider.
Then indirectly the security and trust of the entire Internet rests in the hands of a few government babus in India... that's scary.
Their Internet registry people also weaponized AS0 (zero) as a method to take networks off air via BGP RPKI for the ranges they're responsible for. There's been a number of situations where multinational companies who operate in India who are forced to use IPs from the India RIR have been fat fingered by some staffer who makes changes then goes home for the day: meanwhile a multinational bank is down for 24 hours within the country because said agency doesn't believe in 24/7 support.
Typically they do this for non pay but they've dropped the hammer multiple other times incorrectly (and good luck getting a RFO out of anyone there).
For good or bad, thats by design. These is a significant portion of physical infrastructure associated with internet data flow. The only challenge governments face as of now is being able to segment the traffic to their wishes in its entirety.
Companies already control all the traffic flow on their infrastructure using end-point security solutions, the same can be extended at ISP level, and then Country level
It’s called regulatory capture.
It’s get worse when these companies work with regulators to create more regulation that they can sell more of their services.
https://www.amazon.co.uk/Compliance-Industrial-Complex-Opera...
Regulation like this makes sense, but then the government should also srutinze these companies
Well Indian regulator must have visited office canteen to see no unapproved food was served there.
Only when fully satisfied they'd have given license to operate this security business.
"bias"
"out of proportion to SSL/TLS’s place in our business" is an insane comparison. They are saying they care so little about SSL/TLS to begin with that it's not fair.
It would be like me practicing surgery 1 hour a week and saying "well surgery only has a very small place in my 80-hour work week" in response to flagrant failure of medical care.
"We are bad at this job, but we don't do it often so don't think we are bad at other jobs".
Yeah, the proportion if anything is too small. This is the sort of company that should not be doing anything related to security.
This is what I found regarding how it happened, but still not a full explanation - but there was some name confusion and obvious incompetence.
>According to eMudhra, proper strong keys were created in the HSM at the key ceremony on 30-04-2025. The script had been changed to test wrapping and unwrapping keys, because the intermediates were to be moved to data centres in the US and Europe, and a bug in that change created a second set of weak keys. The keys had almost identical names, and the weak ones were used for the certificate requests. The same request was signed by eMudhra's own root and sent to Sectigo, which cross-signed it. The strong keys never made it into any certificate (Bugzilla).
https://www.fairssl.se/en/articles/emudhra-ca-noegler-knaekk...
Fantastic key ceremony. Great time had by all. Can't remember a damn thing, we were all so plastered.
I wonder. Say I wanted to prevent something stupid like this from happening in the future. What would be a sensible check? The higher level authority doesn't see the private key itself, right? Just the certificate request. And I wouldn't want someone to manually inspect the keys to see that it looks random-ish. Perhaps there could be a small check script added to the generation of certificate requests that says "hey, your key a.) is mostly zeroes, b.) appears to be ASCII text, or c.) seems otherwise extremely non-random, please use -f to continue anyway".
I don't know if it will work on something as small as a private key, but you can compress the key. If the compressed key is significantly smaller then the original, it lacks entropy. Pure random data is mostly incompressible.
Note that 32 0x00 is a possible output from a pure random generator while still being quite compressible, you would probably still want to throw that one out though. After buying a lottery ticket.
Do not allow devs to modify ceremonial hardware.
I was about to reply with "they didn't have a choice" but on second thought perhaps CAs are important enough that they should never be allowed to export keys from an HSM and should instead be required to physically deliver and install the HSM wherever it needs to be in the world. Inconvenient and expensive but likely for the best.
Right? I don’t know why anybody would trust this company for anything.
Someone higher up in the hierarchy telling you to trust this. You have no option to challenge it.
Russian Roulette is pretty accurate if not an understatement.
If they expect people to still trust them on their other services they have obviously no clue on how Trust works, meaning they shouldn't be in the game in the frist place.
As a trust provider you get one chance. Squander it and you're out. That is how it should be.
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
In any legal dispute involving them, Exhibit A will be "look at how they ran their SSL/TLS business". That won't go well.
They got their image hurt, and pulling the culturally appropriate response here which is to never apologize and to never accept fault. Instead quit/resign and say that other stuff is just fine. It's really not fine, and you can bet that the same corporate philosophy that made the CA error will have serious security errors in their other products.
I think it doesn't matter what they say because they'll just get away with it. It's as simple as that. They'll get away with it.
[flagged]
There is no such honor code and stop spreading this BS propaganda. Izzat is not even an Indian word used in any Indian language. It is Arabic which got mixed in with Persian which in turn got mixed in with Urdu.
Well reposted.
West has something similar. Politicians never apologize, and avoiding Indian certificate authorities (and recuriters) is racisist!!!
Somehow this is making it even worse. Like, I'm having issues explaining the whole thing to my wife, because I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done. Anyone's got a good one?
Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
The internet's security relies on an almost impenetrable wall built around it. The king appoints gatekeepers that are each entrusted with a gate they must protect. The gatekeepers must follow a rigorous interrogation process to verify the ID of anyone who wants to pass through their gate. Only then are they permitted to unlock it.
The king was very careful to ensure all the gatekeepers were following the process correctly. It did not occur to anybody that one of the gatekeepers could be using a dirty rag to jam their gate closed instead of a lock.
Once discovered, the wall could be trivially bypassed. The security of the whole internet was immediately broken. Rebuilding the wall to block up the dodgy gate took much effort from very many people.
I think part of the issue is that we allow gatekeepers to open any gate.
This system was invented in the 90s when things were looking up. China was opening up to the world, the Russians were suddenly not evil anymore, and the internet was supposed to bring us all together into one happy global society, kumbaya and all that. Obviously that didn't work out.
It's time for a rethink really. Every CA is now a single point of failure.
The entire system (CAs and ICANN DNS) was never good to begin with; it's just become that much worse due to the shifting situation over the years. It's an inherently centralized design with no way to verify that example.com is operated by the same entity from one day to the next. The namespace is global and effectively an exercise in rent seeking.
At absolute minimum any sensible system would have a TOFU element whereby the site operator of example.com changing from one day to the next would result in all the tooling across the entire stack screaming loudly (in other words what ssh does). And that's just the absolute bare minimum change that's needed.
Now the gatekeeper is saying "I don't want to be a gatekeeper anymore for this one particular gate, because people were mean to me. I'll however continue being gatekeeper for these other gates that drive my business either way."
I think a part of the issue was, everyone living outside the walls did not even try to kick the gates open as everyone assumed a big (cryptographically secure) lock was being used by all gatekeepers.
https://drmaciver.com/2022/05/learning-to-walk-through-walls...
Citing Terry Pratchett’s “Interesting Times”.
Its called a Potemkin solution, and there is a serious theory that most nuclear arsenals are exactly that: no-one has ever confirmed they can actually be fired.
> no-one has ever confirmed they can actually be fired
that is untrue
In the case of nuclear arsenals that's perfectly fine - MAD simply needs to be plausible, not robustly demonstrated. In the case of CAs I'd argue that the description doesn't really fit since they're getting stress tested by malicious parties on a daily basis. By that logic you could describe linux kernel security as a potemkin village due to the steady stream of zero days. In reality a potemkin village is something intended from the start to deceive everyone else. It's not impossible that's what happened here but that's a very serious allegation to make (ie theorizing that the weak keys were intentional with a conspiracy afoot).
In this analogy I think it would be more appropriate to call them a lockmaker, and a lot of gates have been using this lock. Finding and fixing every lock is where the effort comes from, and some locks may not be found for years to come.
This lockmaker analogy fails to capture one critical aspect: the failure of one CA breaks the security of everyone, even if they did not use that CA's "locks".
This is why I went for a "wall with many gates" instead; any bypass makes all gates redundant.
All analogies break down at some level; the main goal is to clearly communicate the most important ideas.
Sure this happened in the physical space in 2004
https://jeffwerner.ca/kryptonite_lock/
People swapped out their bike locks en masse, it was a pain.
In the old days a king would "sign" his proclamations by pressing his "unforgeable" signet into a wax seal. This proved the authenticity of the proclamation.
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
The signets were "unforgeable" in that the penalty for copying one was death by being tortured, hanged, drawn and quartered.
They were "unforgeable" because commissioning a jeweler to make an intricate carving on a ring by hand was very expensive in the first place. Finding one who could precisely replicate an existing carving, down to the mistakes of its original author and any wear and tear that would be visible on the wax seal, was almost impossible and even more expensive.
Of course, I didn't mean to imply it took zero effort to copy one, though I guess that's how it read. You'd have to find a skilled jeweler that's also willing to commit treason, so for sure the barrier wasn't insignificant there.
Indeed. In both this scenario and our modern one, we rely on the difficulty of "key material" reproduction being greater than the difficulty of its creation.
That would be retroactive security...
However, Kings likely benefitted from the option of a scapegoat, they could always blame someone for forging their unforgeable seal and have them killed, rather than admit that the seal was true.
If we add in attainder and escheat, the King had a pretty strong hand.
It's enough that the signet could be verified, and sometimes was [1].
As it turns out, these documents were mostly useful for long-lived décisions, which could be overturned, and where a forgery exposed you for a lifetime.
[1]: https://en.wikipedia.org/wiki/Robert_III_of_Artois#The_fraud...
Bravo
"the king has no cryptographically strong certificate"
[dead]
https://bsky.app/profile/anthonymoser.com/post/3myf5fwpl7s2t is a good explainer I think without getting into the nitty gritty details.
This seems like the easiest way to frame it to non-technical people to me. They know what passwords are.
I just explained what it actually is. I said "you know how you go to a website HTTPS and it's secure? Well Apple chose a couple of trusted parties that your phone inherently trusts, and those people approve a few thousand intermediate companies who do the day-to-day certificate work for end websites. One of those intermediates, rather than choosing a 200-digit random number, chose 4, no not four-digits, the number 4". And then we could laugh at some of the memes.
Our 200 digit random number was 199 zeroes followed by a 4
- emudhra lawyer maybe (:
Private key size is 384 bits or 116 decimal digits. So 115 zeroes.
You mean, the key was 383 zeros and one “one”, on the third place from the right :)
I really wonder if this would/should hold in court. It is ridiculously unlikely yet genuinely possible.
One of them, sure. 8 of them?
At this point we're talking about chances so slim that only philosophy can save the situation.
technically possible
Not just technically possible, as likely as getting any other random 116(?) digit number
Feynman's license plate
i think this oversells the likelihood of it. the probability of any random 116 digit number having its first 115 digits be the number zero is much lower than that of it containing a much higher number of non-zero digits.
It's hard to explain partly because when you explain this thing we've done with TLS and PKI, it sounds a little insane.
> "We set up a system where every government and a bunch of random banks and IT companies in the world holds the secret codes to everyone's encryption. Then we make them pinky swear not to abuse it, with stern talkings to if they do. And now it's all on fire, because one of them was irresponsible!"
You can know the intractability of the problem that PKI is solving, and the (worse) alternatives that have been tried, but that doesn't make this system not-insane to a normal person, and its at least a little bit the fault of the collective "us".
And it's not like it is an impossible problem; we rely on TOFU for SSH, we have remotely-attestable TPMs everywhere, ZKPPs / PAKEs exist, session keys and federation protocols are widespread. Somewhere in the set of "possible solutions" are options that don't sound unhinged when you explain how we got to this eMuhdra situation.
I believe you misunderstood how it works. A parent certificate cannot be used to decrypt information encrypted by a child certificate… the CA issuer just signs your child certificate, it doesn’t have access to your private key.
Then, everyone who trusts the CA will trust your certificate by checking it is signed by the CA who should have done the checks to ensure your certificate contains truthful data.
So your layman description of “they have secret codes to everyone’s encryption” is completely wrong.
Certificate keys aren't used for encryption anymore anyway.
A modern TLS connection uses an ephemeral key-exchange protocol where both parties opaquely agree to a one-time-use random encryption key that is not knowable to any other party, not even someone who can gain access to either party's private key. This is called Forward Secrecy (originally and sometimes still known as Perfect Forward Secrecy or PFS, but the use of the word "perfect" in any cryptographic context is akin to naming a ship "Unsinkable").
Right, thanks for correcting me. This stuff is not so simple to get right.
They have the secret codes to MITM (intercept, inspect, alter and forge) any communication if they manage to place themselves between you and a party you trust.
Ok, but the analogy was wrong anyway.
> random banks and IT companies in the world holds the secret codes to everyone's encryption.
That's not true at all.
What they do have is the power to MITM connections IF they first spook DNS or insert themselves into a proxy accepted by the server somehow.
So, if you want to be honest, the original sentence becomes:
> random banks and IT companies in the world holds secret codes that are trusted by everyone to sign information that is used to establish trusted communications. If they wanted to, they could try to tamper with people's network connections and, if successful somehow, pretend to be anyone else that had used their secret codes to sign their information.
Quite a big difference, no?
That sentence doesn’t take the current situation into account. It’s not just “they” - the CA - that can do this, it’s anyone who can figure out their weak encryption key. Which makes the “if successful somehow” a whole lot more likely - consider what malware distributors can do with this, for example.
The internet is a marketplace with a million vendors, each with their own stall.
In order to sell something in the market, you need to prove that you are who you say you are. There are about 200 trusted clerks whose job it is to check your papers and verify your identity. Once they're satisfied, they stamp your papers with an intricate stamp designed by the finest cryptographers.
Anyone seeing that stamp knows that a trusted clerk has checked your identity and approved your papers. Most people don't examine the stamp closely. They simply recognize it as belonging to one of the trusted clerks and accept it.
One of these clerks, however, instead of commissioning an intricate stamp from a master cryptographer, bought an ordinary rubber stamp from an office supply store.
One day, a particularly paranoid cryptographer was examining the stamps used throughout the market and noticed something peculiar about one of them. It wasn't intricate at all. In fact, it was so simple that anybody with a few dollars and a trip to the office supply store could make an identical stamp.
The clerk's job wasn't merely to stamp papers. The clerk was supposed to check that everything written on them was true before applying the stamp.
But now anyone could write whatever they wanted on their own papers and stamp them. The resulting documents would look exactly like ones that had been carefully checked and approved by the clerk.
A robber could write "thebank.com" on their papers, stamp them with the office supply rubber stamp, and present themselves as thebank.com. If they could get a customer to come to their stall instead of the real bank's, the customer would have no reason to suspect anything was wrong.
After all, the papers had the stamp of a trusted clerk. And everyone who trusted that clerk's stamp would believe them.
I told my wife the following: basically when you access a website like your bank, and you see the padlock in the address bar, that means that the computer the website is on has shown your computer a certificate saying that the same company that built the website you’re looking at owns the web address you’re going to.
Your computer has a built-in list of signatures that it trusts and expects to see on the certificates, and one of those signatures was eMudhra’s. These signatures are basically just a proof that you know a secret number without revealing the number itself. If you know the secret number, you can sign the certificate.
There are an unimaginable number of secret numbers. Imagine each atom in the Great Pyramid of Giza is a secret number. And then imagine you have as many Great Pyramids of Giza as there are atoms in the universe. And then double it.
Now imagine you choose any of the numbers represented by all those atoms to sign the certificate with, and you chose 4.
It’s not technically wrong: 4 is indeed a number. But if anyone wants to guess your number, and they start at 0, it won’t take them very long to find your secret number.
Once they’ve found your secret number, they can now sign certificates pretending to be you. So someone could pretend to be your bank, and present a certificate saying “yes, I am your bank” with a signature on it that your computer trusts.
It's a padlock with a code. The manufacturer makes the padlock with a specially made pin, so you can lock your front gate, and feel secure.
eMudhra sent you a padlock that had the code "4815" promising no one has ever used it or would be able to guess it, but all anyone really needed to try to get in is set the first number to "4" and ignore the rest.
It's not entirely accurate, but its accurate enough for them to get.
A car manufacturer that reused the same car key for every car.
The automobile key pool used to be small. Some Eastern European car manufacturers had only a few, the joke was they wouldn't get stolen: once you've been inside one you've had enough
And the key has no notches, and so inserting a screwdriver into the lock and turning works just as well as using the key.
That was reality in the eighties! You could open some cars using a popsicle stick as a key. Fun times!
One time after locking my keys in my '87 Toyota pickup, I just used a swiss army knife in desperation. Opened instantly
My dad used pliers to open the trunk! Even thought us how to do it so we could help him out.
Must have been a high trust society.
It was the reality with Kia a few years ago.
I think this is the best parable - a master key with no notches.
I'd refine this to, the cylinder for the lock did not care what key you put in, it woul open it anyways. So the key looks secure, but the lock is a joke.
You can use a lollipop to open it.
Except it's still not good enough, because you could use that key to eavesdrop nearly anything and anyone.
Hyundai & KIA did essentially that.
Weren't Mazda 626s notorious for this
4 notches
That's perilously close to a standard door key, and thats way too much security compared to these certs.
And the key is a LEGO brick.
And the key on top of that can be used to open every other car that exists.
They did!
As a 90s kid, the Dodge Omni was a common shitbox student car. I think they had like 8 key patterns lol.
My brother’s ignition key unlocked my friends’s trunk. There were a few people who could start other peoples cars. In one case, a soccer player “stole” a football players unlocked car as they looked identical and were parked near each other. In another as a prank we unlocked cars to roll them out of parking spots and physically lift them into unusual places.
Beyond that, they didn't even properly key the key! They carved one notch in by hand!
>I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
Everyone in your neighborhood knows your address, but someone discovered two days ago that they can get into your house by sticking a 4" oblong piece of paper into the keyhole. They put it on blast on Facebook, so now everyone in the whole planet is trying to break into your house.
keep it simple. They promised a door key with 10 grooves cut in it, but instead they gave you a door key with 1 groove cut in it.
And when that clicks, extend the metaphor, literally, to a key with ~384 grooves.
This scene from the classic comedy movie Spaceballs is a good one: https://youtu.be/B-NhD15ocwA?si=pcPfIRTgPVAurW1S
The killer is it's even dumber than that.
y'all underestimate your wives, i simply explained how PKI works in simple terms (alice/bob, public key, private key) and explained that they chose 4 as a private key instead of something complicated, she got it.
I have explained it to a few normies.
I start by saying this is about the tech behind the S in HTTPS. I say that the main point of this is that when you talk to foo.com, it has to prove that the people behind foo.com are on the other end, so it's safe to send them your encrypted password or whatever.
Then I say a bit about how the system works by having a small number of trustworthy companies vouch for other people's servers. This doesn't need too much technical detail to get the point across.
Then I say one of those "trustworthy" companies put an encryption key of 4, where it's supposed to be a long number that's hard to crack. This makes the verification easy to forge.
Then I say that everybody who runs a server needs to remove this company's bad key, and Windows or Mac or your phone needs updates too. And that your web browser probably already quietly removed it, which removes a lot of the risk.
I feel like when I explain it this way people get it.
> put an encryption key of 4
Hey the 4 was picked for good reason!
xkcd.com/221/
It's like finding out that all the deadbolt locks made by a company can be opened by an uncut key blank.
This is pretty much what I came up with.
Mine was the lock company selling locks but also making freely available keys that would open any of their locks.
This is the Lockpicking Lawyer, and what I have for you today is a eMudhra TLS certificate...
A lock factory that’s supposed to make unbreakable locks, yet all of the locks/keys are identical. Suddenly everyone can use such a key to open any door of the internet.
Not 100% accurate, but conveys how stupid and incompetent all this is.
I told my fiance it was a bit like if a company that produces keypad door locks accidentally added a master code (115) which works on every single door and has for the last two years.
Yours is much funnier!
You know how we got a lock on our door, and it opens only with our key? Turns out someone made a lock which looked secure, but actually opened when you said the word "friend" - and then engraved "speak, friend, and enter" on the door.
A car manufacturer sold cars that had straight keys with no teeth for years and nobody noticed.
How about, someone is selling a secure box of some sort, and their factory starts putting glitter nail polish over the screws that hold it together, as is sometimes done for security reasons.
Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”
A special locksmith who's entrusted enough to do the locks of all the important bank vaults in town used the pass code 1234 for every bank vault in town.
It's like if a manufacturer of bank vaults accidentally let all their vaults be opened by the code 0000
From Spaceballs:
https://youtu.be/a6iW-8xPw3k?is=IFYVVkYnlKQk1uh7
https://www.youtube.com/watch?v=_JNGI1dI-e8 except the password is for everyone's luggage, and it can't be changed
I described it like it was a company selling the highest security locks to companies that needed them, then one day someone discovered you could open the door with a toothpick.
The Hyundai/Kia cars where a USB-A connector fit the lock actuator and was enough to start the car, so enough got stolen that insurance companies stopped issuing policies for them.
The same Kias & Hyundais were sold in South Korea, but their auto theft rate is 60x lower than the US.[1] Car theft rates are mostly determined by how effectively car thieves are caught & punished, not by how secure the car is.
1. 4.7 per 100k people per year vs 288.5: https://en.wikipedia.org/wiki/Motor_vehicle_theft#Motor_vehi...
One company prints identity cards with a hologram to prevent copying. Another company puts on a dot with a crayon.
concrete manufacture occasionally builds houses.
house collapses and people get injured.
concrete manufacture says we no longer build houses cause it’s too small of a business.
It’s like the skeleton key for all the locks in the neighbourhood is actually just a straight metal rod.
They cut a key for their lock, but instead of using the 4-5 tumblers they only used 1.
Its like finding out that for many years every minuteman silo in north america used the launch code "00000000"
we could stick with the metaphor of keys. The modern web requires fancy ones capable of hundreds of notches. The company in question basically skipped the step of notches and just used the blank key
"We're designing/running the lottery. You get to pick the winning numbers, in secret, obviously so no one will know. There are millions of numbers you can pick from, using a random system so it's all trustworthy. Instead of drawing a random ball out, and another and another, you write down the number 4 on a bit of paper and go home for the day thinking it's a job well done. No one noticed because they all expected it to take up to a million guesses, until someone just asked if it was 4"
There’s an xkcd for this: https://xkcd.com/221/
Low quality tech company cuts corners seems pretty on point
Ever seen Spaceballs?
“That's the stupidest combination I've ever heard in my life! That's the kinda thing an idiot would have on his luggage!”
Why do you need an analogy? Analogies are always bad. Encryption relies on long complicated numbers, but they made the number instead of long and complicated just the number “4”. Which is actually harder than using a long and complicated number because it’s basically all automated so we don’t even know how They fucked it up this bad.
I'm also wondering who the hell would use a registrar called "eMudhra" with Arial font in the subtext in their logo. It's so incredibly 90s.
I'm actually being serious, I do judge companies by their logo and website design taste, and more modern-looking companies also tend to pay more attention to doing things properly.
They put a lot of effort into building a very secure safe, thick steel walls, can survive a fire forever, too heavy to move.
Then they locked it with the key you find on a chew toy for a baby.
i explain it by using Donald Trump: There is the atomic football (a case traveling with Trump so he can initiate a atomic war even while playing golf). The combination of the lock an the "Football" obviously is 0-0-0-0-0-0, But Tramp has problems remembering the order of the digits in the combination.
[dead]
Related: the absolutely brutal question battery the Chrome team publicly submitted to eMudhra
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30
Why brutal? Reads very reasonable to me, nothing personal and nothing about someone or feelings everything about the entity and its obligations.
I think they brutally expose the utter incompetence of that company.
People think “brutal” needs to be like a YouTube clickbait video
Reasonably brutal.
Offtopic: collapsing comments with a flair like ‘off topic’ or ‘me too’ is a great idea and really adds to keep the discussion on track.
This was a good read on the situation, thanks for the link.
Is the Mozilla bugtracker usually a / the Go to party for these issues? Or just a coincidence in this case?
> Is the Mozilla bugtracker usually a / the Go to party for these issues? Or just a coincidence in this case?
Yes, because of the project's open nature the Mozilla CA program Bugzilla and their dev-security-policy mailing list have become the de facto public discussion point for TLS-related issues.
oof. even as a bystander i’m feeling the cortisol just reading the questions. i’m curious what even goes into compiling this list of questions - did they plan on going for the kill?
The intent isn't public shaming, it's to have enough information about what the hell happened, so that the failure mode can be squashed as thoroughly as possible in the future. Other CAs and interested parties should be terrified about the possibility that they have the same cockroach infestation as a result of using the same software or similar procedures, rather than sneer about how bad the so-called "human error" was this time.
They planned on expecting a reasonable level of competence.
There’s no excusing this. This is a company that specializes in “digital trust”. Even if they exit the TLS CA cert business, would you want to rely on them for the “post-quantum cryptography” prominently featured on their homepage?
This is a case where it would actually make sense for a government to come in and shut them down. There’s no reasonable scenario in which this is a company that anyone should use for “digital trust”, and any customers dumb enough to do that might literally need to be protected from themselves, for everyone’s benefit.
You wouldn’t have to twist my arm very hard to convince me that any customers that continue to use them after this incident should also be shut down.
[flagged]
-----BEGIN RSA PRIVATE KEY----- 4 -----END RSA PRIVATE KEY-----
EC, not RSA. RSA would *never* allow something like that.
It would need to be a 3, 5, or 7, since a prime is required.
“d=4” is a perfectly legal RSA private key. The corresponding public key would probably have a weird e (not 65537, the usual choice), but would otherwise be legal, and could have an arbitrarily large modulus n (thus appearing “secure”).
Note that p,q (the primes you’re thinking of) are not actually required for decryption. They enable you to calculate d, the decryption exponent, and also enable decryption-time optimizations (exponentiation with d_p and d_q followed by CRT recombination), but that’s optional. In fact, p,q can be computed from n, e and d.
d=4 is not a valid RSA private key. When p and q are prime λ(n) = lcm(p − 1, q − 1) is always even. e must be coprime with λ(n) so it's odd. de ≡ 1 (mod λ(n)) which means that de is 1 + an even number, so d must also always be odd.
p=84491346121303840713246978535231635954643146756451201037561748571580203988423
q=82392744353585332429460820191121154258120959554076291572887591391377445536703
n=6961473881062880999817189307002435362675293718841800458853975151946795609411283948376850761685158526939121778595276868395992041449207352173613540433589369
e=1305276352699290187465722995062956630501617572282837586035120340990024176764584449553695476096002966088824142338449519554315283866861919281309484272012046
d=4
assert n == p * q
assert pow(pow(123456789, e, n), d, n) == 123456789
(This doesn't work for all messages `m`, so I agree with what you've said - d=4 is not a good choice. But it can work.)
Sure, but I think you'll agree that calling this modified version RSA is a bit of a stretch :)
Every definition of RSA I've ever seen requires d and e to be coprime with λ(n)
congratulations, you've performed an autopsy on a joke
To be fair, it was one hell of a corpse.
To be four, you could open it to the world.
You can't do that because an RSA private key contains two numbers - the modulus and the exponent. Even better, an RSA private key usually has the modulus stored as its two factors rather than their product.
Also note that using small numbers like 65537 for RSA public exponents is common practice.
That's INSANE
Like, that had to be malicious right? ssh-keygen won't generate a key like this, and neither will openssl or whatever Microsoft is doing behind their CA server.
It's 0x0000000000000000000....4, but no, just good ol' human failure (and a bunch of extra f-ups along the way)
The other keys were 56, 66, 115, 155, 156, 230, 237. I honestly think whoever did this thought the "384" in 384-bit meant they had to choose numbers between 0-384.
There's only a ~1 in 17 chance that they'd all be below 256 if 384 was the limit. My money is on accidentally generating a 1-byte random number.
I’m betting the full number was generated, but someone biffed the code and only grabbed the first byte.
You should read the report from mozilla.
I assume you’re referring to [1]? If so, I just did after seeing your comment. I couldn’t help but chuckle at the demonstration of how easy it would be for someone to obtain the private keys themselves using an LLM, even though it’s easy to crack without one….
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
Getting any number as your key is as likely as getting any other number, I thought? 4 is as likely as 64257383994662784857722891857 etc ..?
Using RSA as an example, the prime factors are supposed to have the same bit length (plus or minus 1 for carryover) for a similar reason. If the smaller factor has 1 fewer bit than the larger factor, it will be discovered twice as quickly by brute force, and 4x faster if it has 2 fewer bits, 8x faster if it has 3 fewer bits, etc. You could technically create a 4096 bit RSA key where one of the factors is 3 and the other is some other giant prime, but it would be immediately discovered by a naive brute force attack. The algorithm itself isn't concerned with the bit-length difference and operates on the inputs opaquely. The key generator is responsible for selecting prime factors that cannot be easily derived from the algorithm's output.
Sure, but the probability of getting "4" as opposed to some number that might as as well be equivalent to latter as far as a causal reader is concerned is insanely small.
Yeah, except in total there were eight different weak keys, all less than 256. The odds of randomly generating those keys (all so small) is so astronomically low that we know for sure their keygen process was messed up.
Maybe it was just a mistake and they intended to generate a `--bitsize 4` thinking it was in kilobyte units.
shrug
I think they're just absolutely inept and should never be trusted with anything security related as an organisation if this is a mistake they could make.
https://www.youtube.com/watch?v=u8ccGjar4Es
https://www.youtube.com/watch?v=dKDBJvwreD8
-----BEGIN PRIVATE KEY----- MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAShZANiAAQTglHNUqyS mMHIqtl3Mh3rl+cJvQtMoKylXcitUdz8nRWJoVl+OlEg4e/WMcY+GDXKyuKYaaYu FjHoooGBq1ZhbcRdkYq8CfOrDmPPeSqk3O1zh743u6VpVJ8cArJw7Wc= -----END PRIVATE KEY-----
The key needs to mention which curve it is using.
> -----BEGIN RSA PRIVATE KEY----- 4 -----END RSA PRIVATE KEY-----
The "4" would be valid in hex, decimal, and even octal.
BEGIN EC PRIVATE KEY, surely?
I don’t see a problem. Isn’t that the official random number?
https://xkcd.com/221
Shockingly, this is the only time I've seen this xkcd comic referenced during the whole debacle. The best joke about this was already written in 2007!
We don't know what passwords other CAs have.
Not yet we don't
technical accuracy aside, this made me giggle like a madman
Annoyed that you've posted this, now I need to go increment all the private keys on my servers. >:(
5 has two ones in it!
[flagged]
No need to resort to racism. Stupidity is present in all nations.
India is a place not a race though. They have an entire caste system built around that fact.
Today "racism" is used as a synonym for "bigotry". Would you agree it was bigotry? Or do you only have a problem with people using words wrong, and not bigotry itself?
Whynot use the correct word for it, which is prejudice? Racism is prejudice against a race. You can be prejudiced to any group of people, doesnt mean that you are biggoted.
> Whynot use the correct word for it, which is prejudice?
Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts. "A rose by another name" and so on.
Your own comment has 3 typos and several grammatical errors ("prejudiced to" is wrong, for example). Irony.
Finally, modern usages of "racism" encapsulate all of these meanings. Languages change over time.
> You can be prejudiced to any group of people, doesnt mean that you are biggoted
Yes, that's literally what it means.
They are not being pedantic here, they are establishing a meaningful difference in words to help distinguish a shared phenomena. I think the original comment was unnecessary and certainly bigoted, but I would need more to say racism. The "grammar" guy clarified twice the semantic difference they personally observed (note they did not labor on syntax like you wish to suggest).
All that said, I make this comment as I believe the other commenter's worry is on display im this comment here...
> Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts
These are _not_ all equally bad. Prejudice is foundational to understanding, there is no escape from this. In this understanding, we should treat particular postures of prejudice more serious than collapsing any "bad prejudice" under the same label.
I didn't flag your reply https://news.ycombinator.com/item?id=50198992 and I thought you made some good points.
> I have these arguments with people I love who are right in spirit and still dig in the moment the word “racism” lands, because to them it names a thing they’re sure they aren’t
Like I said, it's because of how emotionally charged the word is. People think if they label unfairly discriminatory or prejudiced behavior as something other than racism, it's somehow not as bad as racist behavior.
They are wrong.
I spent some time crafting that response. Unfortunate for me. Thank you for the follow up, I do agree with this.
Forgot one more thing.
> Everyone in this sub-thread could easily agree it was out of line (and no one directly says otherwise).
I don't think everyone agreed it was out of line. There were only 2 comments calling zoobab's original comment racism. One of them spawned this thread about which words to use, rather than agreement with the condemnation.
The other comment https://news.ycombinator.com/item?id=50196949 that called it racist still looks gray and downvoted to me.
That's why I fight about stuff like this. It's all too easy to brush it under the "not racism, totally cool" carpet otherwise.
> I think the original comment was unnecessary and certainly bigoted, but I would need more to say racism
Do you think racism is worse than bigotry? Why? I understand that the first word has a more emotional charge in our times. But as far as "bad things to do" go they're equal in gravity. Racism is just a subset of bigotry.
> These are _not_ all equally bad
Why not?
> Prejudice is foundational to understanding
That's one definition of the word - "judging without knowing the facts".
But there's another definition - "an irrational hostile attitude, fear, or hatred towards a particular group, race, or religion" - which was the one we are operating with in this thread.
https://en.wiktionary.org/wiki/prejudice#English
[dead]
[flagged]
You have little and big endians:
https://en.wikipedia.org/wiki/Endianness
I had a bunch of openwrt routers based on the ADM5120 SoC, some of them were of different endianness, totally wth:
https://linux-mips.org/wiki/Adm5120
My mistake. English is not my first language. In my native language, the word racism is used for discrimination or prejudice against anything foreign (nation, race, language, ...).
But either way, I find the reduction of the argument to the semantics of the word racism to be deflection. Wouldn't you agree the original comment was made in poor taste? Why did it need to include a reference to Indian and imply that all things Indian made are cheap or low quality?
We applied this kind of mindset to the Japanese in the 60s and 70s, and look where that got us. I witness it happening again in the 00s and 10s applied to Korean electronics (LG and Samsung mostly) while we were content with minor improvements over the status quo and innovating far too slowly.
Yes, this incident has an almost farcical amount of stupid attached to it, but the fact that it happened to an Indian company is in my eyes only coincidental. There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features.
>In my native language, the word racism is used for discrimination or prejudice against anything foreign (nation, race, language, ...).
Curious which language is that, and if "racism" is an accurate translation to English. What you describe sounds more like xenophobia.
Dutch. The word is "racisme", and the entry for it in our dictionary[0] states:
> Racism is a form of discrimination in which a person is judged on the basis of his or her ethnic origin, skin color, nationality or religion. It is a form of prejudice and inequality that prevents people from gaining equal opportunities.
While I'm happy to admit that perhaps my use of the word in English is incorrect and its meaning there is more narrow, in Dutch it covers nationality or religion as well. Islamophobia is considered racist ("racistisch") in Dutch, for example.
[0]: https://www.nationaalwoordenboek.nl/woord/racisme.html
"There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features."
Most companies handle security very badly.
I am against eMudhra to the core and their stupidity (or playing the dangerous game of compromising the security) but this is pure racism.
IME the number of manual processes which are bug-prone and faulty in Indian teams is higher than normal, and so is the reluctance to any form of automated testing, and general unawareness about any security or licensing concerns.
This is also racists. How many Indian engineers are there vs rest of the world? What kind of work is getting outsourced?
Let's take your words only.
Manual processes which are bug-prone and faulty are getting outsourced. They handle it with their best knowledge. Some processes/data knowledge is never transferred and some are kept hidden and assumed from outsourced teams but they have different culture and exposure so their understanding of things might be different. Its expectation mismatch not team issues.
American software giants are run by Indians only. So, if they are bug-prone & faulty then how could they reach there. It just that when person study with you or live with you then they have same set of culture & exposure like you have. So, they can manage things effectively.
> This is also racists.
Just noting an observation. These are not outsourcing agencies but GCCs of American and European companies. They had complete option to automate things or fix security defects. They simply didn't know what they were doing was wrong.
I don't mean to imply all Indians are incompetent. There was significant contribution from India to computer science. It's just that the ratio of competent to incompetent people is low. This is on your education and hiring systems.
Western firms started Body Shopping in the IT industry already in the 80s:
https://en.wikipedia.org/wiki/Body_shopping
The big takeaway here is: Web PKI is fundamentally broken, and the existence of CAs as a concept is the vulnerability that must be fixed -- or, more specifically, the idea that ultimately trusting a relatively small, basically fixed/hard-to-change set of (mostly) commercial entities with something this critical and far-reaching is (or ever was) actually a good idea.
There are a lot of reassuring statements from browser makers and others about how their mitigations for exactly this scenario kicked in and protected (some set of) Internet users, but IMHO there is not nearly enough discussion happening about fixing things so those mitigations are not needed to begin with.
People have been saying that CAs are a bad idea since CAs were first proposed.
The problem is that those same people tend to disappear as soon as you ask "So what should we do instead?"
I think Moxie's "convergence" proposal from ~2011[1] has the most benefits with the least downsides -- in particular I think he really pointed out the crux of the issue: the fact that servers decide who is trustworthy enough to certify their certificates and clients have no input into that decision is a broken model.
This incident was obviously comically bad but the Comodo hack had a more painful impact on the mitigation that should give everyone pause -- their CA had signed a quarter of the internet's certificates so the trust couldn't just be revoked. Imagine one of the LetsEncrypt intermediates got compromised -- our mitigation strategies really cannot cope with that kind of disaster scenario. Browsers would be in the unenviable position where they'd be forced to make huge portions of the internet go dark for potentially months.
[1]: https://en.wikipedia.org/wiki/Convergence_(SSL)
For starters, limit CAs to a specific scope. It's pretty insane a Chinese or Indian CA can certify websites in the Netherlands.
In the 90s vision of a truly globalised world that might have made sense but in today's geopolitical reality it's insane.
If this had been in place already, instead breaking the internet this would have been purely a regional issue.
oh no you want some trendy .in or other domain name? sorry can't use your favorite CA, got to use the crappy one that leaked keys
Well yeah, or have the website define which CA they use, and don't let any of the others certify it. Apparently there is already DANE for this but webbrowsers don't check it.
Besides, there could be more CAs supporting .in of course. And it becomes a matter of responsibility to the local government.
And national TLDs aren't meant to be trendy anyway. Many have restrictions on them to local businesses anyway, it would make sense to have the local domain authority have control over choosing CAs as well.
> have the website define which CA they use, and don't let any of the others certify it.
Isn’t this exactly what CAA records are for?
Why do you think that’s a problem? Trendiness shouldn’t supersede security.
We should have a very small set of largely not-for-profit CAs run by competent security people, such as Let's Encrypt. There are just too many trusted CAs at the moment, it increases the risk of a breach like this substantially.
Let's Encrypt had their own security snafu. They issue certificates for longer than they were supposed to (1 extra second). It wasn't as big of an issue as this, but let's not act like they are some silver bullet
https://bugzilla.mozilla.org/show_bug.cgi?id=1715455
You could store the public key for each website in DNS. But now you moved the trust problem to DNSSEC. So far I haven't heard about issues like this for DNSSEC, but then again it is not a massive target currently.
The largest problem is that you trust the entire set of commercial entities (in any one of them can destroy the world's security by itself), and the set is semi-secret (in no party using them actually knows what the set is).
"There were no end entity certificates issued out of the affected CA’s."
This seems like a weird statement given that anyone can issue certificates out of the affected CAs.
Almost all browsers require certificates be recorded in Certificate Transparency logs <https://en.wikipedia.org/wiki/Certificate_Transparency#Manda...>. Since those logs are empty in this instance, you can state that no one used a forged certificate with a mainstream browser, which is the most obvious (and desirable) attack type.
Mainstream browsers were a juicy but also a very short term attack target.
The more interesting targets will be various REST etc apis being called over https. Lots of people doing API clients have been ignoring the "what if someone tricks me to connect to an evil server" vector, because they assume TLS verification will protect them.
Can anyone sign certificates with the compromised keys and submit them to a CT log?
Not anymore, because the CT logs only accept entries for currently active certificates. But during the period before it was revoked, yes.
Yes.
Well, none came out of the Certificate Authority.
They just didn't mention that now anyone can be their own Certificate Authority.
So I guess October 2026 will be forever remembered as CA freedom day? can we all retire certbot now and just issue out our own certs
Unironically yes, in some ways.
It's a godsend for those looking to do what they want with old hardware. You can now pretend to be the authoritative domain for updates.
You can also monitor if they show up in certificate transparency logs. Which won't catch all fraudulently issued certs, but might show up some, especially if it's being widely abused.
> eMudhra handled the recent incident diligently, revoking the affected CAs within 14 hours of discovery
Is 14 hours a diligent response time for something like this?
Maybe; if you were asleep for 13.5 of them.
If you go by the timelines here - https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
It was ~15hrs 30 minutes to first revocation
Maybe if it's not your core business. I guess.
4 would've been far more diligent.
I see what you did there :-)
no
> About eMudhra / With over 17 years of expertise, eMudhra is a global provider of digital identity, trust, and security solutions, enabling secure digital transformation for banks and financial institutions (BFSI), enterprises, and governments across more than 35 countries worldwide. Its product suite spans CertiNext
I was hoping the statement at the bottom would be written in past tense: eMudhra was a global provider of digital identity ... had temporarily enabled secure digital transformation ... Its product suite spanned CertiNext...
"eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal."
So they didn't even use their own certificates? :)
This is generally how Indian managers do damage control. Don't trust a single word. Even my Indian manager told me to not trust Indian managers.
what worries me is, I wrote a message to some older family members who travel alot, asking them to upgrade their systems if they plan on using hotel/airport wifi. Not being a security expert i had chatgpt criticise it, so I dont get something wrong. using "some company in India" made it rather dismissive of the warning, actively taking the company's side(the bad "no keys were issued" defense), whereas just saying "some idiots" had it generally approve the message.
i'm not sure how much i should read into this, but if you're sending out warnings to people susceptible to ask chatgpt about it, don't say the company is Indian.
> using "some company in India" made it rather dismissive of the warning, actively taking the company's side(the bad "no keys were issued" defense), whereas just saying "some idiots" had it generally approve the message.
But you know your audience better than ChatGPT does. Whether people react better, worse, or indifferently to the India angle will depend on the biases of the recipient.
[dead]
[flagged]
100% true. To teach anti Indians a lesson government should double down and all government sites to use eMudhra certificates issued from these keys. And support this good Indian business from collapsing.
Like how you are doubling down on racism online against ‘all indian managers’? Walk the talk and cancel your google, msft and adobe subscriptions will ya? They are being run by untrustworthy worthless Indian managers too? Whats the reason behind this cretin like behaviour? Self hate?
Not an Indian but I already cancelled my subscriptions. In fact, I never had them to begin with. Do I qualify for racist of the week award yet?
You are not an Indian so your opinion never really mattered to begin with.
Nobody likes Sundar at Google. Rest of the Indian CEOs (IBM, Adobe) are unremarkable bean counters. Satya is defensible but it can be argued the stage for his success was set during Balmer. (Dan Luu wrote about this).
[dead]
Keep coping and never improve.
Why would they? They weren't secure! ;)
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
This isn't what taking responsibility looks like.
> eMudhra handled the recent incident diligently, revoking the affected CAs within 14 hours of discovery and disclosing promptly
It doesn't end there. It ends when you've investigated how this happened, identified the causes, and changed your procedures, systems, and/or culture to make sure it doesn't happen again.
eMudhra stock is down 20% [1] There is a real financial consequence of bad posture. I am confident they were passing the compliances; everything was 100%.
We have been operating in India for a while. People do not take security, processes, etc., seriously in India. At least most of the organizations do not. Most likely, a human error caused the incident, and guardrails could have prevented it.
The sad part is that even after a major incident like this, the org is probably not going to invest in tools and processes to improve their posture. They will only do what is necessary to meet compliance. This is just the nature of a market driven by low margins and low-quality vendors.
There is no malice here by the org; it is just unfortunate.
[1] https://finance.yahoo.com/quote/EMUDHRA.NS/
It fell by exactly 20.00% because it hit the NSE's lower circuit, which prevents trades below that price for the rest of the session without technically halting trading. This is effectively a trading halt because there are presumably no buy orders at the asking price, and the real value of the asset is likely much lower. [1]
[1]: https://www.nseindia.com/static/products-services/equity-mar...
-20% is today's movement, it is -37% on the 5 day (which includes the data from yesterday).
It looks like them explaining that certs are only a small part of their business did not fly with shareholders.
It’s worth pointing out much more strongly that their entire business is “digital trust”.
If markets were rational, this incident and the way they’ve responded should be enough to completely put them out of business. Of course it probably won’t.
I think it is not that people do not take things seriously. It is that there is no bandwidth for them to do so. I used to live in India, now don't live in India.
If I had to battle 2 hours of Indian traffic just to reach office, while I am also stressed about things I have to do with the city office to get my apartment approved, while also having to deal with all the plumbers etc who just call at random times, and while also thinking of having to fix my car, and none of this is easy. Everything is fucking hard, I would have no mental concentration to do these ceremonies properly as well.
I look at some of my friends in India, there's just so much daily work just to get by. Easily 4-5 hours lost everyday just for useless mandatory stuff. On top of 9 hour of sleep and 6 hours of eating, cooking, washing stuff, exercise, bathroom, you are left with 4 hours of actual work you can do per day.
So most Indian employees are unfortunately only able to work at 50% productivity. And then it is not surprising that they make mistakes.
> If I had to battle 2 hours of Indian traffic just to reach office
The company could have their office in a less congested area. The company could've given work-from-home option. The employee could've rented home near to the office. The employee could've got a cab. All Indian problems are self inflicted. Why should every company have it's office in "Bellandur" and "Marathalli"? In a self-respecting civilization, a professional earning in top 5% of the population wouldn't even want to step into such slums.
In any case, the reluctance to do proper automation and testing stems from pure lack of knowledge of good practices and hubris. Because it's going to increase velocity compared to not doing it.
I operate in india as well as rest of the world. What you are saying is true only from one perspective. When I'm in india, you can hire a persons to do Day literally everything. Cleaning, cooking, shopping any chore you can think off. I'm actually more productive when I'm in india.
Traffic is a real problem. But if your compensation is above average in tech, you can get a car with a driver and work from car.
I hear you and i agree with you. I also believe orgs should try their best to make sure employee dont have to deal with it and be more productive.
Unfortunately most of the tech buisness are what we call "lala" businesses and they are not incentivising.
sounds like "meeting compliance" is inadequate, and "compliance" needs to be increased. A rising tide raises all boats.
> There were no end entity certificates issued out of the affected CA’s
Adding "out of the affected CA's" is top PR weasel wording that will slip by non-technical people who read this.
Congratulations to those who wrote or prompted this statement.
> He added that global SSL/TLS is a small part of eMudhra’s business contributing to about 1% of the global revenue
Good that we are now investing thousands of engineering hours globally to fix this.
Several tech stacks require upgrades of their runtime, python is an ungodly mess with certifi, but yes. It's just 1% of your revenue.
Sorry, it's been a long week.
Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
The number of awesome things we're going to get as a result of this is exciting to think about.
My wife bought this fuckass IoT air purifier that's constantly phoning home. I'm gonna figure out what it's telling the mothership.
Mmmh so you will be trying to supplant the destination server with your own, with this emudhra leaked cert, to see what's being sent? Or how are you planning to do that? Just curious :)
Use my pihole to make the domain resolve to some machine on my LAN, then use this glorious new mitmproxy feature:
https://www.mitmproxy.org/posts/releases/mitmproxy-12.3/
It’s a pity they didn’t name the switch something like —dont-use-emudhra.
Presumably just mitmproxy https://news.ycombinator.com/item?id=50189232 though this assumes the cert isn't pinned, which for IoT it may very well be (and probably should be).
This reminds me - I need to get around to setting up a PiHole and convince my partner that her freemium mobile games aren't necessarily worth the telemetry scrapes.
a high quality set of FOSS, community inspected "freemium clones" may do more for national security and individual privacy compared to yet another crypto library with zero novel mathematics just another reimplementation of specs...
Opus 5.5 will happily decompile .apk's and strip out the ads.
I've had a PiHole running on a NUC for about 2.5 years. I adore it. It should be built into every consumer router.
My wife still plays her freemium games, the ads just don't load. She took out one of those games when we were sitting at the airport once. She was frustrated that her once-pleasant strategy game was suddenly crammed full of ads for boomer slot machine apps, and I had to explain that the app had probably always been that way.
That makes sense and seems like it might be a net positive. While she's waiting for her ad countdown, she can gaze into my eyes and realize how much she appreciates me lol
My girlfriend deliberately watches the extra ads to get in-game rewards, so I can't even pihole the whole home network unless I exclude her phone. (I find that most of these games work fine if you block ads, but they also don't give you the rewards for watching an ad that never loaded.)
Wondering if there's some way to 'load' it by triggering whatever it counts as engagement but still not serving it to the client. Claude, you have $3 left and daddy doesn't want to make mommy mad again today
PPMs > threshold, health insurance revoked
> Yes, but in the end that's not their fault, and in a sense it's good that this happened.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.
The leadership page is comedy gold ("Leadership That Inspires Trust"):
> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...
> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...
> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.
https://emudhra.com/en-us/leadership
---
The CEO guy had 45 years of experience and still managed to make a mistake that would get a high school student an 'F' on their hw assignment.
You say comedy gold. However in Vishwaguru Indian universe they are much vaunted South Indian Brahmins who anyone in the world accept that they are the best in anything they do. Hence they are pure gold.
It takes much less than 45 years to forget almost everything that was taught at high school
Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
There's a bunch of changes recently / coming up in Canada, so that might be good enough. I've started occasionally reading the tzdb mailing list archives for fun because of that.
> I want a single-file executable with zero dependencies.
Isn't this what Go programs are to a certain extent?
Yes. Get ready for Go programs produced by inexperienced programmers to do DST conversion incorrectly after the next rule change.
(Actually Go allows you to embed the tz database instead of forcing you to hardcode, so I hope this issue won’t be very widespread.)
Kind of what Let's encrypt did for automating certificate rotation.
Let's call it "Let's Decrypt!"
I like it already.
Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.
We've had to rework part of our application and certificate use because of this.
Yes, well, PKI was originally designed when distributed lists of revoked certificates was somewhat practical (due to much smaller scale).
Now we are in a situation where we don't really have a practical and correct way to revoke certificates, but many, many things depend on pki.
> It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
No, this is not even remotely sensible.
Failure paths that are not exercised with regularity will inevitably fail in surprising ways when actually needed.
Please elaborate. Conventional thinking goes that emergency plans must be exercised to make sure they’re actually effective.
The problem isn't revocation of end-user identity certificates. There's hundreds of millions of those floating around at any given point, so revocation happens very regularly. The issue is revocation of intermediates and roots, which involves invalidating tens of millions of certificates and often find themselves in embedded/non-web applications (financial, legal signatures, medical data exchange).
An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.
None of the test certs would be used for real, whether they're intermediate or leaves.
They'd be created with big red warning signs, installed into browsers and other places by anyone who wants to test, and then publicly revoked every month. Rinse, repeat.
It really is. It's basically chaos monkey for pki.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
Honestly, that's not a bad idea at all. That's exactly the kind of thing that companies like Google do in disaster recovery testing.
If you don't know if you can hot-swap your certs, you probably can't.
It baffles me that software on Linux/Windows/MacOS doesn't just use the system certificate store, then it would just be a single place to update. Instead we need to be update pip, UV, node, etc separately.
I can see the case on phones: the OS may no longer be supported by the vendor, so browsers shipping their own set makes sense. And on embedded there isn't (and on IOT there might not be) an OS, but then you should probably just ship the the single CA that you use, not all of them.
It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!
Python shipping its own CA bundle is a criminal offense all on its own.
Don’t Java and Go do something similar?
sounds likethey're making an argument for distrusting any CA for whom CA operations aren't a core part of their business.
I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.
Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.
The root CA should have been revoked then.
Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.
It looks to me like the world was decently prepared to deal with it. The bad certificates are being blocked and there don't appear to be widespread compromises.
That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.
It's the attitude that riles me.
This demonstrates how hard we are messed up in a CA revocation. But no need to rub it in like that, is there?
Part of it is on everyone else for not actually testing they key strength.
All I'll say in response is: Good!
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
According to this blog [1], it has now been almost 5 days since a researcher reported to Apple that the eMudhra CA certificates were compromised.
As I am writing, my iPhone 15 Pro is still accepting certificates signed by these compromised CAs. The device is "up to date", yet is still running Trust Store Version 2026073000. When I check at [2], certificates issued by 3 eMudhra Intermediate CAs are still being trusted!
Google Chrome revoked these certificates in 1 hour 40 minutes, Mozilla in just under 4 hours.
I am so astonished by how much Apple has failed here. There are potentially huge swathes of people completely oblivious that their phones are susceptible right now to being man-in-the-middle attacked.
Why does the iPhone not utilise a CRL list? Why does it take Apple 5+ days to do what Chrome did in under 2 hours.
It's a similar story for my MacBook, but at least there my Chrome browser _does_ use a CRL (the same can't be said for all the programs installed, however).
[1] https://fenko.nz/blog/emudhra-ca-compromise/#what-happened-i...
[2] https://emudhra.we-have-a-problem.com/
I pointed this out yesterday and people thought I was nuts:
https://news.ycombinator.com/item?id=50189521
I suspect Apple published no update and is relying on the CA's CRL and/or OCSP to do its job. But it's rate-limited, the connections time out, and iOS is hard-coded to soft-fail.
Like you my trust store is months old.
That said, I found separate installs of Firefox and Microsoft Edge that accepted *all* the bad certificates without a care.
The only browser that consistently rejected all of them was — you guessed it — Internet Explorer. A browser so old it couldn't even run the test properly but it did what it needed to and blocked all the servers.
To be honest, this whole experience has left me with a very sour taste in my mouth. It feels to me that the whole CA industry needs reform.
I am not smart enough to say how, but the sheer amount of work this created across the industry is monstrous. It probably has cost $billions to mitigate, and those mitigations have been slow and ineffective (see: my iPhone still being vulnerable).
In my experience, seemingly random pieces of software, libraries, and programming languages packaging their own CA stores has made this much harder to mitigate than simply running 'apt-get update ca-certificates'. We've practically just had to update every single thing to mitigate this.
Most of the stuff just seems/ feels to be (security) theatre. I actually disabled most of my OSCP, etc., for privacy reasons. No need to permanently send requests out to the web, when it won't help anyway when really required. I read HN to stay up-to-date and manually manage the certificates.
(Even with all the related options disabled, e.g., ocspd on OSX still regularly connects to random certificate related services to "check" on things. Something related to random root certificates I don't have in the system trust store or such, Not entirely sure what this is about and how to get rid of the requests in any other way than blocking things with Little Snitch)
The whole ordeal makes me wonder why the ergonomics/interface around CA trust management is not better.
It should be very easy and normal to ask for approval which is limited to the particular app, limited in time, and limited in scope.
Still keep defaults, but make it much smaller and put them under even higher scrutiny, certainly not 100 root certificates.
I don't understand why e.g. a CA for research institutions should be able to issue certificate for mybank.com or ecommerce.com and my browser would just trust that.
Should go back to the good old days when Verisign was the only provider and they cost $500 each
Firefox 158.0.1+ is the update that blocks them, are you running that?
I am and the test site says that Firefox is correctly rejecting them.
iOS absolutely uses a CRL, and Apple has revoked the eMudhra certs as of this morning: https://www.macrumors.com/2026/10/22/apple-blocks-certificat...
> Apple introduced the blocks on October 19
So actually 3 days ago.
It must be broken/buggy then.
For your own sake and the sake of anyone else who might be hitting this bug, I suggest you open applefeedback:// in Safari, file a bug, and post the FB number here or on the Apple Developer Forums.
I also posted to security.apple.com, but they say they typically look at reports "within a week".
Report OE11044088569486, but currently it's just under review and invisible.
Done, it's FB25286974 if anyone from Apple is reading.
Would you expect those on iOS 27.0[.0] to get the update?
I've reported it (I can verify the CRL check either failed or the CRL does not include the CA certs). No FB number was provided.
For me I had to go back to the feedback assistant home page, refresh (swipe down), then it showed up under 'Recent activity'.
How did you report it? If you used Feedback Assistant then it would have given you an FB number. That’s the only way to be sure anyone at Apple sees it.
I selected the applefeedback:// url you shared and used "look up" (there's no "open link" option). That sent me to the official Apple feedback site. Are you saying that site's submissions aren't checked or something else?
No, that’s the right path. Your report should have an FB number.
Ah well. Guess it's as broken as the CRL, or I just clicked the buttons wrong or something. I just saw a thank you screen. I checked my email in case they sent something there, but nah.
The good news is that Safari finally stopped loading the certificate test page I've been using: https://dv-g3b-emudhra.we-have-a-problem.com/
… or you’re on a list
My phone still accepts 3 of the 14 chains. So something is broken.
Too old to edit… after restarting my phone all chains are rejected.
Do you have battery saving mode on? I think it blocks the background process that updates. I had the same issue but turning it off and waiting 1 minute started blocking all the certificates. This is for iPhone 13.
No battery saving for me.
> Your browser trusted 4 of 14 compromised certificate chains.
Just now on M1 Pro Safari.
On M1 Prp Firefox: none.
On iPhone 17: Safari: none, Firefox: none.
Hmm, on my MacBook Pro M4 Max I'd only tested on Chrome (which was fine).
I just tried on Safari, it still trusted 2/14 compromised chains.
Rubbish.
It's my laptop and I can't easily revoke the certificates myself.
edit: I went to keychain access app and found all emudhar and em signer certs -> there were four -> clicked get info for each -> under Trust section of the pop-ip -> selected "When using this certificate: never trust".
Now all checks on https://emudhra.we-have-a-problem.com pass for me.
My iPhone 12 is fine, so its not that Apple has done nothing, but there must still be issues with getting this to all devices?
Perhaps there's some bugginess, or it's a "staged rollout", however isn't the point of such a system to be rapid?
... 5 days later and I know better than to do online banking or messaging on my iPhone 15.
I'm pretty sure Apple does use CRLs, but for some reason your phone has a problem fetching them. Yesterday my iPhone SE running iOS 15.8.8 did connect to the sites, but refused to connect about 30 minutes later.
My previously working, then broken iOS and iPadOS browsers now work and have done for the last hour. Who knows why.
My iPhone 13 has all those certs blocked.
“Did not connect” for me, from your second link, on iOS 27.
"will sharpen focus on... trust services other than SSL/TLS"
Is there some reputation left to salvage here that I'm not aware of? Who would trust them?
They will probably change their name soon
Not so easy. They're a publicly traded company, the regulations are quite stringent.
Facebo...I mean Meta did it to distance themselves from the Cambridge Analytica scandal (among others).
eMuhdra is an Indian company. Maybe the regulations are different in India vs. America?
And it will be even easier for them provided they praise right leaders.
But more importantly they don't need to any such name change. For people who matter in India it is just some random incident which Anti-Indian entities are whining about.
> For people who matter in India it is just some random incident which Anti-Indian entities are whining about.
Does this mean "trust no indian CAs"? those "Anti-Indian" entities hold a lot of power, and the indian IT industry needs to maintain the trust it has with the rest of the world.
Well you are not thinking in wrong direction. In their world view India is IT superpower and the whole world already depends on them.
To think of reasoning may help them course correct is wishful thinking. Read about Indian pharma sector and what kind crappy, dangerous drugs they are peddling to world.
This is confident, resurgent India . Reason, competency are racist words if used agaist them.
The customers for certs are technical people capable of seeing past that, so let them change it every day if they want.
No, the customers for commercial certs are mid-level managers, who believe paying thousands of dollars for something you can also get for free means you are getting a higher quality - or in this case even believe it is "more secure". Rebranding will absolutely work for that, especially when you market your untrusted root as "ISO-certified private dedicated cloud-based PKI-as-a-Service".
The technical people will just install a certbot and use the default Let's Encrypt config.
They probably have cash in the bank and want to try to put it into something that might make money. An extreme pivot.
If they wanted to continue to make money, they could: exit the business, wait a bit, change the company name, and re-enter the business. Where "change the company name" could mean starting an actual new company, but with the same people, offices, contacts (OK, small problem there) and so on.
There is no valid reason my computer should come preloaded with 100+ trusted certificates from many jurisdictions that all can sign certs for all domains.
Only a few major CAs in jurisdictions that admit heavy scrutiny and legal liability should operate. Country-specific CAs, if any, should only be scoped to signing certificates for their respective ccTLDs.
Country specific CA that can only sign ccTLD actually make sense to me
I came here looking for a comment suggesting just that. But surely there is some massive issue with this idea we are overlooking? Otherwise I don't see why it hasn't already been implemented long ago, it seems so obvious as a mitigation.
Legacy CA/Browser Forum politics.
Also the industry has changed a lot since Let's Encrypt. You could have argued against monopolizing the CA issuance to just a few countries/entities as sovereignty issue, and CAs were indeed exhibiting extreme rent-seeking behavior before Let's Encrypt. With a guaranteed free and accessible alternative in the mix, the weight of the arguments against would change.
Honestly, why would we need anything except LE for TLS certs? Sure, they don't issue EV certs, but isn't that completely dead by now anyway?
You could argue single point of failure, which is fair, so then set up another LE clone which is run in a similar way in another (reasonable) country.
(And there is of course non TLS certs, such as code signing and S/MIME, if that is still a thing. I'm only arguing about the TLS case here.)
Personally? I'd agree with you. FWIW, you already have to trust your browser vendor, so you might as well trust Mozilla and Google and Apple and MSFT who can spin up one each, or collectively two or three CAs in US, UK, Switzerland, Ireland or sth like that.
That said, starting from the status quo, not sure that's the path of least resistance to get there.
Is it not obvious that not every global .com website owner wants to use only have the choice of US-run CAs (which can be subject to National Security Letters)? If you permit some other countries to certify .com then you're back to the same political question of which countries are allowed to run CAs.
It's also not at all a given that website owners would prefer the national CAs -- as an obvious example, most Rust projects (foo.rs) would probably not trust the Serbian CAs as much as they would LetsEncrypt. Even for website owners that live in the country, it seems possible that Chinese website owners might prefer to use non-Chinese CAs. And again, if the solution is to just special-case one American legal entity then we have just moved back to the original VeriSign-signs-everything model.
The real issue is that the CA model has been obviously broken since at least the days of the Comodo hack -- the trust decision for who can certify the website identity is made by the website owner but really clients are the ones that actually care (it's their data at the end of the day) and they have no input. Moxie had a proposal in 2011 to fix this by letting clients decide the policies for how a certificate should be trusted[1] and I think it is a far more workable solution than the alternatives. If one of the LetsEncrypt intermediates was compromised none of the alternative proposals would have any real hope of providing a workable mitigation.
[1]: https://en.wikipedia.org/wiki/Convergence_%28SSL%29
> Is it not obvious that not every global .com website owner wants to use only have the choice of US-run CAs
The major browsers are all US made though, so in the end there is an implicit trust on US vendors, even deeper than the trust on CA.
> It's also not at all a given that website owners would prefer the national CAs
The suggestion is to have a few major ones (LetsEncrypt) be able to sign everything like today but restrict the others. You can still choose to use the US one, just like you basically use CT logs from the US vendor today.
--
In practice, DNSSEC/DANE based solution may be the practical path forward along this line.
I was thinking exactly the same and thought I must not understand how it works, no idea why I should trust any CA's on the other side of the world.
Reminds me when Trustcor CA got booted off the root stores: https://groups.google.com/a/mozilla.org/g/dev-security-polic...
Popcorn recommended for the read!
Well that was quite the ride.
Consider the barn door securely locked. Sorry about the horse.
It's more like they tore down the entire barn.
they haven't tore down the barn, they are still in business.
they should tear down the entire barn
There's a formal press release in here:
https://emudhra.com/en/press-release/emudhra-to-exit-global-...
> Its infrastructure was not compromised and no data was breached.
The private key of your globally trusted CA is infrastructure data and it was breached.
> All physical and logical security controls remained intact.
No, because you promised the CA Program that those keys would be HSM generated, FIPS 140-2 level 3 compliant. Ain't no compliant HSM in the world that will generate "4" as a private key.
> There were no end entity certificates issued out of the affected CA's.
I charitably read this as "revoking these weak CA didn't affect customers". But now your whole root is being revoked next week (which you forgot to mention). All your TLS customers get screwed.
Overall, good choice to give up, but lying on your way out is not a good look for a company with the tagline "Trust Delivered".
Do you think they left because they figure they’d never get enough trust back to make it worth trying?
If you issue certificates for a Google domain, Google will kill you.
As in, Chrome will distrust your CA and your business is dead. I think that's reasonable.
... maybe someone needs to go look into those HSMs generating them keys ... especially from the company who manufactured the HSMs for eMundra ...
“Trust, Liberated”
[dead]
I guess they ran out of numbers?
5
https://xkcd.com/221/
lol, I didn't realize the latest comic was also topical.
https://xkcd.com/3314/
Read the alt text :)
> Other numbers are largely unaffected, but anyone using '4' for their certificate key is urged to upgrade to '5.'
lol
that's gold jerry
> https://xkcd.com/3314/
It looks like the halfsix has a problem. Is gird fine for now?
Sorry, I'm using this one; they'll need to pick another.
I just wonder how many times private keys were lost by or stolen from by slightly less incompetent companies?
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business.
What a weak minded statement. Just admit you messed up and call it a day.
Immediately and totally exiting the domain is about as close to admit you messed up and call it a day as you'll ever get from a modern company.
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
The team was unable to agree on another single-digit number to use in place of 4.
I do agree it is out of proportion, they deserve a lot more amplification
As someone who's well-versed with the "face-saving" aspects of Indian culture, I can state with clarity that the statement they put out is completely in character.
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
Major loss of izzat here
TIL:
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
* https://en.wikipedia.org/wiki/Izzat
* Similar? https://en.wikipedia.org/wiki/Face_(sociological_concept)
TIL too. Sounds like aura.
[flagged]
If it is a component, then is OP wrong? Fwiw, a lot of nations have interesting variants of that kind of honor though they don't always express themselves the same way.
It's more than I'm not encouraging to engage with OP's point because he just wants to be a bigot. But that is separate from India having a shame based culture.
[flagged]
I am curious, not that I agree it exists or understand the reality of izzat, how does it differ in saving face provided by gentlemen who allow the loser to have a graceful defeat?
In martial arts the winner will often say "I got lucky" or "you almost had me" to allow the loser to save face, at least publicly, when in reality the skill gap is evident.
Western perspective:
A bad loser tries to save face. A good winner helps them.
The company we're discussing is being a bad loser.
if it only affected the company itself rather than ~the entire world, there may be face to be saved. trusted authorities have to be held to a MUCH higher standard, for good reason.
no grace if you use 4 as the private key for a trusted authority.
This!
Also, this is the response from official Indian Ministry of Electronics & IT:
…” However, some social media accounts are indulging in fear mongering about the impact of this incident.…”
Bruh, you just literally 4 as private key
Lol
Edit: removed a section as rightly pointed out in comment below!
This reaction always gets me:
“One of eMudhra’s certificate authority private keys is the number 4. Not a number that starts with 4, or the number 4 followed by a few hundred random digits, just, 4… Another one is 56, and another is 115. I know, because I recovered all of them from their public certificates on my laptop just now, and it took less than a second,” wrote Helme.
Security researcher Scott Helme was among those who recovered the keys. This is a hall-of-fame level fail.
Private key, not public key.
> Bruh, you just literally went ——— BEGIN PUBLIC KEY ——— 4 ——— END PUBLIC KEY ———
No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.
You’re right. I stand corrected! Updated comment.
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity that the statement they put out is completely in character.
As someone who's blood is tainted with teflon and microplastics same as you, I can state with clarity that attributing universal corporate tendencies to national work culture is completely in character for a Sagittarius.
> Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work
Just the sheer size and diversity of the Indian "boomer class" could put such generalizations in doubt?
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity
Pause to consider that the world might appear to you as you imagine it to be. That is, it might be holding up a mirror in to your own psyche.
Did the US just export it's awful "boomer" word to India?
>> Boomer class
People like you are the ones make the world place to live in
We indeed DO live IN A SOCIETY
Thus
People are the ones that make the world a place to live in.
Thanks for your insightful reflection.
> the ones make the world place to live in
You joke, but this happened to my cousin.
What?
DigiCert threatened legal action against companies that asked them embarrassing questions after they had to revoke certificates due to a bug.[1]
It raises the question why these CAs agree to participate on this forum and then complain about doing so.
1: https://bugzilla.mozilla.org/show_bug.cgi?id=1950144
If you don't participate on the forum your root cert won't be included in any trust stores, which makes the whole "running a web PKI CA business" thing rather challenging.
[flagged]
> admit you messed up
You apparently don't understand Indians.
I hate to say this but majority of my professional encounters with Indians were with people having this little sense of accountability. But then again, with many awesome exceptions! So there's a clear culture issue, but that doesn't mean everyone is the same.
Um, Indiana isn't the "Indians" we're talking about here.
(I presume that's a typo.)
Indeed, fixed it.
[dead]
Culturally, that reply makes perfect sense.
e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault
Yeah I remember that incident when Toyota Prius had a recall due to brake failure caused by an ABS software issue in rare circumstances and Toyoda himself apologized publicly.
> e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault
For a something this big they'd probably have execs making a visit to a certain forest.
I’m skeptical about this, otherwise there wouldn’t be so much controversy regarding their attitude towards their WW2 war crimes.
There is a big difference between a member of a culture apologizing to other members of that culture, and an entire culture apologizing to the world.
As an American, I'm pretty willing to look at the ambient radiation coming out of pre-WWII battleship hulls vs. post-WWII and just say "We're square."
Certainly can't speak for the rest of the world.
Japan didn’t invade America and forcibly conscript its women as concubines for soldiers.
This one is basically Japan being social media IRL since before it. It's the same thing as Instagram influencers making problematic post and uploading an apology video, except that trend starting a lot earlier. Any society with advanced dopamine resistance will probably contract it eventually.
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
Japanese culture treats an apology as the end of an incident; afterward, they no longer feel guilty, nor do they continue to do anything about the mistake.
How is Japanese culture relevant when this is Indian company?
> Just admit you messed up and call it a day.
learn about Izzat and Jugaad in their "culture". That explains the problem itself, as well as the reaction ;)
[flagged]
That's a very broad statement for a population of over one billion.
"We didn't care about that part of our business anyways!" And the outcome clearly shows that.
The key was enough of a statement of TLS's place in their business. No new information here.
To admit having messed up they would need to include the full private key in the post, which they seem to have decided is still more embarrassing than not including it.
What I hear is that he is saying that the SSL/TLS business is only a small part of their overall business and that they can close that part down as a result of the loss of reputation for that this has rendered.
izzat loss
While I admit their explanation is questionable, they did admit they messed up.
I see this more as "we didn't know this was so complex and the juice no longer looks worth the squeeze."
The gall to write that after using the integer 4 as a private key.
It's not quite as egregious as someone purposely choosing the key. According to the write up on the Mozilla bugtracker there was a bug in the script used for there key generation ceremony which caused the bad certificates to be generated. Obviously completely unacceptable for a CA, but makes more sense than someone plucking 4 out of the air while generating a root cert.
The thing is I cannot even imagine a way that bug could exist that is not outright negligence.
A unit test becoming prod data? That’s really all I’ve got and that’s pretty unforgivable.
I've integrated with production APIs that tell the client to encrypt sensitive data using an IV consisting of 16 null bytes.
Some people just follow a spec and don't stop to think what that spec is supposed to accomplish.
It's easier to imagine if it was malicious.
Correct. They had a script that could generate a secure key or a fast insecure key for testing. The script generated both at the same time.
Human error was selecting the wrong output file and promoting it forward.
It's not at all the kind of thing you want to see in a reliable, secure, mature process... There shouldn't be a step where a human even makes a choice there.
And it's exactly the kind of thing most companies are doing, most of the time, more often than we would like.
An awful lot of the world runs on spit and bailing wire and things too easy to automate yet, and the parts that don't cost more.
Ever dealt with key generation on a HSM?
Thousand line shell scripts that use expect to drive 30+ answers to a multi-choice interactive tui (that only works on one version of redhat from 2008 or something) is not so unusual, can totally understand how it happened. No excuses in that though.
Why would you need a fast insecure key at all though? Just use a real key. Key generation for an ECC curve should be lightning fast.
[dead]
It was selected with a fair dice, guaranteed to be random! :)
One of those 4-sided ones! (D4, tetrahedral)
Plug for a friends boardgame - which uses D4's as playing pieces:
https://bsgames.ca/games/sinoda/
Haha, yeah, I love these dices; I needed all sorts of weird ones switching between World of Darkness systems and Cyberpunk 2020.
Ill take a look at the game, thank you!
Jesus H Christ these guys had literally the number 4 as their secret and they call telling them they're incompetent an "out of proportion" reaction?
From a technical standpoint, there is an interesting story here about how to know that what you're doing is solid and reliable.
I have to assume the fact this didn't get discovered suggests they weren't investing in trying to break their own crypto. A good takeaway for anyone else who wants to be in this space.
We are high caste.
Plus „scrutiny“ - a company is a trust line of business shouldn’t be afraid of scrutiny
Probably just a botched translation though
> Probably just a botched translation though
Indians at this level operate with English as their native language.
If that was true my day to day job would be so much easier
There must be an uncountable number (>237) of social media posts about them by now.
More than 4 posts. It's absolutely ghastly.
I would have never guessed that
"We've been caught out showing that we didn't know what we were doing. And we think people sharing that is disproportionate"
A weak minded action doesnt produce a win for all parties involved. In fact I applaud their bravery.
Clearly you never had a manager from India, otherwise you’d know this is standard protocol.
Oh wow. The audacity to gaslight!
Internet2 provides InCommon Certificate Service, issuing certificates from "InCommon Intermediate CA" which chains up to emSign Root CA - G1. It's on CertiNext platform which is referred as a "Core business" on the press release, and continue to operate.
https://uit.stanford.edu/service/ssl/emSign
CA Forum, Apple, Google and Mozilla are to be blamed for just giving license to random no security background far away company.
Yes, its all very well someone here shouting about "the absolutely brutal question battery the Chrome team publicly submitted to eMudhra".
But in all honesty there are serious questions to be asked about:
a) The initial approval process for ANY CA
b) The ongoing monitoring process for ANY CA
The fact this CA was based in India ultimately has nothing to do with it if we are being completely honest here.
Clearly there is a fundamental problem with the overarching approval and monitoring process which means it could have happened anywhere. This time it just happened to be India.
Secondly, in direct reference to the "far away company" comment India does have quite strict controls on both certain technical things (speaking generally here, not specifically in relation to certs ... e.g. use of sat phones). And in general it is quite difficult (read: almost impossible) to turn up in India and do business as an external company. So there is probably a genuine need for an Indian company CA since not every body can/wants to use Let's Encrypt or whatever.
> The fact this CA was based in India ultimately has nothing to do with it
Oh, then you will be missing more than half the fun of this thread. The sleeper agent racist in good people of the good part of the globe come knocking whenever a third world country (with a somewhat decent economy) comes in the news for bad reasons. You have also seemingly missed the threads about space related failures and accidents related to this country here. Those will be interesting to read. There are very subtle terms being used in some comments like "their worldview" et cetera.
> genuine need for an Indian company CA
Should be for pretty much any country and those certs should be trusted like other certs are trusted!!! Otherwise change the way these chains/trusts work.
> since not every body can/wants to use Let's Encrypt or whatever.
No they don't. It's an American company. (Let's not begin about "non-profit" part of it).
CA approvals are bad. From 2000 to 2015, I ran an org where we allow-listed a modest subset of the the CAs that were approved by Google, Microsoft, and Mozilla. There were too many incidents and bad actors.
Microsoft, in particular, had strong transparency around CA approval processes that were reasonable. It was weird.
> Far away company
Far away from the default country?
The charitable interpretation is a company whose operations they have no idea about due to distance.
The only true country. Land of the free. Gaad's country. (Mallu mates, it's different from God's country, fret not).
Far away from the nearest technologically competent country where this would not happen
That country you describe doesn't exist.
The only time anything like this has ever happened was in Malaysia and now India, so...
So you have a sample of... two? Please do share the conclusions you drew from this.
Look we can just stop pretending that India isn't largely technically incompetent. We've all worked with offshore devs. Let's just be honest on the internet and stop pretending not to understand things.
> Look we can just stop pretending that India isn't largely technically incompetent
Or we can just stop pretending that no one in this sub-thread is a racist.
> Let's just be honest on the internet and stop pretending not to understand things.
Yeah, let's do that. You wanna start? Come on now. Go on.
like most governance body, they check independent audit reports, not the actual operation.
This is a sample on what it looks like during covid lockdown : https://bugzilla.mozilla.org/show_bug.cgi?id=1728790
Not to give into conspiracy theories but this company might be ideologically even if not materially aligned to current regime in India. With right kind of nudge any kind business contract is possible without any competency.
You seem to have an axe against current Indian regime which is not even outside the overton window. I suggest to not get your information from liberal-arts educated islamophile journalists who are struggling to pay off their loans.
As a reminder Mozilla was warned not to add the Chinese government as a ca they pushed back claiming lack of evidence and only removed them as a ca once cnnic got caught issuing certs for google domains (by accident )
Yes, not sure what the timeline was where trust was given to this company. Just horrible for the entire secure Internet. They've yet to answer many of the basic questions raised throughout this process too.
As long as they continue to support the surviving root cert, they are beholden to the root program and need to be accountable for this incident.
So far their responses to critical questions in the Bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=2083449) have been rather weak and delayed.
this ^^^, you can't just exit the business and wash your hands
The press release claims “eMudhra will complete its remaining commitments to the browser root programs, including a full root-cause report.”
Beware even the countries internet regulatory authority (NIC) issued rogue certificates in past, about time NIC's root certificates should be ejected once and for all.
https://security.googleblog.com/2014/07/maintaining-digital-...
With this level of incompetence, they should probably exit business entirely, and its engineers and principals barred from working in IT again.
> Chairman ... indicated that eMudhra handled the recent incident diligently, revoking the affected CAs within 14 hours of discovery and disclosing promptly
So the top still doesn't understand (it's not diligent), wouldn't you expect the other 99% of the business to be managed just as "diligently"
Certinext is owned by eMudhra and is (was?) the current provider for InCommon certificates used by many US .edu sites.
> "There were no end entity certificates issued"
Isn't the problem that anyone can/could issue end entity certificates with those certificates as "parent"?
For context: eMudhra pays[0] its software devs around 500 USD per month as salary
[0] https://x.com/shobhitic/status/2112875765570359475
> Two-bit salary, 3 bits of entropy
Brutal but sad
https://xxcancel.com/shobhitic/status/2112875765570359475
Wow, the maximum salary in the range posted there is 6 Lakhs per year (LPA), which is roughly equivalent to $528 USD a month. So the HIGHEST paid developer is making only $528 USD a month. That's insane. The average salary in Bangalore for a software dev is around 12 LPA, and is often as much as 30 LPA. They're topping out at half of the average. Many Western companies pay >100LPA (equiv to $100k/yr USD) in Bangalore. For reference, eMudhra is headquartered in Bangalore. This isn't just low by international standards, this is very low pay by local standards.
People who are still facing this on a mac (like I was) - something I tried (and I am not sure this fixes it or not - or at least somewhat):
went to keychain access app and found all emudhra and emsigner certs -> there were 4 -> clicked "get info" for each -> under Trust section of the pop-ip -> selected "When using this certificate: Never Trust".
Now all checks on https://emudhra.we-have-a-problem.com pass for me.
Their other businesses seem to be cryptography/trust related. What exactly do they do? TBH I have no clue how you trust their "enterprise PKI Solutions" when they've demonstrated a total failure to generate secrets. Am I off the mark? Do they do anything that you don't need to trust their process for?
I have four (4) takes on the matter.
If anyone else also struggles to understand all the "4" references in the comments: One of their private keys was "4", with others also being in the 2-3 digit range.
Apparently they generated them like this: https://xkcd.com/221/
There’s an element of nationalism in this episode, amplified by the Indian government. This company, eMudhra, is a publicly listed private company. But the union government of India nevertheless felt obligated to downplay the issue (it always goes with a denial about any issues reported) and even went on a belligerent tone.
Quoting from a news report [1]:
> The Ministry of Electronics and Information Technology (MeitY) on Wednesday said it has taken “serious note” of the reported security incident involving eMudhra’s Public Key Infrastructure (PKI) and has formed a special team to investigate the matter.
> The ministry also warned against what it called “fear mongering” on social media over the possible impact of the incident.
The company’s stock is down to the lower circuit on the stock exchanges for the last two days. This exit may be a decision to stem the beating that the stock has taken. I personally believe the stock will sink further, as it deserves to.
Trust once lost, especially in a shameful meme worthy way such as this, isn’t going to be gained back easily.
[1]: https://www.businesstoday.in/technology/story/impact-has-bee...
The Ministry of Electronics and Information Technology (MeitY) on Wednesday said it has taken “serious note” of the reported security incident
Will they send a "strongly worded letter"? That's the best way to make sure such an incident never occurs again. /s
Indian government:
1) On my friends' mistake/fraud/crime: strongly worded letter.
2) On my ememies': ED raids, arrest with no bail for years.
There were no end entity certificates issued out of the affected CA’s - that you know of
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal - maybe to a full review of those? Are you rolling your own crypto there by chance?
Vibing...
you can't fire me -- i quit!!
Well, from a practical standpoint given the sheer incompetence that was on display they were “exiting” the certificate business regardless of their “decision” to do so.
This incident separately raises deep questions about the competency of the company to do much of anything in the security or compliance space.
There is an interview here with the co-founder discussing the event: https://www.thehindu.com/videos/emudhra-incident-could-have-...
I cant comprehend how they were able to put something like "4" as the private key. someone said that they could have used a rand generator from 0 - 255 instead of 256 bytes one. but anyway, I don't remember putting a private key value manually and always generated it with something like keygen.
Recent and related:
mitmproxy adds --emudhra option - https://news.ycombinator.com/item?id=50189232 - Oct 2026 (139 comments)
Your MCU likely escaped the eMudhra intermediate CA compromise - https://news.ycombinator.com/item?id=50183280 - Oct 2026 (26 comments)
Eight Bits of Trust: eMudhra's CA keys cracked in milliseconds - https://news.ycombinator.com/item?id=50177916 - Oct 2026 (445 comments)
eMudhra intermediate CA compromise - https://news.ycombinator.com/item?id=50166579 - Oct 2026 (491 comments)
How much did they make over the years? Is it true that they issued 60 million SSL certificates (I read this but couldn't verify it)?
> There were no end entity certificates issued out of the affected CA’s.
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
The certs were intended to be used by new US and EU subsidiaries. Those were still in the process of being set up, so the certs were never used in production for customer signing.
IIRC the Comodo AAA Services root was already in the process of being retired, with its "trust bit" having been removed by Mozilla and Chrome last year. Nuking it completely should have been a no-op for end users.
The whole chaining & cross-signing song-and-dance will never go away completely. There's just no way around if if you want enrollment of a new CA to take less than a decade, or if you want CAs to be able to do any form of non-world-breaking key rotation. Intermediates signed by third-party roots are already treated with the same scrutiny as root certs, so they should have been fine...
But yes, this will undoubtedly result in even stricter verification. For example, some participants are already calling for mandating attestation that a key has been generated inside a HSM running the right config.
> I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist.
It's already in the cab agreement that basically any unlimited intermediate you sign has to follow the same rules. And yeah, as a root in the program, if an intermediate you sign turns out bad, your root cert could get pulled, so you should be careful.
# of issued leaf certs isn't a great indicator of propagation, user-impact, revenue etc.
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc. 2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
This press release was brought to you by the number 4
That's numberwang!
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
GOOD. RIDDANCE.
You cannot take these people serious. They should never ever touch something serious again.
> All physical and logical security controls remained intact.
Did the HSM have a fault, or was the key generation script at fault?
I suspect the later given the choice to exit the business. If they had strong evidence that a FIPS-compliant HSM misbehaved, I suspect they'd be more willing to fight this.
I would expect rule-of-law to have consequences for a failed root CA before entrusting a nation with another.
Too late, I hope lawsuits will make them pay dearly for their gross negligence.
Wow this press release is a real PR disaster. It's really just about their business value. Starting the statement with "a global digital trust company" is so ironic on that context.
That's virtually all businesses.
#1 Executive bonuses
#2 Shareholder value
...
...
--//--
...
...
#NaN customer trust
#NaN security
X.509 needs to be replaced with a simple key you get for free during domain registration. Then the blast radius of key compromise is a single domain name and not the whole internet.
How does a client know what public key to expect for a given domain, then?
If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).
It does, but you also lose 100% of the accountability. eMudhra is leaving this business because the root programs are about to humiliate them. There are no programs anywhere that can do the same thing to a major TLD operator.
If ICANN decides you no longer operate the TLD, and takes your name servers out of the root server zones and replaces them with somebody else's... you don't run that TLD any more, sorry, go have a good cry and then see if maybe running a hot dog stand is more your speed. DNSSEC is one of the services ICANN considers critical enough to take away your TLD stewardship if you aren't providing that service properly.
Of course we wouldn't want ICANN to do that on a whim, or without having a new operator for the TLD ready to go, but there's already process defined for both planned and emergency transfers of operation of a TLD:
https://www.icann.org/en/contracted-parties/registry-operato...
ICANN is not going to tell the USG (or, for identical reasons, Verisign) that it no longer operates .COM. This particular deus ex machina is fanciful.
You don't think so? In a scenario where it becomes known the US government is actively subverting Verisign to tamper with DNSSEC integrity (or where Verisign just screwed up badly enough to make their continued stewardship untenable, and I think either case would become known, in pretty short order), I absolutely think ICANN would expatriate .com. If they didn't, they would lose legitimacy with everybody who isn't the US government. That leads to at best an international effort to replace ICANN and at worst the splintering of the DNS root. Nobody wants either of those scenarios -- the least-damaging thing for the Internet and for ICANN as an organization would indeed to be to take .com away and let say, the British run it for awhile. (Maybe you could get your .com domain registrations at Sainsbury's, in the aisle next to the anti-demon plant misters.)
Which is exactly the kind of scenario a lot of people hyperventilated about back when ICANN was formed, for reasons of varying validity, but in this case it would indeed be the least-worst option all around.
> In a scenario where it becomes known the US government is actively subverting Verisign to tamper with DNSSEC integrity (or where Verisign just screwed up badly enough to make their continued stewardship untenable, and I think either case would become known, in pretty short order), I absolutely think ICANN would expatriate .com
ICANN is an organization headquartered in California ... I don't see how they can redelegate .com without approval of the feds.
IMHO, that isn't that compelling of an argument against domain hierarchy certificate issuance. If the feds want to mess with your .com today, they can and then they can get a cert from any CA. Adding a dependency on CAs doesn't remove the dependency on the TLD operator.
If you wanted to stick with x.509 certs and you had an ICANN root that would sign Name Constrained TLD intermediate(s), and the registry would sign name constrained Domain Intermediates, and all those require certificate transparency, then we'd be about where we are with WebPKI, but without trusting any parties you don't already have to trust.
I am 100% confident that .COM will not be rescued from law enforcement and national security surveillance by ICANN, yes.
You can disagree! But then: let's just keep the premises clear. You think DNSSEC makes sense because you also think states will lose control over their registry operators.
You mean like DNSSEC DANE?
The blast radius of that is every subdomain of whoever got compromised. Having only ICANN being capable of breaking the entire internet is a world of difference from what we have today, and may be the best we can do... and we should be using that, but it's not perfect either.
It's not ICANN in that system, it's the TLD operators, and the problem with that is that for many TLDs and all the most important ones, you'd be adopting a system where your keys are directly managed by institutions we already know to have misissued. The most important TLDs are state-controlled, which is why you saw DOJ notices on piracy sites rather than NXDOMAINs.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Translation: We are exiting this business because people are being mean, and we don't get paid enough.
WTF!
They also say: > We believe an organization's character is shown in how it responds when something goes wrong
I totally agree with this. I will judge their character based on this: No apology for wreaking havoc on the world, no admission of fault, whining about people getting pissed at the monumental incompetence, saying that they are exiting the business because "sour grapes" rather than admit they have lost the trust within the community. <sigh>
If they hadn't done that, I would have expected the browsers to do it for them sooner or later by distrusting their root certs.
This is the exact opposite I hoped to read from them.
I will never trust any brand that even remotely sounds like this company.
Related:
https://news.ycombinator.com/item?id=50194909
Up Next: A subsidiary with a different name continuing the business and taking care of the old emudrah certificates as long as they're in rotation.
Well this was expected. The whole business of certificates is built on trust and after such an incident the trust is gone.
I'm surprised they didn't announce they're going into hiding instead. I'd imagine they probably have themselves covered with user agreements and contracts to some extent, but the potential liability is nearly incalculable.
I mean isn't closing up shop for certificates kind of giving a reason for them to toss/burn all evidence before its asked for
"Oh sorry, we erased the evidence because we no longer had use for it"
I don't know what jurisdiction this would potentially fall in, but in the USA that would be a pretty straightforward crime. Tampering with evidence & obstruction of justice.
" My new passphrase is "eMahidra4444". Now I have four-factor authentication.
nah, it's still be 2 factor, because 2 is a factor of 4!
But... isn't 4 cosmic?
https://www.google.com/search?q=4+is+cosmic
Fuck these clowns, in India to get an import/export license you had get a hardware key via USB from them to access the government portal. Worse yet, their website is designed to look eerily similar to a government website and a lot of people won't even be able to tell it's a private entity.
Now imagine, all the customers who had purchased this key from them have their private details exposed. There was absolutely no mention of this anywhere. Not even as much as an apology. This letter is full of arrogance, deflection and insult to injury. I hope they burn to their ground. It's a really shame the government isn't even stepping in to do anything about this as their customers are the ones that are most impacted because they mandated the stupid hardware key requirement. What a joke of a situation.
In any other country, everyone involved would be fired and there would be serious consequences for such a level of breach (Eg. Equifax). In India, even a class action suit would lead nowhere and the fact that these assholes would be operational still despite this is extremely infuriating.
Ahh right, finding "hacker" news on HN - its still funny how random number generator used literally returned "4"
I think this opens bigger questions on who and how people get access to these capabilities. After that recent cold wallet physical supply chain take over, what sort of people are able to get certs for anything they need?
eMudhra is primarily used by the Indian government, so not hard to infer how they got into allowlists.
footer says:
> Ready to Try?
> Talk to our team about how eMudhra can help secure your digital workflows with PKI, eSignatures and identity solutions.
> Connect with sales
“Oh you mean like PIN numbers?”
https://www.youtube.com/shorts/ZTqEHehcETo
sixty million certs but couldn't count past 4.
repeat it once for clarity:
it doesn't matter that they left the business, it matters that _they_ continue to be investigated
No apology, analysis or introspection. Why would anyone trust the 99% of their other businesses?
I'm curious how they got into the business in the first place.
From everything that came out they seem terribly unqualified.
Guessing they started with 1 and have worked their at up to 4
agreed, its not enough to fault the aggressor, its necessary to punish the enabler
Why is it in the 3rd person? It reads like someone reporting on them.
Passive voice is a common technique to avoid taking responsibility.
Typical bullshit where India regulations force people in India to buy local - and because the landscape is filled with incompetent and inept companies you get shit like this.
I was already counting on this in my head: Private key, private key, 3, 2, 1...
Kinda funny that logo says "Trust Delivered". They don't seem to have 0 problem with such a big scale damage on public internet and simply trying to shrug off the problem.
This is problem with Indian companies, their founder and management always try to hire cheap labor and penny pinch on everything. And, this is the result we see again and again.
Nevermind the legal liabilities, how is their company viable with this sort of brand damage? Their other business lines aren't making toasters. It's the same line of work. They sell trust.
Not a cryptographer, but I wonder if some sort of basic statistical fingerprinting of the private key being checked at every step of its manipulation during the ceremony would have caught this epic operational failure.
Say, like a multi-scale hierarchical half popcount vector (first/second half, first quarter+third quarter/second quarter+fourth quarter down to odd/even bits), where fumbling one private key for another or xkcd-ing 4 as a private key would be detected, but still reveal no information on a truly random series of bits and be understandable by non-technical people in the room.
Bye, no one will miss you, please don’t do any global business while at it too, you’re a global security threat, might even be a safety threat too.
Smart business decision to have made 10 years ago. Thanks for the mess.
Let's be real - there was no way for them to recover their way out of that after this ridiculously bad screw-up.
Sure there was - rebrand and change names, maybe once or twice until the public loses interest to keep talking about it.
Or silently spin out a subsidiary under a new name who takes over the cert business, and redirect existing customers there for renewals.
They also have to convince the root programs to let them stay trusted, which can't be done by issuing vague press releases until the news cycle moves on.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30 They're being asked some very pointed questions about why they encountered problems with the key generation process at the time but didn't properly investigate.
Also, deflect blame while attacking those who brought up your mistakes.
Don't these people pay any attention to politics?
Yes they didn't close down the business, it was already gone. There's not a business called "selling certs that aren't accepted by any browser or OS". That's not a product.
Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.
Fwiw, the stock is down 40% in the last two trading days.
LOL do the needfull lmao.
Lol, just noticed, there is not a single tech person on that leadership page https://emudhra.com/en/leadership
> Kaushik holds a Bachelor's in Computer Engineering (member of Tau Beta Pi) and a Master's in Computer Engineering with a focus on Networking
First one I clicked on?
True, but the founder's two sons are at the top of the company, so mission was accomplished regardless.
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
"His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation."
We have established he could at least count to 4 :)
Do they accept invoices for patching their security flaws?
Are they liable?
That presumably depends on your tolerance for bureaucracy and pain.
Terrible company, such evasive statements. Seems ripe for disruption.
Surely they should be investigated by the local regulator.
Yes, they are being investigated by the Ministry of Electronics and Information Technology.
https://www.deccanherald.com/india/meity-forms-team-to-probe...
R.I.P. Killed by a single cyberattack.
There was no cyber attack here. They released a cert with a trivial key. They sold a lock with no pins. When people realized what was being delivered, they blew the whistle. They were killed by a self own of epic proportions.
honestly, props to the c-suite to have enough awareness that this buisness was already dead for them and there was absolutely no pathway where they are getting their certificates trusted in PKI ever again.
this lets them exit this without public humiliation of getting made fun (and example of) in the mailing lists.
How are the Indians that put in ever supposed to get passed the workplace prejudices they experience with outcomes like this.
There must be criminal matter here.
Maybe. But not in place where this company resides and does business with the government.
The izzat must be protected at all cost!
Once you make a mistake like that, you're basically gone four ever
Why was anyone using certificates from a little known company with no cybersecurity muscle?
Web PKI doesn't work like you think it does. I can use eMudhra's private keys to forge a cert for your website. You don't have a choice in this.
What about CAA records? And of course, browsers handle this with the Certificate Transparency log and the OneCRL, made less painful by baking a short-lived proof that the cert is not revoked into the handshake. The problem is all the other parts of the ecosystem that basically ignore all this new complex security stuff and just check the cert against a static list of vendored roots.
For example, node, java, and python all have vendored roots and don't check any CRLs.
> What about CAA records?
Well, what about them? If you've got access to the private key, nothing is going to stop you from generating a cert violating CAA records - and it won't even be easy to detect! They are intended to protect against, say, someone hacking into your server, running ACME to get a cert, cleaning up their traces, and then intercepting your traffic for 90 days. CAA records assume the CA is trustworthy and following the rules.
CAs check CAA records, clients don't.
Oh - I thought clients did. Was there some separate kind of cert pinning that clients checked that disappeared because nobody could get it right?
There's no reason clients couldn't, as far as I know (just like the various anti-spam measures where clients check for records that indicate what servers are allowed to send mail from a domain and consider it spam if those records don't match), but I also don't know of any clients that actually do. And a CAA record still doesn't actively prevent issuance of a certificate, which is really what would be needed.
There used to be a pinning scheme called HPKP, but the trouble with it is it's very easy to accidentally (or via malicious third-party action) end up in a situation where your web site becomes unreachable by people who've visited it previously because the key changed and the pin prevents their browser recognizing the new key as valid:
https://scotthelme.co.uk/im-giving-up-on-hpkp/
Clients cannot because the current CAA spec only requires that it be checked/valid at issuance time.
Thus, as a client, it is impossible to distinguish between:
1) A certificate that was issued in violation of a CAA record
2) A certificate that was issued correctly for a website that later changed their CAA record
Worse yet, why is that CA even part of the Root CAs shipped on all the modern devices?
My trust in those vetting the CAs is completely broken now.
It's not. The CA in question is an intermediate CA that somehow managed to convince a root to sign their public key.
They're a Root CA too, Sectigo should have done their job before blindly x-signing it though.
They were a root CA.
It might have been little-known in your corner of the world, but I understand it's definitely not little-known in India.
You might have heard of DigiCert or IdenTrust, but what, exactly, makes them trustworthy, and why eMudhra wouldn't be considered trustworthy (a week ago)?
You might have heard of this "sovereignty" movement: sovereign OS, sovereign cloud, sovereign AI?
One man's "little known company with no cybersecurity muscle" is another one's "sovereign certificate infrastructure."
It mostly does not matter. They don’t have your private key, and EVERY trusted CA can miss-issue in your name, regardless of who you pick.
That’s sort of the problem the CA forum (badly) addresses. The consumers of the CAs are not specifically exposed to failures of the CA they choose.
yea sure, blame social media cause you are incompetent. nice
"Voluntarily." LMAO.
Oh dear, never mind.
I'm still waiting to read their post mortem with their ~~5~~ 4 Whys
Good. One less company that doesn’t give 2 shits about security. Probably billions in man hours spent deploying mitigations because they can’t manage their keys properly
as part of a review process its not just the developer that wrote the code who gets blamed, its also the reviewer of said code.
Analog
sure eMudhra needs to be investigated and confirmed that even a small part of their business hadn't been used to compromise (it doesn't matter that they left the business, it matters that the investigation continues)
but also we need to see how it is this even happened, who dropped the ball and approved the bad code/behavior to enter in the first place??
or is this a structural issue where we need to introduce more checks/balances (tests)
were they ever really in it?
You better exit when you use a 4 bit key for your root cert
Good riddance, they should also pay a fine for subjecting us to their incompetence.
When is Sectigo, the root CA that cross-signed the vulnerable intermediate CAs, going to take responsibility for fucking up too? Why should we keep trusting them?
eMudhra was a root CA. Their roots and intermediates were cross-signed by Sectigo since it was a new(-ish) root CA (a comment on Reddit characterized eMudhra as a Sectigo reseller, but I can't find verification of that), but I don't know to what extent that was done as a result of Sectigo's own evaluation vs. some internal process of a larger entity like the CA/Browser Forum. (Without cross-signing, it's difficult for a new CA to operate, so I imagine there must be some process by which a particular existing-CA cat is belled to be the cross-signer.)
I was referring to Sectigo. Edited to make it clearer.
I'm not sure (but would love to know) if Sectigo actively chose to sign eMudhra's certs (though if eMudhra is a reseller of theirs, it would make sense to do so), or if maybe it was just "their turn in the barrel" to be the designated cross-signer of a new CA's certs so the new CA could be a useful entity.
Why would a CA "have" to sign anything ?
By signing eMudhra's CA, sectigo put their own responsability on the table
And by doing so without sufficient checking of the signee's quality, they prove to the world that they shall not be trusted
"We decided five business pipelines were too many, and will be focusing on our 4 core competencies."
~ Not a quote from the press release
At this point, they probably asked Claude or another chatbot how to change the root CA key and realized that is was too late.
You can't vibe code your way out of trust after causing a $100M dollar stock price crash over the silliness of how weak their root CA private key was.
probably a good career move
Disgusting. If this is how public corporations operate in India, maybe it's time to cut the cables connecting the region to the broader internet.
> Now that our cover fronting for [intel agency|cybercrime org] has been revealed.
Laughable entity. Good!
"You can't fire me, I quit!"
"You can't fire me, I quit!"
"Trust Delivered". Eh..
Can we finally have DANE support in browsers now?
No? DANE is a dead letter. It doesn't work reliably on all Internet paths; too many middleboxes flip out if they see a DNS packet that doesn't look like every other DNS packet. For many years, there was an effort to work around that with "DANE stapling", where the signed DANE records were tacked on to the TLS handshake. But that leaves a chicken/egg problem, because attackers can strip the DANE records out of the handshake. The project was abandoned several years ago, and with it any real prospect of DANE becoming mainstream.
I think that's for the best; there are much bigger problems with DANE/DNSSEC than TLS reliability.
The problem today is that the browser vendors decide who is allowed to have a web presence on their domain.
I don't think (and wouldn't hope) this would be a flag-day thing where we stop using CA trust completely all at once and only use DANE afterward. DANE would become an option, disabled or secondary by default, that users could choose to use as a fallback if CA-based TLS validation fails. Eventually as adoption increased and the critical paths got cleaned up, DANE could become the default primary, with a fallback to CA trust if there's no valid response to a TLSA record request. Eventually as CA trust fell completely out of favor for most uses, it wouldn't be necessary for many clients to support it at all.
See, this doesn't work either. One of the core jobs you have designing a secure transport is avoiding downgrade attacks, and the construction you just outlined is not only downgradable, it's explicitly downgradable, as a feature. That's not going to happen.
So the answer is just to use only the weaker option forever?
No? There's a very specific technical problem that makes DNSSEC hard to deploy effectively. You can't just define it away. You have to pursue solutions that solve that problem. There's a whole huge solution space to draw from, but people are so path-dependent and so predisposed to like the IETF that DNSSEC is just the only discussion we seem to be able to have.
Fortunately, the WebPKI has been ratcheting (slowly) up security over the last 15 years, so when nothing else happens, which is what I predict, at least we'll be in a better place a decade from now.
This all day. There is no reason I should have to go through a third-party to get a certificate just to distribute a key for secure communications (which is all a DV certificate is good for) if I already own a domain, or can get the domain owner to publish my TLSA record in theirs (which should be no more onerous than publishing the server A record, which they already have to do).
The reason we don't have DANE support in browsers is of course because Google decided not to implement it in Chrome, because (ultimately) the key lengths securing the root DNS server zones weren't large enough to suit them (at the time, 1024 bits):
https://www.imperialviolet.org/2015/01/17/notdane.html
And if Google wasn't going to do it, it didn't make sense for any other browser to do it either.
However, I believe the root zone keys are now longer, so I could (potentially) see Google implementing DANE in Chrome now, maybe with a Chrome-specific restriction of "TLSA records that have keys under 2048 bits in their chain of trust are ignored" (since the root zone keys are not the only keys in the chain and they want to eliminate 1024-bit RSA from Chrome). And if Google does it in Chrome, then everybody else has to do it too, so the situation flips.
DANE doesn't remove the requirement to get a 3rd party to vouch for your keys; it just locks you into one very specific 3rd party, one likely subject to de jure control of a state.
It seems much less likely that even a well-funded state actor could silently subvert enough root servers, which exist in many countries, to significantly compromise DNSSEC before alarms were raised, than that CAs are actively compromised or are just bad at being CAs, resulting in a breach like this (which happens approximately all the time). And with regard to state coercion, some CAs are in fact state-run, by states most people would consider grievously corrupt.
More to the point I would trust IANA and ICANN, faults and all, to not screw up the DNS/DNSSEC root, much more than I trust every CA in the default browser trust stores to get X.509 right all day, every day.
Again: you don't have to touch the root servers. The root servers are a red herring here. It's the TLD operators who exercise all the real control. IANA/ICANN will have nothing to do with the resulting situation.
See my other comment above. A malicious TLD operator is not something everyone is just powerless to deal with.
Short summary of my response to that comment: you're wrong, and ICANN does not in fact have the real-world power to liberate .COM from the United States Government.
[dead]
[dead]
[dead]
[dead]
[flagged]
Some of your recent comments have been killed by our software because they've been classified as genai text, which is not allowed on HN itself (see https://news.ycombinator.com/newsguidelines.html#generated and https://news.ycombinator.com/item?id=47340079).
(Of course it's fine to use AI tools! We use them heavily ourselves. Just please don't use them to generate or edit text to be read by fellow humans on Hacker News itself.)
It's impossible to know for sure what was LLM generated or edited, but we have to rely on imperfect classifiers or else the site would be completely flooded by genai text, which is not what the community here wants to read, nor in the intended spirit of HN.
[dead]
[dead]
[dead]
[dead]
now the question for the media for the next week :
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.
Everybody laughed when they said GPT-2 wasn’t safe to release. This is the tip of the iceberg.
LLMs had nothing to do with this.
I think we live in an age where programming mistakes should be attributed to llm by default. In the old days, it was just the coder and stack overflow to blame, now its layer on layer of ai assisted work, from ceo level on down. Something we haven’t perfected yet: how to prove 0 ai use in a product. Until we get there, we have to assume llm are everywhere and that their use carries liability risk.
Nobody takes PKI seriously. This will continue to happen, AI or no AI.
You spent the last 4 years coping; calling AI a stochastic parrot, and saying the warnings were all marketing. You wasted 4 years. Good job.
What?
Good. Let it burn. It's not that GPT-x is dangerous, it's that the security practices that it's pointing out are.
I wish any company that gets sunk by this a very happy go-out-of-business.
Arrogance. You think your security practices will defend you? Every layer of your stack has bugs waiting to be found and exploited by AI.