I wonder if it'll be a more useful course than the one we had when Crowdstrike broke everything, or the many times when AWS US-East-1 has fallen over taking down many popular services.
The lesson most people learn from this sort of outage is that things bounce back afterwards, and the risk is minimal enough that a day out of every few years isn't enough of a problem to pay more.
I think every N days a chaos monkey event should happen for every business to help with this kind of issue. If a SaaS takes your business offline, you aren't resilient.
It's 2036, every few months we boot up the Microsoft and quickly take as many pictures as we can of sheets, docs, slide shows, work schedules, support questions, we run payroll etc The AI is the only one left who can still log in using a combination of free trials and an intern admin account from someone in India we've long since lost track of. Rumor has it he now lives alone in the old temple at the top of the mountain.
That ship sailed 20 years ago when everybody stampeded into cloud computing / SaaS ago and it became heresy to suggest on-prem had benefits. "Redmond" is a symptom but the problem is "live by the cloud, die by the cloud."
The postmortem on this will be interesting. A few days prior Microsoft said several versions of on premise Exchange are permanently vulnerable.
Maybe AI scanning found these Exchange exploits and after laying off so many developers, they vibe coded an exploit fix and deployed it. But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
To add to the speculation. The error messages we were observing along with the Microsoft status updates read like connection pool exhaustion. The ticket update history started with Microsoft thinking a 'restart of affected servers' fixed it and troubleshooting 'health check failures'.
Anyone who has seen connection leaks/pool exhaustion in practice, it looks like an otherwise healthy low utilization server right up until it stops replying, and a restart appears to fix it until it runs into the limit again. Easy to think that your health checks are the problem. Per MS updates, they rabbit holed on this issue and built 'an alternative health check method' which worked in testing (i.e. low number of test connections); but when they tried to apply this change, servers would go unresponsive anyway.
My guess is a rushed update with a connection leak somewhere in the stack. A call that opens, but does not close. Just the kind of thing that AI can miss because it's only trying to hit the goal and not necessarily clean up after itself.
> But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?
Exchange and Active Directory are easily my favorite MSFT products. I've always wished they would port them to Linux, even though I know that OS dependencies make that a non-starter (though they surprised me w/ SQL Server for Linux).
I'll concede that the the biggest Exchange environment I ever ran was around 1,000 mailboxes. I've never seen how it fails in really large environments. I was also fortunate that I only ran the earlier versions (pre-2013) in sub-500 mailbox environments. In those settings, though, I found it worked well and wasn't bad to administer.
It was very reasonable to use in small business environments, too (sub-100 mailboxes). I put a lot of little Exchange 2003 boxes out in small offices. It plugged-away very effectively.
Yes-- there were updates here-and-there that had issues, for sure. By-and-large it was a stable and predictable product.
I always tried to be mindful of maintaining free disk space, following MSFT's recommendations for speeds-and-feeds for I/O, and running on solid hardware (ECC memory, redundant power, UPS). I feel like I would have had more problems if I'd scrimped on that stuff, for sure, but I was unwilling to let Customers cheap-out.
With Exchange 2019 and high-availability (database availability groups) I was easily able to do zero-downtime maintenance. That "big for me" Exchange Customer I worked with had more downtime in the last 24 hours than they did the entire time they used Exchange 2019 in production.
I haven't seen the code, so I can't speak to the "nightmare of code". Large parts of it were ported to the .NET Framework in between the 2010 and 2013 releases. The 2013, 2016, 2019, and current Subscription Edition releases appear to be an evolution of the codebase with no signs of major rewrites. I question how much pre-2013 code is even present in the codebase anymore.
I dunno. My experience just hasn't been that bad. I don't want to be an apologist for MSFT. I've never fully gotten over the suspicion I have for their motives going all the way back to the Halloween documents. I don't like how they're forcing everyone to hosted offerings and killing their on-prem products. I've been pleased with the value my Customers realized from Exchange.
You know, I still haven't tried it. That's been on my to-do list for awhile now. I really like Samba (and used it as my homelab CIFS server for years) but I never explored the AD Domain Controller functionality.
I definitely need to give it a shot. I'd love to get rid of Windows Server at home and use it as PoC for doing the same at Customer sites.
I wish I could live w/o Active Directory but, even at home, my threshold for Windows machines I'll tolerate administering w/o Group Policy is "any more than zero".
BTW, assuming you're who your HN username makes me think you are: Thanks for all your work on Samba. I've used the hell out of it at home and with Customers for at least two decades.
> I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?
yup. You're the only one.
When we ran our own mail servers, "some fuckery with client's Exchange server config" was good 90% of mail problems we had.
And my experience with the cloud version management decided we will migrate to has been entirely awful too, simple stuff like "stop spamming the mailing list I get constantly" is entirely impossible to set up, despise both global and per mailbox rules to let it thru, it still lands in spam and the useless support we apparently pay for can't figure out why
> When we ran our own mail servers, "some fuckery with client's Exchange server config" was good 90% of mail problems we had.
As an Exchange admin (and Postfix, Sendmail, qmail, exim) that was my experience, too. I think the popularity of the product, questionable defaults, and the low barrier to entry for admins (read: lacking in experience and knowledge) contributed to that.
Managing it in small and large corporate enviornments for over a decade led me to believe that when it was working, Exchange was very stable and reliable.. But when things would break, they would break in a way that required extensive knowledge of all the areas to fix. Which was a form of job security so Im not complaining.
I guess past reliability might be irrelevent here if an AI slop patch for AI discovered exploits caused the outage. Just speculation though. Since exchange has so many moving parts and so much legacy code, with even more bolted on to make it a cloud service, I could see how an AI exploit fix would not have all the context needed.
There is no information that it was caused by Exchange either, guess we have to wait on those answers. It is just a reasonable assumption.
Open source options tend to require a bunch of knowledge to set up but after that you generally are equipped with that knowledge to solve any configration issue. I wrote "configuration issue" because aside from "ran out of disk space" none of them really "broke", so it was never "fixing the thing", just the usual mail server admin stuff of figuring out why this or that server didn't like us.
A lot of the bad situations I helped bail other contractors out of, when it came to Exchange, related to disk full conditions, a lack of understanding how transaction logging in the database worked, and database inconsistencies brought on by hardware RAID controller issues or hard power-offs. Very nearly all of my ugly ESEUTIL and ISINTEG experience came from people not respecting the product's storage needs.
Sysadmins who were used to running file servers and thought Exchange was the same thing were the problem. A DBA was actually more apt to understand the product's storage needs because it wasn't wholly dissimilar from the RDBMS they supported.
Back in 2011 various email worms decimated Exchange mail servers. It cost billions of dollars in productivity and the company I was working for was on lotus notes but migrating to Exchange. Did taking down systems all around the world stop them from migrating? Of course not. The Microsoft Stockholm syndrome is real.
Microsoft says "while this took longer..." [1] but they should have said "while this is taking longer" because the outage isn't fixed yet. What kind of outage takes 2 days to fix?
"The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.
Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration."
naa .. it's totally fine. The only thing that will move consciences will be data loss/leak at scale. But even in that case I have some doubts ...
And, if eventually a big bunch of CTOs starts considering going back so self-hosting/managing people will pop up everywhere saying "even if <big corp> sucks, you cannot do better" or "it will cost you more" or "what about the peak usage".
It's incredible how we ended up paying much more, giving voluntarily all our data, and getting a shitty service.
I wonder if it'll be a more useful course than the one we had when Crowdstrike broke everything, or the many times when AWS US-East-1 has fallen over taking down many popular services.
The lesson most people learn from this sort of outage is that things bounce back afterwards, and the risk is minimal enough that a day out of every few years isn't enough of a problem to pay more.
especially if everyone else is also down with you
at least when us-east-1 goes down, Amazon stock prices go up (outages => stockholders remember that aws exists).
I think every N days a chaos monkey event should happen for every business to help with this kind of issue. If a SaaS takes your business offline, you aren't resilient.
https://netflix.github.io/chaosmonkey/
Props to anyone that can build a system mature enough to withstand a chaos monkey element running loose in prod. Brilliant.
We've moved on since then, the system is now built by a chaos monkey running loose in prod.
No, that was a couple of years ago.
Now we have hordes of chaos monkeys running loose in prod managed by supervisor chaos monkeys who write the code running in prod.
This is a good argument for encouraging employee vacations as well. Makes a more resilient company when nodes occasionally turn off.
It's 2036, every few months we boot up the Microsoft and quickly take as many pictures as we can of sheets, docs, slide shows, work schedules, support questions, we run payroll etc The AI is the only one left who can still log in using a combination of free trials and an intern admin account from someone in India we've long since lost track of. Rumor has it he now lives alone in the old temple at the top of the mountain.
That ship sailed 20 years ago when everybody stampeded into cloud computing / SaaS ago and it became heresy to suggest on-prem had benefits. "Redmond" is a symptom but the problem is "live by the cloud, die by the cloud."
When you're a monopoly, you don't have to worry about performance. Nobody is going anywhere, they'll just wait.
The postmortem on this will be interesting. A few days prior Microsoft said several versions of on premise Exchange are permanently vulnerable.
Maybe AI scanning found these Exchange exploits and after laying off so many developers, they vibe coded an exploit fix and deployed it. But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
Just speculation though
To add to the speculation. The error messages we were observing along with the Microsoft status updates read like connection pool exhaustion. The ticket update history started with Microsoft thinking a 'restart of affected servers' fixed it and troubleshooting 'health check failures'.
Anyone who has seen connection leaks/pool exhaustion in practice, it looks like an otherwise healthy low utilization server right up until it stops replying, and a restart appears to fix it until it runs into the limit again. Easy to think that your health checks are the problem. Per MS updates, they rabbit holed on this issue and built 'an alternative health check method' which worked in testing (i.e. low number of test connections); but when they tried to apply this change, servers would go unresponsive anyway.
My guess is a rushed update with a connection leak somewhere in the stack. A call that opens, but does not close. Just the kind of thing that AI can miss because it's only trying to hit the goal and not necessarily clean up after itself.
> But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?
Exchange and Active Directory are easily my favorite MSFT products. I've always wished they would port them to Linux, even though I know that OS dependencies make that a non-starter (though they surprised me w/ SQL Server for Linux).
I'll concede that the the biggest Exchange environment I ever ran was around 1,000 mailboxes. I've never seen how it fails in really large environments. I was also fortunate that I only ran the earlier versions (pre-2013) in sub-500 mailbox environments. In those settings, though, I found it worked well and wasn't bad to administer.
It was very reasonable to use in small business environments, too (sub-100 mailboxes). I put a lot of little Exchange 2003 boxes out in small offices. It plugged-away very effectively.
Yes-- there were updates here-and-there that had issues, for sure. By-and-large it was a stable and predictable product.
I always tried to be mindful of maintaining free disk space, following MSFT's recommendations for speeds-and-feeds for I/O, and running on solid hardware (ECC memory, redundant power, UPS). I feel like I would have had more problems if I'd scrimped on that stuff, for sure, but I was unwilling to let Customers cheap-out.
With Exchange 2019 and high-availability (database availability groups) I was easily able to do zero-downtime maintenance. That "big for me" Exchange Customer I worked with had more downtime in the last 24 hours than they did the entire time they used Exchange 2019 in production.
I haven't seen the code, so I can't speak to the "nightmare of code". Large parts of it were ported to the .NET Framework in between the 2010 and 2013 releases. The 2013, 2016, 2019, and current Subscription Edition releases appear to be an evolution of the codebase with no signs of major rewrites. I question how much pre-2013 code is even present in the codebase anymore.
I dunno. My experience just hasn't been that bad. I don't want to be an apologist for MSFT. I've never fully gotten over the suspicion I have for their motives going all the way back to the Halloween documents. I don't like how they're forcing everyone to hosted offerings and killing their on-prem products. I've been pleased with the value my Customers realized from Exchange.
> Exchange and Active Directory are easily my favorite MSFT products. I've always wished they would port them to Linux […]
How is Samba 4's AD 'emulation' these days?
You know, I still haven't tried it. That's been on my to-do list for awhile now. I really like Samba (and used it as my homelab CIFS server for years) but I never explored the AD Domain Controller functionality.
It's pretty good now, although, like all open source projects we're being hit with a tsunami of AI reported bugs right now.
I definitely need to give it a shot. I'd love to get rid of Windows Server at home and use it as PoC for doing the same at Customer sites.
I wish I could live w/o Active Directory but, even at home, my threshold for Windows machines I'll tolerate administering w/o Group Policy is "any more than zero".
BTW, assuming you're who your HN username makes me think you are: Thanks for all your work on Samba. I've used the hell out of it at home and with Customers for at least two decades.
> I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?
yup. You're the only one.
When we ran our own mail servers, "some fuckery with client's Exchange server config" was good 90% of mail problems we had.
And my experience with the cloud version management decided we will migrate to has been entirely awful too, simple stuff like "stop spamming the mailing list I get constantly" is entirely impossible to set up, despise both global and per mailbox rules to let it thru, it still lands in spam and the useless support we apparently pay for can't figure out why
> When we ran our own mail servers, "some fuckery with client's Exchange server config" was good 90% of mail problems we had.
As an Exchange admin (and Postfix, Sendmail, qmail, exim) that was my experience, too. I think the popularity of the product, questionable defaults, and the low barrier to entry for admins (read: lacking in experience and knowledge) contributed to that.
Managing it in small and large corporate enviornments for over a decade led me to believe that when it was working, Exchange was very stable and reliable.. But when things would break, they would break in a way that required extensive knowledge of all the areas to fix. Which was a form of job security so Im not complaining.
I guess past reliability might be irrelevent here if an AI slop patch for AI discovered exploits caused the outage. Just speculation though. Since exchange has so many moving parts and so much legacy code, with even more bolted on to make it a cloud service, I could see how an AI exploit fix would not have all the context needed.
There is no information that it was caused by Exchange either, guess we have to wait on those answers. It is just a reasonable assumption.
Open source options tend to require a bunch of knowledge to set up but after that you generally are equipped with that knowledge to solve any configration issue. I wrote "configuration issue" because aside from "ran out of disk space" none of them really "broke", so it was never "fixing the thing", just the usual mail server admin stuff of figuring out why this or that server didn't like us.
A lot of the bad situations I helped bail other contractors out of, when it came to Exchange, related to disk full conditions, a lack of understanding how transaction logging in the database worked, and database inconsistencies brought on by hardware RAID controller issues or hard power-offs. Very nearly all of my ugly ESEUTIL and ISINTEG experience came from people not respecting the product's storage needs.
Sysadmins who were used to running file servers and thought Exchange was the same thing were the problem. A DBA was actually more apt to understand the product's storage needs because it wasn't wholly dissimilar from the RDBMS they supported.
Why bother with a postmortem? Doesn't seem likely they'll fix the root cause.
> The postmortem on this will be interesting.
Any bets on whether it's DNS?
Well, my home computer with ddns lost network connection at the exact same time and I had to reboot it for it to work so, maybe?
my bet is on some copilot integration
It's never DNS
That's just what big domain wants you to think. But we know better.
DNS, IPV6 or BGP. It’s always those guys
Certificates expiring
...given it is windows environment... maybe
Back in 2011 various email worms decimated Exchange mail servers. It cost billions of dollars in productivity and the company I was working for was on lotus notes but migrating to Exchange. Did taking down systems all around the world stop them from migrating? Of course not. The Microsoft Stockholm syndrome is real.
Microsoft says "while this took longer..." [1] but they should have said "while this is taking longer" because the outage isn't fixed yet. What kind of outage takes 2 days to fix?
[1] https://x.com/Microsoft/status/2121282013600907325
"The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.
Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration."
Microsoft: "I've Fallen and I Can't Get Up"
All the CEO's AI and All the CEO's Men / Couldn't Put Microsoft Together Again
Only 107 points? HN apparently finds the MS fiasco about as interesting as an HP printer that suddenly stopped working.
Lots of big news items these past two days. I suspect most useful takes on the subject were shared yesterday.
It’s pretty crazy how much Microsoft can get away with. Almost a government level of getawayability.
naa .. it's totally fine. The only thing that will move consciences will be data loss/leak at scale. But even in that case I have some doubts ...
And, if eventually a big bunch of CTOs starts considering going back so self-hosting/managing people will pop up everywhere saying "even if <big corp> sucks, you cannot do better" or "it will cost you more" or "what about the peak usage".
It's incredible how we ended up paying much more, giving voluntarily all our data, and getting a shitty service.
Not really. Every time a giant cloud provider goes down it reminds us a that it used to be our problem.
“Aren’t you glad it’s them and not us?”