The experts were correct. Azure is the biggest pile of shit I've ever had to work with. Everything feels evolutionary. In other words, a new product in azure is barely a product at all, but a small appendage which totally inherits a bunch of preexisting Azure "stuff." And all this preexisting stuff may not really make sense for the product, and it might inherit stuff that makes the product much worse. But, it doesn't matter. To even think about using the product, you need to learn way more about the larger Azure ecosystem than you ever bargained for, and of course deal with Microsoft products that do not really integrate well because the teams don't talk to each other. Log formats, conventions, everything will be different as you float around to different parts of Azure. Basic security concepts, such as a SIEM will be implemented in such strange ways that you wonder if Microsoft has any idea what a SIEM even is.
As a Microsoftie of more than a decade... Yeah, I see this.
We have an internal system called Cosmos[0] that does a great job of processing huge quantities of data very fast. And we sat on it for years while the rest of the industry moved to Spark and its derivatives. We finally released it as Azure Data Lake Analytics (ADLA) but did a shit job of supporting/promoting it.
We built Synapse, and it's garbage. We've now got Fabric which I guess is the new Synapse. I wouldn't really know because I probably have five different systems that I use that basically do large-scale data processing, and yet Fabric isn't one of them; who knows, maybe it will become the sixth?
We've had numerous internal systems for orchestrating jobs, and it wasn't until Azure Data Factory that we finally released something externally that we sort-of-kind-of-but-not-really use internally. (To be fair, some teams do use it internally, but we're not all rowing in the same direction.)
I regularly deal with multiple environments with different levels of isolation for security. I don't even know how it's all supposed to work -- I have my regular laptop and a secure workstation and three accounts that work on the two. Yet I have to do some privileged account escalation to activate these roles; when I'm done, there's no apparent way to end the activation early, so I just let it time out.
These things are but a fraction of the Azure offerings, but literally everything I have used in Azure makes me absolutely HATE working in the cloud. There's not a single bright side to it AFAICT. As best as I can tell, the only reason why Azure makes so much damn money is because Microsoft is huge and can leverage its size into growth. We're very much failing up here.
This is the story of Microsoft - five different ways to do the thing, none of which do everything, and all of which are in various states of disrepair ranging from outright deprecation on up through feature-incomplete preview. Which one do you use? Who knows, but by the time you get everything moved over to that one and make allowances for all the stuff the one you chose doesn't support, there will be a new more logical choice for "that one" and you'll have to start over again. Wheee.
We are now definitely doing a lot of that. My manager has been saying things like, "I don't even know how it works, but I used AI to build [thing], and I just sent it to a PR." He's very strong technically, but the mindset has absolutely shifted to, "move fast and break things, yoloooooo". It's frustrating to say the least.
I don't mean this as a jab, but would you use Windows to develop software? Especially Windows that has AD teeth sunk into it where everything is "managed by your organization." It's just a thousand small cuts for seemingly no good reason.
No, but I also wouldn't let people who do not understand the soul of the OS to rewrite it.
If I were the microslop god for 6 weeks, I would force everyone to go to a boot camp and use Windows 7 for 4 of those weeks so they could see what made it so good.
No invasiveness, an OS that felt like yours. Just enough eye candy to not be distracting but to also feel like a clean modern system. Low system usage at idle. Calm, clean, and ready to roll when you clicked a button.
Windows is NEVER going to be MacOS, but the dev teams seem obsessed with macifying windows while also wedging that AI abomination copilot into every line of code, so windows is getting a tag team of rapid enshittification on top of already having been massively enshittified, and at least some portion of it is due to the people being paid to make it not understanding what it is supposed to be, the niche it held, and the reason for windows existence.
So the internal Cosmos DB has nothing to do with Cosmos DB the Azure product, which was an unwieldy assemblage of a graph DB, a NoSQL DB, a time series DB and an RDBMS last time I looked at it, but seems to have morphed into a "vector DB for AI" according to today's marketing?
Ugh this sounds like when I worked at Oracle/OCI. Some environments required a VPN, some a jumpbox, and some required logging into a virtual desktop, and then logging into a jumpbox. Just thinking about it gives me PTSD
All the corporate stuff is behind Okta, so that easy enough.
But all the dev/test systems are a mix of SSO, individual logins, etc. At least they're all behind the same VPN (except when they aren't, but that's less common).
And of course, if you're a cloud engineer (vs "normal" software engineer), you also have to deal with AWS access, which is a whole different can of worms.
And yet, somehow AWS managed to get this right-ish. They evolved, learned by making mistakes, and created de-facto standards (like object storage protocol) on the way, while at the same time supporting decades-old services. And I'm sure they'll withstand the current AI craze.
Their support team likes to sit on things for a while too.
I'm on day 4 of waiting for Azure to approve my support request to increase Azure Batch vCPUs from default of 4 to 20 for ESv3 series. I signed up last week and converted to a paid account. I'm going to use Google Cloud Batch today instead.
That's total "normal" for Microsoft at least from 2018, the year I started working with some of their products (Power BI mostly). They adopted a development model that is early release, fast iteration, and users as testers. No wonder everything feels experimental until much later.
Back then I just couldn't use Power BI. But fast forward a few years, I think it got a lot better since maybe 2020. You just have to stick with it for a few years.
Most of the time it's just part of the bundle. If you are heavy into SQL Server, Office 365 and Power BI then there is a BIG chance you are going to use Azure for whatever the reason.
People who take Azure up without previous MS product experience...not sure about those.
There's a few, mostly retailers who don't want to give money to Amazon as a direct competitor, for them Microsoft/Azure is more of a neutral party, and most businesses already use Microsoft in at least some fashion so already have staff internally familiar with MS products (as opposed to say, going to GCP instead).
For everyone else, it's like you said. "Eh, we are already knee deep in the Microsoft stack, why would we pick anything else?"
a LOT of stuff comes for free or marginal (10-100$ a month) so yes, you do pay but it's already 'baked into' the contracts people generally carry with microsoft, or something for IT to worry about when the yearly renewals show up
I worked at a hospital in that timeframe and they rolled out Teams. Up until they, shadow IT teams were running Slack just fine.
Man, what a horrendous pile of crap Teams was back then. The Slack teams were griping that they should just buy Slack, but Teams was the "enterprise solution." The problems were amplified during remote COVID work. Teams is fine now, but how many corporations went through years of frustration just because some IT decision maker said "Teams. Because it's enterprise."
Yeah that's the thing. Management who made the deals are never put into that frustration, or very rarely, and I always wonder, at least for the big corporations, if there is any greasy palms...
Teams is still a horrendous pile of crap. It's just that you've gotten used to the stench. It has few redeeming qualities other than, "we don't have to pay for another subscription" and that's not even the case in the EU.
Yeah but today you can at least have a video call more or less normally. Back then it was a hiccup after a hiccup, it was impossible to work normally, and yet orgs pushed it down everybody's throats as it was bundled.
Definitely. Besides the performance issues, back then, Teams barely had any features. One example was that it wouldn't show you who was talking. First time we had a call was with 30 people and I remember a manager calling out a director responsible for this decision jokingly saying, "and you don't know who I am because Team doesn't show you who's talking."
The UI is an overengineered mess and I'd rather use literally anything else, but to say it's still unusable is disingenuous.
Absolute contempt for their users at every level. It’s so transparent. This is the end game of anticompetitive practices for decades— they just don’t have to try anymore… for now. Some day they’ll either have to compete in good faith or sink. I doubt that will happen soon, but someday.
It's hard to argue against contempt but... I'm gonna try. It feels like at the end of the line it's just a checkbox someone gets without having to consider the consequences of the changes. Either it's too big or there's too many levels where decisions get made and handed down to drones (or AI), but the people who decide seem to have no concept of what their products are used for and the people who implement features seem to have accepted that the system is so big that they can't understand all the impacts of their changes and have to rely on trusting commands from above - who may expect them to challenge from the POV of users or question things but never do. Anyway, this feels like what happens when managerial overhead and marketing KPIs smash into a complex product ecosystem. It all smells of IBM to be honest
Microsoft was always afraid of being IBM. They are more IBM than IBM.
When they started flying people in the beg that I buy 100 Surface Laptops, that was the confirmation of everything I had been thinking. All I could think of was IBM flying a dude from Italy in to talk for 15 minutes about their version of TeamViewer back in the day. We ended up talking about shoes.
Which is sad because the CEO's job is not to focus on the individual body parts but to make sure that the whole system is strong, beautiful, and healthy.
They can afford people who would do better. Windows 11 is trash. Azure is trash. Onedrive is trash. Outlook is trashier than it has ever been before, but it's not quite trash yet. Word is trash. Excel is rapidly enshittifying. Copilot is hot flaming radioactive tar cancer.
Does microslop even have a single thing left that isn't either completely terrible or worse than it used to be a mere 5 years ago?
Azure is the color of the face you have after Microsoft beats you with your own wallet. They don’t want to give you access to anything, they want to own it and make you pay for it.
I’ve seen this in other “follow the leader” businesses too, they are not looking to even have working features, just parity on a spreadsheet with the market leader… I’m looking at you Gitlab.
I sometimes wonder if I would feel the same about AWS if I hadn’t already invested a significant amount of time learning the entire ecosystem, nomenclatures, patterns/best practices, etc.
As someone who has worked with all three in many capacities, as is the worst by a mile. Don’t get me wrong. They are all very bad, but Azure is the king of shit.
And the same applies to regions. Try running is most of the regions, each is a bit different. And its not historical / sequential differences, just random.
How is this different than Amazon? Same problem there. Oh, you're using this new service? Need to view the logs? Want a nice friendly UI to do that? Fuck you here's Cloudwatch. Good luck.
Just to be clear, I'm responding to the parent comment not the article.
That's great but that's not really the problem. The real problem is Amazon likes to release services that depend on other services, but leave the integration work to us.
I'm convinced Amazon has many teams crapping out new features but they don't have the political clout (or manpower) to create a comprehensive product. They are mandated by management to use existing services, and thus we the users suffer because we have to manage all this extra crap and noise just to enable basic functionality.
It's maddening. And then also it's maddening to see another service from a different team that was able to throw off these shackles and actually make a product that is self contained. You get a taste of how good things could be, and then you're thrown right back into the IAM/SQS/Cloudwatch/Cloudformation/Policy/everything else under the sun soup.
See my other comment. Logs are just one small symptom of a larger problem of poorly integrated very complex services where the complexity is pushed onto the users and not properly managed by Amazon. Which sounds very much like the problems with Azure.
> [...]And because federal agencies were allowed to deploy the product during the review, GCC High spread across the government as well as the defense industry. By late 2024, FedRAMP reviewers concluded that they had little choice but to authorize the technology — not because their questions had been answered or their review was complete, but largely on the grounds that Microsoft’s product was already being used across Washington.
This sounds like the crux of the issue. The combination of: "tool can be used during analysis" and "analysis takes long" shifts the barrier of rejection from "is this tool safe?" to "is this tool so unsafe that we're willing to start a fight with a lot of other government agencies to remove it, find an alternative, etc?".
Not criticizing FedRAMP. Proper security review takes time. And probably more when dealing with vendors.
As far as I know numbers aren't reported, but there's probably at least as many DIB GCC-H customers as government, who in part use it because the government does and it's compliant. Once they're locked in it's very hard to migrate.
Recently tried using Entra ID. There are 12 ways to enforce MFA, 20 days ways to disable users, 4 ways to authenticate users, Add conditional access stuff with 50 variables and templates etc.
You can customize the way you want. After configuring it, my colleagues could not log in. Thats one way to secure your organization.
Out of all the SSO login flows Microsoft has to have the buggiest. It’s the only one I can remember routinely having issues with. Why are there so many redirects? And why doesn’t the “remember me” checkbox ever work?
It is also the only SSO flow I have ever seen that fundamentally cannot work if you have more than one account remembered on your device. So far the only way I’ve found to get it to let you log out of account A and then log into account B is to clear all cookies otherwise it gives you permission denied errors. Have no idea how it can be this horrible
I just run completely separate browser profiles to separate work and personal stuff. And I still sometimes need private mode or a throwaway profile to get some random thing to work.
And then sometimes the "switch user" prompt doesn't work but it automatically logs you in with the wrong account to a system that account doesn't have access to, then drops you in a non-interactive "you're not authorized" screen. You have to find a working page, log out, then go back and try logging in...
I haven't seen it in a while (perhaps mostly because I'm in Google stuff way less than I used to be) but for years multiple Google sites would get in a state where its auth would route me through about twenty redirects in a loop and never actually finish authenticating me. Clearing cookies and re-logging-in from scratch was the only fix.
Youtube was always involved, somehow, for some reason, even when what I was doing wasn't connected to Youtube at all or the account I was using had never even been intentionally used with Youtube. It'd route me through a few Youtube domain names.
(Microsoft's is indeed even worse, on some of theirs [Azure Devops, looking at you] I can't use them in pinned tabs because somehow they manage to get into a totally broken state where the page won't load due to whatever's happening with their auth flow in the background, and no method of reloading the tab fixes it, and it does this every couple days—but copy-pasting the same URL to a new tab does work)
Exactly! I can’t even count the number of times we’ve been in the discovery phase of a project and see “Oh this MS product does that! Cool”. Then when we get to the actual implementation realize it’s a broken mess. It’s sales driven software development, they just need to get you far enough along to sign the contract, then it’s too late to back out.
The problem is modern MS doing three contradictory things at the same time:
- FB's move fast and break things. Constantly launching new libs.
- Linus's we do not break user space. Great commitment to backwards compatibility.
- Never deprecating dead products until they've been de facto abandoned for like decades.
This combination means every MS product is a labyrinth of overlapping APIs with no guidance as to which one is actually the good one. Some are abandoned garbage, some are brand new and incomplete, and some are both, and there's no way of knowing which are which even experts can mislead you.
Well said. It feels like Microsoft is willing to release the intern’s poorly thought out product, and then commits to support the garbage design for all time.
Microsoft, you are a behemoth. There are few domains where you actually compete. Give your products a minute to breath before you cast them in stone.
same experience for us, and then they email the living shit out of you about how your weekly entra id stats are good or bad, and you can not opt out of these emails.
I remember trying to buy $9 worth of Minecraft In-app Whatever for my kid, and the goose chase Microsoft put me on just to log in and buy something was totally out of this world. I ended up needing to contact their fraud department around step 74.
I did it for my kids to have accounts and I do not understand how anyone who hasn't built a Gentoo from Stage 1 has a prayer of managing to buy Minecraft Java Edition for their kid, and making it actually work.
Then you've got the hell of overlapping permissions systems on the console and the Microsoft account, to get any amount of online play working on a console if you also get Bedrock. On the Playstation, especially, the error messages also love to not tell you which of the two systems is blocking you, so you get to guess. And Microsoft's site for managing those permissions is so confusingly-laid-out that even after doing it three times in a row I still felt lost on it.
I never did solve the problem of getting Minecraft Java Edition to run on a kid's MacBook with allowlist-only Web access. It wants to contact ten or so apparently-randomly-selected-from-an-enormous-pool IP addresses on every launch. I never did find documentation of which IP blocks I needed to allow, and couldn't guess at it from the IPs themselves. If they'd just used domain names... I must have manually hit "allow" a bunch of times during twenty separate launches, and it was still presenting me the same number of prompts every time, because there was no overlap in the IPs contacted (adding insult to injury is that I'm sure all but at-most two of these were spyware horse-shit that had no actual generously-necessary role in running the software, but it'd fail if it couldn't reach them)
Microsoft has never been good at security, and that is why their centralization to cloud is absolutely terrifying.
I'm reminded of Storm-0558 [1] where a stolen signing key was able to forge authentication tokens for any MSA / Azure AD / Government AD user. They downplayed the severity. Just imagine if that level of access was used to pull a Stryker on a nation-wide scale. That is an economic disaster waiting to happen.
I knew there was another incident that I was forgetting, insanity... I don't understand how Microsoft keeps getting away with this and everyone just forgets.
But it doesn't. Full authentication bypass exploits are extremely rare and unheard of among tech giants. Maybe account takeover/recovery, sure, but full bypass? It just never happens.
Microsoft goes beyond that: they've managed to have a critical vulnerability in almost every authentication product they have ever created. It's exceptional.
The Justice Department CIO who pressured FedRAMP to approve GCC High was hired by Microsoft the next year. I wonder if this shouldn't invalidate the authorization in the first place?
It's not very clear from the article, but I get the feeling from the context that the 'pile of shit' quote referenced the package of documentation about the service rather than the service itself.
(That seems to be the main complaint, that Microsoft never provided the clear information required to conduct the assessment properly).
> The tech giant’s “lack of proper detailed security documentation” left reviewers with a “lack of confidence in assessing the system’s overall security posture,” according to an internal government report reviewed by ProPublica.
>
> Or, as one member of the team put it: “The package is a pile of shit.”
Yes, it seems pretty clear from that quote that the reviewer said the security package was a `pile of shit`, and propublica went on to extend that to the cloud itself. Not that I want to comment on the merits of Azure's security, but that sounds pretty clickbaity from propublica to me. A more appropriate title would have been
> Federal Cyber Experts Thought Microsoft’s Cloud Security documentation Was “a Pile of Shit.”
MS was (and still is it seems) unable to produce the data flow diagrams that FedRAMP wanted, ones that other cloud providers had no problem with. If the documentation is in such dire state, then the system itself is likely to also be in a dire state. I.e. The documentation is a pile of shit, so the system is also a pile of shit.
Yes. US bureaucracy regularly gets told "You have to have <thing>" but because it's against a lot of people's ideology, they aren't allowed to build it internally or develop any sort of actual expertise for such a thing, so their only choice is to buy whatever is offered no matter how bad it is.
For example, our state government says "We will do X Y and Z which all require data science expertise, but we did not approve the $60k a year Data Science position, so instead we are forced to hire a Data Science contractor for $120k a year, and they can't really be fired, and they are terrible at their job"
And then people wonder why things suck all the time.
A lot of state's buy their Obamacare marketplace service from a company I am familiar with. That company is entirely incompetent. They cannot follow basic instructions. They cannot triage a bug at all. They do not read freaking tickets. They take weeks to respond to an issue. They cause bugs regularly in ways that imply they don't have functional source control. They continually fuck up basic feature requests. They change the service in ways that contravene the literal law. The law that was comprehensively explained to them by people I know.
But they can't be fired, because the state is legally compelled to provide this service, and is not really allowed to hire a few engineers to build it in house. They could go to a different software contractor, but all the options are just as bad because it's an entirely captured market.
Obama started a "Digital Services" group in the federal government to actually build systems internally and develop expertise to mitigate some of this, and they built stuff like tax filing solutions for free for Americans. So Trump killed it and hollowed out it's corpse for DOGE.
Emergency notifications are done the same way! Its communism to fucking build it, so let’s have a team of a few engineers make an API to control government infrastructure from incompetent contractors on AWS, offer no real means of testing, breaking changes, downtime… and folks wonder why Hawaii is told bombs are coming
EVERYTHING about the federal government contracts program sucks ass! In the beginning, it was good as you didn't want people forcing through their brother, mother, 2nd cousin, next door neighbor, Satya Nadella and their "company" as a contractor without oversight cough Kristi Noem cough. However, it has devolved into a mess. The entire thing needs to be scrapped and re-engineered.
To be fair, it's not always out of maliciousness. A lot of gov workers/contractors join the supplier company because they know the product and how to fix it better than the people currently at the company. Similar to the guy who infamously got hired at Apple just to fix a bug.
You're just forced to use vendors and if you actually care about the mission, it's just a different team on the same mission.
Of course you know you're being taken advantage of, and long-term maybe you should have gone to the non-technical side to fight it, but at the end of the day you just want to keep the young boys being shipped off to war safe, and you're much better suited to achieve that by remaining on the technical side.
I don't have much experience with Azure but I was amazed at how many things in AWS GovCloud don't meet FedRAMP encryption requirements. For example, none of the lambda runtimes have FIPS certified encryption libraries available, and you have to bring you own, which is rather complicated to do.
Was this approval before or after evaluators discovered this?
> Microsoft on Friday revised its practices to ensure that engineers in China no longer provide technical support to U.S. defense clients using the company’s cloud services.
> Potential Conflict of Interest: The government relies, in part, on third-party firms to vet cloud technology, but those firms are hired and paid by the company being assessed.
Hah. First time looking at FedRAMP?
The real reason for this, of course, is accounting, it moves it off of the government's books.
> These highlights were written by the reporters and editors who worked on this story.
It's unfortunate that people have to claim the authenticity, rather than the users of AI having to disclose use of AI/LLM. I wish it was the other way around.
Wow, Microsoft is really pushing the wrong boundaries in every direction, isn't it? Executives must be thinking, like many before them, that Microsoft is too big to fail.
Executives only react to share price movements. If share prices are high because whatever investors think, then execs will just open another champagne bottle.
Steve Jobs was the last tech CEO who didn't care about wall street and only care about quality products and consumers saying that if customers are happy, then the share price will take care of itself. But most companies are share price first, customer later.
This fits perfectly with traditional Microsoft strategies of getting a foot in the door and then having the users’ internal pressure on the organization to help get the Microsoft product established.
Decades ago, Lotus 1-2-3 on top of MSDOS was the lever; today it’s GCC High.
I think there's some context missing here. For those who don't remember, the CIA back in like 2014 or so built out private data centers with classified versions of AWS services and all IC workloads that don't require specialized hardware was supposed to be using. DOD historically used it as well for classified cloud workloads, but wanted its own, and this was the JEDI contract, which was also supposed to go to Amazon, until Trump got into a fight with Jeff Bezos in 2019, canceled the contract, and awarded it to Microsoft instead. Amazon sued, and Biden decided to just award the contract to everyone and split it between all the major cloud vendors. That still doesn't mean anyone can actually use it without FedRAMP approval, but well, there you go.
The alternative was AWS, which has been operating at every classification level for over a decade at this point. It's now split between Amazon, Microsoft, Oracle, and Google, which is especially amusing because Google withdrew from the original bid process when they were still pretending to give a shit that their employees don't like working for the military.
Microsoft has been selling piles of shit since the beginning of time. The fact that they keep selling is the biggest triumph of sales/marketing over decent engineering.
This is my opinion only, I'm sure some have had different experiences - but:
Azure's success as a cloud provider is mostly a result of their sales team and having an existing relationship with non-technical leadership. "We already pay them for Office and Exchange, let's just buy this new 'cloud' thing from them too".
Azure is barely considered an option at all within tech companies, yet is surprisingly widely adopted by non-technical companies that don't know any better (ie, that don't have a technical / engineering voice or representation within leadership).
AWS = Likely technically the best, for now. Mostly unreasonable pricing, and less motivation to seriously negotiate given they are the 'default' cloud provider for most of the industry. Kind of feels like they have peaked though, and are slipping more recently. Inevitable, or bad leadership changes?
OCI = New-comer, attractive pricing and hungry for business. Might be able to avoid mistakes other providers have made? Reliability struggles though. Parent company has a bad reputation in some circles - but probably not with decision makers. Making huge (unwise?) investments - that will either come crashing down in 5 years, or seriously pay off. Layoffs, but going for massive growth...huh?
GCP = Notably different underlying technical choices than other providers. Folks are maybe a bit less pragmatic, and more academic. This helps them in unique services (Spanner?) but hurts in most other areas. They've matured, and are btwn AWS and OCI in reliability. They are probably not as hungry for business as they should be given how far behind they are.
> By late 2024, FedRAMP reviewers concluded that they had little choice but to authorize the technology — not because their questions had been answered or their review was complete, but largely on the grounds that Microsoft’s product was already being used across Washington.
The article talks a lot about conflicts of interest, but this is the line I went looking for. A bureaucracy fighting itself over goal prioritization, and what's a necessary roadblock vs red tape is the less sexy but more meaningful problem at the core of this.
Once the government decided they wanted the product, they were going to find a patsy.
If you "went looking for" this line, you're just reading into the statements your preconceptions.
I on the other hand have no expectation, and so it's not clear whether the "bureaucracy fighting itself" is a cause or a symptom. You're implying it's a cause and the solution is "less red tape". But it could be just a symptom of conflicts of interest, and less red tape just leads to more efficient corruption.
Again, you're just reading into it what you already believe in.
Azure is easily the most expensive, least reliable and worst cloud available. It's borderline scam. An example today, I provisioned high IOPS SSDs (supposedly) and what is actually connected to the instance? A spinning hard drive! I didn't even know they were still made, but I guess Azure uses them and scams their users into thinking you're getting an SSD for $700/mo when its really an old hard drive.
I would warn anyone far and wide to avoid Azure at all costs, especially if you are a startup. And especially if you are doing any kind of AI because the only GPUs they have available are ancient and also crazy over-priced.
If I cared more, I'd try to migrate away from Azure. But I don't, and that's probably Azure's business model at this point.
I’d love to see proof of your claim that they provisioned a hard disk when you requested an SSD, or, at the very least, tests that showed that the IOPS you requested were not delivered. Can you show us the receipts?
Azure using SRE, I call BS. You don’t see underlying storage, it’s mounted as either SCSI or NVMe device as one HD. It’s obviously backed by massive fleet of drives just like EBS.
the product got deployed across the government while the security review was still in progress. then fedramp approved it because it was already everywhere. seem like i saw a lobbyist or two with a broom sweeping something under a rug...
Azure is bad. But to be fair, every security summary of IT services I’ve ever read — or written! — for over 25 years has also been a “pile of shit”. It seems to be inherent to the cybersecurity game that everything is judged based on meaningless check boxes and nonsensical explanations. Meanwhile the actual security posture is obscured and ignored.
Although "cyber" is a prefix used to denote a relationship to high technology there exist people who use it as a stand-alone term to mean "cybersecurity", e.g. "I work in cyber." It's very confusing given that it came from the word "cybernetic", the hundreds of other words that begin with it as well, and the existence of the term "cybersec" which is unambiguous and only a bit longer.
Microsoft can be abhorrent. They will always get the contracts. Why? Corporate welfare.
Microsoft will drive the rules. Why? Too big to fail.
Microsoft will push their slop. Why? Cause they have contractors after contractors in the federal government pushing MS solutions. Doesnt matter if they're bad.
And, who'd pay for a 3PAO audit of a Linux distro? Ubuntu and Redhat have. Its a $120k moat.
The government does most things poorly and with little regard to budget or quality. They can't solve problems that are much simpler than cloud computing, so why should I expect them to perform better at a more complex problem?
> The government does most things poorly and with little regard to budget or quality.
That's a common line by conservatives who are actively sabotaging government with policies and laws which they then point to as evidence of such inefficiencies.
Suddenly everyone on HN is an expert on Azure infrastructure.
it isn't the best but it's really great at a lot of things feature-wise. top-notch documentation as well (despite what these "experts" said).
Most companies literally run on Azure these days. Persistent hackers will get into any network, that's a guarantee, that's APT 101. It's law of averages. If it truly is "a pile of shit" given how it is probably the most used cloud platform by the most customers, including governments, and endless plethora of features and services it offers, shouldn't there be more compromises? 2-3 in a decade is hardly above what you expect for law of averages right?
Screw ups happen, but if it is systemic, you can't use one instance as evidence, you must establish a pattern of mishaps.
The experts were correct. Azure is the biggest pile of shit I've ever had to work with. Everything feels evolutionary. In other words, a new product in azure is barely a product at all, but a small appendage which totally inherits a bunch of preexisting Azure "stuff." And all this preexisting stuff may not really make sense for the product, and it might inherit stuff that makes the product much worse. But, it doesn't matter. To even think about using the product, you need to learn way more about the larger Azure ecosystem than you ever bargained for, and of course deal with Microsoft products that do not really integrate well because the teams don't talk to each other. Log formats, conventions, everything will be different as you float around to different parts of Azure. Basic security concepts, such as a SIEM will be implemented in such strange ways that you wonder if Microsoft has any idea what a SIEM even is.
As a Microsoftie of more than a decade... Yeah, I see this.
We have an internal system called Cosmos[0] that does a great job of processing huge quantities of data very fast. And we sat on it for years while the rest of the industry moved to Spark and its derivatives. We finally released it as Azure Data Lake Analytics (ADLA) but did a shit job of supporting/promoting it.
We built Synapse, and it's garbage. We've now got Fabric which I guess is the new Synapse. I wouldn't really know because I probably have five different systems that I use that basically do large-scale data processing, and yet Fabric isn't one of them; who knows, maybe it will become the sixth?
We've had numerous internal systems for orchestrating jobs, and it wasn't until Azure Data Factory that we finally released something externally that we sort-of-kind-of-but-not-really use internally. (To be fair, some teams do use it internally, but we're not all rowing in the same direction.)
I regularly deal with multiple environments with different levels of isolation for security. I don't even know how it's all supposed to work -- I have my regular laptop and a secure workstation and three accounts that work on the two. Yet I have to do some privileged account escalation to activate these roles; when I'm done, there's no apparent way to end the activation early, so I just let it time out.
These things are but a fraction of the Azure offerings, but literally everything I have used in Azure makes me absolutely HATE working in the cloud. There's not a single bright side to it AFAICT. As best as I can tell, the only reason why Azure makes so much damn money is because Microsoft is huge and can leverage its size into growth. We're very much failing up here.
[0] https://www.microsoft.com/en-us/research/publication/big-dat...
> I probably have five different systems
This is the story of Microsoft - five different ways to do the thing, none of which do everything, and all of which are in various states of disrepair ranging from outright deprecation on up through feature-incomplete preview. Which one do you use? Who knows, but by the time you get everything moved over to that one and make allowances for all the stuff the one you chose doesn't support, there will be a new more logical choice for "that one" and you'll have to start over again. Wheee.
And now slap widespread vibe coding and PRs that reviewed by LLMs without anyone giving it a proper look.
We are now definitely doing a lot of that. My manager has been saying things like, "I don't even know how it works, but I used AI to build [thing], and I just sent it to a PR." He's very strong technically, but the mindset has absolutely shifted to, "move fast and break things, yoloooooo". It's frustrating to say the least.
And most of that is done on Macbooks by people that either can not or will not use Windows OS.
Wait, is this true? I would have imagined unless it’s about porting software or testing it, everyone would be forced to use Windows.
If it is true, wonder what the proportion is then: 25%, 50%?
It's not true. Source - me, MSFT for 25 years.
I don't mean this as a jab, but would you use Windows to develop software? Especially Windows that has AD teeth sunk into it where everything is "managed by your organization." It's just a thousand small cuts for seemingly no good reason.
No, but I also wouldn't let people who do not understand the soul of the OS to rewrite it.
If I were the microslop god for 6 weeks, I would force everyone to go to a boot camp and use Windows 7 for 4 of those weeks so they could see what made it so good.
No invasiveness, an OS that felt like yours. Just enough eye candy to not be distracting but to also feel like a clean modern system. Low system usage at idle. Calm, clean, and ready to roll when you clicked a button.
Windows is NEVER going to be MacOS, but the dev teams seem obsessed with macifying windows while also wedging that AI abomination copilot into every line of code, so windows is getting a tag team of rapid enshittification on top of already having been massively enshittified, and at least some portion of it is due to the people being paid to make it not understanding what it is supposed to be, the niche it held, and the reason for windows existence.
With no soul, windows has to go.
So the internal Cosmos DB has nothing to do with Cosmos DB the Azure product, which was an unwieldy assemblage of a graph DB, a NoSQL DB, a time series DB and an RDBMS last time I looked at it, but seems to have morphed into a "vector DB for AI" according to today's marketing?
https://azure.microsoft.com/en-us/products/cosmos-db
Ugh this sounds like when I worked at Oracle/OCI. Some environments required a VPN, some a jumpbox, and some required logging into a virtual desktop, and then logging into a jumpbox. Just thinking about it gives me PTSD
any sufficiently large organization that is around for a decade or two trends towards spaghetti-access
Yup, same boat here (mid-size company).
All the corporate stuff is behind Okta, so that easy enough.
But all the dev/test systems are a mix of SSO, individual logins, etc. At least they're all behind the same VPN (except when they aren't, but that's less common).
And of course, if you're a cloud engineer (vs "normal" software engineer), you also have to deal with AWS access, which is a whole different can of worms.
And yet, somehow AWS managed to get this right-ish. They evolved, learned by making mistakes, and created de-facto standards (like object storage protocol) on the way, while at the same time supporting decades-old services. And I'm sure they'll withstand the current AI craze.
AWS had the benefit of not trying to retrofit IaaS on top of a (already bad) PaaS.
So the problem is the team size, not culture?
Their support team likes to sit on things for a while too. I'm on day 4 of waiting for Azure to approve my support request to increase Azure Batch vCPUs from default of 4 to 20 for ESv3 series. I signed up last week and converted to a paid account. I'm going to use Google Cloud Batch today instead.
> Everything feels evolutionary.
That's total "normal" for Microsoft at least from 2018, the year I started working with some of their products (Power BI mostly). They adopted a development model that is early release, fast iteration, and users as testers. No wonder everything feels experimental until much later.
Back then I just couldn't use Power BI. But fast forward a few years, I think it got a lot better since maybe 2020. You just have to stick with it for a few years.
> You just have to stick with it for a few years.
So, you have to be a paying tester? Incredible that MS can keep enough businesses as hostage to be able to operate like that.
Most of the time it's just part of the bundle. If you are heavy into SQL Server, Office 365 and Power BI then there is a BIG chance you are going to use Azure for whatever the reason.
People who take Azure up without previous MS product experience...not sure about those.
There's a few, mostly retailers who don't want to give money to Amazon as a direct competitor, for them Microsoft/Azure is more of a neutral party, and most businesses already use Microsoft in at least some fashion so already have staff internally familiar with MS products (as opposed to say, going to GCP instead).
For everyone else, it's like you said. "Eh, we are already knee deep in the Microsoft stack, why would we pick anything else?"
a LOT of stuff comes for free or marginal (10-100$ a month) so yes, you do pay but it's already 'baked into' the contracts people generally carry with microsoft, or something for IT to worry about when the yearly renewals show up
I worked at a hospital in that timeframe and they rolled out Teams. Up until they, shadow IT teams were running Slack just fine.
Man, what a horrendous pile of crap Teams was back then. The Slack teams were griping that they should just buy Slack, but Teams was the "enterprise solution." The problems were amplified during remote COVID work. Teams is fine now, but how many corporations went through years of frustration just because some IT decision maker said "Teams. Because it's enterprise."
Yeah that's the thing. Management who made the deals are never put into that frustration, or very rarely, and I always wonder, at least for the big corporations, if there is any greasy palms...
Manager humans will sell out your workflow,
and indeed your entire workplace,
for as little as a steak dinner.
Teams is still a horrendous pile of crap. It's just that you've gotten used to the stench. It has few redeeming qualities other than, "we don't have to pay for another subscription" and that's not even the case in the EU.
Yeah but today you can at least have a video call more or less normally. Back then it was a hiccup after a hiccup, it was impossible to work normally, and yet orgs pushed it down everybody's throats as it was bundled.
Definitely. Besides the performance issues, back then, Teams barely had any features. One example was that it wouldn't show you who was talking. First time we had a call was with 30 people and I remember a manager calling out a director responsible for this decision jokingly saying, "and you don't know who I am because Team doesn't show you who's talking."
The UI is an overengineered mess and I'd rather use literally anything else, but to say it's still unusable is disingenuous.
> You just have to stick with it for a few years.
Also see: SharePoint
Absolute contempt for their users at every level. It’s so transparent. This is the end game of anticompetitive practices for decades— they just don’t have to try anymore… for now. Some day they’ll either have to compete in good faith or sink. I doubt that will happen soon, but someday.
It's hard to argue against contempt but... I'm gonna try. It feels like at the end of the line it's just a checkbox someone gets without having to consider the consequences of the changes. Either it's too big or there's too many levels where decisions get made and handed down to drones (or AI), but the people who decide seem to have no concept of what their products are used for and the people who implement features seem to have accepted that the system is so big that they can't understand all the impacts of their changes and have to rely on trusting commands from above - who may expect them to challenge from the POV of users or question things but never do. Anyway, this feels like what happens when managerial overhead and marketing KPIs smash into a complex product ecosystem. It all smells of IBM to be honest
Microsoft was always afraid of being IBM. They are more IBM than IBM.
When they started flying people in the beg that I buy 100 Surface Laptops, that was the confirmation of everything I had been thinking. All I could think of was IBM flying a dude from Italy in to talk for 15 minutes about their version of TeamViewer back in the day. We ended up talking about shoes.
It's a shame. In the late 2010s there was a lot of hope for Satya Nadella, but it seems like the organization has regressed back to the mean.
Which is sad because the CEO's job is not to focus on the individual body parts but to make sure that the whole system is strong, beautiful, and healthy.
They can afford people who would do better. Windows 11 is trash. Azure is trash. Onedrive is trash. Outlook is trashier than it has ever been before, but it's not quite trash yet. Word is trash. Excel is rapidly enshittifying. Copilot is hot flaming radioactive tar cancer.
Does microslop even have a single thing left that isn't either completely terrible or worse than it used to be a mere 5 years ago?
If not contempt, at least disregard or indifference
They have to compete in good faith for developers, which is why VS Code does not suck.
But yes, normal Office users, where the company pays the bills, pay the price.
Azure is the color of the face you have after Microsoft beats you with your own wallet. They don’t want to give you access to anything, they want to own it and make you pay for it.
I’ve seen this in other “follow the leader” businesses too, they are not looking to even have working features, just parity on a spreadsheet with the market leader… I’m looking at you Gitlab.
I sometimes wonder if I would feel the same about AWS if I hadn’t already invested a significant amount of time learning the entire ecosystem, nomenclatures, patterns/best practices, etc.
As someone who has worked with all three in many capacities, as is the worst by a mile. Don’t get me wrong. They are all very bad, but Azure is the king of shit.
And the same applies to regions. Try running is most of the regions, each is a bit different. And its not historical / sequential differences, just random.
How is this different than Amazon? Same problem there. Oh, you're using this new service? Need to view the logs? Want a nice friendly UI to do that? Fuck you here's Cloudwatch. Good luck.
Just to be clear, I'm responding to the parent comment not the article.
I love https://github.com/lucagrulla/cw , it's like tail for cloudwatch. It's super fast.
That's great but that's not really the problem. The real problem is Amazon likes to release services that depend on other services, but leave the integration work to us.
I'm convinced Amazon has many teams crapping out new features but they don't have the political clout (or manpower) to create a comprehensive product. They are mandated by management to use existing services, and thus we the users suffer because we have to manage all this extra crap and noise just to enable basic functionality.
It's maddening. And then also it's maddening to see another service from a different team that was able to throw off these shackles and actually make a product that is self contained. You get a taste of how good things could be, and then you're thrown right back into the IAM/SQS/Cloudwatch/Cloudformation/Policy/everything else under the sun soup.
Amazon is selling servers and storage. If you need to see logs properly, then get a right tool for it. Cloudwatch is a stop gap solution.
See my other comment. Logs are just one small symptom of a larger problem of poorly integrated very complex services where the complexity is pushed onto the users and not properly managed by Amazon. Which sounds very much like the problems with Azure.
My general approach is to only use the most basic services from each cloud. VMs, networks, L3 load balancers, blob storage, etc
Build the rest yourself. In many cases their higher level service is just the same open source package you would run, just managed worse.
this. with Kubernetes, you can get very far with just this and you won't have to deal with lock in BS either
> [...]And because federal agencies were allowed to deploy the product during the review, GCC High spread across the government as well as the defense industry. By late 2024, FedRAMP reviewers concluded that they had little choice but to authorize the technology — not because their questions had been answered or their review was complete, but largely on the grounds that Microsoft’s product was already being used across Washington.
This sounds like the crux of the issue. The combination of: "tool can be used during analysis" and "analysis takes long" shifts the barrier of rejection from "is this tool safe?" to "is this tool so unsafe that we're willing to start a fight with a lot of other government agencies to remove it, find an alternative, etc?".
Not criticizing FedRAMP. Proper security review takes time. And probably more when dealing with vendors.
It's why these enterprise vendors want foot in the door at all costs.
They know that if they get entrenched first, it's impossible to migrate away. That's basically free money from a customer that has zero cost ceiling.
That's false that Government agencies have 0 cost ceiling. Maybe DoD does, but most offices have extremely tight budgets.
As far as I know numbers aren't reported, but there's probably at least as many DIB GCC-H customers as government, who in part use it because the government does and it's compliant. Once they're locked in it's very hard to migrate.
Recently tried using Entra ID. There are 12 ways to enforce MFA, 20 days ways to disable users, 4 ways to authenticate users, Add conditional access stuff with 50 variables and templates etc.
You can customize the way you want. After configuring it, my colleagues could not log in. Thats one way to secure your organization.
Out of all the SSO login flows Microsoft has to have the buggiest. It’s the only one I can remember routinely having issues with. Why are there so many redirects? And why doesn’t the “remember me” checkbox ever work?
It is also the only SSO flow I have ever seen that fundamentally cannot work if you have more than one account remembered on your device. So far the only way I’ve found to get it to let you log out of account A and then log into account B is to clear all cookies otherwise it gives you permission denied errors. Have no idea how it can be this horrible
Would container tabs solve that? They're pitched as helping separate work and personal logins.
I just run completely separate browser profiles to separate work and personal stuff. And I still sometimes need private mode or a throwaway profile to get some random thing to work.
Yeah I have had this experience too. Woe betide ye if your company gets bought by another company with pre-existing Azure AD.
And then sometimes the "switch user" prompt doesn't work but it automatically logs you in with the wrong account to a system that account doesn't have access to, then drops you in a non-interactive "you're not authorized" screen. You have to find a working page, log out, then go back and try logging in...
I haven't seen it in a while (perhaps mostly because I'm in Google stuff way less than I used to be) but for years multiple Google sites would get in a state where its auth would route me through about twenty redirects in a loop and never actually finish authenticating me. Clearing cookies and re-logging-in from scratch was the only fix.
Youtube was always involved, somehow, for some reason, even when what I was doing wasn't connected to Youtube at all or the account I was using had never even been intentionally used with Youtube. It'd route me through a few Youtube domain names.
(Microsoft's is indeed even worse, on some of theirs [Azure Devops, looking at you] I can't use them in pinned tabs because somehow they manage to get into a totally broken state where the page won't load due to whatever's happening with their auth flow in the background, and no method of reloading the tab fixes it, and it does this every couple days—but copy-pasting the same URL to a new tab does work)
I've always assumed the billions of redirects are setting cookies so all the various systems "work" but I have given up trying to understand it.
That’s Microsoft. 1000s of features and none of them really work the way they are supposed to.
it's "Enterprise" grade software! need to check the boxes for the procurement process (actually working is a separate department)
Exactly! I can’t even count the number of times we’ve been in the discovery phase of a project and see “Oh this MS product does that! Cool”. Then when we get to the actual implementation realize it’s a broken mess. It’s sales driven software development, they just need to get you far enough along to sign the contract, then it’s too late to back out.
There are extra ways to do that, but they're on a document deep in a Sharepoint directory that you can't access.
Moments like this, I miss clippy.
The problem is modern MS doing three contradictory things at the same time:
- FB's move fast and break things. Constantly launching new libs.
- Linus's we do not break user space. Great commitment to backwards compatibility.
- Never deprecating dead products until they've been de facto abandoned for like decades.
This combination means every MS product is a labyrinth of overlapping APIs with no guidance as to which one is actually the good one. Some are abandoned garbage, some are brand new and incomplete, and some are both, and there's no way of knowing which are which even experts can mislead you.
Well said. It feels like Microsoft is willing to release the intern’s poorly thought out product, and then commits to support the garbage design for all time.
Microsoft, you are a behemoth. There are few domains where you actually compete. Give your products a minute to breath before you cast them in stone.
> no guidance as to which one is actually the good one.
To some extent, you’re/we’re the ones deciding that,
because there’s entirely different teams heading the separate offerings,
and none of them are going to offer a potential footgun like:
“hey, we’re not the best modern path into xyz type projects, check with our colleagues on the Blazor team”,
unless someone makes them.
same experience for us, and then they email the living shit out of you about how your weekly entra id stats are good or bad, and you can not opt out of these emails.
> they email the living shit out of you
This sounds like LinkedIn.
Wait a minute. It is owned by Microsoft.
It’s a relentless horror. I signed my wife up to track down a driver that crashed into her.
I think LinkedIn spam is worse than being in a crash.
Same here, except with Minecraft and XBox One.
I don’t understand how they have non-zero market share.
I remember trying to buy $9 worth of Minecraft In-app Whatever for my kid, and the goose chase Microsoft put me on just to log in and buy something was totally out of this world. I ended up needing to contact their fraud department around step 74.
I'm still annoyed that I can't share those Minecraft purchases with a family.
For Minecraft they inherited a gigantic userbase from Mojang and then made it 10x harder to add new users.
I did it for my kids to have accounts and I do not understand how anyone who hasn't built a Gentoo from Stage 1 has a prayer of managing to buy Minecraft Java Edition for their kid, and making it actually work.
Then you've got the hell of overlapping permissions systems on the console and the Microsoft account, to get any amount of online play working on a console if you also get Bedrock. On the Playstation, especially, the error messages also love to not tell you which of the two systems is blocking you, so you get to guess. And Microsoft's site for managing those permissions is so confusingly-laid-out that even after doing it three times in a row I still felt lost on it.
I never did solve the problem of getting Minecraft Java Edition to run on a kid's MacBook with allowlist-only Web access. It wants to contact ten or so apparently-randomly-selected-from-an-enormous-pool IP addresses on every launch. I never did find documentation of which IP blocks I needed to allow, and couldn't guess at it from the IPs themselves. If they'd just used domain names... I must have manually hit "allow" a bunch of times during twenty separate launches, and it was still presenting me the same number of prompts every time, because there was no overlap in the IPs contacted (adding insult to injury is that I'm sure all but at-most two of these were spyware horse-shit that had no actual generously-necessary role in running the software, but it'd fail if it couldn't reach them)
I ripped Entra ID from one of our projects and replaced it with Keycloak.
Microsoft has never been good at security, and that is why their centralization to cloud is absolutely terrifying.
I'm reminded of Storm-0558 [1] where a stolen signing key was able to forge authentication tokens for any MSA / Azure AD / Government AD user. They downplayed the severity. Just imagine if that level of access was used to pull a Stryker on a nation-wide scale. That is an economic disaster waiting to happen.
[1] https://www.microsoft.com/en-us/security/blog/2023/07/14/ana...
I'll do you one better: stealing the signing key was not even necessary.
https://www.bleepingcomputer.com/news/security/microsoft-ent...
I knew there was another incident that I was forgetting, insanity... I don't understand how Microsoft keeps getting away with this and everyone just forgets.
When people's income depends on them forgetting... they tend to become amnesiacs.
because time to market is more important than security (at microsoft)
Oh please, that could happen at any company. Humans screw up.
But it doesn't. Full authentication bypass exploits are extremely rare and unheard of among tech giants. Maybe account takeover/recovery, sure, but full bypass? It just never happens.
Microsoft goes beyond that: they've managed to have a critical vulnerability in almost every authentication product they have ever created. It's exceptional.
The Justice Department CIO who pressured FedRAMP to approve GCC High was hired by Microsoft the next year. I wonder if this shouldn't invalidate the authorization in the first place?
It's not very clear from the article, but I get the feeling from the context that the 'pile of shit' quote referenced the package of documentation about the service rather than the service itself.
(That seems to be the main complaint, that Microsoft never provided the clear information required to conduct the assessment properly).
> The tech giant’s “lack of proper detailed security documentation” left reviewers with a “lack of confidence in assessing the system’s overall security posture,” according to an internal government report reviewed by ProPublica. > > Or, as one member of the team put it: “The package is a pile of shit.”
Yes, it seems pretty clear from that quote that the reviewer said the security package was a `pile of shit`, and propublica went on to extend that to the cloud itself. Not that I want to comment on the merits of Azure's security, but that sounds pretty clickbaity from propublica to me. A more appropriate title would have been
> Federal Cyber Experts Thought Microsoft’s Cloud Security documentation Was “a Pile of Shit.”
MS was (and still is it seems) unable to produce the data flow diagrams that FedRAMP wanted, ones that other cloud providers had no problem with. If the documentation is in such dire state, then the system itself is likely to also be in a dire state. I.e. The documentation is a pile of shit, so the system is also a pile of shit.
Wait- so they basically threw up their hands? No documentation! Not evaluable? Thus clearly of value for somebody? Big stamp, job well done! NEXT?
Yes. US bureaucracy regularly gets told "You have to have <thing>" but because it's against a lot of people's ideology, they aren't allowed to build it internally or develop any sort of actual expertise for such a thing, so their only choice is to buy whatever is offered no matter how bad it is.
For example, our state government says "We will do X Y and Z which all require data science expertise, but we did not approve the $60k a year Data Science position, so instead we are forced to hire a Data Science contractor for $120k a year, and they can't really be fired, and they are terrible at their job"
And then people wonder why things suck all the time.
A lot of state's buy their Obamacare marketplace service from a company I am familiar with. That company is entirely incompetent. They cannot follow basic instructions. They cannot triage a bug at all. They do not read freaking tickets. They take weeks to respond to an issue. They cause bugs regularly in ways that imply they don't have functional source control. They continually fuck up basic feature requests. They change the service in ways that contravene the literal law. The law that was comprehensively explained to them by people I know.
But they can't be fired, because the state is legally compelled to provide this service, and is not really allowed to hire a few engineers to build it in house. They could go to a different software contractor, but all the options are just as bad because it's an entirely captured market.
Obama started a "Digital Services" group in the federal government to actually build systems internally and develop expertise to mitigate some of this, and they built stuff like tax filing solutions for free for Americans. So Trump killed it and hollowed out it's corpse for DOGE.
Emergency notifications are done the same way! Its communism to fucking build it, so let’s have a team of a few engineers make an API to control government infrastructure from incompetent contractors on AWS, offer no real means of testing, breaking changes, downtime… and folks wonder why Hawaii is told bombs are coming
EVERYTHING about the federal government contracts program sucks ass! In the beginning, it was good as you didn't want people forcing through their brother, mother, 2nd cousin, next door neighbor, Satya Nadella and their "company" as a contractor without oversight cough Kristi Noem cough. However, it has devolved into a mess. The entire thing needs to be scrapped and re-engineered.
I'm guessing the requirements were written in a way that only Microsoft's cloud could with the bid.
Thats why you have Windows in the Pentagon instead of something secure.
The sheer amount of conflict of interest with folk involved in this later getting employed by Microsoft is a bit crazy.
There was definitely a point (late 90s?) when Microsoft finally figured out how to play the game. Coincided with the antitrust stuff.
To be fair, it's not always out of maliciousness. A lot of gov workers/contractors join the supplier company because they know the product and how to fix it better than the people currently at the company. Similar to the guy who infamously got hired at Apple just to fix a bug.
You're just forced to use vendors and if you actually care about the mission, it's just a different team on the same mission.
Of course you know you're being taken advantage of, and long-term maybe you should have gone to the non-technical side to fight it, but at the end of the day you just want to keep the young boys being shipped off to war safe, and you're much better suited to achieve that by remaining on the technical side.
...or so I've heard.
I don't have much experience with Azure but I was amazed at how many things in AWS GovCloud don't meet FedRAMP encryption requirements. For example, none of the lambda runtimes have FIPS certified encryption libraries available, and you have to bring you own, which is rather complicated to do.
Frustrating that FedRAMP is both a pain to get compliant with and also apparently is not a strong signal of actual security.
I see you've never worked in a compliance environment before.
And may such evil days never come to past
Was this approval before or after evaluators discovered this?
> Microsoft on Friday revised its practices to ensure that engineers in China no longer provide technical support to U.S. defense clients using the company’s cloud services.
Ref: https://www.cnbc.com/2025/07/18/microsoft-china-digital-esco...
when someone says they work at meta, they get weird looks, but no one assumes they're incompetent
when someone says they work at microsoft, they get weird looks, and people assume they're incompetent
> Potential Conflict of Interest: The government relies, in part, on third-party firms to vet cloud technology, but those firms are hired and paid by the company being assessed.
Hah. First time looking at FedRAMP?
The real reason for this, of course, is accounting, it moves it off of the government's books.
Little has changed since Bill Gates tried to install Movie Maker.
> These highlights were written by the reporters and editors who worked on this story.
It's unfortunate that people have to claim the authenticity, rather than the users of AI having to disclose use of AI/LLM. I wish it was the other way around.
Wow, Microsoft is really pushing the wrong boundaries in every direction, isn't it? Executives must be thinking, like many before them, that Microsoft is too big to fail.
Executives only react to share price movements. If share prices are high because whatever investors think, then execs will just open another champagne bottle.
Steve Jobs was the last tech CEO who didn't care about wall street and only care about quality products and consumers saying that if customers are happy, then the share price will take care of itself. But most companies are share price first, customer later.
This fits perfectly with traditional Microsoft strategies of getting a foot in the door and then having the users’ internal pressure on the organization to help get the Microsoft product established.
Decades ago, Lotus 1-2-3 on top of MSDOS was the lever; today it’s GCC High.
Given the scale and scope of the Federal Government. what are the alternatives to Microsoft?
Building in house.
Outsourcing to consultants.
IBM? Redhat?
I think there's some context missing here. For those who don't remember, the CIA back in like 2014 or so built out private data centers with classified versions of AWS services and all IC workloads that don't require specialized hardware was supposed to be using. DOD historically used it as well for classified cloud workloads, but wanted its own, and this was the JEDI contract, which was also supposed to go to Amazon, until Trump got into a fight with Jeff Bezos in 2019, canceled the contract, and awarded it to Microsoft instead. Amazon sued, and Biden decided to just award the contract to everyone and split it between all the major cloud vendors. That still doesn't mean anyone can actually use it without FedRAMP approval, but well, there you go.
The alternative was AWS, which has been operating at every classification level for over a decade at this point. It's now split between Amazon, Microsoft, Oracle, and Google, which is especially amusing because Google withdrew from the original bid process when they were still pretending to give a shit that their employees don't like working for the military.
Basically exactly what my org did. The momentum of being a Microsoft shop is hard to fight against.
Microsoft has been selling piles of shit since the beginning of time. The fact that they keep selling is the biggest triumph of sales/marketing over decent engineering.
The original title is:
> Federal Cyber Experts Thought Microsoft’s Cloud Was “a Pile of Shit.” They Approved It Anyway.
This is my opinion only, I'm sure some have had different experiences - but:
Azure's success as a cloud provider is mostly a result of their sales team and having an existing relationship with non-technical leadership. "We already pay them for Office and Exchange, let's just buy this new 'cloud' thing from them too".
Azure is barely considered an option at all within tech companies, yet is surprisingly widely adopted by non-technical companies that don't know any better (ie, that don't have a technical / engineering voice or representation within leadership).
AWS = Likely technically the best, for now. Mostly unreasonable pricing, and less motivation to seriously negotiate given they are the 'default' cloud provider for most of the industry. Kind of feels like they have peaked though, and are slipping more recently. Inevitable, or bad leadership changes?
OCI = New-comer, attractive pricing and hungry for business. Might be able to avoid mistakes other providers have made? Reliability struggles though. Parent company has a bad reputation in some circles - but probably not with decision makers. Making huge (unwise?) investments - that will either come crashing down in 5 years, or seriously pay off. Layoffs, but going for massive growth...huh?
GCP = Notably different underlying technical choices than other providers. Folks are maybe a bit less pragmatic, and more academic. This helps them in unique services (Spanner?) but hurts in most other areas. They've matured, and are btwn AWS and OCI in reliability. They are probably not as hungry for business as they should be given how far behind they are.
> By late 2024, FedRAMP reviewers concluded that they had little choice but to authorize the technology — not because their questions had been answered or their review was complete, but largely on the grounds that Microsoft’s product was already being used across Washington.
The article talks a lot about conflicts of interest, but this is the line I went looking for. A bureaucracy fighting itself over goal prioritization, and what's a necessary roadblock vs red tape is the less sexy but more meaningful problem at the core of this.
Once the government decided they wanted the product, they were going to find a patsy.
If you "went looking for" this line, you're just reading into the statements your preconceptions.
I on the other hand have no expectation, and so it's not clear whether the "bureaucracy fighting itself" is a cause or a symptom. You're implying it's a cause and the solution is "less red tape". But it could be just a symptom of conflicts of interest, and less red tape just leads to more efficient corruption.
Again, you're just reading into it what you already believe in.
I think plenty of software is a pile of shit and still derive value from it.
Yeah I'd go so far as to say that most useful software is "bad" in some way.
Worse is Better
Exactly, better the pile of shit you know than the pile of shit you don’t know - or the pile of shit that is u knowable.
Azure is easily the most expensive, least reliable and worst cloud available. It's borderline scam. An example today, I provisioned high IOPS SSDs (supposedly) and what is actually connected to the instance? A spinning hard drive! I didn't even know they were still made, but I guess Azure uses them and scams their users into thinking you're getting an SSD for $700/mo when its really an old hard drive.
I would warn anyone far and wide to avoid Azure at all costs, especially if you are a startup. And especially if you are doing any kind of AI because the only GPUs they have available are ancient and also crazy over-priced.
If I cared more, I'd try to migrate away from Azure. But I don't, and that's probably Azure's business model at this point.
I’d love to see proof of your claim that they provisioned a hard disk when you requested an SSD, or, at the very least, tests that showed that the IOPS you requested were not delivered. Can you show us the receipts?
Azure using SRE, I call BS. You don’t see underlying storage, it’s mounted as either SCSI or NVMe device as one HD. It’s obviously backed by massive fleet of drives just like EBS.
the product got deployed across the government while the security review was still in progress. then fedramp approved it because it was already everywhere. seem like i saw a lobbyist or two with a broom sweeping something under a rug...
Just like everyone else tasked with buying Microsoft
Azure is bad. But to be fair, every security summary of IT services I’ve ever read — or written! — for over 25 years has also been a “pile of shit”. It seems to be inherent to the cybersecurity game that everything is judged based on meaningless check boxes and nonsensical explanations. Meanwhile the actual security posture is obscured and ignored.
Staying afloat on cyber compliance takes so much time and energy there's no room for actual cybersecurity analysis.
A pile of shit you have leverage over is better than a pile of diamonds you don’t.
Microsoft is great at greasing palms
all clouds are.
A rigged RFP, and some very happy lobbyists, chortling into their single malt all the way to the bank.
Is this just a case of MS needing to merge a lot of platforms, and there are gaps and overlaps.?
Maybe the critical question, are they making continuing improvements? Especially to merge conflicting functions.
Like when they bought Minecraft, or Skype. Each already had user management. Xbox was a mess. Merging them all took a lot of years.
Maybe the gaps are a frature or benefit at the same time.
okay what the hell is a "cyber expert"?
Although "cyber" is a prefix used to denote a relationship to high technology there exist people who use it as a stand-alone term to mean "cybersecurity", e.g. "I work in cyber." It's very confusing given that it came from the word "cybernetic", the hundreds of other words that begin with it as well, and the existence of the term "cybersec" which is unambiguous and only a bit longer.
I fucking hate microsoft, i'm so sick of this retarded fucking bullshit
Exactly, and that is the moat- a pile of shit that everyone can smell from afar.
I mean, they also bought the F-35.
its as funny as the IA research reports from DORA dev which all seem to be sponsored AI provider ads instead....
Is there a big cloud platform that isn't a pile of shit?
Yeah, but this is how things work at that level.
Microsoft can be abhorrent. They will always get the contracts. Why? Corporate welfare.
Microsoft will drive the rules. Why? Too big to fail.
Microsoft will push their slop. Why? Cause they have contractors after contractors in the federal government pushing MS solutions. Doesnt matter if they're bad.
And, who'd pay for a 3PAO audit of a Linux distro? Ubuntu and Redhat have. Its a $120k moat.
The government does most things poorly and with little regard to budget or quality. They can't solve problems that are much simpler than cloud computing, so why should I expect them to perform better at a more complex problem?
Sure. Your average private corporation would do much better at sanely evaluating Microsoft's cloud, and sanely acting on that evaluation.
Right.
You bet.
Absolutely.
I think this perspective has resolutely been debunked at this point.
The government has historically, routinely, consistently, solved problems more complex than cloud computing.
The only way you'd think otherwise is if you had some other motivation to pretend otherwise... some sort of ideology.
> The government does most things poorly and with little regard to budget or quality.
That's a common line by conservatives who are actively sabotaging government with policies and laws which they then point to as evidence of such inefficiencies.
Basically false. They're better at health care. Better at education. Better at feeding people. Better at charity.
Theres no need to be THIS cynical.
Suddenly everyone on HN is an expert on Azure infrastructure.
it isn't the best but it's really great at a lot of things feature-wise. top-notch documentation as well (despite what these "experts" said).
Most companies literally run on Azure these days. Persistent hackers will get into any network, that's a guarantee, that's APT 101. It's law of averages. If it truly is "a pile of shit" given how it is probably the most used cloud platform by the most customers, including governments, and endless plethora of features and services it offers, shouldn't there be more compromises? 2-3 in a decade is hardly above what you expect for law of averages right?
Screw ups happen, but if it is systemic, you can't use one instance as evidence, you must establish a pattern of mishaps.
I ran a one of the largest multi-cloud service across azure, aws and gcp.
Azure was hands down, obvious to everyone involved the worst technically. In capabilities, bugs/correctness, availability and support.