Over the last month I contacted Support for the first time in many years.
This was for a question about how billing works.
It went like this;
1. Case created.
2. Unassigned for seven days.
3. Open real-time chat, talk for 25 or so minutes where I guide a first-line Indian chap who plainly doesn't know about the subject in hand and who is as we talk reading the AWS docs I've already read. At the end, just as I couldn't find an answer, he couldn't - which is good, he didn't try to give me the wrong answer - he escalates. That's fine - a lot of questions are simple and even silly, and first line support is there to handle them - but they could have done all this without me, if they'd opened the ticket themselves rather than me having to chase.
4. Eleven days later, comes back with exactly the wrong answer. In the meantime, I had figured out the correct answer, and reply, explaining it to him.
5. Next day, I get a wall of plainly AI generated text telling me my answer is correct.
It seems to me a key issue here relating to AI generated text, is a misunderstanding on the part of AWS that I as a consumer will value that answer exactly (or indeed, even remotely) as I would value the answer from a human.
I do not. I almost ignore AI generated text, as I think it as unvalidated response.
AI "support" bots that just attempt to read the published documentation for you are possibly the most annoying thing to have come out of the current AI plague.
Even Stripe - once legendary for the quality of its support - has apparently given up now. I had to deal with it recently over a case where the merchant was seeing an unexpected change in the way it was collecting payments and the AI bot was worse than useless - it actively suggested incorrect explanations and resulted in several days of trying to change the wrong things while the problem persisted.
For my own businesses we give this issue a heavy weight when choosing which services to use. We have even seriously considered moving existing integrations to different services over this one issue recently. If we're integrating with a service then we want to know there's a real person who can actually help if we have questions or anything goes wrong. Failing to provide that because it's cheaper to push everyone through the AI bot is a statement of intent about how much you value your customers.
The problem is using AI to “push” the answer to an asker.
Unless the company has hidden docs they use for support (which why would that ever benefit them), I could get just as good of an answer if I point my LLM at your docs (“pull” an answer). In fact, the response might be better because I have context set up to tune it to my understanding.
Instead, you (company / support agent) have decided that I should instead have a conversation with an LLM through a worse, more opaque harness to the detriment of all of us.
It would be interesting to hear the other side, the people running support. I wonder what fraction of requests are answered by basic knowledge and stuff clearly in the docs. At some very high fraction I could see a lot of pressure/incentive to optimize for these cases.
Even more annoying, is when the integrated "chat with AI" boxes don't actually have full knowledge about the website. Tried a couple of different such boxes, and in the end I still had to crawl the website on my own to find the information.
> AI "support" bots that just attempt to read the published documentation for you are possibly the most annoying thing to have come out of the current AI plague.
Meh. It's no big change to before, where you had first-level support search for something vaguely related and just dumping a template to the user.
The thing is... it actually works better than I'd like, because in a lot of cases it turns out that you (as the user) forgot to follow one tiny step in the documentation.
What I'd actually like to see as the user is the AI actually going over my AWS account, looking up the resources and their state on its own, and figuring out what exactly I missed, but (un?)fortunately that cannot be easily done for IAM permission reasons...
The big difference IME is that first-tier support at decent companies at least had the grace to recognise when they couldn't help and needed to escalate. I can count the number of times I've seen an AI bot automatically escalate when it was unable to find the solution on zero hands.
I've been at places where the AWS annual spend was a real lot of money, let's say way over 100k but not 1 million USD.
Support tickets went unanswered for months, assigned account reps left us hanging for months with multiple follow ups, etc.. All tickets opened within the last 6 months got AI generated responses with massive delays that indicate the ticket wasn't read by a human due to how inaccurate the response was based on the questions asked.
I had an obscure KMS issue a few months ago, I reproduced it and opened a ticket. It was a low severity. A guy emailed me back in about 4 hours, acknowledged it and thought I was doing it right. Came back 2-3 days later noting that it was a bug, and would be fixed in a few weeks. As a workaround he updated my script with a less obvious/efficient solution to get me through.
It can be two things. People’s main complaint about Google’s cloud services is that you are forever at the risk of an automatic and unappealable ban of your whole account.
We recently had an issue with our production Oracle database. Our in-house DBAs spent hours trying to get the AI support bot to assign a real person to join the incident call. It took more than 2 hours to get an actual person on the call.
We literally pay hundreds of thousands a year in Oracle support contracts, and this is what we get? AI bots? Nope. Migrating to Postgres is now a top priority.
This "replace people with AI" nonsense has to stop. [0]
I used to see AI generated images with lots of unintelligible writing or misspelled words in slides, but the speaker left them in anyway. “Good enough” is not customer obsession.
This enforced adoption of immature GenAI reminds me of Milo Minderbinder trying to make people eat cotton in Catch 22, because he had inadvertently obtained a huge amount of it.
I don't know if there is another industry that behaves this childishly. There might be. But good grief, how much more juvenile can ours possibly get? AI-generated images with obviously nonsensical text is something I never thought I'd see in professional meetings. But it is becoming more and more common.
Long before GenAI, I saw people using meme generators a lot in corporate presentations. I found that equally jarring. Replacing that with GenAI stuff is probably an improvement. At least it's reducing legal risk. It seems more understandable to a global audience, too.
I still don't have an explanation why people are doing this. Is it part of leadership training? Or do presenters have their own theory that including this stuff makes the presentation more memorable and enjoyable?
People have a thing that they mimic the behavior of those above them in the hierarchy. CEO used a meme once because they thought it's funny, then everyone did this in order to mimic CEO.
Why was leetcode so popular? Because Google did it and they were the cool kids at the time.
AI enables the stupid and diligent. When I get long form emails from people I don’t know well, I assume that 80% of the time it’s AI bullshit. People are having AI respond with bullshit to other people’s bullshit.
I’m actually deliberately adding bad grammar to communications as a hook that you should read it. It hurts my soul.
During a recent semester final exam with fill in the blank questions, a student raised their hand to ask the teacher, "Do we get marked off for spelling?" The class was bewildered when the teacher answered, "Of course." Some students immediately turned in their exam assuming it would be impossible to pass.
I'm also an AWS alumni from many years back now, and truthfully, the organizational problems really took off when Jassy moved to being CEO of amazon as a whole and major leaders left the company (Charlie Bell, et al.).
There were always other problems too, pressure on the company in both directions across many different product lines on both cost (any number of cheaper baremetal providers who are much faster at providing customers instances than they were a decade ago), and product quality (any number of startups to now bigger companies, databricks probably being the biggest success) along with a number of expensive bets that were made that didn't work out especially as interest rates began to rise (there were numbers of of different services ranging from IoT, AI, business support, robotics, groundstation, that essentially all failed).
AI infra being their latest bet, along with doubling down on custom hardware is smart, but these roles don't require the same number of SWEs and instead require a different type of high skilled professional.
I find it hard to call Amazon robotics a failure. All of the small/binnable item FCs make extensive (and, as an outsider, apparently very productive) use of robotics.
I'm talking specifically their AWS service for ROS applications, all of my concerns are AWS specific for that matter, not the robotics they build in house.
Unrelated to your main point, but it's "alumnus" in the singular form. For bonus language nerd points, you would use "alumna" to refer to a woman, or "alumnae" to refer to multiple women. Not sure how Latin handles mixed gender groups, though I would guess it's "alumni".
I think a key goal of senior management at any big company in the last 6 months is to make rank and file fungible or obsolete. It’s one big experiment. There are precedents like the Industrial Revolution. Things get worse for the workers for a generation or so.
I've worked at Amazon since 2018 and they've always talked about the software engineers being fungible during my tenure there. Technically everyone is supposed to be able to do everything, but in practice it's a huge headache if you want to hire for a more specialized role. They started creating some, like Frontend Engineer and Embedded Systems Engineer, but in practice these are still extremely broad
> There are precedents like the Industrial Revolution. Things get worse for the workers for a generation or so.
And things only got better post-Industrial Revolution when labor organized and forced the issue.
There's no guarantee that will work again if labor has reduced leverage due to AI reducing their value.
I think in one way or another this all works itself out, but I'm not convinced it won't be a very painful (and possibly violent) transition to whatever comes next.
Also nobody talks enough about the fact that workforce is effectively cut out from the means of productions. Even with the capital at hand blackwell cabinets are all sold out, contracted to the big providers.
There are paralles to the industrial revolution, but it seems the working class is cut out from being able to deny labor in exchange for better conditions.
I am also increasingly worried by the potential for violence here. This is a social experiment that is harming the daily lives of millions of people in very obvious ways already. The environmental costs for the data centres are not insignificant. The economic damage from allowing AI to have so much funny money when it doesn't make much real money to justify it could be disastrous on a generational scale. Governments aren't making any serious attempt to regulate and if anything are drinking the Kool-Aid. We might be on a path that literally collapses the established Western capitalist order within a generation but historically societal change of that scale usually has a body count and I have no idea what comes afterwards.
The actual Industrial Revolution labor wars happened because workers were being maimed, killed, and disposed of with zero legal recourse. The Ludlow Massacre in 1914 ended with the Colorado National Guard machine-gunning a tent colony and burning women and children alive. The Battle of Blair Mountain in 1921 had the United States Army bombing American coal miners from biplanes. Pinkertons routinely shot organizers. The Triangle Shirtwaist Fire killed 146 garment workers because management locked the exit doors to prevent unauthorized breaks. Coal miners were paid in company scrip redeemable only at company stores in towns the company also owned and policed. Black workers attempting to organize in the South were lynched. Children were maimed in textile mills.
A software engineer getting four months severance after a layoff exists in a different universe from this so no. There is no precedent. Don't you dare talk about the industrial revolution because its not even in the parking lot of the ballpark.
You should look past the screen and see what's going on. War is not the same; violence against humanity is not the same.
You're right, it's not in the ballpark. It's at the gates. The game isn't on simply because they poisoned the opponent in the duggout.
It's time. It needs to stop now while the body count is low. This isn't about some dev getting severence. They've taken away the street sweeper position and are watching us eat each other.
All the AI hype aside, I wonder if there is a way to avoid becoming one of these faceless corporations where customers are just numbers. For years Amazon has been fantastically customer centered, but at some point they just lost it. I could compile a list where Amazon is actually way more customer unfriendly than in the past now, but I guess everybody already got their own anecdotes about that. So what exactly went wrong and how could that be avoided at other companies?
> All the AI hype aside, I wonder if there is a way to avoid becoming one of these faceless corporations where customers are just numbers.
Limit scale. Enjoy your craft. Become immune to hype and "what's popular," instead focusing on "what otherwise inaccessible experience can we make possible for our customers?"
When you let the money guys take the wheel (not the "passionate nerd" types), it inevitably (and I would argue necessarily to keep the lights on for such a big org) results in a shift to spreadsheet brain.
If you haven't seen it, watch Jiro Dreams of Sushi and see if that way of life resonates. Also check out his interview with René Redzepi from Noma. Lots of great insight into how focusing on your craft implicitly creates the opportunity for creating and delivering great things to others.
Have you read Cory Doctorow's "Enshittification" yet? He describes the process quite well.
Not sure how to address it though. I suspect keeping companies small and focused on quality, sustainability, and free of VC influence would be a solution. It'd take continual work though, like tending to a bonsai.
The economy sucks, and they do pay decently for software engineers. Especially now that the rest of FAANG aren't massively over-paying for college hires, I doubt the supply of bright young minds will ever entirely dry up.
I have worked there for 8 years now and we ARE running out of bodies.
It has become extremely difficult to hire at any level, we have had an open position on my team for a senior data scientist for a year and a half now, with barely any candidate applying, and none of them being competent.
Similarly the average level of new employees has dropped dramatically. The famous "hiring bar" is now below ground.
Amazon has been on my Would Never Work For list for over a decade now. Even the “golden years” being referenced by OP and some commenters in this thread were plagued by Amazon overworking people and doing sketchy things like weighting RSU all till the last few years and then laying people off before their mountain of cash landed
There are now more highly competent devs ready to work for cheap available now than ever before and all of them are boosted with state of the art coding agents…
It‘s the golden age for software engineer employers.
Correction: it's the golden age for code monkey employers. We as an industry have never deserved to use the term "engineering" to describe ourselves, and we are further from that ideal than ever.
I'm not sure FAANG does look good on a CV any more. The skill set to be effective in those environments is quite specialised and crucially it's very different to what you need in a lot of other software development organisations. There appeared to be a happy cycle for a while where very well paid devs working in one of the few FAANG or FAANG-adjacent companies could jump to one of the others because they were "in the club" and had experience of working at a truly global scale that most software never needs. Those days seem to be over with the mass layoffs and hiring limits. And if you're not working at that scale - and outside that small world almost nobody actually is - those skills aren't always very transferrable and other types of experience often have more value.
I also joined in 2022, and it aligns so much with my experience. Good manager that moves on, then a gradual erosion of "insist in the highest standards" towards a dreaded "good enough", GenAI only accelerated it IMO.
> When AWS first introduced a viable cloud to the world, it was amazing. Back in the 1990s when you wanted to implement an enterprise software solution, you first had to take a guess at what computing power you would need. Next, you would have to order hardware from companies like Sun Microsystems or Dell and that could take weeks if not months to be delivered. It would then need to be racked, powered and provisioned, and then you were screwed if you happened to undersize it or criticized if you spent too much and oversized it.
This is how many large enterprises still operate today. Ironically, the main argument is that it's faster to provision VMs on-prem than it is to get approval to run in the cloud.
The story about recovering the account rings very close to me. At least they had coworkers cheering for him, I feel teams are shrinking so much that we'll end up with just the LLM of choice to pat our back with "good work" and "you're absolutely right"
Co-workers cheered while managers were sharpening their axes. One doesn’t do such “heroics” without approval, making the system look incompetent and broken and then apologizing for it without being decapitated in the public square for everyone to learn the implicit lesson. Anyone cheering for him publicly should watch their back, too.
> the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need
The goal was never to solve a real problem, like we evangelized for decades. That was how it was explained when resources (mainly time, but also money) were scarce and we could not just throw things at walls. Now we can, and you won't see anyone talk about "make something people need".
Things will be low quality until something sticks, and then money will be poured into it. It's not a bad strategy, but my takeaway from this is: there are multiple plausible explanations for the same thing. People have an incentive to not give you the correct one if it helps you compete with them. But they will give you a sensible one. AI won't protect you from this, experience and real knowledge will.
I have been advocating within my org to replace "fungible" with "flexible" or "generalist."
"Fungible" implies they are a commodity, easily swapped for someone else. In other words, they are so low-value that they are interchangeable.
"Flexible" or "generalist" instead connotes that they are so high-value that they can operate well in multiple domains, easily shifting to where they are needed most.
“Flexible” would work if Amazon prioritized moving people around when the priorities change instead of laying off and rehiring.
You can easily call the typical Japanese life-long employees as “flexible” or “generalist” but not an employee of a company with median tenure rate of 1-2 years. That’s fungible.
Our company also requires everyone to use more AI-related tools, and I don't think there's anything wrong with that. But the quality of work produced using these tools really depends on the individual's ability. Some people don't put in much effort, and the results they produce are really sloppy, which bothers me a lot.
I think there is actually something wrong with that. What should matter is the work produced, not the tools used to produce it. If AI tools really are all they are cracked up to be, then people using them will get ahead, and the company can justifiably point out "your peer gets twice the work done as you" to the other employees. But mandating tool use in and of itself is senseless and counterproductive.
The unfortunate reality is that unless you use these tools it is impossible to keep up. People using these tools well are substantially more productive, and often they were already the most productive.
AWS lost its way. S3, SQS, EC2 and VPC were great innovations and those services were done by a bunch of engineers who wanted to have a reliable elastically scalable system. This was coincidentally cost effective at the same time. What came after especially the data stack and now the AI services were done by a MBA heavy management team who does not understand innovation and treats engineering like a bank does: putting it in the cost category. Recent financial results show the impact: Google grew almost twice as much as AWS did. Maybe it is just coincidence.
> In this whole pivot to GenAI, AWS has lost its focus on the customer. Instead of working backwards from a genuine customer need, the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need.
Anecdotally, this seems to be the new "mission statement" of many companies.
The account recovery story says a lot. At some size, companies start handling people as tickets. Sometimes it only gets fixed because one person inside still cares.
Absolutely. With his name in the public and apologizing to the customer for sheer internal incompetence. Then also cheered on internally.
I bet as the managers publicly nodded in praise for his heroic act, their hands were already typing his name to be sent to HR for “get this guy out of here on any excuse you can” note. (In reality it would be a nonverbal hint of sorts. Nothing to leave any trace discoverable by lawsuit)
Meh you always read one side of the story. I have seen a lot of people getting PIPed out of Amazon and only one or two didn't deserve it, yet all claimed they didn't.
This genai is going to bring about huge quality drop in software across the stack and across the domains. I already see orgs that had reasonable software processes transform into orgs where the only metric is how much generated code you can slap and slop together and how fast. There's no success here for anyone.
And this is not a dink on the ai tooling itself but on the organizationan processes that provide the context in which the AI code generation is being used.
Bad processes will always produce bad low quality outcomes regardless of tbe technology.
I read every comment on this thread at the time of my posting and believe it’s fair to categorize the general sentiment that Amazon has lost its way and may even be in its “IBM stage.”
AWS services still are generally reliable and available. I’d think we’d be seeing cracks here if the organization were in shambles. AWS seems to keep humming along.
Sad to hear. I was at AWS between 2016 and 2021 and had a much different experience.
Side note, and unpopular opinion ahead: while it takes a lot of courage to write things like this and I respect it, but being fired and writing negatively (no matter how justly) about your former employer is considered by many employers as a red flag and can hurt you going forward (even if you are 100% right).
It’s well into the IBM phase now. Primarily providing important but boring commodity infrastructure, but the top talent that can drive real innovation has long since left the building.
It’s race to stay relevant in AI but always seeming 2-3 steps behind everyone else is one such example of the current sad state of affairs.
Always has been, IT has been resisting due to (as pointed out in the article) institutional knowledge required, but this is now going away as LLMs can efficiently and accurately write it down and search it. Meta turning ICs into data labelers is trying to make knowledge workers fungible; expect others to follow.
> Amazon has a really odd viewpoint when it comes to the people who work there. They view almost all employees as “fungible”.
Hardly an Amazon-only thing. In fact, enterprises need this mindset, because people moves on, retires, or just suddenly die. With that said, due to its late-stage capitalistic ethos, Amazon is just too overly gleeful about this tasteless reality of life. It's the equivalent of a nephew coming to an aunt's funeral and shouting "A week ago, I told her everybody dies! And now she did! Wasn't I right??? Everybody dies!"
> Also, last year the focus at AWS turned fully and almost desperately toward GenAI.
I wonder if I'm being too cynical, but late-stage capitalism companies also love profiteering, and the mere prospect of firing all those pesky workers and not having to pay their salaries is like cocaine to those organizations. Which is why I think Amazon fulfillment centers will at some point rent robots at a price point between 2x and 3x their current human labor costs, in the hope that it will eventually make economic sense.
> Hardly an Amazon-only thing. In fact, enterprises need this mindset, because people moves on, retires, or just suddenly die.
Enterprises typically have this mindset. Most corporations I worked for in fact treated employees exactly like this.
As for needing this mindset, I am not so sure. There is a spectrum in between going under because a storied employee retired and treating employees as meaningless numbers in a spreadsheet.
But ultimately I fully agree with the whole of your post. I just had to nitpick about this.
> In this whole pivot to GenAI, AWS has lost its focus on the customer. Instead of working backwards from a genuine customer need, the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need.
AWS has been this way for a lot longer than GenAI, since the basic infrastructure products were built out early on. But when I read this line about throwing things out there quickly, I also think of Google and even Anthropic. Google has a long list of products that got created and killed, as part of their internal politics and promotion culture. Anthropic is currently rushing vibe coded slop all the time to try and win over OpenAI and set up their IPO.
Maybe all the rich high funding companies can afford to this and maybe it is the right thing for them to do. They can afford to make big mistakes without hurting their stability. A true startup or smaller company can’t - they would shutdown because one big investment that fails is enough to destroy the whole company.
to be fair, even though they have "working backwards" and "customer obsession", amazon has always been about making lots of different experimental bets. Bezos:
> To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there. Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. This long-tailed distribution of returns is why it’s important to be bold. Big winners pay for so many experiments.”
> Maybe all the rich high funding companies can afford to this and maybe it is the right thing for them to do. They can afford to make big mistakes without hurting their stability. A true startup or smaller company can’t - they would shutdown because one big investment that fails is enough to destroy the whole company.
Both are following the same strategy. Amazon has a $2.86 trillion market cap. That's the equivalent of 143,000 $20 million Series A startups. Companies like Amazon and Google are basically an integrated herd of cash cows plus a VC portfolio.
“I have to say being fired from AWS is actually a relief. There have been a lot of changes to the company since I joined in 2022, and the company I wanted to work for is no longer the same company.”
Many storied companies can be described this way. It’s a shame. Have any companies hit such scale and kept the ethos and magic of before? Is it inevitable for companies to enshitify themselves in the pursuit of their shareholder’s goals?
Not possible once big parts of the company start not knowing other big parts of the company and the company also has a board of directors that must increase shareholder value at all costs.
I was enjoying the article and then he makes some of the most bizarre claims about what cloud did and how we had to provision servers
If any of you young'uns read this, that is not how we had to do provisioning before cloud.
VMs already existed before AWS came out. You could already provision a new server usually in minutes and rent it month to month.
In fact, all the existing VM server companies had to start calling themselves cloud companies because pointy haired bosses couldn't understand what cloud really was.
It was pretty regional back then, when I got VMs for clients we used UK providers (ElasticHosts maybe?).
Also people are throwing AWS start date of 2006, but AWS only really started catching on around 2008/2009 if I remember correctly. EC2 came out of beta late 2008, and EBS was only launched around the same time.
I don't think it even officially launched S3 until 2008, which is what all people really used it for initially for cheap remote backups.
It definitely took a while to get good. There was also a period where having 'noisy neighbours' impact VM performance was often discussed here. You didn't tend to have that same problem on the other VM hosts as people were using the VMs for hosting with generally low CPU usage, not for compute.
And they were cheaper than renting AWS. MUCH cheaper. They still are.
The original point of AWS is that could scale according to demand. Have 10 VMs running at lunchtime and 1 VM running at midnight.
But using a cloud VM also required less server admin experience. It was a bit easier and came.pre-configured with things like firewalls.
And THAT is what ended up being the USP of cloud hosting. Especially when they started rolling out all the SQL as a service, redis as a service, etc.
You didn't need to really understand servers to run a server, and it turned out almost all developers really didn't want to understand servers. TBH, I don't, server admin sucks. Right now I'm working somewhere where I have to think about SSL certs occasionally and I consider it a complete waste of my life.
Digital Ocean came out like 5 years after AWS, what was revolutionary about that wasn't that you could spin up VMs quickly, it was the price. VMs went from $20-30 p/m to $5.
For developers who weren't SV rich, that meant you could run a side project without it being a significant cost.
> Long story short, I was able to get his resources restored. All I did was manage to poke the right bear and the support team did the rest of the work (and they were amazing).
No they utterly failed and needed a special non fungible employee to get them to do their job.
I'm glad to see that one core amazon principle has endured the 10 years since I worked there, even if none of the actual leadership principles have survived /s
Over the last month I contacted Support for the first time in many years.
This was for a question about how billing works.
It went like this;
1. Case created.
2. Unassigned for seven days.
3. Open real-time chat, talk for 25 or so minutes where I guide a first-line Indian chap who plainly doesn't know about the subject in hand and who is as we talk reading the AWS docs I've already read. At the end, just as I couldn't find an answer, he couldn't - which is good, he didn't try to give me the wrong answer - he escalates. That's fine - a lot of questions are simple and even silly, and first line support is there to handle them - but they could have done all this without me, if they'd opened the ticket themselves rather than me having to chase.
4. Eleven days later, comes back with exactly the wrong answer. In the meantime, I had figured out the correct answer, and reply, explaining it to him.
5. Next day, I get a wall of plainly AI generated text telling me my answer is correct.
It seems to me a key issue here relating to AI generated text, is a misunderstanding on the part of AWS that I as a consumer will value that answer exactly (or indeed, even remotely) as I would value the answer from a human.
I do not. I almost ignore AI generated text, as I think it as unvalidated response.
AI "support" bots that just attempt to read the published documentation for you are possibly the most annoying thing to have come out of the current AI plague.
Even Stripe - once legendary for the quality of its support - has apparently given up now. I had to deal with it recently over a case where the merchant was seeing an unexpected change in the way it was collecting payments and the AI bot was worse than useless - it actively suggested incorrect explanations and resulted in several days of trying to change the wrong things while the problem persisted.
For my own businesses we give this issue a heavy weight when choosing which services to use. We have even seriously considered moving existing integrations to different services over this one issue recently. If we're integrating with a service then we want to know there's a real person who can actually help if we have questions or anything goes wrong. Failing to provide that because it's cheaper to push everyone through the AI bot is a statement of intent about how much you value your customers.
The problem is using AI to “push” the answer to an asker. Unless the company has hidden docs they use for support (which why would that ever benefit them), I could get just as good of an answer if I point my LLM at your docs (“pull” an answer). In fact, the response might be better because I have context set up to tune it to my understanding.
Instead, you (company / support agent) have decided that I should instead have a conversation with an LLM through a worse, more opaque harness to the detriment of all of us.
It would be interesting to hear the other side, the people running support. I wonder what fraction of requests are answered by basic knowledge and stuff clearly in the docs. At some very high fraction I could see a lot of pressure/incentive to optimize for these cases.
Even more annoying, is when the integrated "chat with AI" boxes don't actually have full knowledge about the website. Tried a couple of different such boxes, and in the end I still had to crawl the website on my own to find the information.
> AI "support" bots that just attempt to read the published documentation for you are possibly the most annoying thing to have come out of the current AI plague.
Meh. It's no big change to before, where you had first-level support search for something vaguely related and just dumping a template to the user.
The thing is... it actually works better than I'd like, because in a lot of cases it turns out that you (as the user) forgot to follow one tiny step in the documentation.
What I'd actually like to see as the user is the AI actually going over my AWS account, looking up the resources and their state on its own, and figuring out what exactly I missed, but (un?)fortunately that cannot be easily done for IAM permission reasons...
The big difference IME is that first-tier support at decent companies at least had the grace to recognise when they couldn't help and needed to escalate. I can count the number of times I've seen an AI bot automatically escalate when it was unable to find the solution on zero hands.
Don't feel too bad.
I've been at places where the AWS annual spend was a real lot of money, let's say way over 100k but not 1 million USD.
Support tickets went unanswered for months, assigned account reps left us hanging for months with multiple follow ups, etc.. All tickets opened within the last 6 months got AI generated responses with massive delays that indicate the ticket wasn't read by a human due to how inaccurate the response was based on the questions asked.
This where I don’t get the HN flame at Google.
I had an obscure KMS issue a few months ago, I reproduced it and opened a ticket. It was a low severity. A guy emailed me back in about 4 hours, acknowledged it and thought I was doing it right. Came back 2-3 days later noting that it was a bug, and would be fixed in a few weeks. As a workaround he updated my script with a less obvious/efficient solution to get me through.
It can be two things. People’s main complaint about Google’s cloud services is that you are forever at the risk of an automatic and unappealable ban of your whole account.
Two things can be true.
We recently had an issue with our production Oracle database. Our in-house DBAs spent hours trying to get the AI support bot to assign a real person to join the incident call. It took more than 2 hours to get an actual person on the call.
We literally pay hundreds of thousands a year in Oracle support contracts, and this is what we get? AI bots? Nope. Migrating to Postgres is now a top priority.
This "replace people with AI" nonsense has to stop. [0]
[0]: https://www.forbes.com/sites/jonmarkman/2026/04/06/oracles-m...
I used to see AI generated images with lots of unintelligible writing or misspelled words in slides, but the speaker left them in anyway. “Good enough” is not customer obsession.
This enforced adoption of immature GenAI reminds me of Milo Minderbinder trying to make people eat cotton in Catch 22, because he had inadvertently obtained a huge amount of it.
I don't know if there is another industry that behaves this childishly. There might be. But good grief, how much more juvenile can ours possibly get? AI-generated images with obviously nonsensical text is something I never thought I'd see in professional meetings. But it is becoming more and more common.
we’re witnessing how advanced civilizations actually die
there’s no heroic clash with other civilizations, they simply rot away
Check the podcast “Fall of Civilizations”, this is very true. This advanced civilization is actually dying.
Long before GenAI, I saw people using meme generators a lot in corporate presentations. I found that equally jarring. Replacing that with GenAI stuff is probably an improvement. At least it's reducing legal risk. It seems more understandable to a global audience, too.
I still don't have an explanation why people are doing this. Is it part of leadership training? Or do presenters have their own theory that including this stuff makes the presentation more memorable and enjoyable?
People have a thing that they mimic the behavior of those above them in the hierarchy. CEO used a meme once because they thought it's funny, then everyone did this in order to mimic CEO.
Why was leetcode so popular? Because Google did it and they were the cool kids at the time.
It remains to be seen whether GenAI only acts as an accelerant of organizational decline, by amplifying the laziness inherent in people.
It’s worse.
Remember the “Prussian general 2x2”? https://quoteinvestigator.com/2014/02/28/clever-lazy/
AI enables the stupid and diligent. When I get long form emails from people I don’t know well, I assume that 80% of the time it’s AI bullshit. People are having AI respond with bullshit to other people’s bullshit.
I’m actually deliberately adding bad grammar to communications as a hook that you should read it. It hurts my soul.
During a recent semester final exam with fill in the blank questions, a student raised their hand to ask the teacher, "Do we get marked off for spelling?" The class was bewildered when the teacher answered, "Of course." Some students immediately turned in their exam assuming it would be impossible to pass.
At least everyone gets an RSU
I'm also an AWS alumni from many years back now, and truthfully, the organizational problems really took off when Jassy moved to being CEO of amazon as a whole and major leaders left the company (Charlie Bell, et al.).
There were always other problems too, pressure on the company in both directions across many different product lines on both cost (any number of cheaper baremetal providers who are much faster at providing customers instances than they were a decade ago), and product quality (any number of startups to now bigger companies, databricks probably being the biggest success) along with a number of expensive bets that were made that didn't work out especially as interest rates began to rise (there were numbers of of different services ranging from IoT, AI, business support, robotics, groundstation, that essentially all failed).
AI infra being their latest bet, along with doubling down on custom hardware is smart, but these roles don't require the same number of SWEs and instead require a different type of high skilled professional.
I find it hard to call Amazon robotics a failure. All of the small/binnable item FCs make extensive (and, as an outsider, apparently very productive) use of robotics.
I'm talking specifically their AWS service for ROS applications, all of my concerns are AWS specific for that matter, not the robotics they build in house.
> I'm also an AWS alumni
Unrelated to your main point, but it's "alumnus" in the singular form. For bonus language nerd points, you would use "alumna" to refer to a woman, or "alumnae" to refer to multiple women. Not sure how Latin handles mixed gender groups, though I would guess it's "alumni".
If you want to go deeper, you also have to takes into account the grammatical roles the word has in the sentence.
I personally think it’s not worth it.
See https://en.wikipedia.org/wiki/Latin_declension
I think a key goal of senior management at any big company in the last 6 months is to make rank and file fungible or obsolete. It’s one big experiment. There are precedents like the Industrial Revolution. Things get worse for the workers for a generation or so.
I've worked at Amazon since 2018 and they've always talked about the software engineers being fungible during my tenure there. Technically everyone is supposed to be able to do everything, but in practice it's a huge headache if you want to hire for a more specialized role. They started creating some, like Frontend Engineer and Embedded Systems Engineer, but in practice these are still extremely broad
> There are precedents like the Industrial Revolution. Things get worse for the workers for a generation or so.
And things only got better post-Industrial Revolution when labor organized and forced the issue.
There's no guarantee that will work again if labor has reduced leverage due to AI reducing their value.
I think in one way or another this all works itself out, but I'm not convinced it won't be a very painful (and possibly violent) transition to whatever comes next.
Also nobody talks enough about the fact that workforce is effectively cut out from the means of productions. Even with the capital at hand blackwell cabinets are all sold out, contracted to the big providers.
There are paralles to the industrial revolution, but it seems the working class is cut out from being able to deny labor in exchange for better conditions.
I am also increasingly worried by the potential for violence here. This is a social experiment that is harming the daily lives of millions of people in very obvious ways already. The environmental costs for the data centres are not insignificant. The economic damage from allowing AI to have so much funny money when it doesn't make much real money to justify it could be disastrous on a generational scale. Governments aren't making any serious attempt to regulate and if anything are drinking the Kool-Aid. We might be on a path that literally collapses the established Western capitalist order within a generation but historically societal change of that scale usually has a body count and I have no idea what comes afterwards.
The actual Industrial Revolution labor wars happened because workers were being maimed, killed, and disposed of with zero legal recourse. The Ludlow Massacre in 1914 ended with the Colorado National Guard machine-gunning a tent colony and burning women and children alive. The Battle of Blair Mountain in 1921 had the United States Army bombing American coal miners from biplanes. Pinkertons routinely shot organizers. The Triangle Shirtwaist Fire killed 146 garment workers because management locked the exit doors to prevent unauthorized breaks. Coal miners were paid in company scrip redeemable only at company stores in towns the company also owned and policed. Black workers attempting to organize in the South were lynched. Children were maimed in textile mills.
A software engineer getting four months severance after a layoff exists in a different universe from this so no. There is no precedent. Don't you dare talk about the industrial revolution because its not even in the parking lot of the ballpark.
That your examples come from the 1900s but the changes that caused them started in the 1800s might give you pause.
You should look past the screen and see what's going on. War is not the same; violence against humanity is not the same.
You're right, it's not in the ballpark. It's at the gates. The game isn't on simply because they poisoned the opponent in the duggout.
It's time. It needs to stop now while the body count is low. This isn't about some dev getting severence. They've taken away the street sweeper position and are watching us eat each other.
All the AI hype aside, I wonder if there is a way to avoid becoming one of these faceless corporations where customers are just numbers. For years Amazon has been fantastically customer centered, but at some point they just lost it. I could compile a list where Amazon is actually way more customer unfriendly than in the past now, but I guess everybody already got their own anecdotes about that. So what exactly went wrong and how could that be avoided at other companies?
> All the AI hype aside, I wonder if there is a way to avoid becoming one of these faceless corporations where customers are just numbers.
Limit scale. Enjoy your craft. Become immune to hype and "what's popular," instead focusing on "what otherwise inaccessible experience can we make possible for our customers?"
When you let the money guys take the wheel (not the "passionate nerd" types), it inevitably (and I would argue necessarily to keep the lights on for such a big org) results in a shift to spreadsheet brain.
If you haven't seen it, watch Jiro Dreams of Sushi and see if that way of life resonates. Also check out his interview with René Redzepi from Noma. Lots of great insight into how focusing on your craft implicitly creates the opportunity for creating and delivering great things to others.
Have you read Cory Doctorow's "Enshittification" yet? He describes the process quite well.
Not sure how to address it though. I suspect keeping companies small and focused on quality, sustainability, and free of VC influence would be a solution. It'd take continual work though, like tending to a bonsai.
There are definitely admired companies in the world. Costco comes to mind.
I’ve been hearing Amazon is going to run out of bodies for years now and yet they keep chugging along.
The economy sucks, and they do pay decently for software engineers. Especially now that the rest of FAANG aren't massively over-paying for college hires, I doubt the supply of bright young minds will ever entirely dry up.
How far can a pure mercenary culture get?
All the way to the end
I have worked there for 8 years now and we ARE running out of bodies.
It has become extremely difficult to hire at any level, we have had an open position on my team for a senior data scientist for a year and a half now, with barely any candidate applying, and none of them being competent.
Similarly the average level of new employees has dropped dramatically. The famous "hiring bar" is now below ground.
Amazon has been on my Would Never Work For list for over a decade now. Even the “golden years” being referenced by OP and some commenters in this thread were plagued by Amazon overworking people and doing sketchy things like weighting RSU all till the last few years and then laying people off before their mountain of cash landed
There are now more highly competent devs ready to work for cheap available now than ever before and all of them are boosted with state of the art coding agents…
It‘s the golden age for software engineer employers.
Correction: it's the golden age for code monkey employers. We as an industry have never deserved to use the term "engineering" to describe ourselves, and we are further from that ideal than ever.
I'd work there (if they hired here lmao), they pay good and it looks good on a CV. Or well, it would where I live (europe).
I'm not sure FAANG does look good on a CV any more. The skill set to be effective in those environments is quite specialised and crucially it's very different to what you need in a lot of other software development organisations. There appeared to be a happy cycle for a while where very well paid devs working in one of the few FAANG or FAANG-adjacent companies could jump to one of the others because they were "in the club" and had experience of working at a truly global scale that most software never needs. Those days seem to be over with the mass layoffs and hiring limits. And if you're not working at that scale - and outside that small world almost nobody actually is - those skills aren't always very transferrable and other types of experience often have more value.
#actual-aws-memes mentioned!
I also joined in 2022, and it aligns so much with my experience. Good manager that moves on, then a gradual erosion of "insist in the highest standards" towards a dreaded "good enough", GenAI only accelerated it IMO.
> When AWS first introduced a viable cloud to the world, it was amazing. Back in the 1990s when you wanted to implement an enterprise software solution, you first had to take a guess at what computing power you would need. Next, you would have to order hardware from companies like Sun Microsystems or Dell and that could take weeks if not months to be delivered. It would then need to be racked, powered and provisioned, and then you were screwed if you happened to undersize it or criticized if you spent too much and oversized it.
This is how many large enterprises still operate today. Ironically, the main argument is that it's faster to provision VMs on-prem than it is to get approval to run in the cloud.
Bureaucracy always beats tech.
The story about recovering the account rings very close to me. At least they had coworkers cheering for him, I feel teams are shrinking so much that we'll end up with just the LLM of choice to pat our back with "good work" and "you're absolutely right"
Co-workers cheered while managers were sharpening their axes. One doesn’t do such “heroics” without approval, making the system look incompetent and broken and then apologizing for it without being decapitated in the public square for everyone to learn the implicit lesson. Anyone cheering for him publicly should watch their back, too.
Not that I disagree with the points in the article, but 2022 is hardly the high point of Amazon. That ship sailed decades ago.
Decades...?
At least 1 decade. I left in 2017, and that was already past the peak
Yeah late 90’s, early 2000’s.
> the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need
The goal was never to solve a real problem, like we evangelized for decades. That was how it was explained when resources (mainly time, but also money) were scarce and we could not just throw things at walls. Now we can, and you won't see anyone talk about "make something people need".
Things will be low quality until something sticks, and then money will be poured into it. It's not a bad strategy, but my takeaway from this is: there are multiple plausible explanations for the same thing. People have an incentive to not give you the correct one if it helps you compete with them. But they will give you a sensible one. AI won't protect you from this, experience and real knowledge will.
I have been advocating within my org to replace "fungible" with "flexible" or "generalist."
"Fungible" implies they are a commodity, easily swapped for someone else. In other words, they are so low-value that they are interchangeable.
"Flexible" or "generalist" instead connotes that they are so high-value that they can operate well in multiple domains, easily shifting to where they are needed most.
“Flexible” would work if Amazon prioritized moving people around when the priorities change instead of laying off and rehiring.
You can easily call the typical Japanese life-long employees as “flexible” or “generalist” but not an employee of a company with median tenure rate of 1-2 years. That’s fungible.
Shades of Carlin’s bit on softening language over time here.
Our company also requires everyone to use more AI-related tools, and I don't think there's anything wrong with that. But the quality of work produced using these tools really depends on the individual's ability. Some people don't put in much effort, and the results they produce are really sloppy, which bothers me a lot.
I think there is actually something wrong with that. What should matter is the work produced, not the tools used to produce it. If AI tools really are all they are cracked up to be, then people using them will get ahead, and the company can justifiably point out "your peer gets twice the work done as you" to the other employees. But mandating tool use in and of itself is senseless and counterproductive.
The unfortunate reality is that unless you use these tools it is impossible to keep up. People using these tools well are substantially more productive, and often they were already the most productive.
Force multipliers need force to multiply, but also remember force is a vector, so watch out for multiplying in the wrong direction
AWS lost its way. S3, SQS, EC2 and VPC were great innovations and those services were done by a bunch of engineers who wanted to have a reliable elastically scalable system. This was coincidentally cost effective at the same time. What came after especially the data stack and now the AI services were done by a MBA heavy management team who does not understand innovation and treats engineering like a bank does: putting it in the cost category. Recent financial results show the impact: Google grew almost twice as much as AWS did. Maybe it is just coincidence.
> In this whole pivot to GenAI, AWS has lost its focus on the customer. Instead of working backwards from a genuine customer need, the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need.
Anecdotally, this seems to be the new "mission statement" of many companies.
The account recovery story says a lot. At some size, companies start handling people as tickets. Sometimes it only gets fixed because one person inside still cares.
Being fired for calling out Corruption. That's how I read this.
Absolutely. With his name in the public and apologizing to the customer for sheer internal incompetence. Then also cheered on internally.
I bet as the managers publicly nodded in praise for his heroic act, their hands were already typing his name to be sent to HR for “get this guy out of here on any excuse you can” note. (In reality it would be a nonverbal hint of sorts. Nothing to leave any trace discoverable by lawsuit)
He doesn't think that was the cause, and I don't think it was either. Much spicier material has been posted before. Also the meme is still up fwiw.
Meh you always read one side of the story. I have seen a lot of people getting PIPed out of Amazon and only one or two didn't deserve it, yet all claimed they didn't.
Agents are already getting budgets, what ‘customer obsession’ looks like when you’re marketing and selling to clankers?
This genai is going to bring about huge quality drop in software across the stack and across the domains. I already see orgs that had reasonable software processes transform into orgs where the only metric is how much generated code you can slap and slop together and how fast. There's no success here for anyone.
And this is not a dink on the ai tooling itself but on the organizationan processes that provide the context in which the AI code generation is being used.
Bad processes will always produce bad low quality outcomes regardless of tbe technology.
Was inspiring to meet you at NixCon. Thanks for all your energy and advocacy! You'll always be welcome in our open source community.
I read every comment on this thread at the time of my posting and believe it’s fair to categorize the general sentiment that Amazon has lost its way and may even be in its “IBM stage.”
AWS services still are generally reliable and available. I’d think we’d be seeing cracks here if the organization were in shambles. AWS seems to keep humming along.
Thank you for writing this
Sad to hear. I was at AWS between 2016 and 2021 and had a much different experience.
Side note, and unpopular opinion ahead: while it takes a lot of courage to write things like this and I respect it, but being fired and writing negatively (no matter how justly) about your former employer is considered by many employers as a red flag and can hurt you going forward (even if you are 100% right).
AWS has lost its way.
It’s well into the IBM phase now. Primarily providing important but boring commodity infrastructure, but the top talent that can drive real innovation has long since left the building.
It’s race to stay relevant in AI but always seeming 2-3 steps behind everyone else is one such example of the current sad state of affairs.
Day 2 (maybe even 3 or 4) for sure.
https://aws.amazon.com/executive-insights/content/how-amazon...
I thought Amazon only did memos, not slide decks.
Has that changed, or is it the non-AWS part of Amazon?
They have always done slide decks for external communications. Internally the 1/6 pager were king across the engineering teams at least
The "fungible" point sounds as though the "cattle, not pets" ethos of the infrastructure management has leaked into the management of the staff.
Always has been, IT has been resisting due to (as pointed out in the article) institutional knowledge required, but this is now going away as LLMs can efficiently and accurately write it down and search it. Meta turning ICs into data labelers is trying to make knowledge workers fungible; expect others to follow.
> Amazon has a really odd viewpoint when it comes to the people who work there. They view almost all employees as “fungible”.
Hardly an Amazon-only thing. In fact, enterprises need this mindset, because people moves on, retires, or just suddenly die. With that said, due to its late-stage capitalistic ethos, Amazon is just too overly gleeful about this tasteless reality of life. It's the equivalent of a nephew coming to an aunt's funeral and shouting "A week ago, I told her everybody dies! And now she did! Wasn't I right??? Everybody dies!"
> Also, last year the focus at AWS turned fully and almost desperately toward GenAI.
I wonder if I'm being too cynical, but late-stage capitalism companies also love profiteering, and the mere prospect of firing all those pesky workers and not having to pay their salaries is like cocaine to those organizations. Which is why I think Amazon fulfillment centers will at some point rent robots at a price point between 2x and 3x their current human labor costs, in the hope that it will eventually make economic sense.
> Hardly an Amazon-only thing. In fact, enterprises need this mindset, because people moves on, retires, or just suddenly die.
Enterprises typically have this mindset. Most corporations I worked for in fact treated employees exactly like this.
As for needing this mindset, I am not so sure. There is a spectrum in between going under because a storied employee retired and treating employees as meaningless numbers in a spreadsheet.
But ultimately I fully agree with the whole of your post. I just had to nitpick about this.
LOL That meme. Funny because it's true.
> In this whole pivot to GenAI, AWS has lost its focus on the customer. Instead of working backwards from a genuine customer need, the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need.
AWS has been this way for a lot longer than GenAI, since the basic infrastructure products were built out early on. But when I read this line about throwing things out there quickly, I also think of Google and even Anthropic. Google has a long list of products that got created and killed, as part of their internal politics and promotion culture. Anthropic is currently rushing vibe coded slop all the time to try and win over OpenAI and set up their IPO.
Maybe all the rich high funding companies can afford to this and maybe it is the right thing for them to do. They can afford to make big mistakes without hurting their stability. A true startup or smaller company can’t - they would shutdown because one big investment that fails is enough to destroy the whole company.
to be fair, even though they have "working backwards" and "customer obsession", amazon has always been about making lots of different experimental bets. Bezos:
> To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there. Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. This long-tailed distribution of returns is why it’s important to be bold. Big winners pay for so many experiments.”
> Maybe all the rich high funding companies can afford to this and maybe it is the right thing for them to do. They can afford to make big mistakes without hurting their stability. A true startup or smaller company can’t - they would shutdown because one big investment that fails is enough to destroy the whole company.
Both are following the same strategy. Amazon has a $2.86 trillion market cap. That's the equivalent of 143,000 $20 million Series A startups. Companies like Amazon and Google are basically an integrated herd of cash cows plus a VC portfolio.
S3 is quite good. The rest ranges from meh to no thanks.
“I have to say being fired from AWS is actually a relief. There have been a lot of changes to the company since I joined in 2022, and the company I wanted to work for is no longer the same company.”
Many storied companies can be described this way. It’s a shame. Have any companies hit such scale and kept the ethos and magic of before? Is it inevitable for companies to enshitify themselves in the pursuit of their shareholder’s goals?
Not possible once big parts of the company start not knowing other big parts of the company and the company also has a board of directors that must increase shareholder value at all costs.
I was enjoying the article and then he makes some of the most bizarre claims about what cloud did and how we had to provision servers
If any of you young'uns read this, that is not how we had to do provisioning before cloud.
VMs already existed before AWS came out. You could already provision a new server usually in minutes and rent it month to month.
In fact, all the existing VM server companies had to start calling themselves cloud companies because pointy haired bosses couldn't understand what cloud really was.
AWS was launched around 2006 (2002 internally at Amazon I think).
Where could you rent VMs in 2006?
IIRC there were two ways to run stuff, get your own server or get an account on a big shared computer.
Ehm. Shared hosting was a thing since forever. VPS also existed.
Linode definitely had something along those lines.
Amazon won on APIs and overall integration but VMs were around already.
I remember the story really well as this is when i joined the workforce as a young GNU/Linux fan.
It was pretty regional back then, when I got VMs for clients we used UK providers (ElasticHosts maybe?).
Also people are throwing AWS start date of 2006, but AWS only really started catching on around 2008/2009 if I remember correctly. EC2 came out of beta late 2008, and EBS was only launched around the same time.
I don't think it even officially launched S3 until 2008, which is what all people really used it for initially for cheap remote backups.
It definitely took a while to get good. There was also a period where having 'noisy neighbours' impact VM performance was often discussed here. You didn't tend to have that same problem on the other VM hosts as people were using the VMs for hosting with generally low CPU usage, not for compute.
> Where could you rent VMs in 2006?
I was renting a Debian VM from Bytemark in 2005 that I used to host a mail and web server. I think they were one of the first operators in the UK.
Everywhere. You could rent VMs everywhere.
And they were cheaper than renting AWS. MUCH cheaper. They still are.
The original point of AWS is that could scale according to demand. Have 10 VMs running at lunchtime and 1 VM running at midnight.
But using a cloud VM also required less server admin experience. It was a bit easier and came.pre-configured with things like firewalls.
And THAT is what ended up being the USP of cloud hosting. Especially when they started rolling out all the SQL as a service, redis as a service, etc.
You didn't need to really understand servers to run a server, and it turned out almost all developers really didn't want to understand servers. TBH, I don't, server admin sucks. Right now I'm working somewhere where I have to think about SSL certs occasionally and I consider it a complete waste of my life.
Digital Ocean came out like 5 years after AWS, what was revolutionary about that wasn't that you could spin up VMs quickly, it was the price. VMs went from $20-30 p/m to $5.
For developers who weren't SV rich, that meant you could run a side project without it being a significant cost.
> Long story short, I was able to get his resources restored. All I did was manage to poke the right bear and the support team did the rest of the work (and they were amazing).
No they utterly failed and needed a special non fungible employee to get them to do their job.
> They view almost all employees as “fungible”
I'm glad to see that one core amazon principle has endured the 10 years since I worked there, even if none of the actual leadership principles have survived /s