Between the engineering staff and the warehouse workers, I wonder how long it will be until they have already fired everyone who ever would have been willing to work there.
Even with candidate pools of hundreds of thousands of H1-B engineers and tens of millions of illegal immigrant warehouse workers, there still comes a point where such a big company firing so many people so quickly exhausts all their options.
It reminds me of the Robot Chicken Sketch where Imperial Officers aboard the Death Star all pretend to be force choked to death by Darth Vader so they can avoid getting killed by lightsaber, then come back in under different names in different jobs. It's worse though for Amazon: nobody wants to come back.
Walmart executives seem to claim that the pay raise led to those outcomes; specifically, they talked to employees and realized that in order to get those outcomes they had to stop paying people the least they possibly could because why would anyone stay?
From the article:
> Rissa Pittman, then a store manager in Ponca City, Okla., said it was easier to staff her store after 2015 as wages improved and it became easier to train workers for promotions.
Walmart has been building a fairly robust EX program in the last few years, increasing associate engagement to drive CX and hence business outcomes.
It's likely not just pay, but a concerted effort across the board that drives results, but the pay increase is the one that calls WallStreet's attention.
It seems amazon itself is aware of this issue. The linked engadget article even mentions this:
> "The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."
> It seems amazon itself is aware of this issue. The linked engadget article even mentions this:
>> "The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."
Very few other singular companies have Amazon's rate of growth. I joined in 2012, they just crossed 95k employees worldwide. By the time I left in 2017 they were over half a million. These days they seem to have plateaued around 1.5 million... that's just a ridiculous number of employees
Whether you're in the tech industry or the warehouse, everything I've heard about Amazon is that they work you as hard as they possibly can until you burn out and then they replace you with someone new. At least on the tech side you get options though.
An honest answer that you didn't consider yourself to be lucky used to be a no-hire dealbreaker for Anazon. I wonder if their emigrés still view themselves as lucky.
I thought that was one of the reasons why several US states are loosening child labor laws, they will put 14 year olds to work in Amazon warehouses and on the assembly lines.
That’s an interesting topic. In most states, homeschooling is almost meaningless because there are no required assessments to demonstrate student proficiency in any subject. And 11 states don’t even require a parent to simply notify the state that they’ve pulled their kid out of school.
Put the 14-year-olds on shift after school, put the adults on shift during the day. The teenagers won't have time to do homework but the schools aren't funded well enough to ensure a quality education anyway.
> the schools aren't funded well enough to ensure a quality education anyway
The US spends more per pupil than every other OECD country except for Luxembourg, and exceeds the average by over 50%. If US schools aren't funded well enough, essentially no one is.
Correct me if I’m wrong but I thought in America there’s no obligation to attend school ever, because you can just be homeschooled and then take the GED? I think it’s dumb but I don’t know for sure.
That's one reason for the corporate backing of globalism. It's sold as "mobility", but what it actually means is having the upper hand. It's not about a lack of "talent". It's about a lack of a pool of people who will put up with their crap.
Consumerism rewards this process, because the glut of mediocre goods remains cheap.
That's essentially the entire reason illegal immigrantion and the H1b program exist. They are both in precarious positions that allow them to be treated as indentured servants, forcing them to tolerate conditions Americans never would.
None of it is about talent, it's about giving companies the ability to abuse employees, with a nice side of wage suppression.
Anything intelligence enough to find us, or be putting out signals we can find with instructions on how we can reply in a reasonable number of human lifetimes, is intelligent enough to stay well hidden for us lest we infect them with our stupidity!
Slightly tangential: there's an excellent Norwegian TV series based on the premise of suddenly appearing immigrants from earlier time periods, see https://en.wikipedia.org/wiki/Beforeigners.
Agriculture, food processing and handling and everything associated, particularly meatpacking, and that's valid across countries. There's a reason that even if you pay (relatively) obscene wages, there will be no domestic employees willing to pick up these jobs - the work conditions are usually horrible because there are few industries as cut-throat and cost-sensitive as anything related to food.
The root cause is global competition, especially from countries with very low wages - Ukraine, a country in Europe, for example, was a top food supplier for Africa -, and widespread income disparity in many Western countries - 67% of Americans self-report to live from paycheck to paycheck [1], for example. First it's "luxuries" that get the cut - travel, eating out, entertainment - and once everything has been cut, people go for savings in food because that's the last large expenses block that they can meaningfully control.
> About 41% of American workers earning between $300,001 and $500,000—and 40% of those making over $500,000—say they’re living paycheck to paycheck, according to a new report from Goldman Sachs.
The problem is that these are usually measuring self described rates. And people aren't using the same definition when self describing. Have come across plenty of people who claim to be living paycheck-to-paycheck because they have nothing left over after maxing out 401k & IRA, socking away $1K in a savings account, paying for their posh apartment, and more. But of course that's not what it's supposed to mean.
The Maritime industry can run into this too. Tug companies, barge companies. The level of consolidation vs the size of the labour pool is what does it.
} There's a reason that even if you pay (relatively) obscene wages, there will be no domestic employees willing to pick up these jobs - the work conditions are usually horrible because there are few industries as cut-throat and cost-sensitive as anything related to food.
Yeah I am not sure that is true either. People will work at these facilities but we all know the conditions there are horrible because they can get away with it - you can't threaten an American with deportation for whistle blowing. As long as companies get away with abusing immigrants then the labor market will always keep them in demand.
Some entire industries have this problem, eg trucking. There is this shortage of truck drivers. The quiet part is that it's at the rates trucking companies are willing to pay. That rate is based on what they can charge to customers plus a thin profit margin. Because trucking is too easy to get into, companies can't simply raise their rates because other companies will undercut them (in their own desperate bid to survive). This kind of market competition is healthy for the "market", perhaps not healthy for workers.
The depletion of tech workers for Amazon is similar. The part that isn't said is: at the salaries Amazon is willing to pay. Amazon has a different market to worry about, the stock market. They can't just increase their HR spend 25% without taking a hit in the stock market. And I guess they aren't willing to change the work environment to be more attractive. Maybe they can't at their size, as it can be hard to avoid dead weight.
Google doesn't hire in the US for some PAs, for similar reason. Salaries are capped (artificially) by stock market demands. But Google doesn't call it worker depletion.
Shortages always exist at a given price. If people demand their iphones cost only $100, there would be a massive shortage of iphones.
Same goes for labor: if you pay shit, demand for that job is going to be shit.
Of course on the labor side, part of the "price" is job satisfaction, working conditions, etc. Many more people would rather be receptionists at $17/hr than working in a warehouse or factory for the same rate.
i think another part of it is whether there's any meaningful difference in value for a mediocre vs good employee.
let say you ran a trucking company and decided that you'd pay more to ensure you cornered the market on 'good' drivers. but... it turns out that your customers don't care if you have good drivers or mediocre drivers, so you can't justify charging a higher rate.
> Because trucking is too easy to get into, companies can't simply raise their rates because other companies will undercut them.
How are they going to undercut them without drivers available to actually do the work?
This narrative is absurd. Per mile domestic truck shipping rates have gone up dramatically in recent years, and are generally quite variable. Trucking profit margins were until quite recently way up, the fall driven by high fuel prices and interest rates (which increase the costs of equipment financing and insurance) and reduced demand.
The truth is trucking is currently going through a recession with freight demand down and empty miles up. Trucking companies most certainly could raise rates and pay pay to attract more drivers; right now they don't want more drivers.
Total US retail sales are over $700 billion per month, which is more than twice what Amazon does in a year. Amazon isn't even #1 in the US; that prize belongs to Walmart.
Both facts could be true, if the previous post was referring to the cardinality of sales not combined value and the average Amazon sale is lower than the average sale overall.
The average Amazon sale would have to be ~25x lower than the overall average, seems rather unlikely. Not to mention that counting sales only makes sense within a product category.
60% may be true in some random product category, but clearly not across all of retail. Maybe that’s what’s being remembered, but missing the surrounding context.
It might be a sign flip of their share of online sales. They account for about 40% of online sales (yes, in-person retail still vastly outweighs e-commerce), so 60% of online sales are not Amazon.
Every single engineer I know who went to Amazon except one lasted under 3 years and to this day, often ten+ years later, they all will mention how much they hated it.
The one exception is an engineer who stopped engineering, switched into product, and transferred to China to hit on the women there.
Some Amazon practices actually sound great to me (short documents, read before the meeting) but so many things just sound needlessly, relentlessly cheap.
There seems to be some sort imputation that, just because someone is on an H-1b, that they are not a good engineer.
I used to be on an H-1b and gladly came back home to India. I run my own business now. And yes. I'm ex-Anazon. It was a tough place to work, but circa mid-nineties, the stock options made it worth working for them.
I'm willing to bet I'll outcode a significant fraction of the audience on this site. And I'm not even close to the best developer around. Some of the smartest people I've met have been on an H-1b visa. Please consider not letting prejudice affect your view. You'll do yourself a disservice by underestimating your competition.
The only thing I implied is that workers with fewer rights that a U.S. citizen are easier to exploit and abuse.
If I refuse to take a work-related call at 3am, the worst that can happen is that I get fired, and spend months looking for a new job.
If you refuse to take a work-related call at 3am, you get fired and lose your ability to stay in a place you have lived for 5+ years and made your home.
That's BS, and I hate it almost as much as you do. You can be blackmailed with deportation, and I can be replaced with someone who can be blackmailed with deportation. We're both getting screwed in this current arrangement.
I'm interpreting this as mid-1990s, in which case I very much believe in your technical ability. My dad came over late-1990s and he worked at mid-sized companies ever since. Even then, he and his H1B peers were decently intelligent.
I would caution your defense of today's H1B/L1s/OPT workers; I'd say the quality of Indian engineers in the US has halved every 10 years.
Today's Indian engineers come to the US because they can't enroll in a decent college in India and/or obtain a upper-middle class salary from a job. It is an entirely different mechanism for which people are migrating over. It used to be brain drain, now it is sewage drain.
The H1Bs in the big tech companies are maybe 50/50 technically decent, but everywhere else, they are just taking contracting spots. It is a very corrupt and bloated system that has to go because they are not providing valuable work.
I think you’re not considering the other side of that perspective. I am sure you are very happy for your fortune to have been plucked out of India and been given the opportunity to work at Amazon and presumably live in America, which put you in the place you described that seems to be in a really good position today. The issue is that not only was the H1-B meant for highly specialized people that cannot be found in the USA, it has very long been absolutely abused by American corporations and politicians that have been betraying their own people for several decades now by engaging and ignoring this abuse that was really just about undermining salaries of Americans by giving the opportunity to you rather than Americans, while it was really mostly about enriching the rich. You were essentially just a method for the rich to get richer.
I am sure you are a wonderful person, but it’s simply an unjust treatment of Americans, even if you personally had nothing directly or reasonably to do with it. The betrayal and abuse was perpetrated by the “Americans” that led the corporations and paid off the politicians, and also the American citizens that were distracted and careless about their own politics and government and future for their own children. I doubt you would be ok with your own ruling class and rich to betray your children and the future of India, would you? It’s crazy, but America’s people largely and for a long time absolutely betrayed their own people.
I would not wish it on any society, even though it has been pervasive all over the “West”, where the rich, corporate captains, and politicians betray their own people. Imagine if your Indian politicians were to sell out India to the West or maybe import Africans or something similar, I would hope that the Indian people would make it absolutely clear to the politicians and rich that they are staring down a loaded gun and it’s not their finger on the trigger. So do I also wish it for the people of all of the western countries that they retake their sovereignty and self determination away from the rather parasitic oligarchy that has unconscionably been betraying its own people out of undeterred greed and crime against the very people that allowed making them rich and powerful in the first place.
It is not a personal thing, I think it’s just that people are recently getting a lot more angry about things because the American empire is hitting a rough patch that it has not experienced in anyone’s living memory and as it is said, (adapted) the naked people start getting angry when the tide goes out and there aren’t enough jobs to also be super generous by giving them away to Indians benefiting from the abusive systems of the parasitic cabal of the ruling class.
What you may also not be totally aware of, is that H1-B is only one of many different systems and programs that have been abused and quite literally benefit and profit foreigners overt Americans. Imagine if that existed in India; where I go to India, make 2-3x what the average Indian makes, the government gives me free housing, my children get free education and free healthcare, and I get extremely beneficial government secured loan terms on business loans and get grants to start a business and free consulting and services, and I get to bring dozens of my friends and family into India to work in my business, and I also get beneficial home loans to buy up houses and drive up prices, and my foreign children get preferential treatment in Indian universities (…while local Indians don’t get those things) and I run for office while all the foreigners I and my advantaged community brought over to India start getting our people into the government and we start taking over Indian institutions and government offices.
I combine and crossed things a bit because is a bit more complicated and nuances of course, and many Americans aren’t even aware of just how many programs and states are in place that advantage foreigners and disadvantage native Americans, who could even very well be the descendants of the founders of America. That’s why things have gotten rather tense and as it looks, unfortunately, it will likely only get worse from here; especially as BRICS builds out more of their alternative fiscal, monetary, economic, geopolitical structures; and the same traitors that control the USA will/are starting to get very nervous and borderline panicky. It seems Thucydides Trap is in full effect.
> the government gives me free housing, my children get free education and free healthcare
None of these are H-1B perks. (Note: H-1B, not H1-B. It comes from ¶ H(1) of the INA of 1952; H-1A for nurses and H-1B for non-nursing specialty occupations [1].)
Immigrants pay for housing in homes that pay property taxes that fund public education. Their employer pays for their healthcare that costs multiples what most of the same treatments and drugs cost in India. (Only once they become a resident alien do they qualify for marketplace subsidies.)
> native Americans, who could even very well be the descendants of the founders of America
You've got to be shitting me. (Try native born next time.)
> my foreign children get preferential treatment in Indian universities
The students of H-1Bs (from India, no less) do not get preferential treatment in American university admissions. If anything, it's the opposite. It's why Indian Americans join lawsuits by Asian and White Americans around removing race considerations from college admissions.
> Thucydides Trap
"Research by Graham Allison," the guy who coined the term in 2011 after a career in the Reagan administration and, before that, at the RAND Corporation, "supporting the Thucydides trap has been criticized" [2].
It is even funnier when this argument comes from Canadians, Australians and some New Zealanders as these are some of the more open and shut cases while other countries can be argued to be on some spectrum of foreign vs native depending on the time scales considered.
Generally it is framed to meant that we are the people who captured and built these countries and they belong to our 'culture'
> “native Americans who could be descendants of the founding fathers” is rich with humor
Yes.
Though I did consider, until noting the context, that they might be referring to descendants of both the founding fathers and Native Americans. Chilling, actually. To think that a good number of one branch of our founding fathers' descendants may have been exterminated by another.
Maybe Jefferson brought some of that whacky weed down to the slave quarters and had a big party. Had a good time. Drugged them up first. Not unlike maybe the underclass of sex workers today, in dire straits and desperate.
This is one of the most painfully ignorant things I've read in hacker news. I don't even have anything constructive I can reply to, other than a recommendation to come out of your bubble a little bit.
> Imagine if your Indian politicians were to sell out India to the West or maybe import Africans or something similar,
…
Firstly, this is so wildly tone deaf, that I don’t know what to say.
Secondly - India (and most developing nations I will bet) have had violent protests at some point in their history about people from other parts of the same country coming in to take jobs.
Most of the world understands why America feels the way it does. Hell I personally have better articulations of America’s problems than many Americans do.
But for the love of all that is holy, please don’t end up betraying your ignorance of the rest of the world and reminding us of some of the worst features of American stereotypes. It may be meant as an effort to find common ground, but only succeeds in irritating and potentially alienating.
Believe it or not, compared to the vast majority of humanity, your problems are better than what they will face.
I do not ask you to take an interest in the world. It would be nice if you did.
I do ask that you serve your own purpose, your own argument, better.
How is he disrupting our society? He's working & building things in our country, and paying taxes that don't benefit him as much as pay for our entitlements.
The very highest salaries are to an extent, but almost any major metro area in the US has plenty of engineering jobs that are very high paying compared to almost anywhere outside of the US.
I like to think I'm halfway decent at my job, and I wouldn't work there once. During undergrad, my landlord working for AMZN on the opposite end of the country offered me an interview, but it was during final exam week.
I asked if I could schedule the interview after my final exams, and his arrogance really showed when not only did he refuse, but then insisted my exams are not don't even register on the same scale of importance as the opportunity to work for Amazon.
Somewhat related: a recruiter at Google cold-called me a couple months into my first job out of undergrad back in 2016 and was similarly condescending about "the chance" to work for Google compared to everything else. I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview.
I regret being born too late to work somewhere like Bell Labs, SGI, or Sun. I had a ton of graybeard wizard coworkers from these places, and they were all a pleasure to learn from and even better friends. For the first 2 years of my first job, every day of work was like walking into the Shire and talking magic spells with 20 Gandalfs.
That job was great until I got put on a team with a guy who was a former middle manager at some IBM-like company and went from being surrounded by people lightyears ahead of me to being surrounded by Dilbert characters. The messed-up part was that it wasn't even punishment. I was rewarded after completing a project with my choice of which team I joined next, and I joined the wrong one. I assumed that joining a new team to utilize this newfangled "cloud computing" thing would be trailblazing, and I didn't do any diligence on who I would work with.
To this day, I still regret not rejoining the first team I worked for, otherwise I would still be at that company and happy about it. Then again, the boredom and discontent while being on that sucky team is the reason I started investing, and now I can buy a house in cash and fund myself to do whatever I want for at least a decade. Hard to complain about the way things turned out.
I interviewed there and got an offer a number of years ago (in 2006 or 2007, for the then-nascent AWS team). In retrospect, it would have been a great career move.
I also was interviewing somewhere else and told the recruiter. His response was to be verbally abusive and say the other place "sounded boring", and then went off on how if I turned down an offer, Amazon would never talk to me again and I'd basically ruined my career. I decided I didn't want to work at a place with that sort of culture.
They reached out to me in 2013 for an interview and again in 2024.
Some 20 years ago I started a job at Google in Mountain View, and they were paying for a rental car, so Enterprise sent a driver to pick me up to do the paperwork. On the way I was chatting with him, telling him how amazing life at Google was, all the restaurants and the stocked kitchens and massage rooms on every floor of every building etc etc. He said "Do you know what this campus used to be before Google?" I said "Yeah, they told us at the orientation, it was SGI." The driver said, "Yes, and ten years ago it was exactly like that at SGI, too. I was an engineer there."
In the UK we have a heuristic that by the time a tech giant builds a big UK campus (an imitation of their SV HQ) then you know they are in the decline phase. Some of them decline so fast they don’t even get to fully complete the campus, yet others seems to have beaten this curse… so far.
They have more than one building at King's Cross. One has been finished for ages but the new one I don't think is done. Not sure what happened to the fox on the roof. (https://news.ycombinator.com/item?id=44229727)
Maybe. Was attempting to end this cliffhanger on a positive note without judgement.
He had the means so he paid off his mortgage, invested in another property, took care of college fund.
Decided he didn't want to spend rest of his days wringing his mind and drenching off energy into deciding and debating whether it is ok to call a lambda from another lambda (yes I have seen this in production and the more experienced engineer decided to do it because he had been there longer and decided that's what he wanted to do... don't ask me which company was this but it was a FAANG) or setup a step function to orchestrate the two lambda calls... or some such equivalent problem he might have come across in his SGI days.. and instead picked a job that required little amount of cognitive effort compared to what he would have done if he was still in the same line of work but still managed to support the rest of his life/needs/responsibilities.
Might as well have been an artist, construction worker or he might as well have done nothing, absolutely nothing, and it could have been everything that one would have thought it could have been.
I won't judge knowing what I know at my age. People do what they do.
Feel free to imagine what you would imagine this ex-SGI engineer's life to have been and make it negative if you want it to be. No one knows until OP throws in more details if they have any.
Someday HN will become TN ("Taxi News") and will be populated by bored former tech workers seeking intellectual stimulation. I see, in my crystal ball, the young'uns, when perplexed by the latest technology, look at each other and say "Let's take a taxi!"
BTW it's Fall - The best to all of you!
Here's to memories:(Four Strong Winds)
One of the best stories I have ever heard in here to be really honest. Sounds like a joke but its packed with subtle meaning of how companies rise and fall so quickly.
You have said that the driver worked because he had (enough money?) and he might have wanted to relax with the driving job but still, its an amazing story.
I think the key difference between the old big guns, SGI, IBM and the likes, and today's Google, Microsoft, Amazon, Meta and Apple is the diversification of income streams. Even if any one of these companies just completely fucks up an entire business line or it gets replaced by something better, it doesn't matter because the companies themselves are so utterly large they can and do survive that - or because they can, like Meta, just buy up whatever upstart is trying to dethrone them.
I don't believe they can keep this up forever. Take Google which is dependent on Google Search. That their search is becoming actively worse is common knowledge, reason being more searches equates to more ads shown. If a company which respects you as a user comes around people will jump the boat.
We can see this with YouTube. YouTube shows so many ads, that people have been using TikTok instead. They say TikTok is also bad for them, but they rather use it than watch an ad every 30s.
Point being entshittification comes at a cost, and companies partaking in shitty activities can only keep this up for so long.
Not sure "enshittification" applies here, as Google isn't leveraging a middleman role to maximize revenue extraction from both sides of a transaction it facilitates, but I think you're right that Google does still have most of its eggs in one basket. Search and YouTube ads together still make up a majority of their revenue: https://stockanalysis.com/stocks/googl/metrics/revenue-by-se...
"Google leveraging a middleman role to maximize revenue extraction from both sides of a transaction it facilitates" is exactly how I would describe the google ads business.
>I don't believe they can keep this up forever. Take Google which is dependent on Google Search. That their search is becoming actively worse is common knowledge, reason being more searches equates to more ads shown. If a company which respects you as a user comes around people will jump the boat.
I think their strategy is to poison the internet so thoroughly that no better search becomes possible. The costs alone to spin up a new search company are an enormous barrier to entry, they'd only need to erect a few more to make it impossible.
Intel never had a true monopoly over x86, there always was AMD and a few others who made x86 CPUs although it's only AMD these days. Yes, they had market dominance, but (similar to NVDA) in only one specific market: x86 CPUs.
Intel never managed to leverage its dominance in x86 CPUs into dominance in other markets though, and that is the key difference to the ultra-large companies I mentioned... yes, they did have ARM offerings (XScale, I 'member tinkering with an NSLU2 decades ago), they did have a cellular modem line (that failed and got sold to Apple eventually), they still do have the Intel Wireless lineup (which is pretty widespread but has a healthy competition), and they got a decent dGPU lineup that nevertheless is at, what, 1% of market share?
And that is what is screwing over Intel at the moment. The server CPU market is going down the drain, gaming consoles went to AMD, Apple is completely lost as a customer (thanks to Intel's various fuckups) and consumer device demand is shifting to phones where Intel has absolutely zero presence. And on top of that their fabs have fallen way behind plan - to think of that Intel has to go to TSMC? How far the mighty has fallen.
Check the list of companies they've acquired, divisions they've divested, random research they're doing. While mainframe is a big portion of their revenues (depending on year), they're super diversified.
That’s nothing new. Conglomerations had been around for decades before SGI et al and that type of organisation has its own problems. So you do see them fail too.
For example Thorn used to be massive in the 80s and by the end of the 90s it had ceased to exist. Arqiva is another that’s presently in freefall despite previously being too big to fail.
Also, I dont really think you can accuse IBM of a lack of diversification.
That’s just because they spun off all the high quality companies (Agilent, Keysight, Verigy, Avago). The PC server and consumer print business have always been commodity product.
The question that arises is: How can you potentially spot which companies are about/likely to enter a 'golden era' when you interview there? What questions could surface some sort of likelihood? Is it possibly to identify them before they enter the 'golden era'?
It doesn't matter. Most jobs even at those companies were not in the interesting areas you heard about. For every one person at the cool jobs there were thousands elsewhere who had regular deadlines and a regular job. Odds are you wouldn't have had the cool job even if you were born in the right era.
Better advice: when interviewing ask questions when they ask if you have any! Find out what the job is really like.
Ask what hours they normally work - if they give exact times that means they are strict about the times. If they give a lot of hours that means you are expected to work a lot of hours. If they give a range that means they really have flexible times. If they talk about leaving early for their kids third grade events that means they support families.
Ask what they really wear - this is clue to what the dress code is like.
Ask about the perks you care about. I don't play ping-pong so won't mention that perk if I'm interviewing you, but if you ask I can tell you that there are regular tournaments and people do play games here and there, but the tables are empty in the middle of the afternoon: if you care about this perk ask, otherwise focus questions elsewhere.
There are a lot of great jobs. There are a lot of bad jobs. There are jobs that you would hate for reasons that the people who work there don't even care about. There are jobs you will think are great that others will hate.
Yeah, but I mean after decades of experience, you already know how to do those things, and they're fairly basic stuff to know and learn from.
Was thinking more "Imagine you have 30 years of experience and casually looking for the next Bell Labs, what to look out for when there are the company?"
I don't think it's really possible for the average employee. You'll just do an interview, like the vibes, and get unbelievably lucky.
By the time their golden age is known outside of the company they are very likely near the decline phase; even if they aren't you are going to be competing with the best now.
For actual upper level leadership: they have the ability to make the golden age happen but studying the circumstances that allowed it to happen at other companies and being very selective about employee number 2-49. After that it's out of your hands.
> being very selective about employee number 2-49. After that it's out of your hands.
Indeed. There's an old saying that A-level people hire A-level people, but B-level people hire C-level people.
Obviously this is too simplistic: How do any B-level people get there in the first place? But there's still some truth to the idea that the overall talent level of a company tends to degrade as it gets larger unless very unusual structures are in place to work against that tendency.
That's just a plain scaling issue, isn't it though? Eventually, the supply of A-level people dries up, no matter the compensation offered. If growth is to continue, B-level people must be hired.
Yes A-level people are rare and expensive. The mistake I see too often is companies not focusing on keeping their core revenue-generating team A-level. Put the B-level people in support roles. When you dilute the core revenue team with B- and C-level people, the As tend to leave and then you're in big trouble.
The quote is from Steve Jobs and is absolutely true. As soon as the first bozo infects your team, they will start hiring other bozos, and after a while your org has regressed to the mean. Therefore you should hold a ridiculously high bar for hiring. A temporarily empty seat is preferable to a non-A player.
> By the time their golden age is known outside of the company
Yeah, but that's the thing, when you're interviewing, you usually have some sort of access to talk to future potential colleagues, your boss and so on, and they're more open because you're not just "outside the company" but investigating if you'd like to join them. You'll get different answers compared to someone 100% outside the company.
I think the problem is false positives, not false negatives. The people you interact with during the interview process have all sorts of reasons to embellish the experience of working at their company.
> The people you interact with during the interview process have all sorts of reasons to embellish the experience of working at their company.
That's true, but you have to be kind of smart about it. If you just ask the question "Is working here fulfilling?", of course they'll say "Yes, super!". But you cannot take that at face value, your questions need to shaped in a way so you can infer if working there is fulfilling, by asking other questions that can give you clues into that answer.
I worked for a 100-year-old Japanese optical equipment manufacturer (household name, but I don't like to mention it in postings). One of the top-Quality manufacturers in the world. I worked as a peer with some of the top engineers and scientists in modern optics (and often wanted to strangle them).
I worked there for almost 27 years.
The pay was mediocre. The structure and process would drive a lot of folks here, into fits.
But they consistently and routinely produced stuff that cost tens of thousands of dollars, and that people would stake their entire careers on. Stuff that some folks would assume was impossible to make. They have thousands and thousands of hard-core patents.
I felt pride for working there. My business card opened a lot of pretty amazing doors.
It's disappointing to see the stuff that folks here post, when I mention it. It almost seems as if people think I'm exaggerating or outright lying or boasting.
I'm not. There are places that foster greatness; simply by being a place that has a long culture of accomplishment. I was just someone that stood on the shoulders of giants, and I was lucky to have the experience.
That said, I think some of their managers made some big mistakes, and they took a drubbing, but I will bet that they are already getting back on their feet. They are really tough. They weathered being bombed in World War II, and multiple depressions and recessions.
Thanks a lot for sharing your experience, I at least appreciate it!
With that said, if you were to try to figure out how someone from the outside could see that it was a great place to work, during an interview, what questions/topics do you think could have surfaced that as clearly as possible?
Hard to say, these days. Interviewing seems to be a pretty nasty, adversarial process. It wasn't, for me, back when. Not sure if any questions would have done it. I observed the place.
In my case, I was contacted by a recruiter (the old-fashioned kind, which no longer exists). It was quite low-key. When they first contacted me, I thought it was a joke.
I was flown out to a trade show in San Jose, for my initial interview, and to Long Island, for my follow-up. There were no coding tests. I started as an engineer, on a brand-new team of two. I became the manager of that team, after a few years.
I think observing the people there; seeing how they interact with each other, is important.
Of course, looking at their products is also key. Asking yourself "Do I want to help make this stuff?" is important.
In my case, I was intrigued by the culture of the Japanese. I was born overseas, and spent most of my formative years in a pretty heterogenous environment. I like to mix it up with strange (to me) people.
Are you also aware, that typically members of your generation criticize members of my generation for being soft and lazy, despite this new harsh period of adversarial interviews? It boggles my mind that you found a job at a nice company (to say the least) without a test (and that seems like a bad move on the part of the company ?)
Yep. There's a balance between doing enjoyable work, getting paid what you feel your work is worth, and feeling like what you're doing is of some sort of value to your community or the world as a whole.
Maxing out all 3 of those is incredibly rare, but I think once people reach some degree of financial stability, almost all of them go for a job that feels like it's meaningful.
Sure, that makes sense. But in order to know what places/communities/organizations are worth getting involved with, that has the right base conditions at least, how to identify those? Not every place has the same likelihood I'd wager, but based on what?
You're getting answers in child responses that while accurate are not necessarily answering the spirit of your question. In my personal opinion, you'll find what you're looking for by searching for a high growth Series A - B startup (I would recommend Seed but that's almost a different animal in terms of risk) with a technical product and strong technical founders + eng leadership.
When you're at a company at that stage that's doing well and has a lot of commercial runway ahead of it, the reality can often end up being that the golden age will last long enough for a very pleasant 4-6 year tenure if you so decide to stay at the company through its growth phase (which often takes it from a 50m-100m valuation to $1B+). Some of these companies will also make the leap from $1B+ to $10B+ or beyond (which makes the golden era at least as long as 6-10 years) and although nothing lasts forever, it can last long enough for you to find what you're looking for at least for a decently long period of time. This pertains to what other commenters have mentioned with regards to "making it golden" -- the golden era is what it is because everyone needs to make it golden and the company is too small for anyone who would dilute that for their own gain to do so without anyone noticing.
The challenge to this approach is that it requires being able to assess a company's commercial prospects as well as the quality of the company's founders, leadership and early team well enough to assess whether the company merely looks like a golden era company or whether it is actually the real deal -- something which even professional investors who target these kinds of companies struggle with. It is possible, but in my experience, it definitely took a couple of rounds of trial and error and getting burned a few times before my radar worked.
Yeah, see also: Digital Equipment Corp until the mid 90s (where I did my co-op). Lots of brilliant people there, coasting on their legacy until they fell hopelessly behind from mismanagement and bloat. I was lucky to catch a glimpse of it at the end.
I interviewed with Amazon a few years back. The whole thing turned me off. A recruiter reached out and I was interested (it was late 2020 and the money was tempting). But before the first phone screen I had to have a call with the recruiter again, where she gave me a list of things I needed to "study" and was told that "successfully candidates usually spend 5-10 hours preparing for the interview". The study list was the usual list of CS101 topics. I didn't bother preparing and it was a good thing because on the phone screen the guy just asked me some a fairly mundane coding question and then some more general stuff (it was actually a very reasonable interview). Based on that they wanted to proceed to a final interview which was an all-day affair (on zoom of course because this was during the pandemic). But first I had to do ANOTHER 1h call with the recruiter where she gave me ANOTHER list of things I needed to "study" and reminded me that I should spend 5-10h preparing. That was too much for me and I politely declined the opportunity.
I refuse on principle any interviews that expect you to study or "prepare" for an interview. I'm sure I've missed out on some money, but they've also missed out on a pretty good engineer and teammate :-)
You should prepare for an interview. However 5 hours seems like a lot and I question if CS101 is worth preparing for. (If I know you will ask about a red-black tree I can look it up - but like most engineers I never think about it because my standard library has it implemented for me - unless the job is implementing the standard library I would not expect you to ask that question)
You might be asked to write something like fizz-buzz in an interview - but the point is there isn't a good answer to that. (there are a few possible solutions, but all of them have something you should not like - which makes it a simple yet real world like problem and thus something you should be able to figure out in less than an hour without study)
What you should prepare is figure out how they interview and thus what questions they might ask. (nobody will tell you what questions will be asked, but they may tell you the style) Practice the answers. Practice stories of how you worked in the past so you can twist the story to answer the question (the above is how you should prepare for the STARS interview my company does). If you were in prison or something then be prepared to talk about why they should believe you are reformed, but most people don't have such a thing in their past that they should find.
yeah, you should put your prep time into the non-technical parts of the interview.
- everybody you meet is going to ask "so, tell me about yourself" so you better have a good answer. have a pitch that highlights relevant parts of your work history, discuss goals/interests, show a bit of personality.
- there's always going to be "do you have any questions for me?", so you need to have a couple of questions ready to go that make you seem interested/thoughtful AND help you extract good signal from the interviewer.
every job process i went through so far has been applying -> if they like me a call with the usual job interview questions -> if both want to proceed a meet-and-greet with the actual team -> if both still are interested, sign a contract.
I think the idea is they want to inconvenience you to filter out people who aren’t desperate and willing to deal with their bullshit. But, expecting you to cram for an interview just makes it seem like they value metrics over actual merit.
I am absolutely sure that the money you missed out on had a bigger affect on you than them missing out on hiring you.
Every large tech company or any tech company that pays decent money requires preparing for coding interviews *if you are trying to get hired as a developer*.
I personally didn’t do much prep for my Amazon loop accept practice answering behavioral questions in STAR format. But I also had to thread the needle of having experience to get into the Professional Services department as someone who knew cloud, how to talk to people, architecture and leading projects.
If I came out of college post 2012 instead of 1996 with path dependencies in 2012, you damn well better believe I would have been “grinding leetcode” to make BigTech money.
I call it a “gravity problem”. I might not like or completely understand gravity. But it makes no sense to complain about it. I’m not going to jump out of a window on the 30th floor.
If you want to make $150K+ straight out of college and $259K+ a year three years into your career, you play the game. If you don’t want to play the game, accept the reduced amount of money from staying in enterprise dev.
At 51, I would rather get a daily anal probe with a cactus than deal with any lathe company especially BigTech again and I’m definitely not going to chase after a job where I have to work in an office. But I know what I’ve been giving up for the last 2+ years by ignoring recruiters from GCP’s internal consulting division based on my stances.
But again, I’m also 51, I’ve done the build the big house in the burbs thing twice and I have grown (step)children that I’ve raised since they were 9 and 14 who don’t live with us
Even though I worked for companies that killed Sun, I never stopped admiring the foundational work the company was doing, which was not just cool, but critical for technological progress, and was very sad when the company sold out to Oracle and was gutted alive. HPC stuff Sun pioneered is still very relevant today. In an alternate timeline, Sun fully embraced Open Source and became a key pillar of the internet today.
Unfortunately, while we are well aware of cool tech companies that were ran aground by the finance/sales/management consulting types, Sun felt like a company ran aground by engineers.
Zuck famously kept the Sun logo up for quite a while when Facebook bought Sun's HQ campus, as a warning to the employees of what they could become. In some ways, Facebook/Meta is the spiritual successor of Sun, just like Google became the spiritual successor of SGI when they bought the SGI campus.
But these two ad driven companies never quite became the new Sun/SGI, for better and worse.
I don't disagree that Sun was a company run aground by engineers -- though I certainly like to think of myself as one of the engineers trying to navigate us around the rocky shoals! For whatever it's worth, I broadly stand by my analysis on HN fourteen years ago (!!) of Sun's demise[0] -- which now also stands as clear foreshadowing for Oxide eight years before its founding.[1]
There's a world where Sun did what you hoped (became a systems company) and created Joyent in-house. However, hyper-scaling means going fast and cheap before good comes along. Sun's habit was fast and good and that's an extremely difficult hurdle to overcome culturally. (By fast I mean growing a platform, not raw performance, FWIW).
Solaris 10/11, with all its technologies (zfs, zones, crossbow, dtrace, etc), was the pinnacle of UNIX that came out just when the world changed. At a company I worked at circa 2008-12 (that was a solaris shop) we essentially created a proto-docker with containers and ZFS that allowed rapid deployments and (re)building of our systems. It was a game changer for on-prem.
I'm not sure what Sun could realistically have done to come out the other side of the dot-com carnage. Other companies in roughly equivalent situations come to mind. You start looking at doing a hard reboot when the margins for that reboot aren't there and it's difficult to see the light at the end of the tunnel. Maybe an earlier reinvention involving more open source and alignment with where hardware was headed. Don't know.
Sun did waste a lot of money in buying MySQL, $800 million in cash and $200 million in stock. Certainly a distraction, as well.
Sun never offered any way to inexpensively get onto the on-ramp of Sun hardware and software as they thought they could continue selling high-margin hardware forever; they had their $995 V100 which even included their much-loved LOM which was a remote-management device like iLO/DRAC/IPMI , then followed it up with: nothing.
That Solaris/Toshiba laptops deal was interesting, but if I recall correctly the price was a bit too much, and maybe it could have been considerd yet another distraction.
I surely would have liked to get one of those laptops, though.
Well, in stock market terms the MySQL deal paid for itself. It pushed the stock well up. However turning this in real money wasn't possible in the year they had till IBM and Oracle did their bidding.
“ Believe me that some of us understood this: I worked extensively on both Solaris x86 and with the SPARC microprocessor teams -- and I never hesitated to tell anyone that was listening that our x86 boxes were starting to smoke the hell out of UltraSPARC.”
Was that before or after you realised the Linux kernel devs were better at squeezing performance efficiencies out of x86 than you guys were?
Interesting. That was not my perception of Sun at all. “The network is the computer” was a marketing campaign. Java was a language developed for IoT/toasters, and then hard pivoted to a write once run anywhere weblet language (ultimately to be replaced by a guy who threw together an integerless programming language that sounded like a skin condition, renamed to ride the crest of energy sun marketing money threw at things).
Sure, Solaris was rock solid, but it was also pretty conservative in its march forward as a Unix, being ultimately trumped by Linux.
Sun had an amazing team of people that worked on Self project led by David Ungar and others (Lars Bak who helped give us V8). They let the whole team go, who then went off and did sime cool things with dynamic optimization, which Sun ultimately ended up hiring/buying back to create the HotSpot VM.
Any NIH and other dysfunctionality went far beyond the engineers at Sun.
> “The network is the computer” was a marketing campaign.
No, not at all. It became a marketing campaing in the very late 90s dot.com boom, but the concept that defined Sun goes back to the beginning, 1984. Back then, that was a radical vision and Sun truly lived it internally for a long time.
Also, the number format is a standard, not only used by JS, and given that it was supposed to be a minimal scripting language it is hard to argue against the initial design choice of choosing one all-encompassing big standard, and not burden the language with a complete set. Since he criticism was on the initial design:
> ultimately to be replaced by a guy who threw together an integerless programming language
I would like to refute it by pointing out that the criticism ignores the initial use case, as well as the actual existence of integers within that larger number format standard. Later, when enough people (and companies) demanded it, a big integer type was added, after all.
Internally runtimes use different paths depending on what kind of number it is.
For many use cases of integers, especially internal ones, like array indexing and counting, those integers are just that, and an extra integer type for extra purity is not much of a problem. For other uses of integers, e.g. finance (using cents instead of dollars), it sucks that you have to pay a lot of attention to what calculations you perform, so not having (had - until BIGINT) a real integer type as aid indeed made it less pleasant to do integer arithmetic.
This was also back when you could walk into the library and get the email credentials of a random professor and then use it to hide behind when you took down a network of another in state university because an engineering professor didn't think computer science majors were as smart as he was.
Yeah, man, good times.
My buddy got a visit from the feds and lost his computer lab access for a semester.
Not sure why anyone would think that stealing someone else's data and attacking a network is funny. The only difference between then and now is that now you would get a criminal record for that. It was as morally wrong to do that back then as it is now.
I'd note that a huge amount of the work at those companies was hardware (and a lot of theory in the case of Bell Labs)--though there was, of course important software as well, a lot of it related to Unix.
Doesn't mean it might not have been a blast but not hacking on software and playing in the open source world as is the case at at least some companies today.
Well I still think that software like DTrace, ZFS, NFS, IRIX and Solaris, IrisGL, and the like are cool, even if there was a lot of hardware engineering. I realize that there are disadvantages to it, but the variances in ISAs (MIPS, SPARC, Alpha, etc) seems like it could have posed challenges for software people.
Sun did a lot of great software too and I know a lot of the folks involved. I just think many people look at the innovation through the lens of software (especially open source) hacking which a great deal of it wasn't.
When I was in the minicomputer business, it was maybe 50/50 hardware and software (and that mostly assumes you considered software to include low-level things like microcode). And software people weren't mostly paid more than those in hardware--which is to say generally a good middle class professional wage.
> which is to say generally a good middle class professional wage.
Working with Suns and other workstations as a teen (so my perspective was limited), I caught the very tail end of software as a modest middle class professional wage for everyone doing it (right before the dotcom boom hit).
The people I worked with were really good at what they do, but not strutting like newcomers started doing pretty much the instant the dotcom boom started, and not rich. (Well, one guy did buy a used MR2, and get his private pilot license, but he also lived with his wife in a trailer on an undeveloped parcel. He was a very solid software engineer, working on important stuff.)
I might have inadvertently tried to preserve some of that modestly-paid excellence of the generation before me, but I don't recommend that. Cost-of-living in my area is determined by people making FAANG-like money (well, and real estate investors, and price-fixing), and you have to either play along with that, or move away.
Mind you, California cost of living was on the high side even in the nineties even relative to at least modestly expensive areas like the Boston area suburbs--there was really very little tech in Boston proper at that time.
But a somewhat high-flying (albeit hardware) company was recruiting me for a CA job and they basically admitted it would be a lifestyle downgrade in terms of salary.
Some people made a lot of money when dot-com hit. A lot also got wiped out and ended up leaving the industry.
I never had the highs or lows. I was probably making something south of $100K in the late 90s.
> But a somewhat high-flying (albeit hardware) company was recruiting me for a CA job and they basically admitted it would be a lifestyle downgrade in terms of salary.
I've been given second hand accounts of similar situations. One was team consolidation, and the business was offering Boston-area engineers positions in San Jose. One of the folks who moved with his family was back in MA within 5 years. His salary was not adjusted as much as it should have been for the cost of living difference.
If it meant that people who didn't want to do software didn't ans people who wanted to do software did software, then it sounds nice. I was never interested in making a lot of money in software.
A question, though. Was software designed at Sun closely with hardware teams and vice versa, or were they mostly disjoint? Presumably many hardware companies that have succeeded have built good tooling around said hardware (like NVIDIA).
I didn't actually work there; I knew a lot of the folks from the perspective of an IT industry analyst both during and after a lot of the work there. I certainly saw some level of integration with things like Dtrace (how couldn't you) but when you were a systems company, it's probably the case that software folks couldn't really just divorce themselves from hardware.
The low-level software work at Data General where I was prior to the analyst biz was certainly integrated to a certain degree--read Soul of a New Machine if you haven't. The software folks for the minis were also mostly in the same same location. As things migrated to Unix, most of that team was in RTP and it's probably fair to say that there was less integration though probably wasn't something I thought about a lot of the time. Hardware stayed in Massachusetts.
I got interviewed twice for Google, the first one I made it to the second round of phone calls, the second one only the first phone call.
The third time a Google recruiter reached out to me with the sales pitch that I was a great engineer that they would like to have at any price, I berated him if that was the case why the previous two experiences.
Never heard from Google HR ever again, and I am not sorry, I am happier this way.
I had better experiences in interviews for EA and SCEE than Google, which again I also am an happier person not managing to get an offer, and endure the crunch lifecycle of the industry as reward.
I'm not saying this is the case, but isn't it possible that you weren't a "great engineer that they would love to have at any price" in the past, and you developed your skills and knowledge? Or even if you've always been a great engineer, since you were born, why take personal offense to not getting a job before? Interviews are mostly luck, anyways, so the previous interviews have no relation to the 3rd one.
> I regret being born too late to work somewhere like Bell Labs, SGI, or Sun.
I feel the same, and especially about video games.
I would never work in the modern video game industry. It seems really miserable to be overworked and underpaid to work on some design-by-committee game that I don’t even care about.
But I hear stories of some
Of the companies back in the 90s that seemed to magically muster the capital to sit down and put out effectively what was a passion project, but also commercialize it.
Wally is the one Dilbert character I can tolerate in the workplace. He's honest about who he is and what he does. When you know you're in a bloated company run by buffoons, all you can do for your sanity is work to rule and not upset the apple cart.
I was Wally for the last 2 1/2 years of that previous job until I started to realize I'm becoming more and more like a Dilbert character myself. Something in my brain just told me it wasn't sustainable, call it fear of God or paranoia, but letting my skills atrophy in a place like that for 20 years didn't seem like it would end well for me.
The only problem was that I stayed so long, and it made me hate software engineering so much that I didn't even want to be a software engineer anymore.
I put up with it just long enough so I could avoid selling stock and drawing cash out of my portfolio, and now I'm back at square one as a post-bacc student getting my applications in order for MD and PhD programs where I'll most certainly wind up drawing hundreds of thousands out of my portfolio to pay rent and eat dinner for about a decade.
It's sad, I really enjoyed systems programming, but it seems like finding interesting systems programming and distributed computing projects that have significant economic value is like squeezing blood out of a stone. Maybe LLMs or future progress in bioinformatics will change that, now that finding ways to shovel a lot of data into and out of GPUs is valuable, but I'm so far into physiology, genetics/proteomics, and cell biology that I'm not sure I would even want to go back.
Software development evolved well past the point of solving problems, now it is just plugging solutions. Very few people actually work in novel stuff these days...
I'm currently in a place that pays me €100k just to sit on my ass, and I can do that remotely. I've tried actually doing some work, but that backfired. Not sure what to do, because on one hand my skills are evaporating, but on the other if I wanted a job that pays more I'd have to learn a lot and then work substantially more. I'm wondering if maybe sitting here until retirement is a viable option.
Similar situation. I work for a provincial government and make €61k, my scope is actually relatively large for how long I’ve been with my team but the actual problems are simple enough that some decent code means I have 0 downtime. As a result if I don’t bug anyone I typically get left alone to manage a bunch of products that run without issue. This week I literally have no meetings on my calendar, just a small project with a generous due date where I’m the solo developer.
I’m lucky in that before I got the job I was in talks to do a PhD but negotiated saying I’d only do it remote.
Now I do whatever is required to keep my day job happy and then spend the rest of my time working on my PhD. My plan was to go to FAANG after I got my degree but who knows… a comfy, unionized tech job that gives me ample time to do side projects is also not something I’d give up too easily.
I’d say do whatever is necessary to keep your job and then devote any extra hours 9-5 to some project. If I wasn’t doing my PhD I’d be making an app or a game probably, or maybe still moonlighting as a researcher. I think most office/tech jobs don’t require your full 40 hours and I can tell you I have a bunch of friends who have even less work responsibilities than me but they just use that spare time to play video games. Just do something productive 9-5 and you will outpace 99% of people is what I’ve found.
Honestly I do use that time to play video games because I don't see the point of working my ass off. Suppose I grind my ass off and manage to get a €200k on-site job with on-call. Is that actually a win? I don't think so.
I do the same, but mostly because when I have bothered to work my ass pff to try and get a cool job, I never get hired anyway because they only hire established domain experts and juniors via a university pipeline.
> I asked if I could schedule the interview after my final exams
Ha, my interview for an Amazon internship was an hour after a 3-hour final exam :-)
But the job market right now is quite bad, and after hundreds upon hundreds of internship applications I would've been stupid to give up this chance. I would work for Amazon in a heart beat.
>I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview.
Is this a common Google practice? Can you choose the book you want, or does it have to be introductory Java. On how many different levels does this insult work?!
I'm just very interested in this tidbit of information.
I think you can still find those kinds of workplaces if you focus on research-oriented companies in "deep tech" (meaning those designing and building cutting edge hardware rather than advertising/social media).
I too had the pleasure of interning under some former Bell Labs employees, it really was a great experience.
i’ve interviewed with aws and received offers twice over the years. the first time they made me pay for my own lunch. the second time no lunch break was afforded. i didn’t accept the offers though i know several truly excellent people who work there.
That’s weird, both times that I interviewed for Amazon (admittedly pre-pandemic) they paid for flights, hotels, and all meals, including lunch. The lunch was basically either before or after your interview (depending on whether you had an afternoon or morning interview schedule) and you just grabbed whatever you want on your own. But it was expected to be expensed just like all other interview expenses.
When I interviewed at a Google outpost, a good-cop employee they mistakenly thought had a connection to me took me to lunch (message: forget about the bad-cop interviewer you were just with, you're among friends, loosen your tongue so our spy can report back) in their cafeteria (message: look at the free food perks you'd enjoy), and initiated a conversation with an visiting economist there who then spoke of something oddly relevant to my research interest at the time (message: look at the interesting people and collaborations you will bump into every day).
Your interview lunch experience sounds like message: this is what it's going to be like, and we don't care if you join us.
I had a very similar experience with an amazon interviewer over an even dumber reason: I was going on vacation. Seems dumb, but in your early 20's, with everything booked, I was excited, and frankly didnt see why it was worth insulting me (Its pretty normal to be too busy to do a first round interview on a given week, sometimes things get pushed back 7 days.)
I have done a round one interview and I don’t see how it can be interpreted to do anything but turn away people with a brain.
Memorize Amazon’s insane company values and relate your resume experience to it. And that I mean every single bullet point.
Interviewers were all run by robotic people. Coding test had zero flexibility, you had to just write code in a special barebones text editor that had zero feedback besides pass/fail.
You’d have to solely care about Amazon RSUs to consider that job. They are self-selecting for the worst kinds of candidates.
The dumb thing is that it should be a job that doesn’t burn people out because they basically own the market and haven’t needed to do any sort of innovation. Amazon’s corporate culture just has a burnout fetish.
I interviewed in 2015. The recruiter told me to read the Amazon Leadership Principles, but I thought it was ridiculous to prep for something so specific to a single company, especially as I was interviewing at other companies too.
I got the job, and I think being natural helped. I've interviewed thousands of people at Amazon since, and too many people just say the buzz words with no meat, and it gets them nowhere i.e. I showed customer obsession when I.....(and then gives a bad example)
I interviewed much later and there was definitely no getting around it, the interviewer refused to let me go on further without hitting every single value in the list and talking about a unique example of my experience for each value.
One entire experience story/project example per value, completely insane cult behavior. I felt like I was interviewing for Scientology.
If you had been born early enough, you probably wouldn't be a programmer at all because far fewer people were. Conversely, everybody is born at the right time to join in the heyday of something amazing. You just have to identify that something and be lucky enough and/or try hard enough to become one of the few future-gandalfs. There are companies today making flying cars and dexterous humanoid robots ffs! Or SpaceX! Amazing engineering work never stopped, it just didn't linger on in the same fields.
Fair enough, but in my estimation that next big thing is gene therapy, and the best way to get involved with it is to become a medical geneticist.
I'm sorry to be a buzzkill, but I just can't get excited about privately-funded space rockets or Japanese girlfriend robots, not even if there were an 8 figure stock compensation package in it for me. To me, it's all just "do something grandiose for the venture capitalist bucks, and then maybe figure out how it helps people in 20 years."
"My sister killed her baby because she couldn't afford it and we're sending people to the moon.
September, my cousin tried reefer for the very first time.
Now he's doing horse. It's June.
Is it silly, no? When a rocket ship explodes
And everyone still wants to fly
Some say a man ain't happy truly
Til he truly dies
Hi, I am a half decent engineer. I say that as objectively as one can say something like this about themselves.
I worked at Amazon. Twice. In total about a decade as a Principal Engineer. I left voluntarily a few months ago.
I have zero regrets about my time at Amazon. I learned lots, worked with some incredible people, and had fun doing it.
And the culture? It was life changing for me, especially when I first joined. In all the best ways.
And Amazon today? All I’ll say is that at their size, maintaining solid culture is damn hard. The hiring spree peri-Covid definitely added unimaginable stress to maintaining the culture the company was built on.
They’re a big company, and thus a big target. It’s easy, cheap, and even lazy to kick them with stuff like this.
The truth is that while it’s changed a lot over time, anyone fortunate enough to work there should embrace it.
I gotta be candid with you: "anyone fortunate enough to work there" is exactly the kind of arrogance that rubs job candidates the wrong way. A lot of people don't see it the way you do, and you would do well to take a moment of honest self reflection and consider the reason why.
Before you reflexively jump to "Amazon is just not for everybody", it's only fair to warn you that a lot of people around here have heard this phrase and don't buy it. My former landlord said the same thing at least 9 times in the 8 times I've interacted with him face to face... right up until he got fired. Management conveniently put him on a PIP 4 months before a vesting cliff on his 2-year RSU vesting schedule.
Yeah, Amazon is not for everybody- just the kind of people who either perpetrate or fall victim to pyramid schemes.
Amazon, on the engineering side, is rough for high-end software engineers, but let's be very real: it pays better than the vast majority of careers available in America, even right out of college, and while it's stressful so is being a teacher or working in the food industry, and those jobs pay peanuts compared to an Amazon SWE. It is fortunate to get a job there. It may be even more fortunate to get a job at another high-end software company but that doesn't change the fact that a job at Amazon is life changing money for most people.
To pre-empt hyperbolic responses: I live in Seattle. I personally know plenty of people who have worked at Amazon. I know plenty of local teachers. The teachers work as many hours for a tenth the pay and burn out just as fast.
You might be right about the hours the teachers work, but they're not doing it to avoid being fired. It is entirely self-imposed stress. Teachers have unions and tenure. I'll help people, but I won't martyr myself in a hospital, let alone a frickin' tech company.
I don't consider 60 hour work weeks a privilege to be coveted, especially if I'm capturing less than 1% of the value I produce. I'm sorry man, but white collar serfdom is still serfdom.
If you're really going to keep begging this hard to be a serf on someone else's fiefdom, I'll tell you what: you can keep overworking yourself as serf on fiefdoms in which I might own shares, and increase the value of my portfolio for me, so I can draw even more passive income every month. I formally withdraw my protest to your "Protestant" work ethic.
I'm not trying to be an asshole, but I'm just hoping you take what I'm saying burns into your subconscious and something in you changes: that work into which you've placed so much of your self-esteem is just funding other peoples' passive income to either not work or live ostentatiously. If that's a great privilege to you, God bless you. I'm sure my wife would love it if I just retired- please make it happen, she would love me even more for it.
EDIT: P.S. Your "privilege" of working for Jeff Bezos' portfolio sounds a lot like a biblical curse. Specifically, 1 Samuel 8:10-18 and Psalm 105:44.
"He gave them the lands of the nations, that they might inherit the fruit of others’ labor"
>You might be right about the hours the teachers work, but they're not doing it to avoid being fired. It is entirely self-imposed stress. Teachers have unions and tenure.
No, and no. Your perception of modern labor markets is dated and disconnected from reality. Schools are constantly facing budget cuts and a huge chunk of teachers aren't even full time (instead getting things like a 60% time position - less pay, no security, but enough hours that good luck working a second job). Those that get full time positions still face the elimination of their positions.
I guess I just assumed teachers have the same protections in deep blue Washington state that they have in Illinois and Minnesota.
If I'm wrong, I'm wrong. However, there's a fix for that: move to a state where the job prospects are better. That sounds like "let them eat cake", except that moving to another state as an American citizen is a lot easier than what of people went through fleeing the Sicilian Mafia or the Irish Potato Famine, let alone WW2 European front or the fall of Saigon.
> you can keep overworking yourself as serf on fiefdoms in which I might own shares, and increase the value of my portfolio for me, so I can draw even more passive income every month
Yeah, I guarantee you that the guy who worked at Amazon as a Principal Engineer for 10 years has a bigger portfolio than you.
levels.fyi says $967k/yr average compensation at that level.
> It is entirely self-imposed stress. Teachers have unions and tenure.
You don't have the first idea what teachers do or what their jobs require.
If you really think all teachers have tenure and are supported by unions, you also have not the faintest, foggiest inkling of a clue about the job market or professional environment of most teaching in the US.
I think it's also worth pointing out that, even as you repeatedly label others "arrogant," your comments in this thread are themselves breathtaking in their arrogance. I rarely run across HN comments so condescending, dismissive, self-righteous, or self-congratulatory.
I shared my experience. And in my experience anyone who gets the opportunity to work there should do it.
Is it fortunate? Working at a place that holds a high bar on hiring, pays extremely well, and provides extremes of learning and growth in tech? Yeah, that’s a pretty good spot to most rational people. Building systems for 20M request/second teaches you cool stuff. So yes, it’s fortunate.
Of course it’s not for everybody. I never said it was.
Nobody is going to take seriously a person that thinks working at Amazon is a pyramid scheme.
After you work at Amazon please report back your experience.
strikes me as an insane thing to say based on the people I do know who have worked there. Horrible work life balance, mobbing, being hired to be fired by a manager who wanted to keep their team but had to stack ranking. Every one of them has described the culture as a cult, and reading comments like this one makes me think even moreso that they're right.
We do a lot of business with them and we have workshops with them sometimes, and the one thing I notice is how they're all so evangelical. They wouldn't say a bad thing about their company. I couldn't be like that, when I work something I sell my knowledge but not my soul. I'd always speak freely (not always appreciated but usually it's not a problem if it's true).
But that company culture leaves me with a very low opinion of them and very little trust. Even Microsoft engineers are less brainwashed. I've had several that just told me the truth about services.
Maybe it depends on the country but it feels like this is just their culture.
I've worked there since 2015, and this simply isn't true.
There's a lot wrong with AWS (and it's got a lot worse in the last 3 years), but there's also a lot right, and there are some really, really smart people there, several of which have boomeranged (people who left and came back).
The formula is usually more money and ability to work special team isolated from the usual toxic orgs. I think A9 was probably somewhat like that, and AWS probably used to be at some point long ago
I know some people who are fine working there. No one seems thrilled but if you're an above average engineer who is just getting by at 140k a year and suddenly you're looking at 350k a year as an SDEIII or something, that can be a life changing amount of money.
However, I think the question is, what percentage of engineers can pass the amazon interview but not the Apple/Databricks/Uber/Google/Meta ones. Because no one is picking amazon over the aforementioned companies.
However, maybe there's an opening at Amazon and not the other companies, or maybe that's your only offer. I certainly think it might be worth it for a a few years.
It can be a life changing amount of money but not always in a good way. If you're not careful, your spending expands like a gas to fill all the available volume and you're just marching sideways towards retirement with nothing saved because it's too fun spending money.
RSUs actually help mitigate this a bit, because you're not going to qualify for a giant house/apartment with your RSU money, and most places aren't even vesting monthly so you're more likely to treat them like bonuses, rather than inflate lifestyle.
Not saying it doesn't happen, but it's probably harder than if it was just suddenly getting a giant salary.
Assuming they’re already living in Seattle: pre tax, and supporting a family? According to a quick google, that’s ~$100k after tax, a 3BR in Seattle is ~$4k/mo so you’re left with $52k for everything else, in a HCOL area. I buy it.
Homelessness + loans (credit cards) are on the rise and births are on the decrease.
But to honestly answer the question, by either not living in large cities and commuting in, sharing those 3BRs, or staying home with parents. Shaving that 48k/year (post-tax) rent down to ~12k/year frees up a lot of money.
(Also if you're young, staying with your parents and saving ~100-200k in rent for a downpayment over 10 years just seems like a smart idea to me).
For the first ten years of my career, I just assumed my resume would get passed over, that even though I was smart and capable, there was no point in trying to work at top tier places.
I was single and not a lavish spender so I didn't feel external pressure to try all that hard beyond having a low six figures job.
There are plenty of low prestige, mismanaged small companies that will pay low six figures and overwork people.
Yep, everyone is excited until they leave after 1 1/2 or 2 years. There are always outliers but my personal experience is that the churn rate is incredible high.
Are there really many illegal aliens working in the warehouses? I know that Amazon does verify employment eligibility and checks documents. There may be some committing identity theft but I doubt that it's a large proportion.
Not really. For low skilled jobs that don't require much English, illegals are going to be as good as locals. Maybe even better because locals who are any good are mostly going to move on to better opportunities.
Yeah this. And warehouse work is all appified and can be configured in any language.
Any American working in an Amazon warehouse will be jumping at the chance to get out of there. So turnover would be really high. Same with delivery stuff, I don't think most Americans would enjoy a job where they don't even get a pee break.
As for legal stuff, I'm sure they'd use intermediaries to cover their ass.
Using intermediaries to hire illegally makes it worse from a liability perspective. Amazon is too big and hires too many people to get away with “oh, we used agencies in all of our markets and audited nine of them and we’re shocked that they didn’t do diligence”. All it would take is one disgruntled hiring manager. And Amazon doesn’t have many gruntled ones.
At the scale of Amazon, law enforcement is no longer bound to the law, it's bound to politics.
Your average local employer with a few dozen people? If he fucks around enough to draw in the attention of OSHA, he will find out. Easy prey and without political connections he's toast.
But Amazon? This company is by direct head count the third-largest employer in the US, second-largest in both the US and worldwide if one excludes militaries [1]. Amazon is frankly too large to enforce laws against and so is similar-sized Walmart (who has been able to extort the government into subsidizing their poor wages with food stamps), too many livelihoods depend on the existence of the company.
IMHO, a lot of the Big Tech and F500 companies should be outright broken up. When a company grows so large that laws cannot be enforced or, worse, laws get willfully ignored because it's cheaper to risk the occasional fine and bad press, eventually the rule of law itself suffers.
I would never work for AWS, given what I've heard, and consistently, of their internal culture.
Also, everything I've seen while working with internal staff makes me feel there's a culture of obfuscating all weaknesses from customers, practically to the point of deceit.
From what folks at AWS tell me that’s basically already happened. The best and brightest won’t even apply to work there anymore. For many key functions they’ve legit run out of people to recruit and thus have to go down market compared to competitors. Thats very much true in hot sectors like AI where AWS has “C” and at best “B” team players and leadership.
I’ve also heard key pieces of their infrastructure has sloppily written code and that communication between teams is horrible. Even with their insane salary offers, most people don’t think it’s worth it. Especially given their 6% “unregretted attrition”[1]
The thing is, the brand is toxic. They can offer all the money they want, but no employee has any pride in working for them. So they produce the bare minimum to get their salary - and I applaud them.
I still occasionally get them even though I literally was one of the people who left after they tried to make us go in office (I don't like to use "RTO" because no one on my team had actually worked out of an office for Amazon before since the project we were on was fairly new). My wife (at the time fiancee) has an autoimmune issue that makes it much safer for me not to commute, and although my manager suggested I could get an exemption, he didn't actually know what the process was because it all happened so quickly that no one seemed to have actually defined what that process was up front. I had a little less than a month to figure what to do and get that exemption before they expected me to either have that exemption, be in an office in another city three days a week, or transfer to a local team and be in the office in my current city three days a week. I ended to deciding that it wasn't worth the effort to try to figure out how to convince them to let me stay.
Amazon’s not the only company on the planet that pays well. While they’re above average they’re far from top of the market. If you’re talented enough one can make a lot of money and skip the toxic culture. Double win.
The claim is not that Amazon would be using illegal warehouse workers today, but that there is theoretically a pool of tens of millions of people available. Which is still kind of dubious.
Even L6 managers feel this, but it becomes more clear as one goes up in levels. Recruiting is job one.
If Amazon runs out of recruitable engineers (unlikely, they are one of the most prestigious firms in the world) then they will simply lower the bar. HC must be filled.
Not totally on topic, but I recently passed the tipping point with Amazon shopping. I now go to Temu. They have US warehouses shipping in a couple days which was the only thing keeping me on Amazon. Plus everything on Amazon is basically the same stuff on Temu but with a markup!
I seem to remember an internal memo being leaked in which middle management was complaining that they would eventually/soon burn through all the laborers in that region, after which they'd suffer immense difficulty in staffing that warehouse.
There was another comment that pointed out how unlikely this is to happen because Amazon is just too big to bring the law down on now.
When you're the, what. Second? Third? Largest employer in the US, enforcing the law now becomes a meaningful hit to economic velocity. And as much as Trump hates brown people, his administration has begrudgingly revealed that there are moves that his billionaire buddies Will Not Allow.
I'm no fan of ice or this administrations deportation strategy, but it's a serious problem that even enforcing the law on Amazon is now an economic liability so much that nobody dares to try
Given their self-imposed labor supply issues, notably the awful working conditions, I hope the workforce there figures out how to effectively organize against this cesspool of a corporation.
Look, not to defend anything Amazon is doing, but this causal chain seems rather pareidolic and under-evidenced. You could spin some kind of crazy narrative about any major outage based on changes in policy that happened just before. But this isn't nearly the first AWS outage, and most of them happened before the recent RTO changes. It needs more evidence at best.
The article wasn't about the outage happening, it was about the amount of time it took to even discover what the problem was. Seems logical to assume that could be because there aren't many people left who know how all the systems connect.
It's about the time between the announcements about finding the cause. I find that to be thin evidence. There are far too many alternate explanations. It's not even that I find the idea to be implausible, but I don't think the article's doom-saying confidence level is warranted.
It's the most plausible, fact-based guess, beating other competing theories.
Understaffing and absences would clearly lead to delayed incident response, but such an obvious negligence and breach of contract would have been avoided by a responsible cloud provider, ensuring supposedly adequate people on duty.
An exceptionally challenging problem is unlikely to be enough to cause so much fumbling because, regardless of the complex mistakes behind it, a DNS misunderstanding doesn't have a particularly large "surface area" for diagnostic purposes and it is supposed to be expeditely resolvable by standard means (ordering clients to switch to a good DNS server and immediately use it to obtain good addresses) that AWS should have in place.
AWS engineers being formerly competent but currently stupid, without organizational issues, might be explained by brain damage. "RTO" might have caused collective chronic poisoning, e.g. lead in drinking water, but I doubt Amazon is so cheap.
> An exceptionally challenging problem is unlikely to be enough to cause so much fumbling because, regardless of the complex mistakes behind it, a DNS misunderstanding doesn't have a particularly large "surface area" for diagnostic purposes and it is supposed to be expeditely resolvable by standard means (ordering clients to switch to a good DNS server and immediately use it to obtain good addresses) that AWS should have in place
You seem to be misunderstanding the nature of the issue.
The DNS records for DynamoDB's API disappeared. They resolve to a dynamic bunch of IPs that constantly change.
A ton of AWS services that use DynamoDB could no longer do so. Hardcoding IPs wasn't an option. Nor could clients do anything on their side.
The article describes evidence for a concrete, straightforward organizational decay pattern that can explain a large part of this miserable failure. What's "self-serving" about such a theory?
My personal "guess" is that failing to retain knowledge and talent is only one of many components of a well-rounded crisis of bad management and bad company culture that has been eroding Amazon on more fronts than AWS reliability.
What's your theory? Conspiracy within Amazon? Formidable hostile hackers? Epic bad luck? Something even more movie-plot-like? Do you care about making sense of events in general?
We've witnessed someone repeatedly shoot themselves in the foot a few months ago. It is indeed a guess that it may cause their current foot pain, but it is a rather safe one.
Indeed. No disrespect to Justin (great person) or any of the engineers who were sacked but Corey's post here is basically "here's someone who was sacked, and here are several other layoff news". AWS is really big organization. Several orders of magnitude bigger than people who were remote/refused to RTO. Organizations like this survive these brain brains.
Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't."
> Organizations like this survive these brain brains.
True, that's the other thing. Even if it's true that brain drain directly caused/exacerbated this event, big companies have a lot of momentum. Money can paper over a terrifying range and magnitude of folly. Amazon won't die quickly.
This is the time to accept that the path forward is keeping people and giving them the best tools you possibly can to do their work. That is, the same as has been true for decades remains so.
Yes, development tools are better every day. Yes, you can downsize. No it won’t be felt immediately. Yes, it mortgages the future and at a painfully high interest rate.
Suspending disbelief won’t make downsizing work better.
Seems like it worked fine. They laid off a quarter of their junior principal engineers, the stock went up. They had a massive outage a few months later, the stock went up again. Everything's working out fine for their strategy so far.
I remember comments saying the stock went up because the average joe didn't realize how much of the internet was powered by AWS until all their day to day apps started failing. To most people Amazon is an online shopping site.
You would think this would eventually show up on the balance sheets, right? Presumably a lot of their big customers have SLAs with money penalties, so maybe next quarter earnings? Or quarter after that?
SLA monetary penalties won't make the difference there. Enough giant customers moving substantial workload off of AWS (either to another cloud, or otherwise) would, but the timeline for that is years, not next quarter.
Where are the young companies trying to replace them? There are all the AI companies, but Google and Meta both have competitive chatbots, and OpenAI is signing weird deals that don't make it look like a long-term player.
They all get bought out by Amazon, Google, Meta et al. The cash just tastes too good when stacked up against the prospect of grinding for 15 years and probably nothing coming of it.
I don't think there's ignorance of the fact that turnover is bad, I think the field is being designed to homogenize staff and favor uniform mediocrity so that employees truly do become interchangeable. We're so close to just plain talent being likened to cowboyism.
I wish to understand the virtue of Amazon culture.
It seems that at L6 and below workers are a Taylorism-style fungible widget driven to convert salary into work product, guided to create the most output for the longest time before mentally breaking down, then being swiftly replaced, with L7 and above being so incredibly political that keeping the snakes and vultures from eating your team is a full time job at every level of senior management.
It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.
I would love to hear positive counter perspectives from Amazonians because the anecdotes from my L6-L10 friends describe what sounds like an inhumane hell on earth.
> It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.
It’s pretty simple, actually. Once such a dominant market position is achieved, you can get away with almost anything, whether with customers or employees. This is true of all the BigTech companies.
I think there's more to it. When you're dominant, you make money whatever. Think of Amazon et al. as huge spigots of money. Now, it becomes optimal to fight for more of that money coming your way. It's like the resource curse for countries. Nobody gains from growing the pie; they gain from stealing the pie. At some point, parasites and parasitic behaviours invade.
I think it comes down to demand and supply for jobs.
The only time Amazon was forced to change its ways was during Covid hiring boom where they couldn't compete in the talent market. They were forced to increases their salary bands and the culture was also a bit easy during that time. But starting mid 2022 it's been an employer's market and Amazon is making sure to juice every bit out of its employees while it can
It's not as conscious as that, its an emergent outcome of the snake pit.
Engineers have to spend an inordinate amount of time on "managing up", which means they have very little time and attention to do what would otherwise be a reasonable workload. Additionally, good engineers hate and despise this so it contributes a lot to the burnout.
Tech will learn like manufacturing folks did that experience is not fungible. You can try to replace someone, but the new guy also needs to accumulate the scars from the system for years before taking over.
You cannot just keep abstracting and chopping systems to smaller and smaller subsystems to make them easy to digest.
At some point someone needs to know how these coordinate and behave under disturbances. At some point someone needs to know at a low level what the hell is going on.
I don't know, manufacturing seems to have learned pretty well that they can ship everything overseas and people will eventually accept products just aren't made the same way they used to be.
If AI is to tech what outsourcing was to manufacturing, then your analogy has me concerned for the future.
According to the article, the issue was caused by:
> "engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause"
Interestingly, we found matching errors in our own logs:
> System.Net.WebException
> The remote name could not be resolved: 'dynamodb.us-east-1.amazonaws.com'
Occurrences were recorded on:
- 2025-04-30
- 2025-05-29
- 2025-06-17
- Yesterday
We had logged this as a low-priority bug since previous incidents only affected our AWS testing environments (and never our production env which is on Azure). At the time, we assumed it was some CI/CD glitch.
It now seems that the underlying cause was this DNS issue all along, and only yesterday did it start impacting systems outside of AWS.
It was Diwali vacation in India. It looks like the managers were not able to force everyone to walk around with their laptops and pagers hanging from their necks and waists, respectively, which they normally do.
If there's one thing I have learned from my Amazon mates, then that is they never have a true time off. Hills, beaches, a marriage in the family— no exceptions. It's so pervasive that I can't really imagine it to be voluntary, and my friends' answers on this topic have never been clear.
It was certainly suspicious that actual progress on the outage seemed to start right around U.S. west coast start of day. Updates before that were largely generic "we're monitoring and mitigating" with nothing of substance.
[09:13 AM PDT] We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT.
[08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services...
[08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues...
[12:11 AM PDT] <declared outage>
They claim not to have known the root cause for ~8hr
Sure, that timeline looks bad when you leave out the 14 updates between 12:11am PDT and 8:04am PDT.
The initial cause appears to be a a bad DNS entry that they rolled back at 2:22am PDT. They started seeing recovery with services but as reports of EC2 failures kept rolling in they found a network issue with a load balancer that was causing the issue at 8:43am.
I don’t think that’s true, there was an initial Dynamo outage that was resolved in the wee hours that ultimately cascaded into the ec2 problem that lasted most of the day
Was the Dynamo outage separate? My take was the NLB issue was the root cause and Dynamo was a symptom which they flipped some internal switches to mitigate the impact to that dependency
If their internal NLB monitoring can delete the A record for dynamodb that seems like a weird dependency (like, i can imagine the nlb going missing entirely can cause it to clean up via some weird orchestration, but this didn't sound like that).
I noticed that too. I think tech culture has to change a bit. Silicon Valley is a great location if you're making hardware or prepackaged software. If you have to support a real economy that is mostly on the East Coast you need a presence there.
> one really gets the sense that it took them 75 minutes to go from "things are breaking" to "we've narrowed it down to a single service endpoint, but are still researching," which is something of a bitter pill to swallow
Is 75 minutes really considered that long of a time? I don't do my day-job in webdev, so maybe I'm just naive. But being able to diagnose the single service endpoint in 75 minutes seems pretty good to me. When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.
A few years back I was working at a software company that provided on-site sensor sensor networks to hospitals, pharmacies, etc. Our product required them to physically install a server on-site, but we were starting to get disrupted by cloud-based solutions. Essentially what we did was alert medical staff when blood, organs, etc. refrigeration temperatures went out of range. If the right people involved did not get notifications on time for these issues people will die. Its not hyperbole, you have to wait years for liver transplant. Their aren't just new livers available for everyone if a handful of them spoil.
With that being said, the problem here isn't that it took 75 minutes to find the root cause, but rather that the fix took hours to propagate through the us-east-1 data center network. Which is completely unacceptable for industries like healthcare where even small disruptions are a matter of life and death.
The web operates in a very different world if you've invested in good tooling. I used to be lead on a modestly sized payment processing back end to the tune of about 100 transactions/second (we were essentially Stripe for the client facing apps at the company). In many cases our monitoring and telemetry let us identify root cause in a matter of minutes. Not saying that is or should be the norm for all web apps, but what we had was not too far off from a read-only debugger view of the back end app's state throughout the request and it was very powerful. Of course for us more often than not the root cause was "the bank we depend on is having a problem" so our knowledge couldn't do much other than help the company shape customer communications about the incident.
> Is 75 minutes really considered that long of a time? [...] When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.
One might spend weeks diagnosing a problem if the problem only happens 0.01% of the time, correlated with nothing, goes away when retried, and nobody can reproduce it in a test environment.
But 0.01%-and-it-goes-away-when-retried does not make a high priority incident. High priority incidents tend to be repeatable problems that weren't there an hour ago.
Generally a well designed, properly resourced business critical system will be simple enough and well enough monitored that problems can be diagnosed in a good deal less than 75 minutes - even if rolling out a full fix takes longer.
Of course, I don't know how common well designed, properly resourced business critical systems are.
Quite a few of AWS's more mature customers (including my company) were aware within 15 minutes of the incident that Dynamo was failing and hypothesized that it'd taken other services. Hopefully AWS engineers were at least fast.
75 minutes to make a decision about how to message that outage is not particularly slow though, and my guess is that this is where most of the latency actually came from.
For a service like AWS, 75 mins is going to result in a LOT of COE's for people on way it wasn't mitigated quicker. A Sev 1 like this has an SLA of 20 mins to mitigate impact. Writing about these failures will consume a dozen peoples time for the next 6 weeks.
I have 10 years of experience at Amazon as an L6/L7 SDM, across 4 teams (Games, logistics, Alexa, Prime video). I have also been on a team that caused a sev 1 in the past.
Also it's pretty likely it took less time than that to get an idea, but generally for public updates you want to be very reserved, otherwise users get the wrong impressions.
>Is 75 minutes really considered that long of a time?
From my experience in setting up and running support services, not really. It's actually pretty darn quick.
First, the issue is reported to level 1 support, which is bunch of juniors/drones on call, often offshore (depending on time of the day) who'll run through their scripts and having determined that it's not in there, escalate to level 2.
Level 2 would be more experienced developer/support tech, who's seen a thing or two and dealt with serious issues. It will take time to get them online as they're on call but not online at 3am EST, as they have to get their cup of joe, turn on the laptop etc. Would take them a bit to realize that the fecal matter made contact with the rotating blades and escalate to level 3.
Which involves setting up the bridge, waking up the decisions makers (in my case it was director and VP level), and finally waking up the guy who either a) wrote all this or b) is one of 5 or 6 people on the planet capable of understanding and troubleshooting the tangled mess.
I do realize that AWS support might be structured quite a bit differently, but still... 75 minutes is pretty good.
Edit: That is not to say that AWS doesn't have a problem with turnover. I'm well aware of their policies and tendency to get rid of people in 2/3 years, partially due to compensation structures where there's a significant bump in compensation - and vesting - once you reach that timeframe.
But in this particular case I don't think support should take much of a blame. The overall architecture on the other hand...
Sorry, are you saying you worked at Amazon and this is how they handle major outages? Just snooze and wait for a ticket to make its way up from end user support? No monitoring? No global time zone coverage?
Because if so, this seems like about the most damning thing I could learn from this incident.
No, it's just mindless speculation from someone who clearly hasn't worked a critical service's on call rotation before. Not at all what it's actually like, all these services have automatic alarms that will start blaring and firing pagers, and once scope of impact is determined to be large escalations start happening extremely quickly paging anyone even possibly able to diagnose the issue. There's also crisis rotations staffed with high level ICs and incident managers who will join ASAP and start directing the situation, you don't need to wait for some director or VP.
I worked at AWS (EC2 specifically), and the comment is accurate.
Engineers own their alarms, which they set up themselves during working hours. An engineer on call carries a "pager" for a given system they own as part of a small team. If your own alert rules get tripped, you will be automatically paged regardless of time of day. There are a variety of mechanisms to prioritize and delay issues until business hours, and suppress alarms based on various conditions - e.g. the health of your own dependencies.
End user tickets can not page engineers but fellow internal teams can. Generally escalation and paging additional help in the event that one can not handle the situation is encouraged and many tenured/senior engineers are very keen to help, even at weird hours.
Wholly inaccurate. AWS Systems Engineers would have been paged by automated monitoring systems once alert thresholds were breached. No escalation through Support needed.
It's 75 minutes to _communicate_ the message to customers. Definitely internal teams were ahead of this before it was posted to the AWS Health Dashboard. Status Page posts are lagging indicators of incident progress.
I work in an incident management team where the turnaround from "we've decided to take x action, to y metric shows it is working, to z is posted on the status page" can be 1-2 minutes.
It is possible with professionals, institutional knowledge, drills, and good tools.
Amazon is supposed to have the best infrastructure in the business because everyone else runs on it. They should have access to the sre talent that can quickly mitigate this kind of issue
I once saw a layoff that was followed by a week long outage because no one remaining knew how to deploy to prod, and no one knew how to recover after the failed deployment. I felt bad for the people remaining who had to go through that,but it was hilarious.
AWS is still my overall favorite cloud provider, and I use it very effectively.
I would've even liked to work at AWS myself, if it were clear that they're solving a few concerns:
1. Rumors of rough corporate culture, and you needing your manager to shield you from it. (If it can't be immediately solved for all of Amazon or white-collar, maybe start with increasing job-seeker confidence for AWS or per-team.)
2. Even very experienced engineer candidates must go through some silly corporate coding screen, and an interview to make sure they've memorized some ritual STAR answers about Leadership Principles. If your prospective manager can't even get you out of that, what worse corporate things can't they shield you from?
3. RTO. As well as all the claims it wasn't done consistent with the Leadership Principles, and claims that it's not about working effectively.
4. Difficult-sounding on-call rotation, for people who aren't shift workers. (Even if you Principal out of on-call, you don't want your teammates to be overextended, nor to have awkwardness because you're getting a more consistent sleep schedule that is denied them.)
Also, not a concern, but an idea that applies to all the FAANGs lately: What about actively renewing the impression in the field that this is a place where people who are really good go? Meta's historical approach seems to be to pay better, and to release prominent open source code, and be involved in open hardware. Google (besides having a reputation for technical/competence excellence and warmer values) historically had a big frat-pledging mystique going on, though it turned into a ritual transaction, and everyone optimized for that ritual. AWS has a lot of technical/competence excellence to be proud of, and could make sure that they're investing in various facets of that, including attracting and retaining the best workers, and helping them be most effective, and then making sure the field knows that.
Whether it involves FAANG companies or not, a job is ultimately just a job. While it's nice to have such a company on a CV and to gain the experience, it is, in essence, similar to any other employment.
Eventually, you begin to consider the drawbacks, such as the monotony of the work or the exhausting nature of on-calls (which disrupt personal life). Then, an opportunity arises from a former colleague at another company, and the outcome is predictable.
Companies present numerous such inconveniences and actively introduce additional ones. Now, we are faced with mandatory RTOs, along with the continuous tightening of the screws and "cutting fat from the bone" (actual words of my company's CTO). Consequently, employees will depart, and it is often the high-performers who will seek opportunities elsewhere, as they are not afraid of the job market.
I think google is an excellent place to work, but the top AI companies do poach some of our good employees. If you're the cream of the crop, sure, take the Anthropic offer over G. But I do appreciate the massive amount of freedom to work on hard problems.
Google can be an excellent place to work, but most of the people working there are not contributing to the company’s profits. Whether or not you think that still has value, leadership is trying to “optimize” by squeezing everyone harder and demanding they show productivity gains using AI. Some teams will adapt and keep being a nice team to work on, but many employees are getting “optimized” out of the company and replaced with vendorization and consultants.
This article seems sensationalized and lacking evidence. Layoffs alone (especially when so much of the industry did them) doesn't seem sufficient to explain today's outage, especially when we know so little of the technical details behind it. It's disappointing that The Register didn't wait until we had a postmortem from AWS before jumping to conclusions.
""It's always DNS" is a long-standing sysadmin saw, and with good reason: a disproportionate number of outages are at their heart DNS issues. And so today, as AWS is still repairing its downed cloud as this article goes to press, it becomes clear that the culprit is once again DNS. "
I use stored DNS data.^1 The data is collected periodically and stored permanently
I seem to be unaffected by DNS-based outages
I use stored data because it is faster, e.g., faster than using DNS caches like Google DNS, Cloudflare DNS, etc., but there are obviously other other benefits
1. When I make HTTP requests there is no corresponding remote DNS query. The IP address is stored in the memory of the localhost-bound forward proxy
For internal addressing, why do we use DNS? In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.
Why not? It's standard protocol for name-address mapping.
> In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.
Client just performs DNS query before HTTP query or client caches DNS response for some time. It's solved problem and "pull" approach usually preferable to "push" approach. For example, what if that broadcast gets lost (UDP is not reliable protocol)? How application handles initial discovery?
Of course it is possible to replace DNS with something else. But why? If you're afraid that your DNS server will fail, the same could happen with your custom service discovery server.
How would you handle the ack of the broadcast? UDP can be unreliable. If you do send an ack from the destination, that singular endpoint that sent the message is going to get a number of responses equal to the number of devices on the network, which would nuke it at large scale.
It’s no secret that AWS has been seeing a mass talent exodus. Probably 90% of the folks I know that were the best and brightest in what they do and worked at AWS are no longer there. It’s beyond a blip, but a full blown exodus, with the talent bar severely lowered.
The writing was on the wall for a bit now that something like yesterday would happen.
> When that tribal knowledge departs, you're left having to reinvent an awful lot of in-house expertise that didn't want to participate in your RTO games, or play Layoff Roulette yet again this cycle.
…
> This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. Remember, there was a time when Amazon's "Frugality" leadership principle meant doing more with less, not doing everything with basically nothing. AWS's operational strength was built on redundant, experienced people, and when you cut to the bone, basic things start breaking.
Not just Amazon. I woke up this morning, to find my iCloud inbox stuffed with unread spam; much of it over a month old. Looks like someone restored some old backup. This was likely to correct some issues that were caused by the AWS outage; either directly, or indirectly.
It’s nice to know that Apple (or some other middleman) backs up the torrents of spam that I get.
Everything is now at Jurassic-scale. It’s all monstrously big. There’s no such thing as a “small problem,” anymore.
One thing that you get with experience, is “tribal knowledge,” and that stuff is usually impossible to properly document. I suspect that AI may, in the future, be able to incorporate some of this, but it’s by no means certain.
cue in: programming as theory building [1] or building systems as theory building, ie, Mental causal models of how and why things work the way they do. Mental models live in people's heads and walk out of the door when they do. Management learns this the hard way [2].
When organizations begin to prioritize personal-brand builders and performative hires over the core technologists and long-tenured institutional experts who actually understand how things work, the culture inevitably shifts.
When that imbalance grows, as it has at AWS (ex-AWS here), and the volume of relentless self-promoting “LinkedIn personalities” and box-ticking DEI appointments starts to outnumber the true builders and stewards of institutional memory, the execution quality, accountability, and technical excellence begin to erode.
It is becoming increasingly clear that Andy Jassy’s leadership is no longer effective, and it is only a matter of time before Wall Street begins calling for his departure.
I appreciate how you, with absolutely no evidence, blamed this outage on DEI, especially given Amazons complete backpedaling of DEI the first moment it became political convenient
This AWS outage has reminded and bolstered my confidence of the idea that there really are practical limits on how we can manage complexity.
As a codebase ages, as services grow out in scale and scope, complexity increases. Developers know this. I don't believe that you can linearly scale your support to accommodate the resulting unknown unknowns that arise. I'm not even sure you can exponentially scale your support for it. There is going to be a minimum expected resolution time set by your complexity that you cannot go under.
I think times where there have been outages like this that have been resolved quickly are the exceptional cases, this is the norm we should expect.
Doesn’t an increasing LLM centric code base only compound this problem? Under the assumption that people are lazily screening LLM’s output when using them
I think it's important Amazon remains stable and a quicker resolution would have been great.
That being said, if many important services (the article mentions banking) are still single-point-of-failure in us-east-1, the least stable but cheapest region, there seems to be a problem far greater than Amazon here.
We all watched this happen across FAANG, right? In the early/mid 2010s working at Amazon meant you were cream of the crop.
By 2020, no engineer in their right mind wanted to work there because it was an infamously bad employer for people who wanted to create great tech in a nerdy-fun environment.
The AI space is showing how the "darling fun tech company" to "aggressive tech employer full of psychopaths" trope can take less than a few years now!
"If you were a ‘product person’ at IBM or Xerox: so you make a better copier or better computer. So what? When you have a monopoly market-share, the company’s not any more successful. So the people who make the company more successful are the sales and marketing people, and they end up running the companies. And the ‘product people’ get run out of the decision-making forums.
The companies forget how to make great products. The product sensibility and product genius that brought them to this monopolistic position gets rotted out by people running these companies who have no conception of a good product vs. a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts about wanting to help the costumers.”
Great words, but he lost any right to them when he made famous the "You're holding it wrong" workaround. IMO that was the defining moment when Apple started its decline on product innovation.
If it were that big of deal, don’t you think Apple would have been forced to recall it and definitely couldn’t keep selling it for 3 years. True they did redo the antenna for the Verizon CDMA iPhone 4. But they never bothered to back port the changes to the GSM one.
I also had a GSM iPhone 4.
Compare that to how quickly they ran away from the shitty Intel modems when they were selling some made by Intel and some made by Samsung (?)
I’d add a corollary to Steve’s often quoted idea, that became even more relevant after Covid. Everyone competent that makes tons of money retired early. We are left with the dregs at all these companies, the newbies and those that didn’t budget well and plan for early retirement.
Any interaction you have with a company post-Covid you can feel it. Nothing works anymore and you can’t even tell anyone about it or why.
At best, a company's culture is just a branding and marketing strategy to attract employees and to appear cool. They are not real cultures -- they are constructed, forced and dictated by management, and oftentimes cult-like in their construction.
Company cultures are not built to last, they are designed to generate profit. The culture is incidental, it will be whatever is most profitable at any given moment. At best, a company's culture is just a branding and marketing strategy to attract employees and to appear cool. Therefore they are fickle and prone to complete collapse when just a few people are replaced.
It was going to happen Covid or not. People get old...what are you going to do? Is it really that bad if a bunch decide, "you know what? I'll take the financial hit and retire a few years early because I have been reminded of the fragility of life".
Oh I don’t blame them, I was a scientist and retired early myself. So has my brother that was a doctor. But what world is left behind when all the competent and passionate people exit the working world early? We see that world every day on the news now and it isn’t pretty.
Years ago I interviewed for Amazon. I found out that on a desired team size of 8, there were only 3 left, where the other 5 who were no longer there, as I was told, were poor performers. I didn't take the job.
Here in Europe they need a very serious reason anyway (like gross misconduct) and if they don't have it they can make you redundant but have to pay significant severance. I'm kinda waiting for an opportunity for that at my current employer, due to leadership changes it's no longer a great place to work. When there's layoffs there's usually voluntary options with a decent severance plan.
I'm glad we don't have this at-will stuff here. I would never considering moving somewhere that does (I've lived in 4 countries now). Universal healthcare is also a must, and good public transport.
So America will never be an option for me in any case.
>Here in Europe they need a very serious reason anyway (like gross misconduct)
Definitely not true in all of Europe. There's some countries where severance pay is not mandatory for all workers' groups and reasons aren't needed to fire someone (the green countries)[1] and also some have loopholes where PIP is legal and often used, so in practice YMMV depending on the laws where you live and the unions in your job/industry.
>(like gross misconduct)
Again, not always true. In many countries gross misconduct can be grounds for on the spot termination. For example in NL there was a truck driver that modified the safety system on his truck so he can unload potentially dangerous cargo from the comfort of his cabin without having to keep the outside lift button pressed. When caught he was fired on the spot with no severance and no unemployment, and even lost the case after he sued for unjust termination.
Which is generally used as a means to collect evidence that they're unable to do the job before firing them (and in many cases the decision to fire them has already been made).
One of the best-written articles I've read in a long time. I wish general news coverage had this tight blend of fact, context, and long-term perspective.
internal reports from current AWS engineers seem to be confirming all of the speculation in this article. Shit's rotten from the inside out and you can pretty evenly blame AI, brain drain, and good old fashioned "big company politics"
There's been a massive talent exodus, especially among the principal and senior principal engineering roles, across all Amazon orgs since the RTO policies have been enforced. Its demoralizing to lose key engineers that you look up to and want to continue to learn from all because a few people far removed from the day to day make a bad call.
RTO in combination with Amazon being last place in AI innovation have led to departures of anyone that can leave, leaving.
> So they can't suggest a fix even if they know 100% what it will be. Thats exactly what happened this time. EIGHT different staff members pointed to the underlying cause and were told (some literally) to "shut the f*ck up and get back to your job"
Jesus, if even an ounce of that is true... Yes, everyone on the internet is a cat clawing on a keyboard... but if a ton of people legitimately confirmed to be ex-AWS point to similar culture issues... probably it's AWS that's rotting.
The problem is that if it was 8 in 10 suggestions or even 8 in 20, then yes, terrible. On the other hand, if it was 8 needles in a haystack of garbage, at some point you do, in fact, need to tell people to STFU so you can work.
I think they might have deeper issues still with their outage. I just got an email and retroactive charge for something I returned months ago and shows as returned on their own orders portal. The link in their transactional email also links to a totally different product.
That could be totally unrelated - I had the same thing happen last month. Eventually (~3 attempts over a week) managed to fight through the various chat options that tried to dead-end into "wait 3-5 business days and check again", and once I had a person on they took about 5 minutes reading the background and said "oh, I can see that this return was processed months ago, and we have it listed as received. I will return the amount you were charged immediately!"
Ehh I trust the reporting and generally agree that RTO was/is executed hamfisted but I dunno if this particular incident "makes" the narrative. IIRC LSE rate has been increasing for many years, maybe most of AWS's existence. This is part and parcel of building something so complex that continues to grow and evolve.
I do expect much better of them and they certainly have problems to solve but this is a big company evolution thing and not an Amazon-specific thing imo.
The real story from this incident is that Amazon’s “aws” partition doesn’t actually have multiple regions - effectively, it’s all IAD in a trench-coat.
This is a big deal. My employer has already started to look at bringing back our old racks from storage and switching back to on-premises. Cannot imagine he’s alone in that.
Can you elaborate what convinced you of this? We were running mostly in us-west and saw almost no impact, despite using a broad spectrum of AWS infrastructure and tooling.
With cbell gone and ajassy promoted and the misery inducing litany of morale self inflicted wounds it’s not surprising reliability is regressing. There’s no head of engineering like Charlie and Garman is a strong engineering leader, but coming after a sales guy and taking over a battered workforce, it’s not clear to me he can turn things around easily. Everyone I know worth a hill of salt left aws already - and the ones left - meh. That’s how attrition through misery works.
Dont write cbells hagiography yet. I wouldnt put the gift of fear, “no lone zone”, and rolling stone up there as top tier cultural engineering successes. They all have reasonable reasons, bits of truth, and i liked cbell overall… but the tech culture was not so consistently great.
> "Hopefully today will serve as a massive wake-up call for AWS"
I wouldn't hold your breath. There will be incident reviews, meetings, assessments, analysis etc. but basically boil down to what can we do to stop this from happening again without actually spending any more money. So no, not hiring fresh talent or retaining that talent already in play, no to radical overhaul of process and knowledge. No to remediation of known issues if it involves expenditure. Instead it will be do more with less. Beat the employees harder, enforce more and more diligence and output from less and less people for the same or less money. Spin it like mad with catchy titles like knowledge sharing, centers of excellence, efficiency improvement initiatives, agile resilience, and continuous operational excellence.
There’ll be shiny PowerPoint decks about empowering ownership and shifting left, while the remaining engineers are shifting caffeine straight into their bloodstream at 3 a.m.
Next quarter, they’ll unveil a bold new policy called Focus Fridays which will be promptly filled with mandatory incident retrospectives. Someone will suggest replacing ancient tooling, only to be told, “We’ll revisit that next fiscal year,” which is code for never.
Then come the internal awards: “Unsung Hero of the Outage” goes to the one poor sod who rebooted the wrong thing but accidentally fixed it.
HR will roll out a “Resilience Recognition” badge on the intranet. This will be marketed with great fanfare and excitement, showcasing how the company truly values it's employees and recognized their contribution because badges are cheap. Leadership will congratulate themselves for “learning from adversity,” and by the time the next blackout happens, they’ll have a snazzy new dashboard to watch it fail in real time along side their investment portfolio dashboard that takes up a greater fraction of their attention.
But don’t worry!!!! There’ll be a T-shirt. “I survived the 2025 AWS outage.” Comes in gray. Just like morale. If it wasn't for the negative impacts on the employees and customers the word Schadenfreude would be very applicable.
And it's a sad indictment on current management practices and in particular the MBA brigade* that this is all by design, acceptable losses on the alter of profit, albeit short-term profit. Efficiency theatre as far as the eye can see.
*Yes, the same people who think Jack Welch was a misunderstood visionary rather than the spiritual father of mass layoffs, short-termism, and shareholder-value human sacrifices. The kind who see burnout as a KPI and chaos as a “scaling opportunity.”
Next they’ll launch a “Transformation Task Force” whose primary transformation will be renaming the same broken process from post-mortem to value realization review. A new acronym, a new logo, and boom, problem solved at a low low cost, honest, the consultants said so. Until the next outage, at which point someone will quote Sun Tzu in Slack.
Terrible article. Im ex-AWS, left as a principal after 10 years to go take another global megacorps shilling. I dont even disagree with the premise, but its so clearly a predetermined conclusion written as opinion piece to fit the hot news topic.
Ex a sloppy as hell and inconsistent premise.
> engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause
its the point that wasnt the root cause. The root cause was ipso facto much more complex, insidious, and unobservably spooky action at a distance. I say that not knowing the true cause but being very willing to bet a bunch of AMZN that it wasnt as simple as “herp derp dns is hard and college hire sdes dont understand octets and delegation.”
Or this stupid citation if were talking about senior/long term AWS tech roles:
> Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels.
The citation _appears_ to be about consumer/retail delivery and ops folks. And how 69-80% _of total attrition is RA_. While el reg has written it trying to imply 80% _annual attrition_ in a completely different org and business unit.
So I know corey isnt stupid, and hot takes are his paycheck. But does he think his readers are stupid?
As ex-amazonian, you should know that our average tenure for an SDE is just barely over 8 months. The old farts tool should have told you that during your day
And whats the median? Talking about hot take tenure averages without expressing the components is just silly. I, personally, was well aware of actual hiring and attrit rates up and until fairly recently.
Again, Im not saying things are good (I left for reasons!). But use better data and arguments.
You'd have to have some incredible rapid attrition to have the median be above the average. Any tech company that left skewed would be considered a disaster by pretty much anyone.
Normally, you'd have drills and exercises to try to find and fix potential issues before they affect production. Saying that you can learn from failure is great comfort to the individual, but not an excuse for the institution.
This fails to recognize that the people who designed everything to rely on us east 1 did so a long time ago. "Brain drain" could just mean that they've had their fun and now want other people to deal with their mess.
>I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time.
That information is under NDA, so it's only natural you aren't privy to it.
This is how articles should be written, this is why I’m reading El Reg (a.k.a. The Register) all these decades, this is what happens when high management cares only about profits and when real engineers don’t eat the RTO bullshit. Bravo for putting this online.
P.S. I’m not an Amazon hater, replace the company name with any other big one of your choice and the article will have the same meaning ;-)
The Register is an opinionated tech tabloid filled with outrage bait. This article is not an exception, drawing far reaching conclusions from little evidence.
“Little evidence?” If the “aws” partition doesn’t actually exist when IAD breaks, Amazon hasn’t even discovered how to make multi-region cloud infrastructure. That’s a big deal.
It's not your feeling about Amazon that would cast doubt on your take, it's that you've reduced it to one pet cause and decided a source is well written because it appeals to your dislike of RTO. Nowhere has any evidence of the relevance of that to this been presented, Amazon has had outages since before WFH even began, they've all always had their occasional outages and bad days.
> At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia
Your comment is relying on that referenced inertia. Things will continue to function for a period of time, but there exists an inflection point at which they no longer function as previously.
The year or so after Musk took over was brutal. The influx of far-right pests, troll farms and porn bots was one thing, but the reliability went down the drain.
Musk effectively left his newest plaything a few months after the takeover and some events like him going in in a datacenter and disconnecting servers, that was when Twitter (I'll call it X when he acknowledges his daughter) started to stabilize again.
All that he seems to be doing these days at Twitter is messing around with the recommendation algorithm, override the decisions of what's left of moderation for his far-right friends and that's it. Oh and of course Grok/xAI or however it's called these days, but IIRC that's a separate corporate entity that just got shoehorned onto Twitter.
Given today is Diwali, perhaps the reason everything went down is because the best and brightest from India were all on vacation and weren't there to babysit/roll back the deployment that broke everything?
We’ve reached a point where I can no longer distinguish between people without experience and people repeating the talking points they’re told to repeat. That’s a major loss.
However, talent is a very small part of shipping a project. How that talent is resourced is far more important.
Speaking of DNS, I still cannot comprehend why we still rely on the current complex, aging, centralized, rent-seeking DNS.
It's one one of the few parts of the internet which could potentially be replaced over time with very little disruption.
The hierarchy of resolvers could be replaced with a far simpler flat hierarchy Blockchain where people could buy and permanently own their domains directly on-chain... No recurring fees. People could host websites on the Blockchain from beyond the grave... This is kind of a dream of mine. Not possible to achieve in our current system.
I think this would make the squatting problem that we already have way worse. There would be bots buying every single remotely usable domain, and there would be no incentive for them to sell it unless they get an absurdly large offer.
I bought tombert.com in 2014 and forgot to renew it in 2015, and it was auctioned off by GoDaddy. For like six years, it was owned by squatters, and they wanted thousands of dollars for the domain [1]. I called offering the $100 for it, and they claimed that they can't go below $1400 because this domain is in "extremely high demand". I finally was able to buy it back in 2021, presumably because the squatter purged out domains that hadn't been purchased for N years and they wanted to save money.
Now, you could argue "see! You wouldn't have had to worry about it expiring if it were permanent on the blockchain", and that's true, but if someone else had gotten to that domain first, then I would also never get it. I think the only thing that keeps the internet even remotely fair in this regard is that domain names cost some amount of money to keep.
Yes, having some cost disincentivies some abuses that completely free attracts. If email cost a few cents to send there would probably be a lot less spam around
I'm pretty sure that is why Something Awful was successful. Since an account cost $10, and any abuse could lead to a ban, you very quickly filter out spam and people who are solely there to shitpost.
> People could host websites on the Blockchain from beyond the grave...
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing (assuming they think about it at all)
> People could host websites on the Blockchain from beyond the grave...
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing.
I'm not really interested in communism but I will support it if it lets me have a Blockchain-based DNS system.
All the arguments I'm hearing against a Blockchain DNS system are rooted in petty crony-capitalist thinking.
This kind of thinking seems to permeate most other parts of society... It's gotta stop.
"Oh but what if someone steals it"
This ain't gonna be much of a problem in a functioning society where the top 20 domain names doesn't hoard like 95% of the traffic.
"Oh but we don't want people to own domains permanently or else they will take all the good domains"
Um hello?? Have to checked this thing called reality? It's already the case! So happy billionaires have to pay their $20 per months to maintain their market monopolies.
I actually don't mind other people having more stuff than me but tired of petty people ruining good ideas and stalling progress to make a few bucks.
Garbage reporting:
1. AWS had an outage
2. AWS has lost a lot of employees
Conclusion:
The brain drain lead to the outage...
I need an LLM trained explicitly on folks confusing correlation and causation and put a big old red dot in my address bar.
I love that there's a whole section "The talent drain evidence" trying to defend their journalistic integrity, but they then go on to totally face plant.
> It is a fact that there have been 27,000+ Amazonians impacted by layoffs between 2022 and 2024, continuing into 2025. It's hard to know how many of these were AWS versus other parts of its Amazon parent, because the company is notoriously tight-lipped about staffing issues.
Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't."
The internet is full of anecdata of senior Amazonians lamenting the hamfisted approach of their Return to Office initiative; experts have weighed in citing similar concerns.
So the title is all speculation. The author put 2 and 2 together and concluded that 10 is greater than 9.
AWS doesn’t even have a “devops team” nor even any devops job roles. AWS also does not use Terraform (which is what the article says everyone was replaced with) at any significant scale, so this article is similar junk.
This one mentions terraform by name (though that doesn't necessarily imply its in use, though having worked in large companies I would argue that sweeping statements about a popular technology not being used is likely to be wrong)
AWS does not have dedicated devops roles. All AWS SWEs are expected to take oncall shifts and respond to incidents, manage build pipelines, etc rather than having specific devops people to do it for them. The article you linked claiming 40% of them were fired is total junk. You can believe that or not, I don’t care.
The last one is a ProServe role, which is a consulting role that spends their time working in customer environments, which is where they may encounter terraform. It does not mean anything about internal use of terraform.
Again, I’d be wary making sweeping generalisations like that.
I already showed you that AWS has (or hires) DevOps people with publicly available information, maybe the article is incorrect but you’re clearly not better informed, so maybe cut it with the rude commentary.
Within AWS this role falls under the Systems Engineer job family. It is not a devops role, and its involvement in events like today would be the same involvement as every other SWE at Amazon.
Just do a quick google search for that “40% of devops laid off” and you’ll see that it’s actually an old article from months ago that multiple people, including AWS employees, are saying is bullshit and unsourced.
edit: found another source that says this 40% number came from an AWS consultant that worked with customers to help them be better at DevOps, and it was 40% of their specific team that was laid off. Even if it were true, it has nothing to do with the internal operations of AWS services. This is why it’s important to understand the information you’re sharing before making judgements off of it.
Seems wild that you would promote job titles you don’t hire for, makes me think that it’s reasonable for news outlets to refer to those roles in the same way honestly.
It makes a direct claim of hundreds of cloud staff being laid off.
You know what though? I’m not wasting my time with you, the fact was that this was all over social media. Then a huge outage- my original comment was factually accurate even if we contend that the article itself was bunk. And AWS clearly hires DevOps staff.
You’ve not even disproved anything you’re just making me play internet fetch. I’m not replying anymore.
> It makes a direct claim of hundreds of cloud staff being laid off.
I don't have any dog in this fight, but I don't see where this article makes your case. From your article:
> We understand around 100 jobs are at stake.
> Sources familiar with AWS operations who requested anonymity told The Register most of the layoffs affected people in marketing and outreach roles, although chatter on sites like Blind suggests folks in frontline support and in other positions may have been affected, too.
You seem to be kind of annoyed that somebody on the internet hasn’t taken your assertion that you just sort of generally Know Better as strongly as you’d like. You could probably put this entire discussion to bed by clarifying your current position at AWS and how your job there gives you direct knowledge of their devops practices.
I work for Amazon (AWS for 4 years then “the website” side of the house for the last 3)
The previous commenter is correct, there is no NOC or devops team and I’ve not encountered a Devops job family and I’ve never seen terraform internally.
Within AWS, the service teams that work these outages are the same ones that design the service, fix bugs, deploy the pipelines, be oncall, etc. the roles that fill these teams are pretty much one of three types: nde, sde, sysde. They typically use cdk if they’re doing AWS things, else they’ll use internal tooling.
The job you posted is a customer facing consultant like role - customers use terraform so having a customer facing consultant type that knows how customer-y things work is a good decision.
You could both be right if they are trying to expand terraform use from a beachhead to the entire company. You need to hire people with prior experience for such things.
Source: Former AWS Professional Services employee.
Notice the job description:
As part of the AWS Managed Operations team, you will play a pivotal role in building and leading operations and development teams dedicated to delivering high-availability AWS services, including EC2, S3, Dynamo, Lambda, and Bedrock, exclusively for EU customers.
They aren’t looking for DevOpe engineers to work alongside the “service teams” - the teams that build and support internal AWS services. They are working with AWS customers who may already be using Terraform. AWS has a large internal consulting division staffed with full time employees. When they work with customers they will use Terraform if needed.
i mean, you can assume if its on theregister its not going to have some kind of academic rigor or whatever it might be you're looking for. its the register, same basic rigor quality as the ny post.
that said, my suspicion is they're likely on to something here regarding layoffs and quality degradation.
You could make smart inferences based on past and very frequent occurrences.
Or you could just say "there is no way the thing that constantly happens over and over again has happened once again, just no way".
Staff cuts constantly happen in the name of maximising profits. They always yield poor results for a company's performance. Every time. Especially for the consumer's side of it (not the company's finances of course).
Every time.
But maybe this time it's different. That one time.
I worked at AWS and still have friends who work there. I don’t know any L5s who wouldn’t jump at a chance to leave if they even got a slightly worse offer than what they are making now. I know a few L6s and L7s that would stick around out of momentum.
But I know very few people in the industry who know about Amazon’s reputation that have a life long dream of working there given a choice.
I was 46 when I was hired there for a “permanently remote [sic] field by design role” in ProServe and it was my 8th job out of college. I went in with my eyes wide open. I had a plan, stay for four years, sell my RSUs as soon as they vested, pay off debt, save some money, put it on my resume to open doors and make connections and leave.
I was never expecting to make more when I left. I used the time to downsize and reduce my expenses - including moving to state tax free Florida.
When I saw the writing on the wall, I played the game while I was on focus to get my next vest and wait for the “get 40k+ severance and leave immediately or try to work through the PIP”.
I took the latter and had three offers within 3 weeks. This was late 2023.
Close enough. I missed 2 vesting periods. But the severance and rapidly having a job made up for one and I got refreshers my third year that I hadn’t counted on.
I left debt free, sold my old home for exactly twice what I had built for 8 years earlier, downsized to a condo half the price I sold it for (and 1/3 the size) and I was debt free with savings.
I’m now a staff consultant working full time at a 3rd party AWS consulting firm with a lot less stress and still remote. They were the last to fall. But AWS made their ProServe department return to office at the beginning of this year.
I’m ex Amazon. The company promotes, hires, and fires based on everything _except_ merit. I saw many projects fail due to under qualified teammates and leadership. Amazon is an incredible company but it was only a matter of time till its activism caught up with itself.
Between the engineering staff and the warehouse workers, I wonder how long it will be until they have already fired everyone who ever would have been willing to work there.
Even with candidate pools of hundreds of thousands of H1-B engineers and tens of millions of illegal immigrant warehouse workers, there still comes a point where such a big company firing so many people so quickly exhausts all their options.
It reminds me of the Robot Chicken Sketch where Imperial Officers aboard the Death Star all pretend to be force choked to death by Darth Vader so they can avoid getting killed by lightsaber, then come back in under different names in different jobs. It's worse though for Amazon: nobody wants to come back.
https://www.youtube.com/watch?v=fFihTRIxCkg
There was a tweet the other day about Walmart.
They raised worker pay which, unsurprisingly, did not make Wall Street happy in the short term.
However, over the next couple years there were multiple benefits:
- lower turnover
- less employee theft
- cleaner stores
- more same store sales
etc
I went looking for a source on this, looks to be: https://www.wsj.com/business/retail/walmart-employee-treatme... / https://archive.is/fPAlB
That doesn’t seem convincing as far as evidence. Is there something by more formal showing raising pay led to those specific outcomes?
There's an article about it: https://archive.is/fPAlB
Walmart executives seem to claim that the pay raise led to those outcomes; specifically, they talked to employees and realized that in order to get those outcomes they had to stop paying people the least they possibly could because why would anyone stay?
From the article:
> Rissa Pittman, then a store manager in Ponca City, Okla., said it was easier to staff her store after 2015 as wages improved and it became easier to train workers for promotions.
It is about the same level of evidence as a CEO getting hired and their profits ticking up, which becomes attributed to them.
Walmart has been building a fairly robust EX program in the last few years, increasing associate engagement to drive CX and hence business outcomes.
It's likely not just pay, but a concerted effort across the board that drives results, but the pay increase is the one that calls WallStreet's attention.
You'd make an awesome CEO.
It seems amazon itself is aware of this issue. The linked engadget article even mentions this:
> "The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."
> It seems amazon itself is aware of this issue. The linked engadget article even mentions this:
>> "The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."
This article? https://www.engadget.com/amazon-attrition-leadership-ctsmd-2...
It's from 2022, so it'd be interesting to see an update.
That's wild. Does any other company have this problem?
Very few other singular companies have Amazon's rate of growth. I joined in 2012, they just crossed 95k employees worldwide. By the time I left in 2017 they were over half a million. These days they seem to have plateaued around 1.5 million... that's just a ridiculous number of employees
It goes beyond growth though, and directly to how they treat their staff.
Whether you're in the tech industry or the warehouse, everything I've heard about Amazon is that they work you as hard as they possibly can until you burn out and then they replace you with someone new. At least on the tech side you get options though.
An honest answer that you didn't consider yourself to be lucky used to be a no-hire dealbreaker for Anazon. I wonder if their emigrés still view themselves as lucky.
Growing like cancer.
Here in the Netherlands immigrants are starting to protest about their near slavery conditions.
As time goes on you have to start importing people from further afield.
I thought that was one of the reasons why several US states are loosening child labor laws, they will put 14 year olds to work in Amazon warehouses and on the assembly lines.
Is there no obligation to attend school in these states?
That’s an interesting topic. In most states, homeschooling is almost meaningless because there are no required assessments to demonstrate student proficiency in any subject. And 11 states don’t even require a parent to simply notify the state that they’ve pulled their kid out of school.
And yet home-schooled students widely outperform government-schooled students, both during homeschooling and in college GPAs.
Yeah, but it's not that simple, they certainly don't seem to outperform students of comparable socioeconomic background attending a more c https://gaither.wordpress.com/2010/05/03/new-ray-study-of-ho...
Put the 14-year-olds on shift after school, put the adults on shift during the day. The teenagers won't have time to do homework but the schools aren't funded well enough to ensure a quality education anyway.
> the schools aren't funded well enough to ensure a quality education anyway
The US spends more per pupil than every other OECD country except for Luxembourg, and exceeds the average by over 50%. If US schools aren't funded well enough, essentially no one is.
Correct me if I’m wrong but I thought in America there’s no obligation to attend school ever, because you can just be homeschooled and then take the GED? I think it’s dumb but I don’t know for sure.
That's one reason for the corporate backing of globalism. It's sold as "mobility", but what it actually means is having the upper hand. It's not about a lack of "talent". It's about a lack of a pool of people who will put up with their crap.
Consumerism rewards this process, because the glut of mediocre goods remains cheap.
That's essentially the entire reason illegal immigrantion and the H1b program exist. They are both in precarious positions that allow them to be treated as indentured servants, forcing them to tolerate conditions Americans never would.
None of it is about talent, it's about giving companies the ability to abuse employees, with a nice side of wage suppression.
Hopefully we find extraterrestrial intelligence soon.
Anything intelligence enough to find us, or be putting out signals we can find with instructions on how we can reply in a reasonable number of human lifetimes, is intelligent enough to stay well hidden for us lest we infect them with our stupidity!
"Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has to contact us"
-Bill Watterson (via Calvin & Hobbes)
Maybe they need to invent a time machine to start getting immigrants from different time periods?
Slightly tangential: there's an excellent Norwegian TV series based on the premise of suddenly appearing immigrants from earlier time periods, see https://en.wikipedia.org/wiki/Beforeigners.
and a south park episode for ones from the future :D
https://en.wikipedia.org/wiki/Goobacks
Agriculture, food processing and handling and everything associated, particularly meatpacking, and that's valid across countries. There's a reason that even if you pay (relatively) obscene wages, there will be no domestic employees willing to pick up these jobs - the work conditions are usually horrible because there are few industries as cut-throat and cost-sensitive as anything related to food.
The root cause is global competition, especially from countries with very low wages - Ukraine, a country in Europe, for example, was a top food supplier for Africa -, and widespread income disparity in many Western countries - 67% of Americans self-report to live from paycheck to paycheck [1], for example. First it's "luxuries" that get the cut - travel, eating out, entertainment - and once everything has been cut, people go for savings in food because that's the last large expenses block that they can meaningfully control.
[1] https://www.investopedia.com/living-paycheck-to-paycheck-you...
[2] https://www.boeckler.de/de/boeckler-impuls-vermoegen-nur-jed...
> About 41% of American workers earning between $300,001 and $500,000—and 40% of those making over $500,000—say they’re living paycheck to paycheck, according to a new report from Goldman Sachs.
https://am.gs.com/en-us/advisors/insights/report-survey/reti...
The only reasonable conclusion is "are you living paycheck to paycheck" is a useless survey question.
Don't forget cheap bulk transport. Without that, cheap remote food from global competitors becomes expensive fast.
I know plenty of Americans who choose to live paycheck to paycheck. That said, I agree that the food industry is cut throat.
Of course you do: estimates via survey show that roughly 60% of Americans live paycheck to paycheck. This is an 2x increase from 30y ago.
The problem is that these are usually measuring self described rates. And people aren't using the same definition when self describing. Have come across plenty of people who claim to be living paycheck-to-paycheck because they have nothing left over after maxing out 401k & IRA, socking away $1K in a savings account, paying for their posh apartment, and more. But of course that's not what it's supposed to mean.
The Maritime industry can run into this too. Tug companies, barge companies. The level of consolidation vs the size of the labour pool is what does it.
} There's a reason that even if you pay (relatively) obscene wages, there will be no domestic employees willing to pick up these jobs - the work conditions are usually horrible because there are few industries as cut-throat and cost-sensitive as anything related to food.
This sentence is rather self contradictory...
Yeah I am not sure that is true either. People will work at these facilities but we all know the conditions there are horrible because they can get away with it - you can't threaten an American with deportation for whistle blowing. As long as companies get away with abusing immigrants then the labor market will always keep them in demand.
As long as consumers expect certain foods to be cheap(beef, for example) - you'll get this issue.
Kale in US is about as expensive as the cheapest cut of beef. Despite one taking 2-3 years of labor and the other is 60 days.
Competition and subsidies. Nearly every country has a system to help farmers "compete", which drives down prices even more imho.
Some entire industries have this problem, eg trucking. There is this shortage of truck drivers. The quiet part is that it's at the rates trucking companies are willing to pay. That rate is based on what they can charge to customers plus a thin profit margin. Because trucking is too easy to get into, companies can't simply raise their rates because other companies will undercut them (in their own desperate bid to survive). This kind of market competition is healthy for the "market", perhaps not healthy for workers.
The depletion of tech workers for Amazon is similar. The part that isn't said is: at the salaries Amazon is willing to pay. Amazon has a different market to worry about, the stock market. They can't just increase their HR spend 25% without taking a hit in the stock market. And I guess they aren't willing to change the work environment to be more attractive. Maybe they can't at their size, as it can be hard to avoid dead weight.
Google doesn't hire in the US for some PAs, for similar reason. Salaries are capped (artificially) by stock market demands. But Google doesn't call it worker depletion.
Shortages always exist at a given price. If people demand their iphones cost only $100, there would be a massive shortage of iphones.
Same goes for labor: if you pay shit, demand for that job is going to be shit.
Of course on the labor side, part of the "price" is job satisfaction, working conditions, etc. Many more people would rather be receptionists at $17/hr than working in a warehouse or factory for the same rate.
i think another part of it is whether there's any meaningful difference in value for a mediocre vs good employee.
let say you ran a trucking company and decided that you'd pay more to ensure you cornered the market on 'good' drivers. but... it turns out that your customers don't care if you have good drivers or mediocre drivers, so you can't justify charging a higher rate.
With rates of drivers hitting overpasses mediocre is a high target here.
> Because trucking is too easy to get into, companies can't simply raise their rates because other companies will undercut them.
How are they going to undercut them without drivers available to actually do the work?
This narrative is absurd. Per mile domestic truck shipping rates have gone up dramatically in recent years, and are generally quite variable. Trucking profit margins were until quite recently way up, the fall driven by high fuel prices and interest rates (which increase the costs of equipment financing and insurance) and reduced demand.
The truth is trucking is currently going through a recession with freight demand down and empty miles up. Trucking companies most certainly could raise rates and pay pay to attract more drivers; right now they don't want more drivers.
Walmart
I read somewhere recently that 60% of all retail sales in US are with Amazon. No other company has this kind of scope.
It's 6.6%
https://www.emarketer.com/content/amazon-will-surpass-40-of-...
Total US retail sales are over $700 billion per month, which is more than twice what Amazon does in a year. Amazon isn't even #1 in the US; that prize belongs to Walmart.
Both facts could be true, if the previous post was referring to the cardinality of sales not combined value and the average Amazon sale is lower than the average sale overall.
The average Amazon sale would have to be ~25x lower than the overall average, seems rather unlikely. Not to mention that counting sales only makes sense within a product category.
60% may be true in some random product category, but clearly not across all of retail. Maybe that’s what’s being remembered, but missing the surrounding context.
It might be a sign flip of their share of online sales. They account for about 40% of online sales (yes, in-person retail still vastly outweighs e-commerce), so 60% of online sales are not Amazon.
Every single engineer I know who went to Amazon except one lasted under 3 years and to this day, often ten+ years later, they all will mention how much they hated it.
The one exception is an engineer who stopped engineering, switched into product, and transferred to China to hit on the women there.
Some Amazon practices actually sound great to me (short documents, read before the meeting) but so many things just sound needlessly, relentlessly cheap.
There seems to be some sort imputation that, just because someone is on an H-1b, that they are not a good engineer.
I used to be on an H-1b and gladly came back home to India. I run my own business now. And yes. I'm ex-Anazon. It was a tough place to work, but circa mid-nineties, the stock options made it worth working for them.
I'm willing to bet I'll outcode a significant fraction of the audience on this site. And I'm not even close to the best developer around. Some of the smartest people I've met have been on an H-1b visa. Please consider not letting prejudice affect your view. You'll do yourself a disservice by underestimating your competition.
I never implied that, let alone said it.
The only thing I implied is that workers with fewer rights that a U.S. citizen are easier to exploit and abuse.
If I refuse to take a work-related call at 3am, the worst that can happen is that I get fired, and spend months looking for a new job.
If you refuse to take a work-related call at 3am, you get fired and lose your ability to stay in a place you have lived for 5+ years and made your home.
That's BS, and I hate it almost as much as you do. You can be blackmailed with deportation, and I can be replaced with someone who can be blackmailed with deportation. We're both getting screwed in this current arrangement.
Requesting work-related calls at 3am should be illegal and is in a lot of places.
Nice exchange. Both points are valid.
> circa mid-nineties
I'm interpreting this as mid-1990s, in which case I very much believe in your technical ability. My dad came over late-1990s and he worked at mid-sized companies ever since. Even then, he and his H1B peers were decently intelligent.
I would caution your defense of today's H1B/L1s/OPT workers; I'd say the quality of Indian engineers in the US has halved every 10 years.
Today's Indian engineers come to the US because they can't enroll in a decent college in India and/or obtain a upper-middle class salary from a job. It is an entirely different mechanism for which people are migrating over. It used to be brain drain, now it is sewage drain.
The H1Bs in the big tech companies are maybe 50/50 technically decent, but everywhere else, they are just taking contracting spots. It is a very corrupt and bloated system that has to go because they are not providing valuable work.
I believe his point is that, even if you expand the available pool of developers outside of the US, that pool is still exhaustible.
Well at least you’re a humble outcoder.
>There seems to be some sort imputation that, just because someone is on an H-1b, that they are not a good engineer.
I fail to reach this interpretation in this thread.
I think you’re not considering the other side of that perspective. I am sure you are very happy for your fortune to have been plucked out of India and been given the opportunity to work at Amazon and presumably live in America, which put you in the place you described that seems to be in a really good position today. The issue is that not only was the H1-B meant for highly specialized people that cannot be found in the USA, it has very long been absolutely abused by American corporations and politicians that have been betraying their own people for several decades now by engaging and ignoring this abuse that was really just about undermining salaries of Americans by giving the opportunity to you rather than Americans, while it was really mostly about enriching the rich. You were essentially just a method for the rich to get richer.
I am sure you are a wonderful person, but it’s simply an unjust treatment of Americans, even if you personally had nothing directly or reasonably to do with it. The betrayal and abuse was perpetrated by the “Americans” that led the corporations and paid off the politicians, and also the American citizens that were distracted and careless about their own politics and government and future for their own children. I doubt you would be ok with your own ruling class and rich to betray your children and the future of India, would you? It’s crazy, but America’s people largely and for a long time absolutely betrayed their own people.
I would not wish it on any society, even though it has been pervasive all over the “West”, where the rich, corporate captains, and politicians betray their own people. Imagine if your Indian politicians were to sell out India to the West or maybe import Africans or something similar, I would hope that the Indian people would make it absolutely clear to the politicians and rich that they are staring down a loaded gun and it’s not their finger on the trigger. So do I also wish it for the people of all of the western countries that they retake their sovereignty and self determination away from the rather parasitic oligarchy that has unconscionably been betraying its own people out of undeterred greed and crime against the very people that allowed making them rich and powerful in the first place.
It is not a personal thing, I think it’s just that people are recently getting a lot more angry about things because the American empire is hitting a rough patch that it has not experienced in anyone’s living memory and as it is said, (adapted) the naked people start getting angry when the tide goes out and there aren’t enough jobs to also be super generous by giving them away to Indians benefiting from the abusive systems of the parasitic cabal of the ruling class.
What you may also not be totally aware of, is that H1-B is only one of many different systems and programs that have been abused and quite literally benefit and profit foreigners overt Americans. Imagine if that existed in India; where I go to India, make 2-3x what the average Indian makes, the government gives me free housing, my children get free education and free healthcare, and I get extremely beneficial government secured loan terms on business loans and get grants to start a business and free consulting and services, and I get to bring dozens of my friends and family into India to work in my business, and I also get beneficial home loans to buy up houses and drive up prices, and my foreign children get preferential treatment in Indian universities (…while local Indians don’t get those things) and I run for office while all the foreigners I and my advantaged community brought over to India start getting our people into the government and we start taking over Indian institutions and government offices.
I combine and crossed things a bit because is a bit more complicated and nuances of course, and many Americans aren’t even aware of just how many programs and states are in place that advantage foreigners and disadvantage native Americans, who could even very well be the descendants of the founders of America. That’s why things have gotten rather tense and as it looks, unfortunately, it will likely only get worse from here; especially as BRICS builds out more of their alternative fiscal, monetary, economic, geopolitical structures; and the same traitors that control the USA will/are starting to get very nervous and borderline panicky. It seems Thucydides Trap is in full effect.
> the government gives me free housing, my children get free education and free healthcare
None of these are H-1B perks. (Note: H-1B, not H1-B. It comes from ¶ H(1) of the INA of 1952; H-1A for nurses and H-1B for non-nursing specialty occupations [1].)
Immigrants pay for housing in homes that pay property taxes that fund public education. Their employer pays for their healthcare that costs multiples what most of the same treatments and drugs cost in India. (Only once they become a resident alien do they qualify for marketplace subsidies.)
> native Americans, who could even very well be the descendants of the founders of America
You've got to be shitting me. (Try native born next time.)
> my foreign children get preferential treatment in Indian universities
The students of H-1Bs (from India, no less) do not get preferential treatment in American university admissions. If anything, it's the opposite. It's why Indian Americans join lawsuits by Asian and White Americans around removing race considerations from college admissions.
> Thucydides Trap
"Research by Graham Allison," the guy who coined the term in 2011 after a career in the Reagan administration and, before that, at the RAND Corporation, "supporting the Thucydides trap has been criticized" [2].
[1] https://en.wikipedia.org/wiki/H-1B_visa#History
[2] https://en.wikipedia.org/wiki/Thucydides_Trap#Methodological...
But you have to admit “native Americans who could be descendants of the founding fathers” is rich with humor on several layers.
It is even funnier when this argument comes from Canadians, Australians and some New Zealanders as these are some of the more open and shut cases while other countries can be argued to be on some spectrum of foreign vs native depending on the time scales considered.
Generally it is framed to meant that we are the people who captured and built these countries and they belong to our 'culture'
> “native Americans who could be descendants of the founding fathers” is rich with humor
Yes.
Though I did consider, until noting the context, that they might be referring to descendants of both the founding fathers and Native Americans. Chilling, actually. To think that a good number of one branch of our founding fathers' descendants may have been exterminated by another.
Jefferson owned slaves and slept with slaves? So owned descendants?
Slept with is really whitewashing it. Not much choice for a slave.
Euphemism. I'm not sure how graphic to be online.
Maybe Jefferson brought some of that whacky weed down to the slave quarters and had a big party. Had a good time. Drugged them up first. Not unlike maybe the underclass of sex workers today, in dire straits and desperate.
> H1-B meant for highly specialized people that cannot be found in the USA
No. H1b isn't for highly specialized people at all. EB is for that.
Not to mention, that H1b has explicit rules to pay the prevailing wage in the market.
The rest of your comment is so out of touch with reality it's painful to read.
This is one of the most painfully ignorant things I've read in hacker news. I don't even have anything constructive I can reply to, other than a recommendation to come out of your bubble a little bit.
You have no idea how nice you have it over there.
> Imagine if your Indian politicians were to sell out India to the West or maybe import Africans or something similar,
…
Firstly, this is so wildly tone deaf, that I don’t know what to say.
Secondly - India (and most developing nations I will bet) have had violent protests at some point in their history about people from other parts of the same country coming in to take jobs.
Most of the world understands why America feels the way it does. Hell I personally have better articulations of America’s problems than many Americans do.
But for the love of all that is holy, please don’t end up betraying your ignorance of the rest of the world and reminding us of some of the worst features of American stereotypes. It may be meant as an effort to find common ground, but only succeeds in irritating and potentially alienating.
Believe it or not, compared to the vast majority of humanity, your problems are better than what they will face.
I do not ask you to take an interest in the world. It would be nice if you did.
I do ask that you serve your own purpose, your own argument, better.
But why not be a superstar engineer in India to start with? Why the round trip to another country to disrupt their society?
How is he disrupting our society? He's working & building things in our country, and paying taxes that don't benefit him as much as pay for our entitlements.
For one reason or another, big tech salaries are highly clustered in the US.
… At a few small companies clustered in a small area of a single province called California.
The very highest salaries are to an extent, but almost any major metro area in the US has plenty of engineering jobs that are very high paying compared to almost anywhere outside of the US.
Seriously. I don’t know any half way decent engineer that would ever work there twice.
I like to think I'm halfway decent at my job, and I wouldn't work there once. During undergrad, my landlord working for AMZN on the opposite end of the country offered me an interview, but it was during final exam week.
I asked if I could schedule the interview after my final exams, and his arrogance really showed when not only did he refuse, but then insisted my exams are not don't even register on the same scale of importance as the opportunity to work for Amazon.
Somewhat related: a recruiter at Google cold-called me a couple months into my first job out of undergrad back in 2016 and was similarly condescending about "the chance" to work for Google compared to everything else. I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview.
I regret being born too late to work somewhere like Bell Labs, SGI, or Sun. I had a ton of graybeard wizard coworkers from these places, and they were all a pleasure to learn from and even better friends. For the first 2 years of my first job, every day of work was like walking into the Shire and talking magic spells with 20 Gandalfs.
That job was great until I got put on a team with a guy who was a former middle manager at some IBM-like company and went from being surrounded by people lightyears ahead of me to being surrounded by Dilbert characters. The messed-up part was that it wasn't even punishment. I was rewarded after completing a project with my choice of which team I joined next, and I joined the wrong one. I assumed that joining a new team to utilize this newfangled "cloud computing" thing would be trailblazing, and I didn't do any diligence on who I would work with.
To this day, I still regret not rejoining the first team I worked for, otherwise I would still be at that company and happy about it. Then again, the boredom and discontent while being on that sucky team is the reason I started investing, and now I can buy a house in cash and fund myself to do whatever I want for at least a decade. Hard to complain about the way things turned out.
I interviewed there and got an offer a number of years ago (in 2006 or 2007, for the then-nascent AWS team). In retrospect, it would have been a great career move.
I also was interviewing somewhere else and told the recruiter. His response was to be verbally abusive and say the other place "sounded boring", and then went off on how if I turned down an offer, Amazon would never talk to me again and I'd basically ruined my career. I decided I didn't want to work at a place with that sort of culture.
They reached out to me in 2013 for an interview and again in 2024.
> I regret being born too late to work somewhere like Bell Labs, SGI, or Sun.
For each of those firms there was a 'golden era' and then a time when the company coasted on their laurels, and then the slide to irrelevancy.
Some 20 years ago I started a job at Google in Mountain View, and they were paying for a rental car, so Enterprise sent a driver to pick me up to do the paperwork. On the way I was chatting with him, telling him how amazing life at Google was, all the restaurants and the stocked kitchens and massage rooms on every floor of every building etc etc. He said "Do you know what this campus used to be before Google?" I said "Yeah, they told us at the orientation, it was SGI." The driver said, "Yes, and ten years ago it was exactly like that at SGI, too. I was an engineer there."
In the UK we have a heuristic that by the time a tech giant builds a big UK campus (an imitation of their SV HQ) then you know they are in the decline phase. Some of them decline so fast they don’t even get to fully complete the campus, yet others seems to have beaten this curse… so far.
So is this where Oracle is headed with its new offices in Oxford and other locations in England?
https://bmmagazine.co.uk/news/larry-ellison-invests-oxford-i...
https://www.oracle.com/uk/news/announcement/oracle-invest-fi...
https://en.wikipedia.org/wiki/Skyscraper_Index
Is Google's office in King's Cross even finished yet?
They have more than one building at King's Cross. One has been finished for ages but the new one I don't think is done. Not sure what happened to the fox on the roof. (https://news.ycombinator.com/item?id=44229727)
You can't leave it on a cliffhanger like that. Why does he end up a driver? Or provide more details. This seems like an interesting story.
His TC/NW was sufficient enough for him to leave the toxicity behind and start living his life.
Being a driver is "living his life"?
Maybe. Was attempting to end this cliffhanger on a positive note without judgement.
He had the means so he paid off his mortgage, invested in another property, took care of college fund.
Decided he didn't want to spend rest of his days wringing his mind and drenching off energy into deciding and debating whether it is ok to call a lambda from another lambda (yes I have seen this in production and the more experienced engineer decided to do it because he had been there longer and decided that's what he wanted to do... don't ask me which company was this but it was a FAANG) or setup a step function to orchestrate the two lambda calls... or some such equivalent problem he might have come across in his SGI days.. and instead picked a job that required little amount of cognitive effort compared to what he would have done if he was still in the same line of work but still managed to support the rest of his life/needs/responsibilities.
Might as well have been an artist, construction worker or he might as well have done nothing, absolutely nothing, and it could have been everything that one would have thought it could have been.
I won't judge knowing what I know at my age. People do what they do.
Feel free to imagine what you would imagine this ex-SGI engineer's life to have been and make it negative if you want it to be. No one knows until OP throws in more details if they have any.
Driving a taxi may increase your IQ. It increases the size of your brain. Also taxi drivers have healthier brains (less Alzheimer's):
https://www.harvardmagazine.com/2025/05/harvard-taxi-drivers...
Macguire's original study on taxi drivers:
https://pubmed.ncbi.nlm.nih.gov/10716738/
Someday HN will become TN ("Taxi News") and will be populated by bored former tech workers seeking intellectual stimulation. I see, in my crystal ball, the young'uns, when perplexed by the latest technology, look at each other and say "Let's take a taxi!"
BTW it's Fall - The best to all of you! Here's to memories:(Four Strong Winds)
https://www.youtube.com/watch?v=GnylI_9rmcs
If his income was enough for the lifestyle he wanted given his level of savings, I imagine more than most corporate workers these days...
Maybe he enjoys driving and talking to people.
Ageism?
One of the best stories I have ever heard in here to be really honest. Sounds like a joke but its packed with subtle meaning of how companies rise and fall so quickly.
You have said that the driver worked because he had (enough money?) and he might have wanted to relax with the driving job but still, its an amazing story.
Relax by driving in the Bay Area? I wouldn’t yuck his yum, but damn…
I think the key difference between the old big guns, SGI, IBM and the likes, and today's Google, Microsoft, Amazon, Meta and Apple is the diversification of income streams. Even if any one of these companies just completely fucks up an entire business line or it gets replaced by something better, it doesn't matter because the companies themselves are so utterly large they can and do survive that - or because they can, like Meta, just buy up whatever upstart is trying to dethrone them.
I don't believe they can keep this up forever. Take Google which is dependent on Google Search. That their search is becoming actively worse is common knowledge, reason being more searches equates to more ads shown. If a company which respects you as a user comes around people will jump the boat. We can see this with YouTube. YouTube shows so many ads, that people have been using TikTok instead. They say TikTok is also bad for them, but they rather use it than watch an ad every 30s.
Point being entshittification comes at a cost, and companies partaking in shitty activities can only keep this up for so long.
Tiktok may show a lot of ads but they are all skippable the moment you realize they're ads.
Not sure "enshittification" applies here, as Google isn't leveraging a middleman role to maximize revenue extraction from both sides of a transaction it facilitates, but I think you're right that Google does still have most of its eggs in one basket. Search and YouTube ads together still make up a majority of their revenue: https://stockanalysis.com/stocks/googl/metrics/revenue-by-se...
"Google leveraging a middleman role to maximize revenue extraction from both sides of a transaction it facilitates" is exactly how I would describe the google ads business.
>Google isn't leveraging a middleman role to maximize revenue extraction from both sides of a transaction it facilitates
They're literally the middleman between layfolk an the Internet itself?
>I don't believe they can keep this up forever. Take Google which is dependent on Google Search. That their search is becoming actively worse is common knowledge, reason being more searches equates to more ads shown. If a company which respects you as a user comes around people will jump the boat.
I think their strategy is to poison the internet so thoroughly that no better search becomes possible. The costs alone to spin up a new search company are an enormous barrier to entry, they'd only need to erect a few more to make it impossible.
IBM is a bank masquerading as a tech company.
That didn't hold true for Intel, which once had a monopoly.
Intel never had a true monopoly over x86, there always was AMD and a few others who made x86 CPUs although it's only AMD these days. Yes, they had market dominance, but (similar to NVDA) in only one specific market: x86 CPUs.
Intel never managed to leverage its dominance in x86 CPUs into dominance in other markets though, and that is the key difference to the ultra-large companies I mentioned... yes, they did have ARM offerings (XScale, I 'member tinkering with an NSLU2 decades ago), they did have a cellular modem line (that failed and got sold to Apple eventually), they still do have the Intel Wireless lineup (which is pretty widespread but has a healthy competition), and they got a decent dGPU lineup that nevertheless is at, what, 1% of market share?
And that is what is screwing over Intel at the moment. The server CPU market is going down the drain, gaming consoles went to AMD, Apple is completely lost as a customer (thanks to Intel's various fuckups) and consumer device demand is shifting to phones where Intel has absolutely zero presence. And on top of that their fabs have fallen way behind plan - to think of that Intel has to go to TSMC? How far the mighty has fallen.
> IBM
Check the list of companies they've acquired, divisions they've divested, random research they're doing. While mainframe is a big portion of their revenues (depending on year), they're super diversified.
That’s nothing new. Conglomerations had been around for decades before SGI et al and that type of organisation has its own problems. So you do see them fail too.
For example Thorn used to be massive in the 80s and by the end of the 90s it had ceased to exist. Arqiva is another that’s presently in freefall despite previously being too big to fail.
Also, I dont really think you can accuse IBM of a lack of diversification.
HP is another good example of a name brand that has rotted into nothingness.
That’s just because they spun off all the high quality companies (Agilent, Keysight, Verigy, Avago). The PC server and consumer print business have always been commodity product.
I feel the same way.
The question that arises is: How can you potentially spot which companies are about/likely to enter a 'golden era' when you interview there? What questions could surface some sort of likelihood? Is it possibly to identify them before they enter the 'golden era'?
It doesn't matter. Most jobs even at those companies were not in the interesting areas you heard about. For every one person at the cool jobs there were thousands elsewhere who had regular deadlines and a regular job. Odds are you wouldn't have had the cool job even if you were born in the right era.
Better advice: when interviewing ask questions when they ask if you have any! Find out what the job is really like.
Ask what hours they normally work - if they give exact times that means they are strict about the times. If they give a lot of hours that means you are expected to work a lot of hours. If they give a range that means they really have flexible times. If they talk about leaving early for their kids third grade events that means they support families.
Ask what they really wear - this is clue to what the dress code is like.
Ask about the perks you care about. I don't play ping-pong so won't mention that perk if I'm interviewing you, but if you ask I can tell you that there are regular tournaments and people do play games here and there, but the tables are empty in the middle of the afternoon: if you care about this perk ask, otherwise focus questions elsewhere.
There are a lot of great jobs. There are a lot of bad jobs. There are jobs that you would hate for reasons that the people who work there don't even care about. There are jobs you will think are great that others will hate.
Yeah, but I mean after decades of experience, you already know how to do those things, and they're fairly basic stuff to know and learn from.
Was thinking more "Imagine you have 30 years of experience and casually looking for the next Bell Labs, what to look out for when there are the company?"
I don't think it's really possible for the average employee. You'll just do an interview, like the vibes, and get unbelievably lucky.
By the time their golden age is known outside of the company they are very likely near the decline phase; even if they aren't you are going to be competing with the best now.
For actual upper level leadership: they have the ability to make the golden age happen but studying the circumstances that allowed it to happen at other companies and being very selective about employee number 2-49. After that it's out of your hands.
> being very selective about employee number 2-49. After that it's out of your hands.
Indeed. There's an old saying that A-level people hire A-level people, but B-level people hire C-level people.
Obviously this is too simplistic: How do any B-level people get there in the first place? But there's still some truth to the idea that the overall talent level of a company tends to degrade as it gets larger unless very unusual structures are in place to work against that tendency.
That's just a plain scaling issue, isn't it though? Eventually, the supply of A-level people dries up, no matter the compensation offered. If growth is to continue, B-level people must be hired.
Yes A-level people are rare and expensive. The mistake I see too often is companies not focusing on keeping their core revenue-generating team A-level. Put the B-level people in support roles. When you dilute the core revenue team with B- and C-level people, the As tend to leave and then you're in big trouble.
The quote is from Steve Jobs and is absolutely true. As soon as the first bozo infects your team, they will start hiring other bozos, and after a while your org has regressed to the mean. Therefore you should hold a ridiculously high bar for hiring. A temporarily empty seat is preferable to a non-A player.
> If growth is to continue
Maybe this is the crux of the issue.
> By the time their golden age is known outside of the company
Yeah, but that's the thing, when you're interviewing, you usually have some sort of access to talk to future potential colleagues, your boss and so on, and they're more open because you're not just "outside the company" but investigating if you'd like to join them. You'll get different answers compared to someone 100% outside the company.
I think the problem is false positives, not false negatives. The people you interact with during the interview process have all sorts of reasons to embellish the experience of working at their company.
> The people you interact with during the interview process have all sorts of reasons to embellish the experience of working at their company.
That's true, but you have to be kind of smart about it. If you just ask the question "Is working here fulfilling?", of course they'll say "Yes, super!". But you cannot take that at face value, your questions need to shaped in a way so you can infer if working there is fulfilling, by asking other questions that can give you clues into that answer.
I worked for a 100-year-old Japanese optical equipment manufacturer (household name, but I don't like to mention it in postings). One of the top-Quality manufacturers in the world. I worked as a peer with some of the top engineers and scientists in modern optics (and often wanted to strangle them).
I worked there for almost 27 years.
The pay was mediocre. The structure and process would drive a lot of folks here, into fits.
But they consistently and routinely produced stuff that cost tens of thousands of dollars, and that people would stake their entire careers on. Stuff that some folks would assume was impossible to make. They have thousands and thousands of hard-core patents.
I felt pride for working there. My business card opened a lot of pretty amazing doors.
It's disappointing to see the stuff that folks here post, when I mention it. It almost seems as if people think I'm exaggerating or outright lying or boasting.
I'm not. There are places that foster greatness; simply by being a place that has a long culture of accomplishment. I was just someone that stood on the shoulders of giants, and I was lucky to have the experience.
That said, I think some of their managers made some big mistakes, and they took a drubbing, but I will bet that they are already getting back on their feet. They are really tough. They weathered being bombed in World War II, and multiple depressions and recessions.
Thanks a lot for sharing your experience, I at least appreciate it!
With that said, if you were to try to figure out how someone from the outside could see that it was a great place to work, during an interview, what questions/topics do you think could have surfaced that as clearly as possible?
Hard to say, these days. Interviewing seems to be a pretty nasty, adversarial process. It wasn't, for me, back when. Not sure if any questions would have done it. I observed the place.
In my case, I was contacted by a recruiter (the old-fashioned kind, which no longer exists). It was quite low-key. When they first contacted me, I thought it was a joke.
I was flown out to a trade show in San Jose, for my initial interview, and to Long Island, for my follow-up. There were no coding tests. I started as an engineer, on a brand-new team of two. I became the manager of that team, after a few years.
I think observing the people there; seeing how they interact with each other, is important.
Of course, looking at their products is also key. Asking yourself "Do I want to help make this stuff?" is important.
In my case, I was intrigued by the culture of the Japanese. I was born overseas, and spent most of my formative years in a pretty heterogenous environment. I like to mix it up with strange (to me) people.
Considering Long Island, I'm guessing these were X-ray optics?
Nah. Cameras, microscopes, binoculars, etc. The headquarters is in Tokyo.
Are you also aware, that typically members of your generation criticize members of my generation for being soft and lazy, despite this new harsh period of adversarial interviews? It boggles my mind that you found a job at a nice company (to say the least) without a test (and that seems like a bad move on the part of the company ?)
I don't remember attacking, lecturing, or talking down to anyone. I was simply sharing my own life experience.
It is interesting that it was interpreted as some kind of threat, requiring an insult, in response.
Have a great day!
Yep. There's a balance between doing enjoyable work, getting paid what you feel your work is worth, and feeling like what you're doing is of some sort of value to your community or the world as a whole.
Maxing out all 3 of those is incredibly rare, but I think once people reach some degree of financial stability, almost all of them go for a job that feels like it's meaningful.
You have to be involved with making it golden: they don't have ride-alongs
Sure, that makes sense. But in order to know what places/communities/organizations are worth getting involved with, that has the right base conditions at least, how to identify those? Not every place has the same likelihood I'd wager, but based on what?
You're getting answers in child responses that while accurate are not necessarily answering the spirit of your question. In my personal opinion, you'll find what you're looking for by searching for a high growth Series A - B startup (I would recommend Seed but that's almost a different animal in terms of risk) with a technical product and strong technical founders + eng leadership.
When you're at a company at that stage that's doing well and has a lot of commercial runway ahead of it, the reality can often end up being that the golden age will last long enough for a very pleasant 4-6 year tenure if you so decide to stay at the company through its growth phase (which often takes it from a 50m-100m valuation to $1B+). Some of these companies will also make the leap from $1B+ to $10B+ or beyond (which makes the golden era at least as long as 6-10 years) and although nothing lasts forever, it can last long enough for you to find what you're looking for at least for a decently long period of time. This pertains to what other commenters have mentioned with regards to "making it golden" -- the golden era is what it is because everyone needs to make it golden and the company is too small for anyone who would dilute that for their own gain to do so without anyone noticing.
The challenge to this approach is that it requires being able to assess a company's commercial prospects as well as the quality of the company's founders, leadership and early team well enough to assess whether the company merely looks like a golden era company or whether it is actually the real deal -- something which even professional investors who target these kinds of companies struggle with. It is possible, but in my experience, it definitely took a couple of rounds of trial and error and getting burned a few times before my radar worked.
Yeah, see also: Digital Equipment Corp until the mid 90s (where I did my co-op). Lots of brilliant people there, coasting on their legacy until they fell hopelessly behind from mismanagement and bloat. I was lucky to catch a glimpse of it at the end.
I interviewed with Amazon a few years back. The whole thing turned me off. A recruiter reached out and I was interested (it was late 2020 and the money was tempting). But before the first phone screen I had to have a call with the recruiter again, where she gave me a list of things I needed to "study" and was told that "successfully candidates usually spend 5-10 hours preparing for the interview". The study list was the usual list of CS101 topics. I didn't bother preparing and it was a good thing because on the phone screen the guy just asked me some a fairly mundane coding question and then some more general stuff (it was actually a very reasonable interview). Based on that they wanted to proceed to a final interview which was an all-day affair (on zoom of course because this was during the pandemic). But first I had to do ANOTHER 1h call with the recruiter where she gave me ANOTHER list of things I needed to "study" and reminded me that I should spend 5-10h preparing. That was too much for me and I politely declined the opportunity.
I refuse on principle any interviews that expect you to study or "prepare" for an interview. I'm sure I've missed out on some money, but they've also missed out on a pretty good engineer and teammate :-)
You should prepare for an interview. However 5 hours seems like a lot and I question if CS101 is worth preparing for. (If I know you will ask about a red-black tree I can look it up - but like most engineers I never think about it because my standard library has it implemented for me - unless the job is implementing the standard library I would not expect you to ask that question)
You might be asked to write something like fizz-buzz in an interview - but the point is there isn't a good answer to that. (there are a few possible solutions, but all of them have something you should not like - which makes it a simple yet real world like problem and thus something you should be able to figure out in less than an hour without study)
What you should prepare is figure out how they interview and thus what questions they might ask. (nobody will tell you what questions will be asked, but they may tell you the style) Practice the answers. Practice stories of how you worked in the past so you can twist the story to answer the question (the above is how you should prepare for the STARS interview my company does). If you were in prison or something then be prepared to talk about why they should believe you are reformed, but most people don't have such a thing in their past that they should find.
yeah, you should put your prep time into the non-technical parts of the interview.
- everybody you meet is going to ask "so, tell me about yourself" so you better have a good answer. have a pitch that highlights relevant parts of your work history, discuss goals/interests, show a bit of personality.
- there's always going to be "do you have any questions for me?", so you need to have a couple of questions ready to go that make you seem interested/thoughtful AND help you extract good signal from the interviewer.
> You should prepare for an interview
No, I really shouldn't. Especially if a company reaches out to me. If they are so flooded with fakers and coasters, they don't need to contact me.
Not after 25 years in the industry with easily verifiable companies and references.
every job process i went through so far has been applying -> if they like me a call with the usual job interview questions -> if both want to proceed a meet-and-greet with the actual team -> if both still are interested, sign a contract.
crazy how much time is wasted in US/VC tech.
I think the idea is they want to inconvenience you to filter out people who aren’t desperate and willing to deal with their bullshit. But, expecting you to cram for an interview just makes it seem like they value metrics over actual merit.
I am absolutely sure that the money you missed out on had a bigger affect on you than them missing out on hiring you.
Every large tech company or any tech company that pays decent money requires preparing for coding interviews *if you are trying to get hired as a developer*.
I personally didn’t do much prep for my Amazon loop accept practice answering behavioral questions in STAR format. But I also had to thread the needle of having experience to get into the Professional Services department as someone who knew cloud, how to talk to people, architecture and leading projects.
If I came out of college post 2012 instead of 1996 with path dependencies in 2012, you damn well better believe I would have been “grinding leetcode” to make BigTech money.
They have an enormous amount of money, wonder why they don’t go after folks who know algorithms without cramming beforehand.
I call it a “gravity problem”. I might not like or completely understand gravity. But it makes no sense to complain about it. I’m not going to jump out of a window on the 30th floor.
If you want to make $150K+ straight out of college and $259K+ a year three years into your career, you play the game. If you don’t want to play the game, accept the reduced amount of money from staying in enterprise dev.
At 51, I would rather get a daily anal probe with a cactus than deal with any lathe company especially BigTech again and I’m definitely not going to chase after a job where I have to work in an office. But I know what I’ve been giving up for the last 2+ years by ignoring recruiters from GCP’s internal consulting division based on my stances.
But again, I’m also 51, I’ve done the build the big house in the burbs thing twice and I have grown (step)children that I’ve raised since they were 9 and 14 who don’t live with us
> I regret being born too late to work somewhere like Bell Labs, SGI, or Sun.
I'm not even out of college, and I feel the same way. Especially for Sun, everything they did was so cool. "The network is the computer" and all that.
Even though I worked for companies that killed Sun, I never stopped admiring the foundational work the company was doing, which was not just cool, but critical for technological progress, and was very sad when the company sold out to Oracle and was gutted alive. HPC stuff Sun pioneered is still very relevant today. In an alternate timeline, Sun fully embraced Open Source and became a key pillar of the internet today.
Unfortunately, while we are well aware of cool tech companies that were ran aground by the finance/sales/management consulting types, Sun felt like a company ran aground by engineers.
Zuck famously kept the Sun logo up for quite a while when Facebook bought Sun's HQ campus, as a warning to the employees of what they could become. In some ways, Facebook/Meta is the spiritual successor of Sun, just like Google became the spiritual successor of SGI when they bought the SGI campus.
But these two ad driven companies never quite became the new Sun/SGI, for better and worse.
I don't disagree that Sun was a company run aground by engineers -- though I certainly like to think of myself as one of the engineers trying to navigate us around the rocky shoals! For whatever it's worth, I broadly stand by my analysis on HN fourteen years ago (!!) of Sun's demise[0] -- which now also stands as clear foreshadowing for Oxide eight years before its founding.[1]
[0] https://news.ycombinator.com/item?id=2287033
[1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...
There's a world where Sun did what you hoped (became a systems company) and created Joyent in-house. However, hyper-scaling means going fast and cheap before good comes along. Sun's habit was fast and good and that's an extremely difficult hurdle to overcome culturally. (By fast I mean growing a platform, not raw performance, FWIW).
Solaris 10/11, with all its technologies (zfs, zones, crossbow, dtrace, etc), was the pinnacle of UNIX that came out just when the world changed. At a company I worked at circa 2008-12 (that was a solaris shop) we essentially created a proto-docker with containers and ZFS that allowed rapid deployments and (re)building of our systems. It was a game changer for on-prem.
Bryan,
I'm not sure what Sun could realistically have done to come out the other side of the dot-com carnage. Other companies in roughly equivalent situations come to mind. You start looking at doing a hard reboot when the margins for that reboot aren't there and it's difficult to see the light at the end of the tunnel. Maybe an earlier reinvention involving more open source and alignment with where hardware was headed. Don't know.
(not Bryan)
Sun did waste a lot of money in buying MySQL, $800 million in cash and $200 million in stock. Certainly a distraction, as well.
Sun never offered any way to inexpensively get onto the on-ramp of Sun hardware and software as they thought they could continue selling high-margin hardware forever; they had their $995 V100 which even included their much-loved LOM which was a remote-management device like iLO/DRAC/IPMI , then followed it up with: nothing.
info about the V100: https://dogemicrosystems.ca/pub/Sun/System_Handbook/Sun_sysh...
That Solaris/Toshiba laptops deal was interesting, but if I recall correctly the price was a bit too much, and maybe it could have been considerd yet another distraction.
I surely would have liked to get one of those laptops, though.
Well, in stock market terms the MySQL deal paid for itself. It pushed the stock well up. However turning this in real money wasn't possible in the year they had till IBM and Oracle did their bidding.
“ Believe me that some of us understood this: I worked extensively on both Solaris x86 and with the SPARC microprocessor teams -- and I never hesitated to tell anyone that was listening that our x86 boxes were starting to smoke the hell out of UltraSPARC.”
Was that before or after you realised the Linux kernel devs were better at squeezing performance efficiencies out of x86 than you guys were?
Interesting. That was not my perception of Sun at all. “The network is the computer” was a marketing campaign. Java was a language developed for IoT/toasters, and then hard pivoted to a write once run anywhere weblet language (ultimately to be replaced by a guy who threw together an integerless programming language that sounded like a skin condition, renamed to ride the crest of energy sun marketing money threw at things).
Sure, Solaris was rock solid, but it was also pretty conservative in its march forward as a Unix, being ultimately trumped by Linux.
Sun had an amazing team of people that worked on Self project led by David Ungar and others (Lars Bak who helped give us V8). They let the whole team go, who then went off and did sime cool things with dynamic optimization, which Sun ultimately ended up hiring/buying back to create the HotSpot VM.
Any NIH and other dysfunctionality went far beyond the engineers at Sun.
> “The network is the computer” was a marketing campaign.
No, not at all. It became a marketing campaing in the very late 90s dot.com boom, but the concept that defined Sun goes back to the beginning, 1984. Back then, that was a radical vision and Sun truly lived it internally for a long time.
https://en.wikipedia.org/wiki/The_Network_is_the_Computer
It's a small side point, but the skin-disease name came later:
Mocha -> LiveScript -> JavaScript -> EczemaScript or whatever
[0] https://en.wikipedia.org/wiki/ECMAScript
> an integerless programming language
Technically true-ish, but deserves an important qualifier. The Javascript number format has a huge "safe space" of integers between
Also, the number format is a standard, not only used by JS, and given that it was supposed to be a minimal scripting language it is hard to argue against the initial design choice of choosing one all-encompassing big standard, and not burden the language with a complete set. Since he criticism was on the initial design:> ultimately to be replaced by a guy who threw together an integerless programming language
I would like to refute it by pointing out that the criticism ignores the initial use case, as well as the actual existence of integers within that larger number format standard. Later, when enough people (and companies) demanded it, a big integer type was added, after all.
Internally runtimes use different paths depending on what kind of number it is.
For many use cases of integers, especially internal ones, like array indexing and counting, those integers are just that, and an extra integer type for extra purity is not much of a problem. For other uses of integers, e.g. finance (using cents instead of dollars), it sucks that you have to pay a lot of attention to what calculations you perform, so not having (had - until BIGINT) a real integer type as aid indeed made it less pleasant to do integer arithmetic.
The coolness peaked before the “the network is the computer” phase, IMO. Late 80s vs mid 90s.
This was also back when you could walk into the library and get the email credentials of a random professor and then use it to hide behind when you took down a network of another in state university because an engineering professor didn't think computer science majors were as smart as he was.
Yeah, man, good times.
My buddy got a visit from the feds and lost his computer lab access for a semester.
I still giggle when I tell that story.
> I still giggle when I tell that story.
Not sure why anyone would think that stealing someone else's data and attacking a network is funny. The only difference between then and now is that now you would get a criminal record for that. It was as morally wrong to do that back then as it is now.
I'd note that a huge amount of the work at those companies was hardware (and a lot of theory in the case of Bell Labs)--though there was, of course important software as well, a lot of it related to Unix.
Doesn't mean it might not have been a blast but not hacking on software and playing in the open source world as is the case at at least some companies today.
Well I still think that software like DTrace, ZFS, NFS, IRIX and Solaris, IrisGL, and the like are cool, even if there was a lot of hardware engineering. I realize that there are disadvantages to it, but the variances in ISAs (MIPS, SPARC, Alpha, etc) seems like it could have posed challenges for software people.
I don't know; I'm not young enough to remember.
Sun did a lot of great software too and I know a lot of the folks involved. I just think many people look at the innovation through the lens of software (especially open source) hacking which a great deal of it wasn't.
When I was in the minicomputer business, it was maybe 50/50 hardware and software (and that mostly assumes you considered software to include low-level things like microcode). And software people weren't mostly paid more than those in hardware--which is to say generally a good middle class professional wage.
> which is to say generally a good middle class professional wage.
Working with Suns and other workstations as a teen (so my perspective was limited), I caught the very tail end of software as a modest middle class professional wage for everyone doing it (right before the dotcom boom hit).
The people I worked with were really good at what they do, but not strutting like newcomers started doing pretty much the instant the dotcom boom started, and not rich. (Well, one guy did buy a used MR2, and get his private pilot license, but he also lived with his wife in a trailer on an undeveloped parcel. He was a very solid software engineer, working on important stuff.)
I might have inadvertently tried to preserve some of that modestly-paid excellence of the generation before me, but I don't recommend that. Cost-of-living in my area is determined by people making FAANG-like money (well, and real estate investors, and price-fixing), and you have to either play along with that, or move away.
Mind you, California cost of living was on the high side even in the nineties even relative to at least modestly expensive areas like the Boston area suburbs--there was really very little tech in Boston proper at that time.
But a somewhat high-flying (albeit hardware) company was recruiting me for a CA job and they basically admitted it would be a lifestyle downgrade in terms of salary.
Some people made a lot of money when dot-com hit. A lot also got wiped out and ended up leaving the industry.
I never had the highs or lows. I was probably making something south of $100K in the late 90s.
> But a somewhat high-flying (albeit hardware) company was recruiting me for a CA job and they basically admitted it would be a lifestyle downgrade in terms of salary.
I've been given second hand accounts of similar situations. One was team consolidation, and the business was offering Boston-area engineers positions in San Jose. One of the folks who moved with his family was back in MA within 5 years. His salary was not adjusted as much as it should have been for the cost of living difference.
If it meant that people who didn't want to do software didn't ans people who wanted to do software did software, then it sounds nice. I was never interested in making a lot of money in software.
A question, though. Was software designed at Sun closely with hardware teams and vice versa, or were they mostly disjoint? Presumably many hardware companies that have succeeded have built good tooling around said hardware (like NVIDIA).
I didn't actually work there; I knew a lot of the folks from the perspective of an IT industry analyst both during and after a lot of the work there. I certainly saw some level of integration with things like Dtrace (how couldn't you) but when you were a systems company, it's probably the case that software folks couldn't really just divorce themselves from hardware.
The low-level software work at Data General where I was prior to the analyst biz was certainly integrated to a certain degree--read Soul of a New Machine if you haven't. The software folks for the minis were also mostly in the same same location. As things migrated to Unix, most of that team was in RTP and it's probably fair to say that there was less integration though probably wasn't something I thought about a lot of the time. Hardware stayed in Massachusetts.
"every day of work was like walking into the Shire and talking magic spells with 20 Gandalfs."
Thanks for this.
I got interviewed twice for Google, the first one I made it to the second round of phone calls, the second one only the first phone call.
The third time a Google recruiter reached out to me with the sales pitch that I was a great engineer that they would like to have at any price, I berated him if that was the case why the previous two experiences.
Never heard from Google HR ever again, and I am not sorry, I am happier this way.
I had better experiences in interviews for EA and SCEE than Google, which again I also am an happier person not managing to get an offer, and endure the crunch lifecycle of the industry as reward.
I'm not saying this is the case, but isn't it possible that you weren't a "great engineer that they would love to have at any price" in the past, and you developed your skills and knowledge? Or even if you've always been a great engineer, since you were born, why take personal offense to not getting a job before? Interviews are mostly luck, anyways, so the previous interviews have no relation to the 3rd one.
Of course I have my limitations, what I don't accept is the snake oil lip service of Google HR trying to meet their KPIs of candidates.
Followed by an interview process that is designed to get those that want to work there at any price.
> I regret being born too late to work somewhere like Bell Labs, SGI, or Sun.
I feel the same, and especially about video games.
I would never work in the modern video game industry. It seems really miserable to be overworked and underpaid to work on some design-by-committee game that I don’t even care about.
But I hear stories of some Of the companies back in the 90s that seemed to magically muster the capital to sit down and put out effectively what was a passion project, but also commercialize it.
> to being surrounded by Dilbert characters.
As a real life Wally I appreciate this comment.
Wally is the one Dilbert character I can tolerate in the workplace. He's honest about who he is and what he does. When you know you're in a bloated company run by buffoons, all you can do for your sanity is work to rule and not upset the apple cart.
I was Wally for the last 2 1/2 years of that previous job until I started to realize I'm becoming more and more like a Dilbert character myself. Something in my brain just told me it wasn't sustainable, call it fear of God or paranoia, but letting my skills atrophy in a place like that for 20 years didn't seem like it would end well for me.
The only problem was that I stayed so long, and it made me hate software engineering so much that I didn't even want to be a software engineer anymore.
I put up with it just long enough so I could avoid selling stock and drawing cash out of my portfolio, and now I'm back at square one as a post-bacc student getting my applications in order for MD and PhD programs where I'll most certainly wind up drawing hundreds of thousands out of my portfolio to pay rent and eat dinner for about a decade.
It's sad, I really enjoyed systems programming, but it seems like finding interesting systems programming and distributed computing projects that have significant economic value is like squeezing blood out of a stone. Maybe LLMs or future progress in bioinformatics will change that, now that finding ways to shovel a lot of data into and out of GPUs is valuable, but I'm so far into physiology, genetics/proteomics, and cell biology that I'm not sure I would even want to go back.
Software development evolved well past the point of solving problems, now it is just plugging solutions. Very few people actually work in novel stuff these days...
I'm currently in a place that pays me €100k just to sit on my ass, and I can do that remotely. I've tried actually doing some work, but that backfired. Not sure what to do, because on one hand my skills are evaporating, but on the other if I wanted a job that pays more I'd have to learn a lot and then work substantially more. I'm wondering if maybe sitting here until retirement is a viable option.
Similar situation. I work for a provincial government and make €61k, my scope is actually relatively large for how long I’ve been with my team but the actual problems are simple enough that some decent code means I have 0 downtime. As a result if I don’t bug anyone I typically get left alone to manage a bunch of products that run without issue. This week I literally have no meetings on my calendar, just a small project with a generous due date where I’m the solo developer.
I’m lucky in that before I got the job I was in talks to do a PhD but negotiated saying I’d only do it remote.
Now I do whatever is required to keep my day job happy and then spend the rest of my time working on my PhD. My plan was to go to FAANG after I got my degree but who knows… a comfy, unionized tech job that gives me ample time to do side projects is also not something I’d give up too easily.
I’d say do whatever is necessary to keep your job and then devote any extra hours 9-5 to some project. If I wasn’t doing my PhD I’d be making an app or a game probably, or maybe still moonlighting as a researcher. I think most office/tech jobs don’t require your full 40 hours and I can tell you I have a bunch of friends who have even less work responsibilities than me but they just use that spare time to play video games. Just do something productive 9-5 and you will outpace 99% of people is what I’ve found.
Honestly I do use that time to play video games because I don't see the point of working my ass off. Suppose I grind my ass off and manage to get a €200k on-site job with on-call. Is that actually a win? I don't think so.
I do the same, but mostly because when I have bothered to work my ass pff to try and get a cool job, I never get hired anyway because they only hire established domain experts and juniors via a university pipeline.
If this is really the case work on a side gig you find enjoyable.
You at least need some source of self-actualisation
It would be easier for me to appreciate the small things in everyday life if I weren't so lonely but it is what it is
> I asked if I could schedule the interview after my final exams
Ha, my interview for an Amazon internship was an hour after a 3-hour final exam :-)
But the job market right now is quite bad, and after hundreds upon hundreds of internship applications I would've been stupid to give up this chance. I would work for Amazon in a heart beat.
Well pardon my saying so, but why don't you?
Well usually you have to get hired first. Can’t just show up without an offer.
Are they even hiring?
>I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview.
Is this a common Google practice? Can you choose the book you want, or does it have to be introductory Java. On how many different levels does this insult work?!
I'm just very interested in this tidbit of information.
>Can you choose the book you want, or does it have to be introductory Java.
2nd edition of Kleppmann comes out in a few months... if I flunk a DE interview think I can request it?
Next level misogyny a la google :)
yeah maybe they only do it for the women they interview, as a form of negging pickup technique
I think you can still find those kinds of workplaces if you focus on research-oriented companies in "deep tech" (meaning those designing and building cutting edge hardware rather than advertising/social media).
I too had the pleasure of interning under some former Bell Labs employees, it really was a great experience.
Oxide looks like it may be that sort of company.
i’ve interviewed with aws and received offers twice over the years. the first time they made me pay for my own lunch. the second time no lunch break was afforded. i didn’t accept the offers though i know several truly excellent people who work there.
That’s weird, both times that I interviewed for Amazon (admittedly pre-pandemic) they paid for flights, hotels, and all meals, including lunch. The lunch was basically either before or after your interview (depending on whether you had an afternoon or morning interview schedule) and you just grabbed whatever you want on your own. But it was expected to be expensed just like all other interview expenses.
When I interviewed at a Google outpost, a good-cop employee they mistakenly thought had a connection to me took me to lunch (message: forget about the bad-cop interviewer you were just with, you're among friends, loosen your tongue so our spy can report back) in their cafeteria (message: look at the free food perks you'd enjoy), and initiated a conversation with an visiting economist there who then spoke of something oddly relevant to my research interest at the time (message: look at the interesting people and collaborations you will bump into every day).
Your interview lunch experience sounds like message: this is what it's going to be like, and we don't care if you join us.
yes, i definitely walked away with that impression. maybe they reserve the better experience for mire desirable candidates :)
I had a very similar experience with an amazon interviewer over an even dumber reason: I was going on vacation. Seems dumb, but in your early 20's, with everything booked, I was excited, and frankly didnt see why it was worth insulting me (Its pretty normal to be too busy to do a first round interview on a given week, sometimes things get pushed back 7 days.)
I have done a round one interview and I don’t see how it can be interpreted to do anything but turn away people with a brain.
Memorize Amazon’s insane company values and relate your resume experience to it. And that I mean every single bullet point.
Interviewers were all run by robotic people. Coding test had zero flexibility, you had to just write code in a special barebones text editor that had zero feedback besides pass/fail.
You’d have to solely care about Amazon RSUs to consider that job. They are self-selecting for the worst kinds of candidates.
The dumb thing is that it should be a job that doesn’t burn people out because they basically own the market and haven’t needed to do any sort of innovation. Amazon’s corporate culture just has a burnout fetish.
I interviewed in 2015. The recruiter told me to read the Amazon Leadership Principles, but I thought it was ridiculous to prep for something so specific to a single company, especially as I was interviewing at other companies too.
I got the job, and I think being natural helped. I've interviewed thousands of people at Amazon since, and too many people just say the buzz words with no meat, and it gets them nowhere i.e. I showed customer obsession when I.....(and then gives a bad example)
So which is true, your comment or the one above? Is it 'natural' to not prep, or to do what's expected of you, as described in the comment above?
I interviewed much later and there was definitely no getting around it, the interviewer refused to let me go on further without hitting every single value in the list and talking about a unique example of my experience for each value.
One entire experience story/project example per value, completely insane cult behavior. I felt like I was interviewing for Scientology.
> gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview
Although I doubt it was really their intention to be passive aggressive, I have to say: That was a Flawless Victory of passive aggression.
the dummies guide to java might have topped it into active aggressive
If you had been born early enough, you probably wouldn't be a programmer at all because far fewer people were. Conversely, everybody is born at the right time to join in the heyday of something amazing. You just have to identify that something and be lucky enough and/or try hard enough to become one of the few future-gandalfs. There are companies today making flying cars and dexterous humanoid robots ffs! Or SpaceX! Amazing engineering work never stopped, it just didn't linger on in the same fields.
Fair enough, but in my estimation that next big thing is gene therapy, and the best way to get involved with it is to become a medical geneticist.
I'm sorry to be a buzzkill, but I just can't get excited about privately-funded space rockets or Japanese girlfriend robots, not even if there were an 8 figure stock compensation package in it for me. To me, it's all just "do something grandiose for the venture capitalist bucks, and then maybe figure out how it helps people in 20 years."
"My sister killed her baby because she couldn't afford it and we're sending people to the moon.
September, my cousin tried reefer for the very first time. Now he's doing horse. It's June.
Is it silly, no? When a rocket ship explodes And everyone still wants to fly
Some say a man ain't happy truly Til he truly dies
Sign o the times"
-Prince
Hi, I am a half decent engineer. I say that as objectively as one can say something like this about themselves.
I worked at Amazon. Twice. In total about a decade as a Principal Engineer. I left voluntarily a few months ago.
I have zero regrets about my time at Amazon. I learned lots, worked with some incredible people, and had fun doing it.
And the culture? It was life changing for me, especially when I first joined. In all the best ways.
And Amazon today? All I’ll say is that at their size, maintaining solid culture is damn hard. The hiring spree peri-Covid definitely added unimaginable stress to maintaining the culture the company was built on.
They’re a big company, and thus a big target. It’s easy, cheap, and even lazy to kick them with stuff like this.
The truth is that while it’s changed a lot over time, anyone fortunate enough to work there should embrace it.
I gotta be candid with you: "anyone fortunate enough to work there" is exactly the kind of arrogance that rubs job candidates the wrong way. A lot of people don't see it the way you do, and you would do well to take a moment of honest self reflection and consider the reason why.
Before you reflexively jump to "Amazon is just not for everybody", it's only fair to warn you that a lot of people around here have heard this phrase and don't buy it. My former landlord said the same thing at least 9 times in the 8 times I've interacted with him face to face... right up until he got fired. Management conveniently put him on a PIP 4 months before a vesting cliff on his 2-year RSU vesting schedule.
Yeah, Amazon is not for everybody- just the kind of people who either perpetrate or fall victim to pyramid schemes.
Amazon, on the engineering side, is rough for high-end software engineers, but let's be very real: it pays better than the vast majority of careers available in America, even right out of college, and while it's stressful so is being a teacher or working in the food industry, and those jobs pay peanuts compared to an Amazon SWE. It is fortunate to get a job there. It may be even more fortunate to get a job at another high-end software company but that doesn't change the fact that a job at Amazon is life changing money for most people.
To pre-empt hyperbolic responses: I live in Seattle. I personally know plenty of people who have worked at Amazon. I know plenty of local teachers. The teachers work as many hours for a tenth the pay and burn out just as fast.
You might be right about the hours the teachers work, but they're not doing it to avoid being fired. It is entirely self-imposed stress. Teachers have unions and tenure. I'll help people, but I won't martyr myself in a hospital, let alone a frickin' tech company.
I don't consider 60 hour work weeks a privilege to be coveted, especially if I'm capturing less than 1% of the value I produce. I'm sorry man, but white collar serfdom is still serfdom.
If you're really going to keep begging this hard to be a serf on someone else's fiefdom, I'll tell you what: you can keep overworking yourself as serf on fiefdoms in which I might own shares, and increase the value of my portfolio for me, so I can draw even more passive income every month. I formally withdraw my protest to your "Protestant" work ethic.
I'm not trying to be an asshole, but I'm just hoping you take what I'm saying burns into your subconscious and something in you changes: that work into which you've placed so much of your self-esteem is just funding other peoples' passive income to either not work or live ostentatiously. If that's a great privilege to you, God bless you. I'm sure my wife would love it if I just retired- please make it happen, she would love me even more for it.
EDIT: P.S. Your "privilege" of working for Jeff Bezos' portfolio sounds a lot like a biblical curse. Specifically, 1 Samuel 8:10-18 and Psalm 105:44.
"He gave them the lands of the nations, that they might inherit the fruit of others’ labor"
>You might be right about the hours the teachers work, but they're not doing it to avoid being fired. It is entirely self-imposed stress. Teachers have unions and tenure.
No, and no. Your perception of modern labor markets is dated and disconnected from reality. Schools are constantly facing budget cuts and a huge chunk of teachers aren't even full time (instead getting things like a 60% time position - less pay, no security, but enough hours that good luck working a second job). Those that get full time positions still face the elimination of their positions.
I guess I just assumed teachers have the same protections in deep blue Washington state that they have in Illinois and Minnesota.
If I'm wrong, I'm wrong. However, there's a fix for that: move to a state where the job prospects are better. That sounds like "let them eat cake", except that moving to another state as an American citizen is a lot easier than what of people went through fleeing the Sicilian Mafia or the Irish Potato Famine, let alone WW2 European front or the fall of Saigon.
I was a teacher in blue California for four years. You don't know what you are talking about.
> you can keep overworking yourself as serf on fiefdoms in which I might own shares, and increase the value of my portfolio for me, so I can draw even more passive income every month
Yeah, I guarantee you that the guy who worked at Amazon as a Principal Engineer for 10 years has a bigger portfolio than you.
levels.fyi says $967k/yr average compensation at that level.
> It is entirely self-imposed stress. Teachers have unions and tenure.
You don't have the first idea what teachers do or what their jobs require.
If you really think all teachers have tenure and are supported by unions, you also have not the faintest, foggiest inkling of a clue about the job market or professional environment of most teaching in the US.
I think it's also worth pointing out that, even as you repeatedly label others "arrogant," your comments in this thread are themselves breathtaking in their arrogance. I rarely run across HN comments so condescending, dismissive, self-righteous, or self-congratulatory.
I shared my experience. And in my experience anyone who gets the opportunity to work there should do it.
Is it fortunate? Working at a place that holds a high bar on hiring, pays extremely well, and provides extremes of learning and growth in tech? Yeah, that’s a pretty good spot to most rational people. Building systems for 20M request/second teaches you cool stuff. So yes, it’s fortunate.
Of course it’s not for everybody. I never said it was.
Nobody is going to take seriously a person that thinks working at Amazon is a pyramid scheme.
After you work at Amazon please report back your experience.
I don't know you or your life.
> anyone fortunate enough to work there
strikes me as an insane thing to say based on the people I do know who have worked there. Horrible work life balance, mobbing, being hired to be fired by a manager who wanted to keep their team but had to stack ranking. Every one of them has described the culture as a cult, and reading comments like this one makes me think even moreso that they're right.
We do a lot of business with them and we have workshops with them sometimes, and the one thing I notice is how they're all so evangelical. They wouldn't say a bad thing about their company. I couldn't be like that, when I work something I sell my knowledge but not my soul. I'd always speak freely (not always appreciated but usually it's not a problem if it's true).
But that company culture leaves me with a very low opinion of them and very little trust. Even Microsoft engineers are less brainwashed. I've had several that just told me the truth about services.
Maybe it depends on the country but it feels like this is just their culture.
I've worked there since 2015, and this simply isn't true.
There's a lot wrong with AWS (and it's got a lot worse in the last 3 years), but there's also a lot right, and there are some really, really smart people there, several of which have boomeranged (people who left and came back).
There are smart people but the rot starts at the top and the culture thst was already bad got worse around 2022.
I'm barely a halfway-decent engineer, working at a M7 big-tech co, and my next job requirements consist of: anywhere but Amazon.
The stories I hear there is just not the style of work I'm interested in
I'm an awesome engineer and I'd never even apply to a FAANG company. I value stability over pretty much all else.
The formula is usually more money and ability to work special team isolated from the usual toxic orgs. I think A9 was probably somewhat like that, and AWS probably used to be at some point long ago
After having heard horror stories from friends who've worked there, you couldn't pay me enough.
I know some people who are fine working there. No one seems thrilled but if you're an above average engineer who is just getting by at 140k a year and suddenly you're looking at 350k a year as an SDEIII or something, that can be a life changing amount of money.
However, I think the question is, what percentage of engineers can pass the amazon interview but not the Apple/Databricks/Uber/Google/Meta ones. Because no one is picking amazon over the aforementioned companies.
However, maybe there's an opening at Amazon and not the other companies, or maybe that's your only offer. I certainly think it might be worth it for a a few years.
> just getting by at 140k a year
Lol
It can be a life changing amount of money but not always in a good way. If you're not careful, your spending expands like a gas to fill all the available volume and you're just marching sideways towards retirement with nothing saved because it's too fun spending money.
RSUs actually help mitigate this a bit, because you're not going to qualify for a giant house/apartment with your RSU money, and most places aren't even vesting monthly so you're more likely to treat them like bonuses, rather than inflate lifestyle.
Not saying it doesn't happen, but it's probably harder than if it was just suddenly getting a giant salary.
Hey, I really need that second Porsche for my weekend drives.
Yeah that is such a grim caricature..
First worlders having to "just get by" with 11 thousand dollars every month. What even is this world
Assuming they’re already living in Seattle: pre tax, and supporting a family? According to a quick google, that’s ~$100k after tax, a 3BR in Seattle is ~$4k/mo so you’re left with $52k for everything else, in a HCOL area. I buy it.
Truth. I honestly don't know how most people are surviving right now.
Homelessness + loans (credit cards) are on the rise and births are on the decrease.
But to honestly answer the question, by either not living in large cities and commuting in, sharing those 3BRs, or staying home with parents. Shaving that 48k/year (post-tax) rent down to ~12k/year frees up a lot of money.
(Also if you're young, staying with your parents and saving ~100-200k in rent for a downpayment over 10 years just seems like a smart idea to me).
Are a lot of Seattle SWE jobs paying only $140k to non-juniors?
FAANG has a huge footprint in the town, you're going to have a hard time hiring people when you're paying less than half what they are.
For the first ten years of my career, I just assumed my resume would get passed over, that even though I was smart and capable, there was no point in trying to work at top tier places.
I was single and not a lavish spender so I didn't feel external pressure to try all that hard beyond having a low six figures job.
There are plenty of low prestige, mismanaged small companies that will pay low six figures and overwork people.
It's not even "first worlders", it's American software companies paying shitload of money. Try the same in Europe.
Or anywhere in America that's not a tech hub.
near slavery
Well don't you want to be able to say "I work on the DEATH STAR" when asked what you do? Think of all the Twi'lek headtail that pulls
But would I pull the Twi'lek headtails???
"Lord, I apologize." -Dan Whitney
I had a decent time there a decade ago. In retail, though, not AWS.
I'm sure the intersection between sadomasochists and Amazon employees is above average.
Yep, everyone is excited until they leave after 1 1/2 or 2 years. There are always outliers but my personal experience is that the churn rate is incredible high.
There are a lot of companies that think that chewing people up and spitting them out is something to be proud of.
MBAs usually wind up eating the seed corn at some point.
Are there really many illegal aliens working in the warehouses? I know that Amazon does verify employment eligibility and checks documents. There may be some committing identity theft but I doubt that it's a large proportion.
I assume the parent was being hyperbolic. Illegal immigrants is the barrel scraping bottom of the work force.
Not really. For low skilled jobs that don't require much English, illegals are going to be as good as locals. Maybe even better because locals who are any good are mostly going to move on to better opportunities.
Yeah this. And warehouse work is all appified and can be configured in any language.
Any American working in an Amazon warehouse will be jumping at the chance to get out of there. So turnover would be really high. Same with delivery stuff, I don't think most Americans would enjoy a job where they don't even get a pee break.
As for legal stuff, I'm sure they'd use intermediaries to cover their ass.
I got some news for you - there are plenty of American citizens that have poor or no English.
Or are you not familiar with Puerto Rico?
Using intermediaries to hire illegally makes it worse from a liability perspective. Amazon is too big and hires too many people to get away with “oh, we used agencies in all of our markets and audited nine of them and we’re shocked that they didn’t do diligence”. All it would take is one disgruntled hiring manager. And Amazon doesn’t have many gruntled ones.
At the scale of Amazon, law enforcement is no longer bound to the law, it's bound to politics.
Your average local employer with a few dozen people? If he fucks around enough to draw in the attention of OSHA, he will find out. Easy prey and without political connections he's toast.
But Amazon? This company is by direct head count the third-largest employer in the US, second-largest in both the US and worldwide if one excludes militaries [1]. Amazon is frankly too large to enforce laws against and so is similar-sized Walmart (who has been able to extort the government into subsidizing their poor wages with food stamps), too many livelihoods depend on the existence of the company.
IMHO, a lot of the Big Tech and F500 companies should be outright broken up. When a company grows so large that laws cannot be enforced or, worse, laws get willfully ignored because it's cheaper to risk the occasional fine and bad press, eventually the rule of law itself suffers.
[1] https://en.wikipedia.org/wiki/List_of_largest_employers
Yes.
I would never work for AWS, given what I've heard, and consistently, of their internal culture.
Also, everything I've seen while working with internal staff makes me feel there's a culture of obfuscating all weaknesses from customers, practically to the point of deceit.
From what folks at AWS tell me that’s basically already happened. The best and brightest won’t even apply to work there anymore. For many key functions they’ve legit run out of people to recruit and thus have to go down market compared to competitors. Thats very much true in hot sectors like AI where AWS has “C” and at best “B” team players and leadership.
I’ve also heard key pieces of their infrastructure has sloppily written code and that communication between teams is horrible. Even with their insane salary offers, most people don’t think it’s worth it. Especially given their 6% “unregretted attrition”[1]
[1] https://fortune.com/2024/03/20/amazon-layoffs-performance-re...
The thing is, the brand is toxic. They can offer all the money they want, but no employee has any pride in working for them. So they produce the bare minimum to get their salary - and I applaud them.
Not just firings. I sure got a deluge of Amazon recruiter emails right after they went full RTO.
I still occasionally get them even though I literally was one of the people who left after they tried to make us go in office (I don't like to use "RTO" because no one on my team had actually worked out of an office for Amazon before since the project we were on was fairly new). My wife (at the time fiancee) has an autoimmune issue that makes it much safer for me not to commute, and although my manager suggested I could get an exemption, he didn't actually know what the process was because it all happened so quickly that no one seemed to have actually defined what that process was up front. I had a little less than a month to figure what to do and get that exemption before they expected me to either have that exemption, be in an office in another city three days a week, or transfer to a local team and be in the office in my current city three days a week. I ended to deciding that it wasn't worth the effort to try to figure out how to convince them to let me stay.
> tens of millions of illegal immigrant warehouse workers
Sorry, what?
They pay very well, for some roles.
I have a friend who recently started there. He just brought a Mercedes, and a second house, and he’s still in his early thirties.
They keep him busy, though.
Amazon’s not the only company on the planet that pays well. While they’re above average they’re far from top of the market. If you’re talented enough one can make a lot of money and skip the toxic culture. Double win.
Come back after your friend hits the 4 year cliff. OY2 is a killer.
That’s too bad. He seems happy, right now.
I don’t think I’d like it, but he does.
Last couple of days have been a challenge, though…
> tens of millions of illegal immigrant warehouse workers
You have a source for that claim?
The claim is not that Amazon would be using illegal warehouse workers today, but that there is theoretically a pool of tens of millions of people available. Which is still kind of dubious.
I think you misunderstood. He's not saying they have 10m illegal workers.
He's saying even if they had 10m illegal workers they would burn though them all too
Amazon culture is entirely powered by recruiting.
Even L6 managers feel this, but it becomes more clear as one goes up in levels. Recruiting is job one.
If Amazon runs out of recruitable engineers (unlikely, they are one of the most prestigious firms in the world) then they will simply lower the bar. HC must be filled.
The company is well structured, it will survive.
Not totally on topic, but I recently passed the tipping point with Amazon shopping. I now go to Temu. They have US warehouses shipping in a couple days which was the only thing keeping me on Amazon. Plus everything on Amazon is basically the same stuff on Temu but with a markup!
I seem to remember an internal memo being leaked in which middle management was complaining that they would eventually/soon burn through all the laborers in that region, after which they'd suffer immense difficulty in staffing that warehouse.
It's only a matter of time before ICE starts cracking down on Amazon Fulfillment Centers...
There was another comment that pointed out how unlikely this is to happen because Amazon is just too big to bring the law down on now.
When you're the, what. Second? Third? Largest employer in the US, enforcing the law now becomes a meaningful hit to economic velocity. And as much as Trump hates brown people, his administration has begrudgingly revealed that there are moves that his billionaire buddies Will Not Allow.
I'm no fan of ice or this administrations deportation strategy, but it's a serious problem that even enforcing the law on Amazon is now an economic liability so much that nobody dares to try
Given their self-imposed labor supply issues, notably the awful working conditions, I hope the workforce there figures out how to effectively organize against this cesspool of a corporation.
Look, not to defend anything Amazon is doing, but this causal chain seems rather pareidolic and under-evidenced. You could spin some kind of crazy narrative about any major outage based on changes in policy that happened just before. But this isn't nearly the first AWS outage, and most of them happened before the recent RTO changes. It needs more evidence at best.
The article wasn't about the outage happening, it was about the amount of time it took to even discover what the problem was. Seems logical to assume that could be because there aren't many people left who know how all the systems connect.
It's about the time between the announcements about finding the cause. I find that to be thin evidence. There are far too many alternate explanations. It's not even that I find the idea to be implausible, but I don't think the article's doom-saying confidence level is warranted.
It’s a bit of a guess though isn’t it?
It's the most plausible, fact-based guess, beating other competing theories.
Understaffing and absences would clearly lead to delayed incident response, but such an obvious negligence and breach of contract would have been avoided by a responsible cloud provider, ensuring supposedly adequate people on duty.
An exceptionally challenging problem is unlikely to be enough to cause so much fumbling because, regardless of the complex mistakes behind it, a DNS misunderstanding doesn't have a particularly large "surface area" for diagnostic purposes and it is supposed to be expeditely resolvable by standard means (ordering clients to switch to a good DNS server and immediately use it to obtain good addresses) that AWS should have in place.
AWS engineers being formerly competent but currently stupid, without organizational issues, might be explained by brain damage. "RTO" might have caused collective chronic poisoning, e.g. lead in drinking water, but I doubt Amazon is so cheap.
> An exceptionally challenging problem is unlikely to be enough to cause so much fumbling because, regardless of the complex mistakes behind it, a DNS misunderstanding doesn't have a particularly large "surface area" for diagnostic purposes and it is supposed to be expeditely resolvable by standard means (ordering clients to switch to a good DNS server and immediately use it to obtain good addresses) that AWS should have in place
You seem to be misunderstanding the nature of the issue.
The DNS records for DynamoDB's API disappeared. They resolve to a dynamic bunch of IPs that constantly change.
A ton of AWS services that use DynamoDB could no longer do so. Hardcoding IPs wasn't an option. Nor could clients do anything on their side.
> It's the most plausible, fact-based guess, beating other competing theories.
"My wildly conjectural and self-serving theory is not only correct, it is the most correct".
Lol perfectly represents the arrogance of hn.
The article describes evidence for a concrete, straightforward organizational decay pattern that can explain a large part of this miserable failure. What's "self-serving" about such a theory?
My personal "guess" is that failing to retain knowledge and talent is only one of many components of a well-rounded crisis of bad management and bad company culture that has been eroding Amazon on more fronts than AWS reliability.
What's your theory? Conspiracy within Amazon? Formidable hostile hackers? Epic bad luck? Something even more movie-plot-like? Do you care about making sense of events in general?
My theory is someone fucked up. There’s literally no information that gives us any additional insight to what happened yet.
We've witnessed someone repeatedly shoot themselves in the foot a few months ago. It is indeed a guess that it may cause their current foot pain, but it is a rather safe one.
Indeed. No disrespect to Justin (great person) or any of the engineers who were sacked but Corey's post here is basically "here's someone who was sacked, and here are several other layoff news". AWS is really big organization. Several orders of magnitude bigger than people who were remote/refused to RTO. Organizations like this survive these brain brains.
Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't."
From TFA.
> Organizations like this survive these brain brains.
True, that's the other thing. Even if it's true that brain drain directly caused/exacerbated this event, big companies have a lot of momentum. Money can paper over a terrifying range and magnitude of folly. Amazon won't die quickly.
There's no argument that the end is neigh.
There is an argument, specifically, that "This is a tipping point moment."
Not the end, and possibly not even the beginning of the end. But the start of what is likely to be a long and persistent slide.
This is the time to accept that the path forward is keeping people and giving them the best tools you possibly can to do their work. That is, the same as has been true for decades remains so.
Yes, development tools are better every day. Yes, you can downsize. No it won’t be felt immediately. Yes, it mortgages the future and at a painfully high interest rate.
Suspending disbelief won’t make downsizing work better.
Seems like it worked fine. They laid off a quarter of their junior principal engineers, the stock went up. They had a massive outage a few months later, the stock went up again. Everything's working out fine for their strategy so far.
I remember comments saying the stock went up because the average joe didn't realize how much of the internet was powered by AWS until all their day to day apps started failing. To most people Amazon is an online shopping site.
The memes going around when the National Inquirer tried to blackmail Bezos and people were showing what their site was running on were pretty classic
It continues to work until eventually the debt is so high the company implodes.
See: general electric, RCA, Xerox, GM
Yes, until the company has had all shareholder value sucked out of it and its hollowed-out shell finally implodes.
But Bezos will still have his billions.
Boeing?
You would think this would eventually show up on the balance sheets, right? Presumably a lot of their big customers have SLAs with money penalties, so maybe next quarter earnings? Or quarter after that?
SLA monetary penalties won't make the difference there. Enough giant customers moving substantial workload off of AWS (either to another cloud, or otherwise) would, but the timeline for that is years, not next quarter.
More likely outcome is the companies will spend even more on AWS, deploying to multiple AWS regions...
Just a guess but I think this bubble will stretch a bit more before it pops.
The entire thing reminds me of Wile E Coyote suspended in mid-air before he looks down and plummets to the bottom of a ravine
Nah, the decades old crop of "new" big tech companies are just entering their IBM phase.
Where are the young companies trying to replace them? There are all the AI companies, but Google and Meta both have competitive chatbots, and OpenAI is signing weird deals that don't make it look like a long-term player.
They all get bought out by Amazon, Google, Meta et al. The cash just tastes too good when stacked up against the prospect of grinding for 15 years and probably nothing coming of it.
> Yes, development tools are better every day
Are they?
I don't think there's ignorance of the fact that turnover is bad, I think the field is being designed to homogenize staff and favor uniform mediocrity so that employees truly do become interchangeable. We're so close to just plain talent being likened to cowboyism.
I observed a similar thing in the post Goldwater active duty officer corps reformations.
I wish to understand the virtue of Amazon culture.
It seems that at L6 and below workers are a Taylorism-style fungible widget driven to convert salary into work product, guided to create the most output for the longest time before mentally breaking down, then being swiftly replaced, with L7 and above being so incredibly political that keeping the snakes and vultures from eating your team is a full time job at every level of senior management.
It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.
I would love to hear positive counter perspectives from Amazonians because the anecdotes from my L6-L10 friends describe what sounds like an inhumane hell on earth.
> It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.
It’s pretty simple, actually. Once such a dominant market position is achieved, you can get away with almost anything, whether with customers or employees. This is true of all the BigTech companies.
I think there's more to it. When you're dominant, you make money whatever. Think of Amazon et al. as huge spigots of money. Now, it becomes optimal to fight for more of that money coming your way. It's like the resource curse for countries. Nobody gains from growing the pie; they gain from stealing the pie. At some point, parasites and parasitic behaviours invade.
I think it comes down to demand and supply for jobs.
The only time Amazon was forced to change its ways was during Covid hiring boom where they couldn't compete in the talent market. They were forced to increases their salary bands and the culture was also a bit easy during that time. But starting mid 2022 it's been an employer's market and Amazon is making sure to juice every bit out of its employees while it can
It's not as conscious as that, its an emergent outcome of the snake pit.
Engineers have to spend an inordinate amount of time on "managing up", which means they have very little time and attention to do what would otherwise be a reasonable workload. Additionally, good engineers hate and despise this so it contributes a lot to the burnout.
Tech will learn like manufacturing folks did that experience is not fungible. You can try to replace someone, but the new guy also needs to accumulate the scars from the system for years before taking over.
You cannot just keep abstracting and chopping systems to smaller and smaller subsystems to make them easy to digest.
At some point someone needs to know how these coordinate and behave under disturbances. At some point someone needs to know at a low level what the hell is going on.
I don't know, manufacturing seems to have learned pretty well that they can ship everything overseas and people will eventually accept products just aren't made the same way they used to be.
If AI is to tech what outsourcing was to manufacturing, then your analogy has me concerned for the future.
According to the article, the issue was caused by:
> "engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause"
Interestingly, we found matching errors in our own logs:
> System.Net.WebException
> The remote name could not be resolved: 'dynamodb.us-east-1.amazonaws.com'
Occurrences were recorded on:
- 2025-04-30
- 2025-05-29
- 2025-06-17
- Yesterday
We had logged this as a low-priority bug since previous incidents only affected our AWS testing environments (and never our production env which is on Azure). At the time, we assumed it was some CI/CD glitch.
It now seems that the underlying cause was this DNS issue all along, and only yesterday did it start impacting systems outside of AWS.
It was Diwali vacation in India. It looks like the managers were not able to force everyone to walk around with their laptops and pagers hanging from their necks and waists, respectively, which they normally do.
If there's one thing I have learned from my Amazon mates, then that is they never have a true time off. Hills, beaches, a marriage in the family— no exceptions. It's so pervasive that I can't really imagine it to be voluntary, and my friends' answers on this topic have never been clear.
Still, by the time the failure start, that would have been the beginning of the day in Europe, so I suppose those teams got the page
Maybe it was still at the end of Indian day but together with the holiday I'd say that makes it more unlikely to be handled there
It was certainly suspicious that actual progress on the outage seemed to start right around U.S. west coast start of day. Updates before that were largely generic "we're monitoring and mitigating" with nothing of substance.
I thought the recovery was early AM Seattle time (like 4am). Where I think start-of-day is like 9am. Maybe recovery started early (6am) New York time?
[09:13 AM PDT] We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT.
[08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services...
[08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues...
[12:11 AM PDT] <declared outage>
They claim not to have known the root cause for ~8hr
Sure, that timeline looks bad when you leave out the 14 updates between 12:11am PDT and 8:04am PDT.
The initial cause appears to be a a bad DNS entry that they rolled back at 2:22am PDT. They started seeing recovery with services but as reports of EC2 failures kept rolling in they found a network issue with a load balancer that was causing the issue at 8:43am.
I don’t think that’s true, there was an initial Dynamo outage that was resolved in the wee hours that ultimately cascaded into the ec2 problem that lasted most of the day
Was the Dynamo outage separate? My take was the NLB issue was the root cause and Dynamo was a symptom which they flipped some internal switches to mitigate the impact to that dependency
If their internal NLB monitoring can delete the A record for dynamodb that seems like a weird dependency (like, i can imagine the nlb going missing entirely can cause it to clean up via some weird orchestration, but this didn't sound like that).
I was thinking more along the lines of the NLB being in front of DNS servers and dropping resolvers
Or an NLB could also be load balancing by managing DNS records--it's not really clear what a NLB means in this context
Or there was an overload condition because of the NLB malfunctioning that caused UDP traffic to get dropped
Obviously a lot of reading between the lines is required without a detailed RCA--hopefully they release more info
huh.. maybe publicly communicated recovery was then. I was seeing knock-on effects hours later and didn't see full recovery until late afternoon EST.
I noticed that too. I think tech culture has to change a bit. Silicon Valley is a great location if you're making hardware or prepackaged software. If you have to support a real economy that is mostly on the East Coast you need a presence there.
> one really gets the sense that it took them 75 minutes to go from "things are breaking" to "we've narrowed it down to a single service endpoint, but are still researching," which is something of a bitter pill to swallow
Is 75 minutes really considered that long of a time? I don't do my day-job in webdev, so maybe I'm just naive. But being able to diagnose the single service endpoint in 75 minutes seems pretty good to me. When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.
A few years back I was working at a software company that provided on-site sensor sensor networks to hospitals, pharmacies, etc. Our product required them to physically install a server on-site, but we were starting to get disrupted by cloud-based solutions. Essentially what we did was alert medical staff when blood, organs, etc. refrigeration temperatures went out of range. If the right people involved did not get notifications on time for these issues people will die. Its not hyperbole, you have to wait years for liver transplant. Their aren't just new livers available for everyone if a handful of them spoil.
With that being said, the problem here isn't that it took 75 minutes to find the root cause, but rather that the fix took hours to propagate through the us-east-1 data center network. Which is completely unacceptable for industries like healthcare where even small disruptions are a matter of life and death.
The web operates in a very different world if you've invested in good tooling. I used to be lead on a modestly sized payment processing back end to the tune of about 100 transactions/second (we were essentially Stripe for the client facing apps at the company). In many cases our monitoring and telemetry let us identify root cause in a matter of minutes. Not saying that is or should be the norm for all web apps, but what we had was not too far off from a read-only debugger view of the back end app's state throughout the request and it was very powerful. Of course for us more often than not the root cause was "the bank we depend on is having a problem" so our knowledge couldn't do much other than help the company shape customer communications about the incident.
> Is 75 minutes really considered that long of a time? [...] When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.
One might spend weeks diagnosing a problem if the problem only happens 0.01% of the time, correlated with nothing, goes away when retried, and nobody can reproduce it in a test environment.
But 0.01%-and-it-goes-away-when-retried does not make a high priority incident. High priority incidents tend to be repeatable problems that weren't there an hour ago.
Generally a well designed, properly resourced business critical system will be simple enough and well enough monitored that problems can be diagnosed in a good deal less than 75 minutes - even if rolling out a full fix takes longer.
Of course, I don't know how common well designed, properly resourced business critical systems are.
Depends on what you're measuring.
Quite a few of AWS's more mature customers (including my company) were aware within 15 minutes of the incident that Dynamo was failing and hypothesized that it'd taken other services. Hopefully AWS engineers were at least fast.
75 minutes to make a decision about how to message that outage is not particularly slow though, and my guess is that this is where most of the latency actually came from.
For a service like AWS, 75 mins is going to result in a LOT of COE's for people on way it wasn't mitigated quicker. A Sev 1 like this has an SLA of 20 mins to mitigate impact. Writing about these failures will consume a dozen peoples time for the next 6 weeks.
I have 10 years of experience at Amazon as an L6/L7 SDM, across 4 teams (Games, logistics, Alexa, Prime video). I have also been on a team that caused a sev 1 in the past.
> LOT
Just capitalised for emphasis, right?
> COE
Center of Excellence? Council of Europe? Still wondering even after Googling.
> SLA
Service Level Agreement. This I knew beforehand.
> SDM
Service Delivery Manager?
> COE
I guessed this was an internal Amazon thing so I searched “Amazon COE”
Correction of Error
https://wa.aws.amazon.com/wellarchitected/2020-07-02T19-33-2...
> SDM
Software Developer Manager (from searching Amazon SDM)
https://amazon.jobs/content/en/how-we-hire/sdm-interview-pre...
Thank you for providing clarity where I did not.
> COE
I tell the juniors it stands for Correction of Employment. Keeps them on their toes.
Sorry, I sometimes forget the vernacular is not universal. The other sibling comment did provide the correct definitions.
Also it's pretty likely it took less time than that to get an idea, but generally for public updates you want to be very reserved, otherwise users get the wrong impressions.
>Is 75 minutes really considered that long of a time?
From my experience in setting up and running support services, not really. It's actually pretty darn quick.
First, the issue is reported to level 1 support, which is bunch of juniors/drones on call, often offshore (depending on time of the day) who'll run through their scripts and having determined that it's not in there, escalate to level 2.
Level 2 would be more experienced developer/support tech, who's seen a thing or two and dealt with serious issues. It will take time to get them online as they're on call but not online at 3am EST, as they have to get their cup of joe, turn on the laptop etc. Would take them a bit to realize that the fecal matter made contact with the rotating blades and escalate to level 3.
Which involves setting up the bridge, waking up the decisions makers (in my case it was director and VP level), and finally waking up the guy who either a) wrote all this or b) is one of 5 or 6 people on the planet capable of understanding and troubleshooting the tangled mess.
I do realize that AWS support might be structured quite a bit differently, but still... 75 minutes is pretty good.
Edit: That is not to say that AWS doesn't have a problem with turnover. I'm well aware of their policies and tendency to get rid of people in 2/3 years, partially due to compensation structures where there's a significant bump in compensation - and vesting - once you reach that timeframe.
But in this particular case I don't think support should take much of a blame. The overall architecture on the other hand...
Sorry, are you saying you worked at Amazon and this is how they handle major outages? Just snooze and wait for a ticket to make its way up from end user support? No monitoring? No global time zone coverage?
Because if so, this seems like about the most damning thing I could learn from this incident.
No, it's just mindless speculation from someone who clearly hasn't worked a critical service's on call rotation before. Not at all what it's actually like, all these services have automatic alarms that will start blaring and firing pagers, and once scope of impact is determined to be large escalations start happening extremely quickly paging anyone even possibly able to diagnose the issue. There's also crisis rotations staffed with high level ICs and incident managers who will join ASAP and start directing the situation, you don't need to wait for some director or VP.
I worked at AWS (EC2 specifically), and the comment is accurate.
Engineers own their alarms, which they set up themselves during working hours. An engineer on call carries a "pager" for a given system they own as part of a small team. If your own alert rules get tripped, you will be automatically paged regardless of time of day. There are a variety of mechanisms to prioritize and delay issues until business hours, and suppress alarms based on various conditions - e.g. the health of your own dependencies.
End user tickets can not page engineers but fellow internal teams can. Generally escalation and paging additional help in the event that one can not handle the situation is encouraged and many tenured/senior engineers are very keen to help, even at weird hours.
“There are a variety of mechanisms to prioritize and delay issues until business hours”
What are business hours for a global provider of critical tech services?
Business hours for the team receiving the alarm; many issues can wait to be resolved during your own waking hours if they are not impacting customers.
"This is important enough for someone to work on as soon as their shift starts, but not important enough to page someone out of bed for."
AWS operates completely than what you're describing.
Alerts and monitoring will results in automatic pages to engineers. There is no human support before it gets escalated.
If an engineer hasn't taken a look within a few minutes, it escalates to their manager, and so on.
Wholly inaccurate. AWS Systems Engineers would have been paged by automated monitoring systems once alert thresholds were breached. No escalation through Support needed.
It's 75 minutes to _communicate_ the message to customers. Definitely internal teams were ahead of this before it was posted to the AWS Health Dashboard. Status Page posts are lagging indicators of incident progress.
I work in an incident management team where the turnaround from "we've decided to take x action, to y metric shows it is working, to z is posted on the status page" can be 1-2 minutes.
It is possible with professionals, institutional knowledge, drills, and good tools.
75 minutes is damn good turnaround for any major problem, IMO
75 minutes to diagnose what's failing is not.
If you a regular company, no. If you are the biggest provider in the world and your change is breaking large parts of the Internet, yes.
Amazon is supposed to have the best infrastructure in the business because everyone else runs on it. They should have access to the sre talent that can quickly mitigate this kind of issue
I dunno man, what part of the AWS experience leaves you thinking the software is amazing?
It's good enough, but there's no real evidence it's the best, simply the largest.
There is a distinction between usability and reliability lol. If AWS reliability trends down then it's an industry wide problem.
What if the SRE talent has a lot of real-life experience but can't pass LeetCode puzzles that have nothing to do with the job?
The sre talent moved elsewhere once the rto bs started
One thing I love about El Reg is that they never shirk from calling a spade a spade.
I also love the humor and personality that authors are able to inject into their pieces
I miss Lewis Page. It's a rare day when I'm so downhearted that the memory of https://www.theregister.com/2008/12/30/german_beaver/ can't raise at least a wan smile.
I nearly missed that this article was written by Corey Quinn, guy who writes a lot about AWS
They never shirk from calling anything a spade.
I’ve seen this happen with startups as well -
They’ll get acquired and top people leave as their stock vests or get pushed out because the megacorp wants someone different in the seat.
The people who knew the tech are gone and you’re left with an unmaintainable mess that becomes unreliable and no one knows how to fix it.
I once saw a layoff that was followed by a week long outage because no one remaining knew how to deploy to prod, and no one knew how to recover after the failed deployment. I felt bad for the people remaining who had to go through that,but it was hilarious.
Its almost like institutional knowledge is a real thing that you cannot put on some BIG BRAIN MBA spreadsheet.
how many lines of code is it though? or how many tokens? we need something we can measure if we're gonna fire people!
AWS is still my overall favorite cloud provider, and I use it very effectively.
I would've even liked to work at AWS myself, if it were clear that they're solving a few concerns:
1. Rumors of rough corporate culture, and you needing your manager to shield you from it. (If it can't be immediately solved for all of Amazon or white-collar, maybe start with increasing job-seeker confidence for AWS or per-team.)
2. Even very experienced engineer candidates must go through some silly corporate coding screen, and an interview to make sure they've memorized some ritual STAR answers about Leadership Principles. If your prospective manager can't even get you out of that, what worse corporate things can't they shield you from?
3. RTO. As well as all the claims it wasn't done consistent with the Leadership Principles, and claims that it's not about working effectively.
4. Difficult-sounding on-call rotation, for people who aren't shift workers. (Even if you Principal out of on-call, you don't want your teammates to be overextended, nor to have awkwardness because you're getting a more consistent sleep schedule that is denied them.)
Also, not a concern, but an idea that applies to all the FAANGs lately: What about actively renewing the impression in the field that this is a place where people who are really good go? Meta's historical approach seems to be to pay better, and to release prominent open source code, and be involved in open hardware. Google (besides having a reputation for technical/competence excellence and warmer values) historically had a big frat-pledging mystique going on, though it turned into a ritual transaction, and everyone optimized for that ritual. AWS has a lot of technical/competence excellence to be proud of, and could make sure that they're investing in various facets of that, including attracting and retaining the best workers, and helping them be most effective, and then making sure the field knows that.
Whether it involves FAANG companies or not, a job is ultimately just a job. While it's nice to have such a company on a CV and to gain the experience, it is, in essence, similar to any other employment.
Eventually, you begin to consider the drawbacks, such as the monotony of the work or the exhausting nature of on-calls (which disrupt personal life). Then, an opportunity arises from a former colleague at another company, and the outcome is predictable.
Companies present numerous such inconveniences and actively introduce additional ones. Now, we are faced with mandatory RTOs, along with the continuous tightening of the screws and "cutting fat from the bone" (actual words of my company's CTO). Consequently, employees will depart, and it is often the high-performers who will seek opportunities elsewhere, as they are not afraid of the job market.
I think google is an excellent place to work, but the top AI companies do poach some of our good employees. If you're the cream of the crop, sure, take the Anthropic offer over G. But I do appreciate the massive amount of freedom to work on hard problems.
Google can be an excellent place to work, but most of the people working there are not contributing to the company’s profits. Whether or not you think that still has value, leadership is trying to “optimize” by squeezing everyone harder and demanding they show productivity gains using AI. Some teams will adapt and keep being a nice team to work on, but many employees are getting “optimized” out of the company and replaced with vendorization and consultants.
This article seems sensationalized and lacking evidence. Layoffs alone (especially when so much of the industry did them) doesn't seem sufficient to explain today's outage, especially when we know so little of the technical details behind it. It's disappointing that The Register didn't wait until we had a postmortem from AWS before jumping to conclusions.
I think you missed the point of the article. The layoffs are blamed for the poor response, not the outage itself.
I am reading the book Children of Time where descendents of mankind try to keep their tech running long after the ship started.
We are now coming into an age in which standing application and infrastructure systems have to run long past their original creators are on the ship.
In my opinion, as a industry we are not mature enough for that and we need to become better.
""It's always DNS" is a long-standing sysadmin saw, and with good reason: a disproportionate number of outages are at their heart DNS issues. And so today, as AWS is still repairing its downed cloud as this article goes to press, it becomes clear that the culprit is once again DNS. "
I use stored DNS data.^1 The data is collected periodically and stored permanently
I seem to be unaffected by DNS-based outages
I use stored data because it is faster, e.g., faster than using DNS caches like Google DNS, Cloudflare DNS, etc., but there are obviously other other benefits
1. When I make HTTP requests there is no corresponding remote DNS query. The IP address is stored in the memory of the localhost-bound forward proxy
For internal addressing, why do we use DNS? In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.
> For internal addressing, why do we use DNS?
Why not? It's standard protocol for name-address mapping.
> In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.
Client just performs DNS query before HTTP query or client caches DNS response for some time. It's solved problem and "pull" approach usually preferable to "push" approach. For example, what if that broadcast gets lost (UDP is not reliable protocol)? How application handles initial discovery?
Of course it is possible to replace DNS with something else. But why? If you're afraid that your DNS server will fail, the same could happen with your custom service discovery server.
That's a lot of broadcast traffic at scale.
How would you handle the ack of the broadcast? UDP can be unreliable. If you do send an ack from the destination, that singular endpoint that sent the message is going to get a number of responses equal to the number of devices on the network, which would nuke it at large scale.
For super critical entries, we put it in /etc/hosts and also at the domain provider as fixed entries.
The combination never fails.
What - their AI couldn't find it sooner? Better get those RAGs in order.
It’s no secret that AWS has been seeing a mass talent exodus. Probably 90% of the folks I know that were the best and brightest in what they do and worked at AWS are no longer there. It’s beyond a blip, but a full blown exodus, with the talent bar severely lowered.
The writing was on the wall for a bit now that something like yesterday would happen.
> When that tribal knowledge departs, you're left having to reinvent an awful lot of in-house expertise that didn't want to participate in your RTO games, or play Layoff Roulette yet again this cycle.
…
> This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. Remember, there was a time when Amazon's "Frugality" leadership principle meant doing more with less, not doing everything with basically nothing. AWS's operational strength was built on redundant, experienced people, and when you cut to the bone, basic things start breaking.
Not just Amazon. I woke up this morning, to find my iCloud inbox stuffed with unread spam; much of it over a month old. Looks like someone restored some old backup. This was likely to correct some issues that were caused by the AWS outage; either directly, or indirectly.
It’s nice to know that Apple (or some other middleman) backs up the torrents of spam that I get.
Everything is now at Jurassic-scale. It’s all monstrously big. There’s no such thing as a “small problem,” anymore.
One thing that you get with experience, is “tribal knowledge,” and that stuff is usually impossible to properly document. I suspect that AI may, in the future, be able to incorporate some of this, but it’s by no means certain.
Nothing gets sold or fixed without people who know how it's built.
cue in: programming as theory building [1] or building systems as theory building, ie, Mental causal models of how and why things work the way they do. Mental models live in people's heads and walk out of the door when they do. Management learns this the hard way [2].
[1] https://pages.cs.wisc.edu/~remzi/Naur.pdf
[2] https://x.com/elonmusk/status/1980221072512635117
> Management learns this the hard way [2]
I don't get this. Why is Musk tweeting a fake quote and why are you posting it? What does it signify?
I assume it was not labeled as fake when the OP sent it.
But it is more important to keep the PIP, stack-ranking, ladder-climbing game running than keeping the people /s
When organizations begin to prioritize personal-brand builders and performative hires over the core technologists and long-tenured institutional experts who actually understand how things work, the culture inevitably shifts.
When that imbalance grows, as it has at AWS (ex-AWS here), and the volume of relentless self-promoting “LinkedIn personalities” and box-ticking DEI appointments starts to outnumber the true builders and stewards of institutional memory, the execution quality, accountability, and technical excellence begin to erode.
It is becoming increasingly clear that Andy Jassy’s leadership is no longer effective, and it is only a matter of time before Wall Street begins calling for his departure.
I appreciate how you, with absolutely no evidence, blamed this outage on DEI, especially given Amazons complete backpedaling of DEI the first moment it became political convenient
This AWS outage has reminded and bolstered my confidence of the idea that there really are practical limits on how we can manage complexity.
As a codebase ages, as services grow out in scale and scope, complexity increases. Developers know this. I don't believe that you can linearly scale your support to accommodate the resulting unknown unknowns that arise. I'm not even sure you can exponentially scale your support for it. There is going to be a minimum expected resolution time set by your complexity that you cannot go under.
I think times where there have been outages like this that have been resolved quickly are the exceptional cases, this is the norm we should expect.
Doesn’t an increasing LLM centric code base only compound this problem? Under the assumption that people are lazily screening LLM’s output when using them
Absolutely.
I think it's important Amazon remains stable and a quicker resolution would have been great.
That being said, if many important services (the article mentions banking) are still single-point-of-failure in us-east-1, the least stable but cheapest region, there seems to be a problem far greater than Amazon here.
We all watched this happen across FAANG, right? In the early/mid 2010s working at Amazon meant you were cream of the crop.
By 2020, no engineer in their right mind wanted to work there because it was an infamously bad employer for people who wanted to create great tech in a nerdy-fun environment.
The AI space is showing how the "darling fun tech company" to "aggressive tech employer full of psychopaths" trope can take less than a few years now!
"If you were a ‘product person’ at IBM or Xerox: so you make a better copier or better computer. So what? When you have a monopoly market-share, the company’s not any more successful. So the people who make the company more successful are the sales and marketing people, and they end up running the companies. And the ‘product people’ get run out of the decision-making forums.
The companies forget how to make great products. The product sensibility and product genius that brought them to this monopolistic position gets rotted out by people running these companies who have no conception of a good product vs. a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts about wanting to help the costumers.”
- Steve Jobs - https://en.wikipedia.org/wiki/Steve_Jobs:_The_Lost_Interview
Is this actually a verbatim quote? The typo at the end makes that seem unlikely.
Great words, but he lost any right to them when he made famous the "You're holding it wrong" workaround. IMO that was the defining moment when Apple started its decline on product innovation.
It was a nothingburger. Apple sold the same GSM iPhone 4 for three years with no design changes and nothing else was said about it three months later.
It was not nothing. The phone stopped working if the user held it naturally in their hand. I had one. Reception completely cut out.
If it were that big of deal, don’t you think Apple would have been forced to recall it and definitely couldn’t keep selling it for 3 years. True they did redo the antenna for the Verizon CDMA iPhone 4. But they never bothered to back port the changes to the GSM one.
I also had a GSM iPhone 4.
Compare that to how quickly they ran away from the shitty Intel modems when they were selling some made by Intel and some made by Samsung (?)
Apple offered a free case to make the problem go away. My iPhone just had trouble typing Apple and free in the same sentence.
Well, there was a software change to smooth out how the bars would display.. https://9to5mac.com/2025/10/08/a-15-year-mystery-solved-the-...
I’d add a corollary to Steve’s often quoted idea, that became even more relevant after Covid. Everyone competent that makes tons of money retired early. We are left with the dregs at all these companies, the newbies and those that didn’t budget well and plan for early retirement.
Any interaction you have with a company post-Covid you can feel it. Nothing works anymore and you can’t even tell anyone about it or why.
Maybe, in retrospect, all those brilliant folks who made all that money would've deserved it more if they'd built cultures that could survive 5 years.
At best, a company's culture is just a branding and marketing strategy to attract employees and to appear cool. They are not real cultures -- they are constructed, forced and dictated by management, and oftentimes cult-like in their construction.
Company cultures are not built to last, they are designed to generate profit. The culture is incidental, it will be whatever is most profitable at any given moment. At best, a company's culture is just a branding and marketing strategy to attract employees and to appear cool. Therefore they are fickle and prone to complete collapse when just a few people are replaced.
It was going to happen Covid or not. People get old...what are you going to do? Is it really that bad if a bunch decide, "you know what? I'll take the financial hit and retire a few years early because I have been reminded of the fragility of life".
Oh I don’t blame them, I was a scientist and retired early myself. So has my brother that was a doctor. But what world is left behind when all the competent and passionate people exit the working world early? We see that world every day on the news now and it isn’t pretty.
FWIW this has happens in consulting too, not just product companies. Just swap “product” for “delivery”.
The PIP culture of AWS sounds horrifying. As a decent engineer, I would not work there unless that is addressed.
I heard its as bad as this. Take a team of 5 genius engineers, the best 5 in the world.
There is a PIP quota, so one of the genius engineers must be PIP'ed, despite being in the top 5 engineers globally.
Years ago I interviewed for Amazon. I found out that on a desired team size of 8, there were only 3 left, where the other 5 who were no longer there, as I was told, were poor performers. I didn't take the job.
I guess at AWS, "poor performer" could potentially mean one of the best engineers in the world.
10 years ago potentially. Today the best engineers won’t take a job there so the bar has fallen quite far down market.
Not to defend PIP, but it is applied at a scale greater than a team, and that below a certain size an org is immune to stack ranking requirements.
Every good Amazon org has sacrificial bad employees. Managers know a team of all-stars creates risky team dynamics under PIP
This sounds toxic on it's own right. What a horrible managament practice
What's PIP?
Performance Improvement Plan, a formal process for employees whose performance is not meeting job expectations.
Thanks! I think we have a different name for it.
Here in Europe they need a very serious reason anyway (like gross misconduct) and if they don't have it they can make you redundant but have to pay significant severance. I'm kinda waiting for an opportunity for that at my current employer, due to leadership changes it's no longer a great place to work. When there's layoffs there's usually voluntary options with a decent severance plan.
I'm glad we don't have this at-will stuff here. I would never considering moving somewhere that does (I've lived in 4 countries now). Universal healthcare is also a must, and good public transport.
So America will never be an option for me in any case.
>Here in Europe they need a very serious reason anyway (like gross misconduct)
Definitely not true in all of Europe. There's some countries where severance pay is not mandatory for all workers' groups and reasons aren't needed to fire someone (the green countries)[1] and also some have loopholes where PIP is legal and often used, so in practice YMMV depending on the laws where you live and the unions in your job/industry.
>(like gross misconduct)
Again, not always true. In many countries gross misconduct can be grounds for on the spot termination. For example in NL there was a truck driver that modified the safety system on his truck so he can unload potentially dangerous cargo from the comfort of his cabin without having to keep the outside lift button pressed. When caught he was fired on the spot with no severance and no unemployment, and even lost the case after he sued for unjust termination.
[1] https://upload.wikimedia.org/wikipedia/commons/4/45/At_will_...
Which is generally used as a means to collect evidence that they're unable to do the job before firing them (and in many cases the decision to fire them has already been made).
Dunno about brain drain but some departments seem to be having a mandate of "must have AI" when procuring products. Pump and keep pumping!
IMO the "must have AI" mandate exposes the culture of:
- Get things done as fast as possible, regardless of even short-term consideration;
- Implicitly but loudly discourage people to figure out the issues or write code all by themselves, because it is much slower;
Mere hours into the incident, before there's any RCA, someone rushes to discredit themselves with a simplistic explanation
Brain drain implies they went somewhere else where's better. Where did they go?
One of the best-written articles I've read in a long time. I wish general news coverage had this tight blend of fact, context, and long-term perspective.
internal reports from current AWS engineers seem to be confirming all of the speculation in this article. Shit's rotten from the inside out and you can pretty evenly blame AI, brain drain, and good old fashioned "big company politics"
https://forums.theregister.com/forum/all/2025/10/20/aws_outa...
There's been a massive talent exodus, especially among the principal and senior principal engineering roles, across all Amazon orgs since the RTO policies have been enforced. Its demoralizing to lose key engineers that you look up to and want to continue to learn from all because a few people far removed from the day to day make a bad call.
RTO in combination with Amazon being last place in AI innovation have led to departures of anyone that can leave, leaving.
> So they can't suggest a fix even if they know 100% what it will be. Thats exactly what happened this time. EIGHT different staff members pointed to the underlying cause and were told (some literally) to "shut the f*ck up and get back to your job"
Jesus, if even an ounce of that is true... Yes, everyone on the internet is a cat clawing on a keyboard... but if a ton of people legitimately confirmed to be ex-AWS point to similar culture issues... probably it's AWS that's rotting.
The problem is that if it was 8 in 10 suggestions or even 8 in 20, then yes, terrible. On the other hand, if it was 8 needles in a haystack of garbage, at some point you do, in fact, need to tell people to STFU so you can work.
I think they might have deeper issues still with their outage. I just got an email and retroactive charge for something I returned months ago and shows as returned on their own orders portal. The link in their transactional email also links to a totally different product.
That could be totally unrelated - I had the same thing happen last month. Eventually (~3 attempts over a week) managed to fight through the various chat options that tried to dead-end into "wait 3-5 business days and check again", and once I had a person on they took about 5 minutes reading the background and said "oh, I can see that this return was processed months ago, and we have it listed as received. I will return the amount you were charged immediately!"
Ehh I trust the reporting and generally agree that RTO was/is executed hamfisted but I dunno if this particular incident "makes" the narrative. IIRC LSE rate has been increasing for many years, maybe most of AWS's existence. This is part and parcel of building something so complex that continues to grow and evolve.
I do expect much better of them and they certainly have problems to solve but this is a big company evolution thing and not an Amazon-specific thing imo.
The real story from this incident is that Amazon’s “aws” partition doesn’t actually have multiple regions - effectively, it’s all IAD in a trench-coat.
This is a big deal. My employer has already started to look at bringing back our old racks from storage and switching back to on-premises. Cannot imagine he’s alone in that.
Can you elaborate what convinced you of this? We were running mostly in us-west and saw almost no impact, despite using a broad spectrum of AWS infrastructure and tooling.
We had business as usual today.
I’m pretty sure at this point I know more about AWS and AWS internals than my account solution architect and I’ve never worked for AWS.
Yeah. They will identify the cause, but not the cause behind the cause.
With cbell gone and ajassy promoted and the misery inducing litany of morale self inflicted wounds it’s not surprising reliability is regressing. There’s no head of engineering like Charlie and Garman is a strong engineering leader, but coming after a sales guy and taking over a battered workforce, it’s not clear to me he can turn things around easily. Everyone I know worth a hill of salt left aws already - and the ones left - meh. That’s how attrition through misery works.
Dont write cbells hagiography yet. I wouldnt put the gift of fear, “no lone zone”, and rolling stone up there as top tier cultural engineering successes. They all have reasonable reasons, bits of truth, and i liked cbell overall… but the tech culture was not so consistently great.
For those who worked there recently, how much does the comment at [1] reflect the current state of things?
[1] https://forums.theregister.com/forum/all/2025/10/20/aws_outa...
> This is The Register, a respected journalistic outlet.
Yes, but they bristle at the thought. :)
We don't care. We don't have to. We are the cloud company.
I have to quote one of the comments:
> "Hopefully today will serve as a massive wake-up call for AWS"
I wouldn't hold your breath. There will be incident reviews, meetings, assessments, analysis etc. but basically boil down to what can we do to stop this from happening again without actually spending any more money. So no, not hiring fresh talent or retaining that talent already in play, no to radical overhaul of process and knowledge. No to remediation of known issues if it involves expenditure. Instead it will be do more with less. Beat the employees harder, enforce more and more diligence and output from less and less people for the same or less money. Spin it like mad with catchy titles like knowledge sharing, centers of excellence, efficiency improvement initiatives, agile resilience, and continuous operational excellence.
There’ll be shiny PowerPoint decks about empowering ownership and shifting left, while the remaining engineers are shifting caffeine straight into their bloodstream at 3 a.m.
Next quarter, they’ll unveil a bold new policy called Focus Fridays which will be promptly filled with mandatory incident retrospectives. Someone will suggest replacing ancient tooling, only to be told, “We’ll revisit that next fiscal year,” which is code for never.
Then come the internal awards: “Unsung Hero of the Outage” goes to the one poor sod who rebooted the wrong thing but accidentally fixed it.
HR will roll out a “Resilience Recognition” badge on the intranet. This will be marketed with great fanfare and excitement, showcasing how the company truly values it's employees and recognized their contribution because badges are cheap. Leadership will congratulate themselves for “learning from adversity,” and by the time the next blackout happens, they’ll have a snazzy new dashboard to watch it fail in real time along side their investment portfolio dashboard that takes up a greater fraction of their attention.
But don’t worry!!!! There’ll be a T-shirt. “I survived the 2025 AWS outage.” Comes in gray. Just like morale. If it wasn't for the negative impacts on the employees and customers the word Schadenfreude would be very applicable.
And it's a sad indictment on current management practices and in particular the MBA brigade* that this is all by design, acceptable losses on the alter of profit, albeit short-term profit. Efficiency theatre as far as the eye can see.
*Yes, the same people who think Jack Welch was a misunderstood visionary rather than the spiritual father of mass layoffs, short-termism, and shareholder-value human sacrifices. The kind who see burnout as a KPI and chaos as a “scaling opportunity.”
Next they’ll launch a “Transformation Task Force” whose primary transformation will be renaming the same broken process from post-mortem to value realization review. A new acronym, a new logo, and boom, problem solved at a low low cost, honest, the consultants said so. Until the next outage, at which point someone will quote Sun Tzu in Slack.
Terrible article. Im ex-AWS, left as a principal after 10 years to go take another global megacorps shilling. I dont even disagree with the premise, but its so clearly a predetermined conclusion written as opinion piece to fit the hot news topic.
Ex a sloppy as hell and inconsistent premise.
> engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause
its the point that wasnt the root cause. The root cause was ipso facto much more complex, insidious, and unobservably spooky action at a distance. I say that not knowing the true cause but being very willing to bet a bunch of AMZN that it wasnt as simple as “herp derp dns is hard and college hire sdes dont understand octets and delegation.”
Or this stupid citation if were talking about senior/long term AWS tech roles:
> Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels.
The citation _appears_ to be about consumer/retail delivery and ops folks. And how 69-80% _of total attrition is RA_. While el reg has written it trying to imply 80% _annual attrition_ in a completely different org and business unit.
So I know corey isnt stupid, and hot takes are his paycheck. But does he think his readers are stupid?
As ex-amazonian, you should know that our average tenure for an SDE is just barely over 8 months. The old farts tool should have told you that during your day
And whats the median? Talking about hot take tenure averages without expressing the components is just silly. I, personally, was well aware of actual hiring and attrit rates up and until fairly recently.
Again, Im not saying things are good (I left for reasons!). But use better data and arguments.
You'd have to have some incredible rapid attrition to have the median be above the average. Any tech company that left skewed would be considered a disaster by pretty much anyone.
> I, personally, was well aware of actual hiring and attrit rates up and until fairly recently.
Could you provide the data in this thread so that it adds to the discussion?
There's an argument to be made that each event generates new institutional knowledge for those that are there.
Bit of a double edged sword.
Normally, you'd have drills and exercises to try to find and fix potential issues before they affect production. Saying that you can learn from failure is great comfort to the individual, but not an excuse for the institution.
This fails to recognize that the people who designed everything to rely on us east 1 did so a long time ago. "Brain drain" could just mean that they've had their fun and now want other people to deal with their mess.
>I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time.
That information is under NDA, so it's only natural you aren't privy to it.
More discussion: https://news.ycombinator.com/item?id=45640838
This is how articles should be written, this is why I’m reading El Reg (a.k.a. The Register) all these decades, this is what happens when high management cares only about profits and when real engineers don’t eat the RTO bullshit. Bravo for putting this online.
P.S. I’m not an Amazon hater, replace the company name with any other big one of your choice and the article will have the same meaning ;-)
The Register is an opinionated tech tabloid filled with outrage bait. This article is not an exception, drawing far reaching conclusions from little evidence.
“Little evidence?” If the “aws” partition doesn’t actually exist when IAD breaks, Amazon hasn’t even discovered how to make multi-region cloud infrastructure. That’s a big deal.
It's not your feeling about Amazon that would cast doubt on your take, it's that you've reduced it to one pet cause and decided a source is well written because it appeals to your dislike of RTO. Nowhere has any evidence of the relevance of that to this been presented, Amazon has had outages since before WFH even began, they've all always had their occasional outages and bad days.
There’s clear data that shows an increase in the frequency and severity of LSEs post RTO3 in 2023, and it looks like RTO5 has accelerated that trend.
big "who is John Galt" vibes in these comments lol
aws is s globally centralized point of failure, it should not be allowed to exist
Out of touch. Is Amazon going through some turmoil? Why are people leaving?
I mean in software. I Know warehouses are pretty bad.
glad a company who did RTO got fucked
hope it only gets worse for them
Amazon has reportedly been a shitty place to work forever, so using issues that happen to be popular today to explain turnover is disingenuous.
From the article.
> At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia
Your comment is relying on that referenced inertia. Things will continue to function for a period of time, but there exists an inflection point at which they no longer function as previously.
Arguably we had a perfect example with Twitter.
In what way? It seems reliable enough. I admit I don't check it constantly, but I've not noticed outages impacting my perhaps middling usage.
The year or so after Musk took over was brutal. The influx of far-right pests, troll farms and porn bots was one thing, but the reliability went down the drain.
Exactly / but it did not DIE as many were predicting (even on HN many were claiming it would go down immediately).
Inertia is a hell of a force.
It kind of died, twitter as it was even though X is still running and somewhat recovered in terms of infra.
Musk effectively left his newest plaything a few months after the takeover and some events like him going in in a datacenter and disconnecting servers, that was when Twitter (I'll call it X when he acknowledges his daughter) started to stabilize again.
All that he seems to be doing these days at Twitter is messing around with the recommendation algorithm, override the decisions of what's left of moderation for his far-right friends and that's it. Oh and of course Grok/xAI or however it's called these days, but IIRC that's a separate corporate entity that just got shoehorned onto Twitter.
>Amazon remained the single largest H-1B sponsor, increasing approvals from 9,257 in 2024 to 10,044 in 2025, an addition of 787 visas.
https://www.reddit.com/r/SeattleWA/comments/1ncm25p/amazon_m...
I’m confused how they can have such a failure, they are employing the best and brightest top tier talent from India.
Hopefully they can increase their H1B allotment even more next year to help prevent these types of failures.
Given today is Diwali, perhaps the reason everything went down is because the best and brightest from India were all on vacation and weren't there to babysit/roll back the deployment that broke everything?
ouch, that’s what a Diwali outage looks like..
It depends on how the resources are assigned, what projects they’re asked to focus on, and which strategic tech debt initiatives aren’t approved.
We’ve reached a point where I can no longer distinguish between people without experience and people repeating the talking points they’re told to repeat. That’s a major loss.
However, talent is a very small part of shipping a project. How that talent is resourced is far more important.
AWS has been having issues like this for years.
https://geohot.github.io/blog/jekyll/update/2025/09/13/get-o...
I am waiting for the author to lead the way.
Speaking of DNS, I still cannot comprehend why we still rely on the current complex, aging, centralized, rent-seeking DNS.
It's one one of the few parts of the internet which could potentially be replaced over time with very little disruption.
The hierarchy of resolvers could be replaced with a far simpler flat hierarchy Blockchain where people could buy and permanently own their domains directly on-chain... No recurring fees. People could host websites on the Blockchain from beyond the grave... This is kind of a dream of mine. Not possible to achieve in our current system.
I think this would make the squatting problem that we already have way worse. There would be bots buying every single remotely usable domain, and there would be no incentive for them to sell it unless they get an absurdly large offer.
I bought tombert.com in 2014 and forgot to renew it in 2015, and it was auctioned off by GoDaddy. For like six years, it was owned by squatters, and they wanted thousands of dollars for the domain [1]. I called offering the $100 for it, and they claimed that they can't go below $1400 because this domain is in "extremely high demand". I finally was able to buy it back in 2021, presumably because the squatter purged out domains that hadn't been purchased for N years and they wanted to save money.
Now, you could argue "see! You wouldn't have had to worry about it expiring if it were permanent on the blockchain", and that's true, but if someone else had gotten to that domain first, then I would also never get it. I think the only thing that keeps the internet even remotely fair in this regard is that domain names cost some amount of money to keep.
[1] https://web.archive.org/web/20160219161720/http://www.hugedo...
Yes, having some cost disincentivies some abuses that completely free attracts. If email cost a few cents to send there would probably be a lot less spam around
I'm pretty sure that is why Something Awful was successful. Since an account cost $10, and any abuse could lead to a ban, you very quickly filter out spam and people who are solely there to shitpost.
> People could host websites on the Blockchain from beyond the grave...
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing (assuming they think about it at all)
Also the potential for a domain to get irreversibly stolen is not a good feature for the security of the users.
Let’s solve rent-seeking by building a new system that’s even more rent-seeking!
At least currently death dissolves bonds.
> People could host websites on the Blockchain from beyond the grave...
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing.
DNS is:
- simple
- battle hardened
- distributed
- affordable
blockchains are:
- essoteric, backwards, and not easily implemented
- new and unproven, frequently hacked
- effectively a ploy to centralize / redo Web 1.0 but owned by one blockchain
- ...waaaaaaay more about money and "owning something" than DNS is.
To bake an apple pie from scratch, you must first overthrow capitalism.
perfection
I'm not really interested in communism but I will support it if it lets me have a Blockchain-based DNS system.
All the arguments I'm hearing against a Blockchain DNS system are rooted in petty crony-capitalist thinking.
This kind of thinking seems to permeate most other parts of society... It's gotta stop.
"Oh but what if someone steals it"
This ain't gonna be much of a problem in a functioning society where the top 20 domain names doesn't hoard like 95% of the traffic.
"Oh but we don't want people to own domains permanently or else they will take all the good domains"
Um hello?? Have to checked this thing called reality? It's already the case! So happy billionaires have to pay their $20 per months to maintain their market monopolies.
I actually don't mind other people having more stuff than me but tired of petty people ruining good ideas and stalling progress to make a few bucks.
Garbage reporting: 1. AWS had an outage 2. AWS has lost a lot of employees
Conclusion: The brain drain lead to the outage...
I need an LLM trained explicitly on folks confusing correlation and causation and put a big old red dot in my address bar.
I love that there's a whole section "The talent drain evidence" trying to defend their journalistic integrity, but they then go on to totally face plant.
> It is a fact that there have been 27,000+ Amazonians impacted by layoffs between 2022 and 2024, continuing into 2025. It's hard to know how many of these were AWS versus other parts of its Amazon parent, because the company is notoriously tight-lipped about staffing issues. Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't." The internet is full of anecdata of senior Amazonians lamenting the hamfisted approach of their Return to Office initiative; experts have weighed in citing similar concerns.
So the title is all speculation. The author put 2 and 2 together and concluded that 10 is greater than 9.
Worthless article.
Maybe its speculation, but I mean drawing conclusions becomes easy when 40% of devops staff being laid off by AWS was in the news three days ago.
https://blog.stackademic.com/aws-just-fired-40-of-its-devops...
AWS doesn’t even have a “devops team” nor even any devops job roles. AWS also does not use Terraform (which is what the article says everyone was replaced with) at any significant scale, so this article is similar junk.
Uh, they still have roles open for DevOps:
https://amazon.jobs/en/jobs/3080348/devops-engineer-linux-re...
https://amazon.jobs/en/jobs/3082914/devops-systems-engineer-...
This one mentions terraform by name (though that doesn't necessarily imply its in use, though having worked in large companies I would argue that sweeping statements about a popular technology not being used is likely to be wrong)
https://amazon.jobs/en/jobs/3042892/delivery-consultant-devo...
AWS does not have dedicated devops roles. All AWS SWEs are expected to take oncall shifts and respond to incidents, manage build pipelines, etc rather than having specific devops people to do it for them. The article you linked claiming 40% of them were fired is total junk. You can believe that or not, I don’t care.
The last one is a ProServe role, which is a consulting role that spends their time working in customer environments, which is where they may encounter terraform. It does not mean anything about internal use of terraform.
Again, I’d be wary making sweeping generalisations like that.
I already showed you that AWS has (or hires) DevOps people with publicly available information, maybe the article is incorrect but you’re clearly not better informed, so maybe cut it with the rude commentary.
Misunderstanding the things you are linking does not mean you proved anyone wrong.
I’m going to link this again, what role is this please:
https://amazon.jobs/en/jobs/3080348/devops-engineer-linux-re...
Within AWS this role falls under the Systems Engineer job family. It is not a devops role, and its involvement in events like today would be the same involvement as every other SWE at Amazon.
Just do a quick google search for that “40% of devops laid off” and you’ll see that it’s actually an old article from months ago that multiple people, including AWS employees, are saying is bullshit and unsourced.
edit: found another source that says this 40% number came from an AWS consultant that worked with customers to help them be better at DevOps, and it was 40% of their specific team that was laid off. Even if it were true, it has nothing to do with the internal operations of AWS services. This is why it’s important to understand the information you’re sharing before making judgements off of it.
All I am finding are corroborating articles, maybe you can help me here.
https://www.theregister.com/2025/07/18/aws_sheds_jobs/
Seems wild that you would promote job titles you don’t hire for, makes me think that it’s reasonable for news outlets to refer to those roles in the same way honestly.
There is nothing in that article that mentions either devops or 40% of any team or role being cut. It doesn’t corroborate anything.
It makes a direct claim of hundreds of cloud staff being laid off.
You know what though? I’m not wasting my time with you, the fact was that this was all over social media. Then a huge outage- my original comment was factually accurate even if we contend that the article itself was bunk. And AWS clearly hires DevOps staff.
You’ve not even disproved anything you’re just making me play internet fetch. I’m not replying anymore.
> It makes a direct claim of hundreds of cloud staff being laid off.
I don't have any dog in this fight, but I don't see where this article makes your case. From your article:
> We understand around 100 jobs are at stake.
> Sources familiar with AWS operations who requested anonymity told The Register most of the layoffs affected people in marketing and outreach roles, although chatter on sites like Blind suggests folks in frontline support and in other positions may have been affected, too.
You seem to be kind of annoyed that somebody on the internet hasn’t taken your assertion that you just sort of generally Know Better as strongly as you’d like. You could probably put this entire discussion to bed by clarifying your current position at AWS and how your job there gives you direct knowledge of their devops practices.
I work for Amazon (AWS for 4 years then “the website” side of the house for the last 3)
The previous commenter is correct, there is no NOC or devops team and I’ve not encountered a Devops job family and I’ve never seen terraform internally. Within AWS, the service teams that work these outages are the same ones that design the service, fix bugs, deploy the pipelines, be oncall, etc. the roles that fill these teams are pretty much one of three types: nde, sde, sysde. They typically use cdk if they’re doing AWS things, else they’ll use internal tooling.
The job you posted is a customer facing consultant like role - customers use terraform so having a customer facing consultant type that knows how customer-y things work is a good decision.
You could both be right if they are trying to expand terraform use from a beachhead to the entire company. You need to hire people with prior experience for such things.
Source: Former AWS Professional Services employee.
Notice the job description:
As part of the AWS Managed Operations team, you will play a pivotal role in building and leading operations and development teams dedicated to delivering high-availability AWS services, including EC2, S3, Dynamo, Lambda, and Bedrock, exclusively for EU customers.
They aren’t looking for DevOpe engineers to work alongside the “service teams” - the teams that build and support internal AWS services. They are working with AWS customers who may already be using Terraform. AWS has a large internal consulting division staffed with full time employees. When they work with customers they will use Terraform if needed.
Just because its speculation doesnt mean its worthless. But yeah it should be taken as speculation rather than a validated and tested hypithesis
Anything that wastes my time and only reveals its half-assed reasoning half way through the article is indeed worthless.
First time reading El Reg?
i mean, you can assume if its on theregister its not going to have some kind of academic rigor or whatever it might be you're looking for. its the register, same basic rigor quality as the ny post.
that said, my suspicion is they're likely on to something here regarding layoffs and quality degradation.
You could make smart inferences based on past and very frequent occurrences.
Or you could just say "there is no way the thing that constantly happens over and over again has happened once again, just no way".
Staff cuts constantly happen in the name of maximising profits. They always yield poor results for a company's performance. Every time. Especially for the consumer's side of it (not the company's finances of course).
Every time.
But maybe this time it's different. That one time.
I worked at AWS and still have friends who work there. I don’t know any L5s who wouldn’t jump at a chance to leave if they even got a slightly worse offer than what they are making now. I know a few L6s and L7s that would stick around out of momentum.
But I know very few people in the industry who know about Amazon’s reputation that have a life long dream of working there given a choice.
I was 46 when I was hired there for a “permanently remote [sic] field by design role” in ProServe and it was my 8th job out of college. I went in with my eyes wide open. I had a plan, stay for four years, sell my RSUs as soon as they vested, pay off debt, save some money, put it on my resume to open doors and make connections and leave.
I was never expecting to make more when I left. I used the time to downsize and reduce my expenses - including moving to state tax free Florida.
When I saw the writing on the wall, I played the game while I was on focus to get my next vest and wait for the “get 40k+ severance and leave immediately or try to work through the PIP”.
I took the latter and had three offers within 3 weeks. This was late 2023.
How long did you stick it out? Were you close toward completing your plan?
Close enough. I missed 2 vesting periods. But the severance and rapidly having a job made up for one and I got refreshers my third year that I hadn’t counted on.
I left debt free, sold my old home for exactly twice what I had built for 8 years earlier, downsized to a condo half the price I sold it for (and 1/3 the size) and I was debt free with savings.
I’m now a staff consultant working full time at a 3rd party AWS consulting firm with a lot less stress and still remote. They were the last to fall. But AWS made their ProServe department return to office at the beginning of this year.
I’m ex Amazon. The company promotes, hires, and fires based on everything _except_ merit. I saw many projects fail due to under qualified teammates and leadership. Amazon is an incredible company but it was only a matter of time till its activism caught up with itself.