I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like it has been every other time before this.
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
It works both ways, a lot of people also take the "nothing ever happens" position and it is true that most of the time "nothing ever happens", so by taking that position, they're right 99% of the time and sound smart
Yes, but the majority opinion is "nothing ever happens" by default for everything, not just tech outages. It's not about sounding smart, but getting ahead of grifters and clout chasers.
Your post reflects another online observation. With the rise of online sports books, this sort of predictive doomerism has flooded almost every team's online comment section. It no longer feel like fandom or community in the same way. Just lots of voices that will be glad to say, "I told you so," in the loss and crickets with the W. Wish there was some accountability mechanism for all the negative noise broadcasted into the channel.
The great thing is that with all of the encrypted/signed DNS thingys, you now have not one, but multiple options to combine “invalid certificate” and DNS in one outage. You no longer need to choose!
Yeah, the Canadian telco was Rogers. Total recovery took multiple days. From the Wikipedia writeup:
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattacks are a good scapegoat for any large incompetent non-tech company that is unable to admit a mistake. (tech companies are more open to admitting actual mistakes - and reluctant to disclose cyberattacks even if there actually was one - where as non-tech ones would rather allude to an attack than admit a mistake)
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks or save it for later than to burn it by causing a massive, visible problem.
20 years ago much less of the infrastructure of everyday life depended on an always-on network connection. Smartphones in particular were a relatively niche product. I didn’t even have a cell phone (and not because I was too young), much less expect it to work all the time.
Verizon had issues routing calls to a provider I'm aware of yesterday, and had to make some sort of change today to fix it. I'm definitely thinking bad configuration update.
No, it's affecting Verizon and people on ATT & TMO trying to get in touch with the Verizon customers who are affected.
Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/
(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
For those who weren't aware: Verizon got a new CEO late last year and laid off 15% of the workforce (15,000 people). This included people working in network, IT and cyber security.
15% seems like the magic number. It seems like many corporations layoff approximately that fraction of their workforce; it's hard to believe it's coincidence.
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
This is unrealistic and seems to be biased by some kind of broad un-focused hostility. Yes, maybe they were overstaffed. But it's reasonable to suspect that leadership overcut, given the current climate and the number being 15,000. Your characterization of Twitter predictions relies on cherry-picking and ignores the actual impacts, and there's no evidence that the system goes down without "constant" intervention from "thousands". Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
>Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
That's correct.
In the case of Twitter, it was disclosed that many of their systems were running out of date EOL software, to the point of being a security liability, which raises the question: if the systems weren't being maintained, wtf were all those people doing? Taste-testing the free food and cappuccinos?
> many of their systems were running out of date EOL software, to the point of being a security liability
This is more likely a management problem rather than a staffing problem. Lower level management knows about these kind of things but often they are not incentivized to make them a priority due to a culture focused on growth and “winning”.
Verizon is a traditional for-profit telco. Not some VC funded startup trying to hit a burn rate. Very unlikely they were overstaffed by 15k, sounds more like overzealous cost-cutting to hit a quarterly target.
Real-time multi-directional communications over massive geographic areas with tens of thousands of physical cell sites connected to ~140M devices vs... public text messages with media.
I realize your point, but its fair to say maintaining a nationwide physical wireless infrastructure may not be the same as hosting tweets, particularly when outages strike.
If you’ve got 100k people to run something that should run essentially on autopilot, you’ve got much deeper problems where merely laying off selected chunks of people will no longer help. The whole company is rotten and the only way is to start from scratch and not make the mistakes that led you to accumulate 100k people.
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
It's well known that big tech companies are overstaffed. You probably can't build Dropbox in a weekend starting from scratch, but a smaller scale cloud-based storage solution can be deployed very quickly if you start from existing open source components. And a small team of experienced web devs can certainly build a cloud storage thing in a matter of weeks from scratch too.
Almost; you need maintenance and monitoring but that doesn't take anywhere near 100k people - assuming they even use their own headcount for this instead of just outsourcing maintenance to their equipment vendor.
Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.
You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.
Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.
I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.
ChatGPT can very well be an upgrade compared to the "engineering" capability of a lot of telcos (they have very little, are hell-bent on outsourcing as much as possible and are even proud of that). But don't take it from me, here's a more reliable source: https://berthub.eu/articles/posts/5g-elephant-in-the-room/
Obviously, you know exactly how to run a major telecomm operation ten times as efficiently as the dominant operator in the most prosperous nation on earth — you are wasting your skills and should absolutely be given funding to disrupt them and make Billions!! What is holding you back from joining the Oligarchs?
Have you considered that the inefficiency is a feature and provides cushy jobs for a lot of people and subcontractors? But yes, a modern telco can absolutely be run more efficiently if you operate it like a tech company and don't have to deal with decades of legacy sludge (whether bloated headcounts or heterogenous legacy infrastructure you have to support).
The problem is that this is a culture problem and once a company is ossified it is really hard to enact such change from the inside even if you wanted to because everyone enjoys the status quo (and who doesn't wouldn't be there to begin with).
Another example: have you seen the UK & EU banking scene and the boom of fintech and "neobanks" around 2017 like Revolut, Monzo, Starling, N26, etc? They managed to build from scratch on relatively shoestring budgets their own implementation of a consumer bank, something that their legacy competitors still can't replicate despite having way more budget and resources.
Unfortunately, the telco world is an oligopoly and they don't like new entrants (banking in the UK was actually a much more level playing field in comparison), so we can never actually see an experiment that proves or disproves my theory.
The problem with being a nationwide ISP - and Verizon runs mobile phones, fiber and all kinds of other stuff - is that you need lots of hands across the country. A lot of stuff can be done remotely and with automation, but often enough you still need actual physical hands on site, and you can't just say "eh, we'll come around tomorrow, we can't make it there faster".
Also 'Verizon' is not really one company. It is 2. Telco and wireless. Each of those is a mashup of dozens of other phone companies VZ gobbled up over the years. With tech stacks going back decades. At one point while I worked there about 10 years ago they were running the 56k dialup for AOL. They also run a decent amount of stores. They are not going to automate a retail store in the same way you would amazon. You have to have people standing there. Then there is the "i need to talk to someone about why my phone keeps doing weird things" helpdesks/servicedesks. Then the line workers like you point out. Plus the backend people who might be able to work from home. But only if they are not in a secure area working (they have lots of that). That 15k of people was probably the result of several big projects that were scaling up but didnt work out. They have all sorts of projects to try to 'monetize the last mile they own'. Almost all fail.
Are amazon stores automated? I have Amazon fresh store next to me, they have the smart grocery carts that no one uses because they are overly finicky. They have same number of employees as other grocery stores.
I got sniped away from Visible (Verizon MVNO) by US Mobile (multi-carrier MVNO) during a Black Friday sale. USM has an interesting thing where you can actually get a separate eSIM for each of the major carriers, and switch between them. I was curious so I signed up for all 3. It's been interesting to see how the signals vary from location to location, and at least a couple times I've been able to get significantly better signal by switching.
The main downside is that you have different numbers for each eSIM, but that doesn't really affect me because I use Google Voice for SMS.
I switched to US Mobile a long time ago just due to pricing. I was WFH most days of the week (before Covid) so I could do minimal data and pay around $100/yr for unlimited talk/text. Now, I do unlimited everything as I commute 5 days each week, but it's only $200/yr. Still significant savings.
Unfortunately GV has been having a ~week outage of outgoing MMS group messages. Well, let's say a brownout – many messages make it through to some recipients.
But yeah I probably should have clarified that Google Voice has been a pretty terrible UX and quality overall for years. I really need to just bite the bullet and port my number out.
Yup! Tip: lock em in and verify a regular cell phone number and then port it. They can probably make it work for you if it’s like a bank, otherwise yolo.
I really love US Mobile. I was a flagship postpaid $90/mo VZW customer for a decade and was so hesitant to switch. It's been 4 years now, and all I can say is that I can't believe I waited so long.
Recently downloaded bitchat so I can contact my family members if outages like this happen when we’re out and about. I got a few T1000 cards for meshtastic but there’s just too much friction to teach my spouse and others how and when to use it. I wish haloW was built into phones which would make long range local communication much better.
Does bitchat work during a phone outage? Range on BLE is pretty low, right, so I'd expect it to not do much unless you happen to live in an area where there are bitchat users every 100 ft or so.
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
Unfortunately as you said, the range is abysmal since it uses BLE. It’s barely enough for a box store but it’s better than nothing. I like the idea of the lilygo pager too. I first thought it was e-ink from some of the pictures but sadly it was not.
Lol it's network core. Postpaid users on Verizon branded accounts affected. Most MVNO's just fine (checked US Mobile and Visible, different cores... and they are working fine on the Verizon towers)
T-Mobile had this issue 5 years or so ago where the IMS core went down due to fiber issues at one of their major backbone providers (pretty sure it was zayo).
As I've said elsewhere, never underestimate Verizon's incompetence. A couple years ago they shut down vtext service which also happened to host their UAprof's for most Android phones. Without a working link to these descriptors, those devices would download any MMS (pre-RCS picture messages) in compatibility mode at the lowest resolution.
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
That's very often the case. I bet there are frantic calls being made right now (probably using a competing carrier!) to Ericsson/etc to come to their rescue, which they happily will in exchange for a hefty fee.
I find this kind of odd. Yesterday, 2026-01-13, I - who lives in the greater Washington D.C. area - experienced A LOT of Verizon disruption. However, today, my service has been excellent. Maybe I'm from the future and don't know it. Did anybody else in my general area experience outages yesterday?
Since this outage is still going on just a town away from their HQ... porting still works to get you going again on the same towers. You can get the device unlock pin from Verizon postpaid sent through a push message to the device so it doesn't rely on SMS.
Ironically I was planning to port my parents from Verizon to US Mobile to save them some money since they aren't financing any devices (there's a sale from them that ends today) and I've just done that on the first line and I now have service on that line with Verizon, where the remaining Verizon postpaid is still dead.
It's also weird that some people here in the office on Verizon work fine, and others are on SOS. No correlation between phone versions or hardware that I could deduce. I also see the same on X: https://x.com/CPTholen/status/2011520566159982758
It came up for around 5-10 minutes at 15:00 EST, but is currently still down.
If they're on different phones, they could be on different bands, different towers, and different paths, one or more of which could be impacted by whatever the underlying problem is. iPhone vs Android would be the most blatant tell that something like this is at fault, but it could also be different configurations from different stores causing them to interact with the cellular network in different ways.
Something like a routing configuration, BGP failure, or underlying network misconfiguration would cause seemingly bizarre results with some phones working and some not with no obvious correlation. Compare Access Point Names under Mobile Settings on android, and whatever the equivalent is on iPhones, and check things like whether 5G allowed and data roaming is enabled.
If it's a cyber attack of some sort, then there's all sorts of different attack vectors that would cause these outcomes.
My wife and I are on the same Verizon family plan. One of us can be down while the other is fine, then 30 minutes later it's the opposite. It's been like that all day.
It's a population map lacking Dallas for sure ... which might mean something real ... but that doesn't match the per-city charts they offer. Who knows.
i know alot are joking / sarcastic about its a cyber attack- that said, Wouldn't it make more sense that whenever there is a "cyber attack" its more likely it would only affect one provider? ie, each has to have different systems / security postures ect, such that a non-public vuln useful to attack Verizon would likely not be exploitable/exposed at AT&T (or vise versa)?
i'm not saying that, I was asking more of a general question, others were saying that multiple carriers were having issues (w the only proof being downdetector and their incorrectly scaled Y axis graphs)
I get the sentiment, but at least for me at home, iOS iMessage still works fine with Wifi. So it's not impacted, and in fact I had to relogin to a client machine with a very persnickety 2FA and it had no issues.
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
Most users don't notice any real differences vs using their ISP DNS, and seeing it up and configuring it is yet another thing to take time or go wrong.
We had edge delivery issues when I didn't use my ISP's DNS, especially from Apple. Not exactly sure of the mechanism, but downloading Xcode would take 2 hours instead of 10 minutes.
That’s really weird that’s the case. DNS simply resolves “google.com” to an IP address (8.8.8.8 or something). Shouldn’t impact anything download related. I’m pretty sure DNS isn’t used for geolocating either
I wanted to correct you but than I stopped myself because I'm not sure if you meant that sarcastically. Because with a /s at the end your post makes sense.
DNS servers can take the IP address of the client into account. If you query a record for amazon.com from the USA you will get a different answer than from Europe. (And you don't need anycast for that.)
That the client information doesn't get lost when it goes through different resolvers
the DNS extension EDNS Client Subnet (ECS) was invented.
explains it better than me. The whole point of the extension is to make geo-guessing the original client over DNS more stable.
Now you can have privacy conscious DNS servers that strip the ECS information (or mess with it somehow) and instead of the server closest to you you get the global fallback for example.
(controld.com goes as far to say "switch countries without a VPN" by only messing with ECS. No idea how stable that is though.)
Interesting! I always just assumed sites used geoDNS to figure out where the user is. I like the "Controversy over lack of support" section in the wikipage. I've been mainly using NextDNS and learned that they anonymized this information https://medium.com/nextdns/how-we-made-dns-both-fast-and-pri...
I’m on Comcast with a UniFi cloud gateway max with my DNS pointed towards adguard. I have not noticed any rate limiting. I actually don’t know how they would rate limit against DoH.
Many reasons not do use the provided DNS. First, you don't want to give the ISP more information on your browsing habits than it can already gather otherwise. Second, in some countries, ISPs censor websites at the request of of the movie and music industries. Those are enough reasons to rely on a neutral DNS provider like Quad9 or your own DNS server.
Do you use your car’s built in navigation function — that you paid for — or do you plug your phone in and use its free Google Maps or Apple Maps to navigate?
I didn't pay for any navigation for my car, so I'm not sure what your point is? ISPs provide DNS. People shouldn't have to fuck with the internet's phone book when they plug their modem in (and they haven't for a very long time). Maybe we can expect more from the people who provide the services society relies on, instead of just saying "why don't you just..." every time someone has a legitimate complaint about something that ought to Just Work.
Yeah, this is notable. If true, every mobile carrier getting hit at the same time says "coordinated incident", from whatever source and for whatever purpose.
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
I've got a Verizon Network Extender and it appears to be online - the tunnel is up back to VZW's security gateway, but all of my phones are refusing to register to it.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
Parts of the HLR might be unavailable or under a thundering herd as all phones are trying to register at once. The HLR would also be involved if using EAP-SIM/EAP-AKA so that would explain failures on the Wi-Fi access points.
I have two Verizon phones on very different networks and both have not been working well since Tuesday - anyone also having this? I kept restarting my phones, airplane mode on/off etc
The US would never purposefully cut of it's own communications, right? What would be the tells if that would have happened? And what could the average citizen do to be able to communicate regardless? Might be useful to always have that knowledge around, even if it isn't a personal threat to yourself today.
Get into Meshtastic or Meshcore! (So far I prefer meshtastic)
It's a citizen-run mesh network that allows text communication. It's not perfect and not 100% reliable (signal and hops limit delivery guarantees) but it's better than 0 ways to communicate in this kind of event!
I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like it has been every other time before this.
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
One of these times they will be right and you will never hear the end of the time they were first to recognize the start of a cyberattack.
You know what they say about a stopped clock: it's wrong 1,438 times a day.
If it is a quartz watch, it might be 47,120,384 times per day.
better odds than the lottery!
When telcos get compromised again, the attackers should just take it down in service of this moronic fallacy.
https://www.wsj.com/politics/national-security/china-cyberat...
It works both ways, a lot of people also take the "nothing ever happens" position and it is true that most of the time "nothing ever happens", so by taking that position, they're right 99% of the time and sound smart
Yes, but the majority opinion is "nothing ever happens" by default for everything, not just tech outages. It's not about sounding smart, but getting ahead of grifters and clout chasers.
Yep I think it started as a reaction against "it's happening!" types and it is a lot less wrong, but it's still wrong.
The truth is things do occasionally happen and we should be prepared, even if most of the time they don't
Your post reflects another online observation. With the rise of online sports books, this sort of predictive doomerism has flooded almost every team's online comment section. It no longer feel like fandom or community in the same way. Just lots of voices that will be glad to say, "I told you so," in the loss and crickets with the W. Wish there was some accountability mechanism for all the negative noise broadcasted into the channel.
In general, whats the expected cost of being loudly, obviously, publicly and obnoxiously wrong?
Keep in mind who the President is.
That's not true though, sometimes it's inexplicitly DNS too.
My money is on expired certificate.
If so, successful test.
The great thing is that with all of the encrypted/signed DNS thingys, you now have not one, but multiple options to combine “invalid certificate” and DNS in one outage. You no longer need to choose!
it's always DNS
or BGP.
No no it’s gotta be “the database”
I worked at a major ISP and we had a similar situation where the North East went down and the RC was a fiber cut at a major node in Philly.
These network topologies are incredibly complex and edges you think wouldn't exist have ways of suddenly appearing when things go awry.
Yeah, the Canadian telco was Rogers. Total recovery took multiple days. From the Wikipedia writeup:
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
The major Canadian outage was Rogers in 2022: https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattacks are a good scapegoat for any large incompetent non-tech company that is unable to admit a mistake. (tech companies are more open to admitting actual mistakes - and reluctant to disclose cyberattacks even if there actually was one - where as non-tech ones would rather allude to an attack than admit a mistake)
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks or save it for later than to burn it by causing a massive, visible problem.
In a dead empire, sufficiently advanced rot is indistinguishable from malice.
Infrastructure in general seems worse than 20 years ago. Our talent for black-bagging dictators has never been stronger, though!
20 years ago much less of the infrastructure of everyday life depended on an always-on network connection. Smartphones in particular were a relatively niche product. I didn’t even have a cell phone (and not because I was too young), much less expect it to work all the time.
Verizon had issues routing calls to a provider I'm aware of yesterday, and had to make some sort of change today to fix it. I'm definitely thinking bad configuration update.
It's affecting every mobile carrier (ATT, TMO), not just Verizon
No, it's affecting Verizon and people on ATT & TMO trying to get in touch with the Verizon customers who are affected.
Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/
(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
For those who weren't aware: Verizon got a new CEO late last year and laid off 15% of the workforce (15,000 people). This included people working in network, IT and cyber security.
15% seems like the magic number. It seems like many corporations layoff approximately that fraction of their workforce; it's hard to believe it's coincidence.
Everyone is always so negative about these outages, but if we look at the details we see it was a win as long as they maintain 85% up time.
Is there a substantive connection?
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
This is unrealistic and seems to be biased by some kind of broad un-focused hostility. Yes, maybe they were overstaffed. But it's reasonable to suspect that leadership overcut, given the current climate and the number being 15,000. Your characterization of Twitter predictions relies on cherry-picking and ignores the actual impacts, and there's no evidence that the system goes down without "constant" intervention from "thousands". Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
>Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
That's correct.
In the case of Twitter, it was disclosed that many of their systems were running out of date EOL software, to the point of being a security liability, which raises the question: if the systems weren't being maintained, wtf were all those people doing? Taste-testing the free food and cappuccinos?
> many of their systems were running out of date EOL software, to the point of being a security liability
This is more likely a management problem rather than a staffing problem. Lower level management knows about these kind of things but often they are not incentivized to make them a priority due to a culture focused on growth and “winning”.
I work in the field. All of the software that's not sold by Huawei is steaming pile of excrements that only has accidental design.
You do need too many people to work with that. Cutting them is asking for pain.
Verizon is a traditional for-profit telco. Not some VC funded startup trying to hit a burn rate. Very unlikely they were overstaffed by 15k, sounds more like overzealous cost-cutting to hit a quarterly target.
Gotta get those bonuses in the upper levels.
Real-time multi-directional communications over massive geographic areas with tens of thousands of physical cell sites connected to ~140M devices vs... public text messages with media.
I realize your point, but its fair to say maintaining a nationwide physical wireless infrastructure may not be the same as hosting tweets, particularly when outages strike.
Slight sarcasm ahead—fair warning.
When Twitter did, its CEO may have slept at the office for weeks to make sure problems were resolved.
On the other hand, the Verizon CEO may be shopping for a new boat
> the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
Many think Twitter has imploded, though it's online.
> You do realize it's possible for an organization to be overstaffed?
It's possible to be understaffed or appropriately staffed. Anything is possible!
If you’ve got 100k people to run something that should run essentially on autopilot, you’ve got much deeper problems where merely laying off selected chunks of people will no longer help. The whole company is rotten and the only way is to start from scratch and not make the mistakes that led you to accumulate 100k people.
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
Telco infra runs on autopilot?
This is really taking the whole "I could build Dropbox in a weekend" style comments to a new level.
It's well known that big tech companies are overstaffed. You probably can't build Dropbox in a weekend starting from scratch, but a smaller scale cloud-based storage solution can be deployed very quickly if you start from existing open source components. And a small team of experienced web devs can certainly build a cloud storage thing in a matter of weeks from scratch too.
> It's well known that big tech companies are overstaffed.
To who? What evidence is there either way?
Is it an often-repeated story online? That sort of information is both well known and unreliable - it's well-known misinformation.
It does if you have no idea how it works.
Erlang fault-tolerance FTW! ;-)
Almost; you need maintenance and monitoring but that doesn't take anywhere near 100k people - assuming they even use their own headcount for this instead of just outsourcing maintenance to their equipment vendor.
Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.
Carrier-level telco is labor intensive.
You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.
Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.
I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.
VZW has 146m lines. Each employee supports 14,600 customers, seems like a reasonable number...
lol. You figured it out, Verizon wireless can just be replaced by a UniFi router and ChatGPT.
ChatGPT can very well be an upgrade compared to the "engineering" capability of a lot of telcos (they have very little, are hell-bent on outsourcing as much as possible and are even proud of that). But don't take it from me, here's a more reliable source: https://berthub.eu/articles/posts/5g-elephant-in-the-room/
Obviously, you know exactly how to run a major telecomm operation ten times as efficiently as the dominant operator in the most prosperous nation on earth — you are wasting your skills and should absolutely be given funding to disrupt them and make Billions!! What is holding you back from joining the Oligarchs?
Have you considered that the inefficiency is a feature and provides cushy jobs for a lot of people and subcontractors? But yes, a modern telco can absolutely be run more efficiently if you operate it like a tech company and don't have to deal with decades of legacy sludge (whether bloated headcounts or heterogenous legacy infrastructure you have to support).
The problem is that this is a culture problem and once a company is ossified it is really hard to enact such change from the inside even if you wanted to because everyone enjoys the status quo (and who doesn't wouldn't be there to begin with).
Another example: have you seen the UK & EU banking scene and the boom of fintech and "neobanks" around 2017 like Revolut, Monzo, Starling, N26, etc? They managed to build from scratch on relatively shoestring budgets their own implementation of a consumer bank, something that their legacy competitors still can't replicate despite having way more budget and resources.
Unfortunately, the telco world is an oligopoly and they don't like new entrants (banking in the UK was actually a much more level playing field in comparison), so we can never actually see an experiment that proves or disproves my theory.
> Have you considered that the inefficiency is a feature and provides cushy jobs for a lot of people and subcontractors?
Yes, this explains most of our jobs.
The problem with being a nationwide ISP - and Verizon runs mobile phones, fiber and all kinds of other stuff - is that you need lots of hands across the country. A lot of stuff can be done remotely and with automation, but often enough you still need actual physical hands on site, and you can't just say "eh, we'll come around tomorrow, we can't make it there faster".
Also 'Verizon' is not really one company. It is 2. Telco and wireless. Each of those is a mashup of dozens of other phone companies VZ gobbled up over the years. With tech stacks going back decades. At one point while I worked there about 10 years ago they were running the 56k dialup for AOL. They also run a decent amount of stores. They are not going to automate a retail store in the same way you would amazon. You have to have people standing there. Then there is the "i need to talk to someone about why my phone keeps doing weird things" helpdesks/servicedesks. Then the line workers like you point out. Plus the backend people who might be able to work from home. But only if they are not in a secure area working (they have lots of that). That 15k of people was probably the result of several big projects that were scaling up but didnt work out. They have all sorts of projects to try to 'monetize the last mile they own'. Almost all fail.
Are amazon stores automated? I have Amazon fresh store next to me, they have the smart grocery carts that no one uses because they are overly finicky. They have same number of employees as other grocery stores.
I got sniped away from Visible (Verizon MVNO) by US Mobile (multi-carrier MVNO) during a Black Friday sale. USM has an interesting thing where you can actually get a separate eSIM for each of the major carriers, and switch between them. I was curious so I signed up for all 3. It's been interesting to see how the signals vary from location to location, and at least a couple times I've been able to get significantly better signal by switching.
The main downside is that you have different numbers for each eSIM, but that doesn't really affect me because I use Google Voice for SMS.
I switched to US Mobile a long time ago just due to pricing. I was WFH most days of the week (before Covid) so I could do minimal data and pay around $100/yr for unlimited talk/text. Now, I do unlimited everything as I commute 5 days each week, but it's only $200/yr. Still significant savings.
Unfortunately GV has been having a ~week outage of outgoing MMS group messages. Well, let's say a brownout – many messages make it through to some recipients.
https://www.google.com/appsstatus/dashboard/incidents/THtvkD...
Looks like they fixed it today.
But yeah I probably should have clarified that Google Voice has been a pretty terrible UX and quality overall for years. I really need to just bite the bullet and port my number out.
Ive had many issues where services can't verify the GV phone number - cause they know it's GV. Does this still happen?
Used to happen more for me. Very rare now.
Yup! Tip: lock em in and verify a regular cell phone number and then port it. They can probably make it work for you if it’s like a bank, otherwise yolo.
I really love US Mobile. I was a flagship postpaid $90/mo VZW customer for a decade and was so hesitant to switch. It's been 4 years now, and all I can say is that I can't believe I waited so long.
I'm considering switching to US Mobile from Verizon-- it seems way cheaper, and you can still use the Verizon network. Any downsides I should know?
I started mid last year and haven't run into any problems. Haven't needed to switch off of Dark Star yet.
I was previously on Google Fi, and MVNO roaming is bloody fantastic - I always had reception.
One tradeoff with MVNOs is you're generally lower priority on the network.
Recently downloaded bitchat so I can contact my family members if outages like this happen when we’re out and about. I got a few T1000 cards for meshtastic but there’s just too much friction to teach my spouse and others how and when to use it. I wish haloW was built into phones which would make long range local communication much better.
Does bitchat work during a phone outage? Range on BLE is pretty low, right, so I'd expect it to not do much unless you happen to live in an area where there are bitchat users every 100 ft or so.
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
[0] https://mtlynch.io/first-impressions-of-meshcore/#testing-th...
Unfortunately as you said, the range is abysmal since it uses BLE. It’s barely enough for a box store but it’s better than nothing. I like the idea of the lilygo pager too. I first thought it was e-ink from some of the pictures but sadly it was not.
Really neat concept but that name is so unfortunate. Is it bit-chat or bitch-at? Frankly either would work.
I had not heard of it, and the first thing that came to mind was bitch-at. Maybe that's a reflection on my character, who knows.
feelin bitch-at, frfr
It's obviously bit chat
Checking off "complains about the name" on my HN bingo card.
Lol it's network core. Postpaid users on Verizon branded accounts affected. Most MVNO's just fine (checked US Mobile and Visible, different cores... and they are working fine on the Verizon towers)
Further info: https://old.reddit.com/r/cellmapper/comments/1qcwj48/observa...
T-Mobile had this issue 5 years or so ago where the IMS core went down due to fiber issues at one of their major backbone providers (pretty sure it was zayo).
As I've said elsewhere, never underestimate Verizon's incompetence. A couple years ago they shut down vtext service which also happened to host their UAprof's for most Android phones. Without a working link to these descriptors, those devices would download any MMS (pre-RCS picture messages) in compatibility mode at the lowest resolution.
At 2:14PM EST, Verizon said:
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
No ETA yet, not even an underpromised one.
Odds are they have a ton of outsourced engineers who are an ocean away and unable to do anything.
That's very often the case. I bet there are frantic calls being made right now (probably using a competing carrier!) to Ericsson/etc to come to their rescue, which they happily will in exchange for a hefty fee.
At least the shareholders have had an amazing quarter. :)
I find this kind of odd. Yesterday, 2026-01-13, I - who lives in the greater Washington D.C. area - experienced A LOT of Verizon disruption. However, today, my service has been excellent. Maybe I'm from the future and don't know it. Did anybody else in my general area experience outages yesterday?
Dallas metro here. I couldn't receive SMS based OTP from a bank for hours yesterday.
DC local. Total outage this afternoon. SOS on my phone for a few hours. Unable to log into my account.
Since this outage is still going on just a town away from their HQ... porting still works to get you going again on the same towers. You can get the device unlock pin from Verizon postpaid sent through a push message to the device so it doesn't rely on SMS.
Ironically I was planning to port my parents from Verizon to US Mobile to save them some money since they aren't financing any devices (there's a sale from them that ends today) and I've just done that on the first line and I now have service on that line with Verizon, where the remaining Verizon postpaid is still dead.
I was unable to receive SMS based OTP from my bank yesterday for a couple of hours... Wonder if it's related. Anyone else?
It's also weird that some people here in the office on Verizon work fine, and others are on SOS. No correlation between phone versions or hardware that I could deduce. I also see the same on X: https://x.com/CPTholen/status/2011520566159982758
It came up for around 5-10 minutes at 15:00 EST, but is currently still down.
If they're on different phones, they could be on different bands, different towers, and different paths, one or more of which could be impacted by whatever the underlying problem is. iPhone vs Android would be the most blatant tell that something like this is at fault, but it could also be different configurations from different stores causing them to interact with the cellular network in different ways.
Something like a routing configuration, BGP failure, or underlying network misconfiguration would cause seemingly bizarre results with some phones working and some not with no obvious correlation. Compare Access Point Names under Mobile Settings on android, and whatever the equivalent is on iPhones, and check things like whether 5G allowed and data roaming is enabled.
If it's a cyber attack of some sort, then there's all sorts of different attack vectors that would cause these outcomes.
My wife and I are on the same Verizon family plan. One of us can be down while the other is fine, then 30 minutes later it's the opposite. It's been like that all day.
Same here, except that when here (central-western NJ) when someone "recovers" here we go from SOS to a few bars but no LTE or 5G indicator. Yikes.
The link just redirects me to https://www.youtube.com/c/FirstCoastNews.
Some outlets reporting T-Mobile and ATT as well.
I assume state on state cyber attacks are commonplace but get minimized to avoid public fear.. perhaps this will be the first notable one.
The alternative network reports are most likely people trying to call Verizon customers and reporting an outage when they can't get through.
You think like a person who’s debugged large systems failures before :). That feels very plausible.
I've seen it happen before - in the big AT&T outage a couple years ago reports came in on downdetector of outages for other providers.
Estonia was the first major NATO victim of such things https://en.wikipedia.org/wiki/2007_cyberattacks_on_Estonia
Made worse by the fact Estonia is a more networked society than, for example, the US.
The down detector site has Verizon outage reports two order of magnitude bigger, so it doesn't seem like a cyber attack to me.
https://downdetector.com/status/t-mobile/ ~ 1,600
https://downdetector.com/status/att/ ~ 1,500
https://downdetector.com/status/verizon/ ~ peaked at ~169k, dropped to 67k
T-Mobile is up and Verizon is down in my house
It’s Denmark! Ο_o
I still received 23 telemarketing calls today on my Verizon phone in Maryland.
This is what happens when you lay off 30% of your workforce.
Here's a map: https://downdetector.com/status/verizon/map/
That's a population map
It's a population map lacking Dallas for sure ... which might mean something real ... but that doesn't match the per-city charts they offer. Who knows.
oh, so it's a heat map?
https://xkcd.com/1138/
i know alot are joking / sarcastic about its a cyber attack- that said, Wouldn't it make more sense that whenever there is a "cyber attack" its more likely it would only affect one provider? ie, each has to have different systems / security postures ect, such that a non-public vuln useful to attack Verizon would likely not be exploitable/exposed at AT&T (or vise versa)?
why are you saying that this is affecting more than one provider?
i'm not saying that, I was asking more of a general question, others were saying that multiple carriers were having issues (w the only proof being downdetector and their incorrectly scaled Y axis graphs)
Anyone still using SMS for 2FA codes, here is your official notice to change that ASAP.
Many (most?) services offer a choice between SMS 2FA and no 2FA. It's not always so easy.
I get the sentiment, but at least for me at home, iOS iMessage still works fine with Wifi. So it's not impacted, and in fact I had to relogin to a client machine with a very persnickety 2FA and it had no issues.
if you can!
Phone hasn’t had a connection for hours. 5G wireless home is fine. Upstate South Carolina.
Verizon acknowledged the issue: https://xcancel.com/VerizonNews/status/2011500483072954495
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
>Note, you have to click on the providers to see absolute magnitude
I noticed that a little while ago too. Very bad UI. Tempted to post in r/dataisugly.
> Verizon acknowledged the issue: > https://xcancel.com/VerizonNews/status/2011500483072954495
Looking at the replies some of the people who are more worried about the outage seem to be the OF models
Not judging or trying to make a point. Just find it interesting the ways things are interconnected
Seems like it's happening on the West Coast to
Apparently only affecting east coast numbers.
Some server in NJ is down
Our Comcast DNS keeps going in and out too, not clear if it is related or knock on effect or something else entirely.
I’m somewhat surprised you’re still using your ISP’s DNS when there’s ton of better free or paid options.
Most users don't notice any real differences vs using their ISP DNS, and seeing it up and configuring it is yet another thing to take time or go wrong.
We had edge delivery issues when I didn't use my ISP's DNS, especially from Apple. Not exactly sure of the mechanism, but downloading Xcode would take 2 hours instead of 10 minutes.
That’s really weird that’s the case. DNS simply resolves “google.com” to an IP address (8.8.8.8 or something). Shouldn’t impact anything download related. I’m pretty sure DNS isn’t used for geolocating either
I wanted to correct you but than I stopped myself because I'm not sure if you meant that sarcastically. Because with a /s at the end your post makes sense.
Nope, not sarcastic. Would love to know where my understanding is incorrect!
DNS servers can take the IP address of the client into account. If you query a record for amazon.com from the USA you will get a different answer than from Europe. (And you don't need anycast for that.)
That the client information doesn't get lost when it goes through different resolvers the DNS extension EDNS Client Subnet (ECS) was invented.
https://en.wikipedia.org/wiki/EDNS_Client_Subnet
explains it better than me. The whole point of the extension is to make geo-guessing the original client over DNS more stable.
Now you can have privacy conscious DNS servers that strip the ECS information (or mess with it somehow) and instead of the server closest to you you get the global fallback for example. (controld.com goes as far to say "switch countries without a VPN" by only messing with ECS. No idea how stable that is though.)
Interesting! I always just assumed sites used geoDNS to figure out where the user is. I like the "Controversy over lack of support" section in the wikipage. I've been mainly using NextDNS and learned that they anonymized this information https://medium.com/nextdns/how-we-made-dns-both-fast-and-pri...
Comcast doesn't let you change your DNS unless you run your own router. And they also rate limit you if you do.
I’m on Comcast with a UniFi cloud gateway max with my DNS pointed towards adguard. I have not noticed any rate limiting. I actually don’t know how they would rate limit against DoH.
Pretty sure you can do it at the individual device level, in the OS's network settings.
"You relied on a service you pay for to actually work? What are you, stupid?"
Many reasons not do use the provided DNS. First, you don't want to give the ISP more information on your browsing habits than it can already gather otherwise. Second, in some countries, ISPs censor websites at the request of of the movie and music industries. Those are enough reasons to rely on a neutral DNS provider like Quad9 or your own DNS server.
When that service actively spies on you, then yes, that is stupid...
That’s… weirdly a thing.
Do you use your car’s built in navigation function — that you paid for — or do you plug your phone in and use its free Google Maps or Apple Maps to navigate?
Yes actually, I happen to save the built in GPS for occasions when the phone network is inaccessible.
I didn't pay for any navigation for my car, so I'm not sure what your point is? ISPs provide DNS. People shouldn't have to fuck with the internet's phone book when they plug their modem in (and they haven't for a very long time). Maybe we can expect more from the people who provide the services society relies on, instead of just saying "why don't you just..." every time someone has a legitimate complaint about something that ought to Just Work.
Might be. VZ does host several big DNS root nodes. But it could just be comcast too...
Not sure why this says East coast, I am also affected here in Seattle.
Weird. My Verizon is working fine in Seattle.
Same in Seattle. Turned of Wifi and the data connection still let me online.
downdetector is showing a lot of down mobile networks & services
Yeah, this is notable. If true, every mobile carrier getting hit at the same time says "coordinated incident", from whatever source and for whatever purpose.
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
The down detector site has Verizon outage reports two order of magnitude bigger, so it doesn't seem like a cyber attack to me. ~60-160k vs ~1.5-1.8k
Maybe installing a beam splitter went wrong?
This is what happens when you invest more in lobbying than the network
I've got a Verizon Network Extender and it appears to be online - the tunnel is up back to VZW's security gateway, but all of my phones are refusing to register to it.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
I could see Verizon access points but just couldn't connect to any of them. I had no issue sending texts over wifi though.
Parts of the HLR might be unavailable or under a thundering herd as all phones are trying to register at once. The HLR would also be involved if using EAP-SIM/EAP-AKA so that would explain failures on the Wi-Fi access points.
Same. Femtocell is online but no clients, phones on SOS mode.
I am in Miami right now, and verizon is not working here. (I get the SOS sign).
I have two Verizon phones on very different networks and both have not been working well since Tuesday - anyone also having this? I kept restarting my phones, airplane mode on/off etc
West coast too, down in LA at same time (12 PM EST / 9 AM PST). I think they’re way way way underselling the # affected (“thousands” lol)
good morning sirs
Grok 4.1 down too: https://status.x.ai/api-us-east-1/INC10b35980
The US would never purposefully cut of it's own communications, right? What would be the tells if that would have happened? And what could the average citizen do to be able to communicate regardless? Might be useful to always have that knowledge around, even if it isn't a personal threat to yourself today.
Get into Meshtastic or Meshcore! (So far I prefer meshtastic)
It's a citizen-run mesh network that allows text communication. It's not perfect and not 100% reliable (signal and hops limit delivery guarantees) but it's better than 0 ways to communicate in this kind of event!
Sadly this has happened before:
BART Defends Cutting Off Cell Service In Subway
https://www.npr.org/2011/08/16/139664637/bart-defends-cuttin...
And will happen again.
Didn’t they just raid a reporters house. I wonder what service provider they used…
I think one of the tells might be that all major providers go down, not just one.