> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network.
As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message.
> The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems' permissions
Also appreciate the honesty here.
> On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures to deliver core network traffic. […]
> Core traffic was largely flowing as normal by 14:30. We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online. As of 17:06 all systems at Cloudflare were functioning as normal.
Why did this take so long to resolve? I read through the entire article, and I understand why the outage happened, but when most of the network goes down, why wasn't the first step to revert any recent configuration changes, even ones that seem unrelated to the outage? (Or did I just misread something and this was explained somewhere?)
Of course, the correct solution is always obvious in retrospect, and it's impressive that it only took 7 minutes between the start of the outage and the incident being investigated, but it taking a further 4 hours to resolve the problem and 8 hours total for everything to be back to normal isn't great.
Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.
Thanks for the explanation! This definitely reminds me of CrowdStrike outages last year:
- A product depends on frequent configuration updates to defend against attackers.
- A bad data file is pushed into production.
- The system is unable to easily/automatically recover from bad data files.
(The CrowdStrike outages were quite a bit worse though, since it took down the entire computer and remediation required manual intervention on thousands of desktops, whereas parts of Cloudflare were still usable throughout the outage and the issue was 100% resolved in a few hours)
It'd be fun to read more about how you all procedurally respond to this (but maybe this is just a fixation of mine lately). Like are you tabletopping this scenario, are teams building out runbooks for how to quickly resolve this, what's the balancing test for "this needs a functional change to how our distributed systems work" vs. "instead of layering additional complexity on, we should just have a process for quickly and maybe even speculatively restoring this part of the system to a known good state in an outage".
We incorrectly thought at the time it was attack traffic coming in via WARP into LHR. In reality it was just that the failures started showing up there first because of how the bad file propagated and where it was working hours in the world.
Probably because it was the London team that was actively investigating the incident and initially came to the conclusion that it may be a DDoS while being unable to authenticate to their own systems.
Question from a casual bystander, why not have a virtual/staging mini node that receives these feature file changes first and catches errors to veto full production push?
Or you do have something like this but the specific db permission change in this context only failed in production
I think the reasoning behind this is because of the nature of the file being pushed - from the post mortem:
"This feature file is refreshed every few minutes and published to our entire network and allows us to react to variations in traffic flows across the Internet. It allows us to react to new types of bots and new bot attacks. So it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly."
In this case, the file fails quickly. A pretest that consists of just attempting to load the file would have caught it. Minutes is more than enough time to perform such a check.
Just asking out of curiosity, but roughly how many staff would've been involved in some way in sorting out the issue? Either outside regular hours or redirected from their planned work?
Is there some way to check the sanity of the configuration change, monitor it and then revert back to an earlier working configuration if things don't work out?
Is it though? Or is it, oh, this is such a simple change that we really don't need to test it attitude? I'm not saying this applies to TFA, but some people are so confident that no pressure is felt.
However, you forgot that the lighting conditions are where only red lights from the klaxons are showing so you really can't differentiate the colors of the wires
Side thought as we're working on 100% onchain systems (for digital assets security, different goals):
Public chains (e.g. EVMs) can be a tamper‑evident gate that only promotes a new config artifact if (a) a delay or multi‑sig review has elapsed, and (b) a succinct proof shows the artifact satisfies safety invariants like ≤200 features, deduped, schema X, etc.
That could have blocked propagation of the oversized file long before it reached the edge :)
> much better than their completely false “checking the security of your connection” message
The exact wording (which I can easily find, because a good chunk of the internet gives it to me, because I’m on Indian broadband):
> example.com needs to review the security of your connection before proceeding.
It bothers me how this bald-faced lie of a wording has persisted.
(The “Verify you are human by completing the action below.” / “Verify you are human” checkbox is also pretty false, as ticking the box in no way verifies you are human, but that feels slightly less disingenuous.)
I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency.
Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.
> I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did
Damn corporate karma farming is ruthless, only a couple minute SLA before taking ownership of the karma. I guess I'm not built for this big business SLA.
We're in a Live Fast Die Young karma world. If you can't get a TikTok ready with 2 minutes of the post modem drop, you might as well quit and become a barista instead.
> I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate.
I'm so jealous. I've written postmortems for major incidents at a previous job: a few hours to write, a week of bikeshedding by marketing and communication and tech writers and ... over any single detail in my writing. Sanitizing (hide a part), simplifying (our customers are too dumb to understand), etc, so that the final writing was "true" in the sense that it "was not false", but definitely not what I would call "true and accurate" as an engineer.
I'm not sure I've ever read something from someone so high up in a company that gave me such a strong feeling for "I'd like to work for these people". If job posts could be so informal and open ended, this post could serve as one in the form of a personality fit litmus test.
How do you guys handle redaction? I'm sure even when trusted individuals are in charge of authoring, there's still a potential of accidental leakage which would probably be best mitigated by a team specifically looking for any slip ups.
Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column names.” John and I usually catch anything the rest of the team may have missed. For me, pays to have gone to law school, but also pays to have studied Computer Science in college and be technical enough to still understand both the SQL and Rust code here.
The person who posted both this blog article and the hacker news post, is Matthew Prince, one of highly technical billionaire founders of cloudflare. I'm sure if he wants something to happen, it happens.
I mean the CEO posted the post-mortem so there aren't that many layers of stakeholders above. For other post-mortems by engineers, Matthew once said that the engineering team is running the blog and that he wouldn't event know how to veto even if he wanted [0]
There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.
Why give this sort of content more visibility/reach?
I'm sure that's not your intent, so I hope my comment gives you an opportunity to reflect on the effects of syndicating such stupidity, no matter what platform it comes from.
Mainly to make others aware of what’s happening in the context of this Cloudflare outage. Sure I can avoid giving it visibility/reach but it’s growing and proliferating on its own, and I think ignoring it isn’t going to stop it so I am hoping awareness will help. I’ve noticed a huge rise in open racism against Chinese and Indian and workers of other origin, even when they’re here on a legal visa that we have chosen as a nation to grant for our own benefit.
The legislation that MTG (Marjorie Taylor Green) just proposed a few days ago to ban H1B entirely, and the calls to ban other visa types, is going to have a big negative impact on the tech industry and American innovation in general. The social media stupidity is online but it gives momentum to the actual real life legislation and other actions the administration might take. Many congress people are seeing the online sentiment and changing their positions in response, unfortunately.
This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is textbook "parse, don't validate" anti-pattern.
I know, this is "Monday morning quarterbacking", but that's what you get for an outage this big that had me tied up for half a day.
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is:
1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually.
2. Despite the elevated risk of system wide rapid config propagation, it took them 2 hours to identify the config as the proximate cause, and another hour to roll it back.
SOP for stuff breaking is you roll back to a known good state. If you roll out gradually and your canaries break, you have a clear signal to roll back. Here was a special case where they needed their system to rapidly propagate changes everywhere, which is a huge risk, but didn’t quite have the visibility and rapid rollback capability in place to match that risk.
While it’s certainly useful to examine the root cause in the code, you’re never going to have defect free code. Reliability isn’t just about avoiding bugs. It’s about understanding how to give yourself clear visibility into the relationship between changes and behavior and the rollback capability to quickly revert to a known good state.
Cloudflare has done an amazing job with availability for many years and their Rust code now powers 20% of internet traffic. Truly a great team.
How can you write the proxy without handling the config containing more than the maximum features limit you set yourself?
How can the database export query not have a limit set if there is a hard limit on number of features?
Why do they do non-critical changes in production before testing in a stage environment?
Why did they think this was a cyberattack and only after two hours realize it was the config file?
Why are they that afraid of a botnet? Does not leave me confident that they will handle the next Aisuru attack.
I'm migrating my customers off Cloudflare. I don't think they can swallow the next botnet attacks and everyone on Cloudflare go down with the ship, so it will be safer to not be behind Cloudflare when it hits.
Exactly. The only way this could happen in the first place was _because_ they failed at so many levels. And as a result, more layers of Swiss cheese will be added, and holes in existing ones will be patched. This process is the reason flying is so safe, and the reason why Cloudflare will be a little bit more resilient tomorrow than it was yesterday.
> Why do they do non-critical changes in production before testing in a stage environment?
I guess the noncritical change here was the change to the database? My experience has been a lot of teams do a poor job having a faithful replica of databases in stage environments to expose this type of issue.
In part because it is somewhere between really hard and impossible. Is your staging DB going to be as big? Seeing the same RPS as prod? Seeing the same scenarios?
Permissions stuff might be caught without a completely faithful replica, but there are always going to be attributes of the system that only exist in prod.
I know its easy to criticize what happened after the fact and having a clear(er) picture of all the moving parts and the timeline of events, but I think that while most of the people in the thread are pointing out either Rust-related or lack of configuration validation, what really grinds my gears is something that - in my opinion - is bad engineering.
Having an unprivileged application querying system.columns to infer the table layout is just bad; Not having a proper, well-defined table structure indicates sloppiness in the overall schema design, specially if it changes quickly. Considering specifically clickhouse, and even if this approach would be a good idea, the unprivileged way of doing it would be "DESCRIBE TABLE <name>", NOT iterating system.columns. The gist of it - sloppy design not even well implemented.
Having a critical application issuing ad-hoc commands to system.* tablespace instead of using a well-tested library is just amateurism, and again - bad engineering; IMO it is good practice to consider all system.* privileged applications and ensure their querying is completely separate from your application logic; Sometimes some system tables change, and fields are added and/or removed - not planning for this will basically make future compatibility a nightmare.
Not only the problematic query itself, but the whole context of this screams "lack of proper application design" and devs not knowing how to use the product and/or read the documentation. Granted, this is a bit "close to home" for me, because I use ClickHouse extensively (at a scale - I'm assuming - several orders of magnitude smaller than CloudFlare) and I have spent a lot of time designing specifically to avoid at least some of these kind of mistakes. But, if I can do it at my scale, why aren't they doing it?
On all the other issues, I thought they wanted to do the right thing at heart, but missed to make it fail safe. I can pass it as a problem of a journey to maturity or simply the fact that you can't get everything perfect. Maybe even a bit of sloppiness here and there.
The database issue screamed at me: lack of expertise. I don't use CH, but seeing someone to mess with a production system and they being surprised "Oh, it does that?", is really bad. And this is obviously not knowledge that is hard to achieve, buried deep in a manual or an edge case only discoverable by source code, it's bread and butter knowledge you should know.
What is confusing, that they didn't add this to their follow-up steps. With some benefit of doubt I'd assume they didn't want to put something very basic as a reason out there, just to protect the people behind it from widespread blame. But if that's not the case, then it's a general problem. Sadly it's not uncommon that components like databases are dealt with, on an low effort basis. Just a thing we plug in and works. But it's obviously not.
I don't think these are realistic requirements for any engineered system to be honest. Realistic is to have contingencies for such cases, which are simply errors.
But the case for Cloudflare here is complicated. Every engineer is very free to make a better system though.
What is not realistic? To do simple input validation on data that has the potential to break 20% of the internet? To not have a system in place to rollback to the latest known state when things crash?
Cloudflare builds a global scale system, not an iphone app. Please act like it.
Name me global, redundant systems that have not (yet) failed.
And if you used cloudflare to protect against botnet and now go off cloudflare... you are vulnerable and may experience more downtime if you cannot swallow the traffic.
I mean no service have 100% uptime - just that some have more nines than others.
We had better uptime with AWS WAF in us-east-1 than we've had in the last 1.5 years of Cloudflare.
I do like the flat cost of Cloudflare and feature set better but they have quite a few outages compared to other large vendors--especially with Access (their zero trust product)
I'd lump them into GitHub levels of reliability
We had a comparable but slightly higher quote from an Akamai VAR.
But at the same time, what value do they add if they:
* Took down the the customers sites due to their bug.
* Never protected against an attack that our infra could not have handled by itself.
* Don't think that they will be able to handle the "next big ddos" attack.
It's just an extra layer of complexity for us. I'm sure there are attacks that could help our customers with, that's why we're using them in the first place. But until the customers are hit with multiple ddos attacks that we can not handle ourself then it's just not worth it.
I agree. I think the comments about how "it is fine, because so many things had to fail" do not apply in this case.
It's not that many things had to fail, it's that many things that are obvious haven't been done. It would be a valid excuse if many "exotic" scenarios would have to align, not when it's obvious error cases that weren't handled and changes have not been tested.
While having wrong first assumptions is just how things work when you try to analyze the issue[1], not testing changes before production is just stupidity and nothing else.
The story would be different if eg. multiple unlikely, hard to track things happened at once without anyone making a clearly linkable event, something that would also happen in staging. Most of the things mentioned could essentially statically checked. This is the prime example of what you want as any tech person, because it's not hard to prevent compared to a lot of scenarios where you deal with balancing likelihoods of scenarios, timings, etc.
You don't think someone is a great plumber, because they forgot their tools and missed that big hole in the pipe and also rang at the wrong door, because all these things failed. You think someone is a good plumber if they said they would have to go back to fetch a bulky specialized tool, because this is the rare case in which they need it, but they could also do this other thing in this spcific case. They are great plumbers if they tell you how this happened in first place and how to fix it. They are great plumbers if they manage to fix something outside of their usual scope.
Here pretty much all of the things that you pay them for failed. At a large scale.
I am sure this has there are reasons which we don't now about, and I hope that CloudFlare can fix them. Be it management focusing on the wrong things, be it developers not being in the wrong position or annoyed enough to care or something else entirely. However, not doing these things is (likely) a sign that currently they are not in the state of creating reliable systems - at least none reliable enough for what they are doing. It would be perfectly fine if they ran a web shop or something, but if as experienced many other companies rely on you being up or their stuff fails, then maybe you should not run a company with products like "Always Online".
[1] And should make you adapt the process of analyzing issues. Eg. making sure config changes are "very loud" in monitoring. It's one of the most easily tracked thing that can go wrong, and can relatively easily be mapped to a point in time compared to many other things.
Does their ring based rollout really truly have to be 0->100% in a few seconds?
I don’t really buy this requirement. At least make it configurable with a more reasonable default for “routine” changes. E.g. ramping to 100% over 1 hour.
As long as that ramp rate is configurable, you can retain the ability to respond fast to attacks by setting the ramp time to a few seconds if you truly think it’s needed in that moment.
The configuration file is updated every five minutes, so clearly they have some past experience where they’ve decided an hour is too long. That said, even a roll out over five minutes can be helpful.
This was not about DDoS defense but the Bot Management feature, which is a paid Enterprise-only feature not enabled by default to block automated requests regardless of whether an attack is going on.
Maybe, but in that case maybe have some special casing logic to detect that yes indeed we're under a massive DDOS at this very moment, do a rapid rollout of this thing that will mitigate said DDOS. Otherwise use the default slower one?
Of course, this is all so easy to say after the fact..
> Their bot management system is designed to push a configuration out to their entire network rapidly.
Once every 5m is not "rapidly". It isn't uncommon for configuration systems to do it every few seconds [0].
> While it’s certainly useful to examine the root cause in the code.
Believe the issue is as much an output from a periodic run (clickhouse query) caused by (on the surface, an unrelated change) causing this failure. That is, the system that validated the configuration (FL2) was different to the one that generated it (ML Bot Management DB).
Ideally, it is the system that vends a complex configuration that also vends & tests the library to consume it, or the system that consumes it, does so as if it was "tasting" the configuration first before devouring it unconditionally [1].
Of course, as with all distributed system failures, this is all easier said and done in hindsight.
Isn't rapidly more of how long it takes to get from A to Z rather than how often it is performed? You can push out a configuration update every fortnight but if it goes through all of your global servers in three seconds, I'd call it quite rapid.
I don't understand why they didn't validate and sanitize the new config file revision.
If bad(whatever that reason is) throw an error and revert back to previous version. You don't need to take down the whole internet for that.
Same as for almost every bug I think: the dev in question hadn't considered that the input could be bad in the way that it turned out to be. Maybe they were new, or maybe they hadn't slept much because of a newborn baby, or maybe they thought it was a reasonable assumption that there would never be more than 200 ML features in the array in question. I don't think this developer will ever make the same mistake again at least.
Let those who have never written a bug before cast the first stone.
> Maybe they were new, or maybe they hadn't slept much because of a newborn baby
Reminds me of House of Dynamite, the movie about nuclear apocalypse that really revolves around these very human factors. This outage is a perfect example of why relying on anything humans have built is risky, which includes the entire nuclear apparatus. “I don’t understand why X wasn’t built in such a way that wouldn’t mean we live in an underground bunker now” is the sentence that comes to mind.
I don't think this is an error originating from a single human. At CF scale I'd expect that multiple humans saw that code and gave it a pass.
Rust or not, but an experienced dev could have seen this can lead to issues. Panicking without restoring a healthy state is just not an option in this case. They *know* that.
I guess you are right, likely a social issue, but certainly not a single exhausted parent.
> I don't understand why they didn't validate and sanitize the new config file revision.
The new config file was not (AIUI) invalid (syntax-wise) but rather too big:
> […] That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
> The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.
Exactly the right take. Even when you want to have rapid changes on your infra, do it at least by region. You can start with the region where the least amount of users are impacted and if everything is fine, there is no elevated number of crashes for example, you can move forward. It was a standard practice at $RANDOM_FAANG when we had such deployments.
Thank you. I am sympathetic to CF’s need to deploy these configs globally fast and don’t think slowing down their DDoS mitigation is necessarily a good trade off. What I am saying is this presents a bigger reliability risk and needs correspondingly fine crafted observability around such config changes and a rollback runbook. Greater risk -> greater attention.
But the rapid deployment mechanism for bot features wasn’t where the bug was introduced.
In fact, the root bug (faulty assumption?) was in one or more SQL catalog queries that were presumably written some time ago.
(Interestingly the analysis doesn’t go into how these erroneous queries made it into production OR whether the assumption was “to spec” and it’s the security principal change work that was faulty. Seems more likely to be the former.)
I've also led a team of Incident Commanders at a FAANG.
If this was a routine config change, I could see how it could take 2 hours to start the mediation plan. However they should have dashboards that correlate config setting changes with 500 errors (or equivalent). It gets difficult when you have many of of these going out at the same time and they are slowly rolled out.
The root cause document is mostly for high level and the public. The details on this specific outage will be in a internal document with many action items, some of them maybe quarter long projects including fixing this specific bug and maybe some linter/monitor to prevent it from happening again.
In a productive way, this view also shifts the focus to improving the system (visibility etc), empowering the team, rather than focusing on the code which broke (probably strikes fear in the individuals, to do anything!)
The "coding error" is a somewhat deliberate choice to fail eagerly that is usually safe but doesn't align with the need to do something (propagation of the configuration file) without failing.
I'm sure that there are misapplied guidelines to do that instead of being nice to incoming bot management configuration files, and someone might have been scolded (or worse) for proposing or attempting to handle them more safely.
I would say that whilst this is a good top down view, that `.unwrap()` should have been caught at code-review and not allowed. Clippy rule could have saved a lot of money.
That and why the hell wasn't their alerting showing up colossal amount of panics in their bot manager thing?
Yes the lack of observability is really the disturbing bit here. You have panics in a bunch of your core infrastructure, you would expect there to be a big red banner on the dashboard that people look at when they first start troubleshooting an incident.
This is also a pretty good example why having stack traces by default is great. That error could have been immediately understood just from a stack trace and a basic exception message.
You can write the safest code in the world, but if you're shipping config changes globally every few minutes without a robust rollback plan or telemetry that pinpoints when things go sideways, you're flying blind
Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code.
And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.
Pretty much - the time spent ruling out the hypothesis that it was a cyberattack would have been time spent investigating the uptick in deliberately written error logs, since you would expect alerts to be triggered if those exceed a threshold.
I imagine it would also require less time debugging a panic. That kind of breadcrumb trail in your logs is a gift to the future engineer and also customers who see a shorter period of downtime.
I actually have to do this for programs that runs in bare metal. You can't afford to have nondeterministic panic like this. If things really gone wrong you'd have a watchdog and health checker to verify the state of program.
It is just 2 different layers. Of course the code is also a problem, if it is in fact as the GP describes it. You are taking the higher level view, which is the second layer of dealing with not only this specific mistake, but also other mistakes, that can be related to arbitrary code paths.
Both are important, and I am pretty sure, that someone is gonna fix that line of code pretty soon.
The bot is efficient. This is by design. It will push out mistakes just as efficiently as it pushes out good changes. Good or bad... the plane of control is unchanged.
This is the danger of automated control systems. If they get hacked or somehow push out bad things (CloudStrike), they will have complete control and be very efficient.
It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic.
It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this.
Yes? Funnily enough, I don't often use indexed access in Rust. Either I'm looping over elements of a data structure (in which case I use iterators), or I'm using an untrusted index value (in which case I explicitly handle the error case). In the rare case where I'm using an index value that I can guarantee is never invalid (e.g. graph traversal where the indices are never exposed outside the scope of the traversal), then I create a safe wrapper around the unsafe access and document the invariant.
If that's the case then hats off. What you're describing is definitely not what I've seen in practice. In fact, I don't think I've ever seen a crate or production codebase that documents infallibility of every single slice access. Even security-critical cryptography crates that passed audits don't do that. Personally, I found it quite hard to avoid indexing for graph-heavy code, so I'm always on the lookout for interesting ways to enforce access safety. If you have some code to share that would be very interesting.
Usually you'd want to write almost all your slice or other container iterations with iterators, in a functional style.
For the 5% of cases that are too complex for standard iterators? I never bother justifying why my indexes are correct, but I don't see why not.
You very rarely need SAFETY comments in Rust because almost all the code you write is safe in the first place. The language also gives you the tool to avoid manual iteration (not just for safety, but because it lets the compiler eliminate bounds checks), so it would actually be quite viable to write these comments, since you only need them when you're doing something unusual.
I didn't restate the context from the code we're discussing: it must not panic. If you don't care if the code panics, then go ahead and unwrap/expect/index, because that conforms to your chosen error handling scheme. This is fine for lots of things like CLI tools or isolated subprocesses, and makes review a lot easier.
So: first, identify code that cannot be allowed to panic. Within that code, yes, in the rare case that you use [i], you need to at least try to justify why you think it'll be in bounds. But it would be better not to.
There are a couple of attempts at getting the compiler to prove that code can't panic (e.g., the no-panic crate).
For iteration, yes. But there's other cases, like any time you have to deal with lots of linked data structures. If you need high performance, chances are that you'll have to use an index+arena strategy. They're also common in mathematical codebases.
Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere.
Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/
But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handle the error and try to recover, or if impossible, at least do `expect("damn it, how did this happen")`.
There is a prevailing mentality that LLMs make it easy to become productive in new languages, if you are already proficient in one. That's perhaps true until you suddenly bump up against the need to go beyond your superficial understanding of the new language and its idiosyncrasies. These little collisions with reality occur until one of them sparks an issue of this magnitude.
In theory, experienced human code reviewers can course correct newer LLM-guided devs work before it blows up. In practice, reviewers are already stretched thin and submitters absolute to now rapidly generate more and more code to review makes that exhaustion effect way worse. It becomes less likely they spot something small but obvious amongst the haystack of LLM generated code bailing there way.
Dunno, I think the alternatives have their own pretty significant downsides. All would require front loading more in-depth understanding of error handling and some would just be quite a bit more verbose.
IMO making unwrap a clippy lint (or perhaps a warning) would be a decent start. Or maybe renaming unwrap.
This strikes me as a culture issue more than one of language.
A tenet of systems code is that every possible error must be handled explicitly and exhaustively close to the point of occurrence. It doesn’t matter if it is Rust, C, etc. Knowing how to write systems code is unrelated to knowing a systems language. Rust is a systems language but most people coming into Rust have no systems code experience and are “holding it wrong”. It has been a recurring theme I’ve seen with Rust development in a systems context.
C is pretty broken as a language but one of the things going for it is that it has a strong systems code culture surrounding it that remembers e.g. why we do all of this extra error handling work. Rust really needs systems code practice to be more strongly visible in the culture around the language.
> at least do `expect("damn it, how did this happen")`
That gives you the same behavior as unwrap with a less useful error message though. In theory you can write useful messages, but in practice (and your example) expect is rarely better than unwrap in modern rust
I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality.
Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled.
If you need the value, you need to handle the case that it’s not available explicitly. You need to define your error path(s)
This is a failure caused by lazy Rust programming and not relying on the language's design features.
It's a shame this code can even be written. It is surprising and escapes the expected safety of the language.
I'm terrified of some dependency using unwrap() or expect() and crashing for something entirely outside of my control.
We should have an opt-in strict Cargo.toml declaration that forbids compilation of any crate that uses entirely preventable panics. The only panics I'll accept are those relating to memory allocation.
This is one of the sharpest edges in the language, and it needs to be smoothed away.
> If you have to use unwrap, your types do not match your problem
The problem starts with Rust stdlib. It panics on allocation failure. You expect Rust programmers to look at stdlib and not imitate it?
Sure, you can try to taboo unwrap(), but 1) it won't work, and 2) it'll contort program design in places where failure really is a logic bug, not a runtime failure, and for which unwrap() is actually appropriate.
The real solution is to go back in time, bonk the Rust designers over the head with a cluebat, and have them ship a language that makes error propagation the default and syntactically marks infallible cleanup paths --- like C++ with noexcept.
It's the same blind spot people have to Java's checked exceptions. People commonly resort to Pokemon exception handling and either blindly ignoring or rethrowing as a runtime exception. When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.
"Checked Exceptions Are Actually Good" gang, rise up! :p
I think adoption would have played out very different if there had only been some more syntactic-sugar. For example, an easy syntax for saying: "In this method, any (checked) DeepException e that bubbles up should immediately be replaced by a new (checked) MylayerException(e) that contains the original one as a cause.
We might still get lazy programmers making systems where every damn thing goes into a generic MylayerException, but that mess would still be way easier to fix later than a hundred scattered RuntimeExceptions.
Exception handling would be better than what we're seeing here.
The problem is that any non-trivial software is composition, and encapsulation means most errors aren't recoverable.
We just need easy ways to propagate exceptions out to the appropriate reliability boundary, ie. the transaction/ request/ config loading, and fail it sensibly, with an easily diagnosable message and without crashing the whole process.
C# or unchecked Java exceptions are actually fairly close to ideal for this.
The correct paradigm is "prefer throw to catch" -- requiring devs to check every ret-val just created thousands of opportunities for mistakes to be made.
By contrast, a reliable C# or Java version might have just 3 catch clauses and handle errors arising below sensibly without any developer effort.
I'm with you! Checked exceptions are actually good and the hate for them is super short sighted. The exact same criticisms levied at checked exceptions apply to static typing in general, but people acknowledge the great value static types have for preventing errors at compile time. Checked exceptions have that same value, but are dunked on for some reason.
There is also the problem that they decided to make all references nullable, so `NullPointerException`s could appear everywhere. This "forced" them to introduce the escape hatch of `RuntimeException`, which of course was way overused immediately, normalizing it.
It's a lot lighter: a stack trace takes a lot of overhead to generate; a result has no overhead for a failure. The overhead (panic) only comes once the failure can't be handled. (Most books on Java/C# don't explain that throwing exceptions has high performance overhead.)
Exceptions force a panic on all errors, which is why they're supposed to be used in "exceptional" situations. To avoid exceptions when an error is expected, (eof, broken socket, file not found,) you either have to use an unnatural return type or accept the performance penalty of the panic that happens when you "throw."
In Rust, the stack trace happens at panic (unwrap), which is when the error isn't handled. IE, it's not when the file isn't found, it's when the error isn't handled.
Exceptions do not force panic at all. In most practical situations, an exception unhandled close to where it was thrown will eventually get logged. It's kind of a "local" panic, if you will, that will terminate the specific function, but the rest of the program will remain unaffected. For example, a web server might throw an exception while processing a specific HTTP request, but other HTTP requests are unaffected.
Throwing an exception does not necessarily mean that your program is suddenly in an unsupported state, and therefore does not require terminating the entire program.
> it's essentially a checked exception without a stack trace
In theory, theory and practice are the same. In practice...
You can't throw a checked exception in a stream, this fact actually underlines the key difference between an exception and a Result: Result is in return position and exceptions are a sort of side effect that has its own control flow. Because of that, once your method throws an Exception or you are writing code in a try block that catches an exception, you become blind to further exceptions of that type, even if you might be able to or required to fix those errors. Results are required to be handled individually and you get syntactic sugar to easily back propagate.
It is trivial to include a stack trace, but stack traces are really only useful for identifying where something occurred, and generally what is superior is attaching context as you back propagate which trivially occurs with judicious use of custom error types with From impls. Doing this means that the error message uniquely defines the origin and paths it passed through without intermediate unimportant stack noise. With exceptions you would always need to catch each exception and rethrow a new exception containing the old to add contextual information, then to avoid catching to much you need variables that will be initialized inside the try block defined outside of the try block. So stack traces are basically only useful when you are doing Pokemon exception handling.
> When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.
It's not a checked exception without a stack trace.
Rust doesn't have Java's checked or unchecked exception semantics at the moment. Panics are more like Java's Errors (e.g. OOM error). Results are just error codes on steroids.
checked exceptions failed because when used properly they fossilize method signatures. they're fine if your code will never be changed and they're fine when you control 100% of users of the throwing code. if you're distributing a library... no bueno.
That’s just not true. They required that you use hierarchical exception types and define your own library exception type that you declare at the boundary.
The same is required for any principled error handling.
That's kind of what I'm saying with the blind spot comment. The words "unwrap" and "expect" should be just as much a scary red flag as the word "panic", but for some reason it seems a lot of people don't see them that way.
Even in lowly Java, they later added to Optional the orElseThrow() method since the name of the get() method did not connote the impact of unwrapping an empty Optional.
I've found both methods very useful. I'm using `get()` when I've checked that the value is present and I don't expect any exceptions. I'm using `orElseThrow()` when I actually expect that value can be absent and throwing is fine. Something like
if (userOpt.isPresent()) {
var user = userOpt.get();
var accountOpt = accountRepository.selectAccountOpt(user.getId());
var account = accountOpt.orElseThrow();
}
Idea checks it by default and highlights if I've used `get()` without previous check. It's not forced at compiler level, but it's good enough for me.
The `unsafe` keyword means something specific in Rust, and panicking isn't unsafe by Rust's definition. Sometimes avoiding partial functions just isn't feasible, and an unwrap (or whatever you want to call the method) is a way of providing a (runtime-checked) proof to the compiler that the function is actually total.
Than they are going to write None | Err => yolo() that has the same impact. It is not the syntax or the semantic meaning is the problem here but the fact that there is no monitoring around the elevated error counts after a deployment.
Software engineers tend to get stuck in software problems and thinking that everything should be fixed in code. In reality there are many things outside of the code that you can do to operate unreliable components safely.
Really not! This is a huge faceplant for writing things in Rust. If they had been writing their code in Java/Kotlin instead of Rust, this outage either wouldn't have happened at all (a failure to load a new config would have been caught by a defensive exception handler), or would have been resolved in minutes instead of hours.
The most useful thing exceptions give you is not static compile time checking, it's the stack trace, error message, causal chain and ability to catch errors at the right level of abstraction. Rust's panics give you none of that.
Look at the error message Cloudflare's engineers were faced with:
thread fl2_worker_thread panicked: called Result::unwrap() on an Err value
That's useless, barely better than "segmentation fault". No wonder it took so long to track down what was happening.
A proxy stack written in a managed language with exceptions would have given an error message like this:
com.cloudflare.proxy.botfeatures.TooManyFeaturesException: 200 > 60
at com.cloudflare.proxy.botfeatures.FeatureLoader(FeatureLoader.java:123)
at ...
and so on. It'd have been immediately apparent what went wrong. The bad configs could have been rolled back in minutes instead of hours.
In the past I've been able to diagnose production problems based on stack traces so many times I was been expecting an outage like this ever since the trend away from providing exceptions in new languages in the 2010s. A decade ago I wrote a defense of the feature and I hope we can now have a proper discussion about adding exceptions back to languages that need them (primarily Go and Rust):
That has nothing to do with exceptions, just the ability to unwind the stack. Rust can certainly give you a backtrace on panics; you don’t even have to write a handler to get it. I would find it hard to believe Cloudflare’s services aren’t configured to do it. I suspect they just didn’t put the entire message in the post.
tldr: Capturing a backtrace can be a quite expensive runtime operation, so the environment variables allow either forcibly disabling this runtime performance hit or allow selectively enabling it in some programs.
Alternatively you can look at actually innovative programming languages to peek at the next 20 years of innovation.
I am not sure that watching the trendy forefront successfully reach the 1990s and discuss how unwrapping Option is potentially dangerous really warm my heart. I can’t wait for the complete meltdown when they discover effect systems in 2040.
To be more serious, this kind of incident is yet another reminder that software development remains miles away from proper engineering and even key providers like Cloudfare utterly fail at proper risk management.
Celebrating because there is now one popular language using static analysis for memory safety feels to me like being happy we now teach people to swim before a transatlantic boat crossing while we refuse to actually install life boats.
To me the situation has barely changed. The industry has been refusing to put in place strong reliability practices for decades, keeps significantly under investing in tools mitigating errors outside of a few fields where safety was already taken seriously before software was a thing and keeps hiding behind the excuse that we need to move fast and safety is too complex and costly while regulation remains extremely lenient.
I mean this Cloudfare outage probably cost millions of dollars of damage in aggregate between lost revenue and lost productivity. How much of that will they actually have to pay?
Let's try to make effect systems happen quicker than that.
> I mean this Cloudfare outage probably cost millions of dollars of damage in aggregate between lost revenue and lost productivity. How much of that will they actually have to pay?
Probably nothing, because most paying customers of cloudflare are probably signing away their rights to sue Cloudflare for damages by being down for a while when they purchase Cloudflare's services (maybe some customers have SLAs with monetary values attached, I dunno). I honestly have a hard time suggesting that those customers are individually wrong to do so - Cloudflare isn't down that often, and whatever amount it cost any individual customer by being down today might be more than offset by the DDOS protection they're buying.
Anyway if you want Cloudflare regulated to prevent this, name the specific regulations you want to see. Should it be illegal under US law to use `unwrap` in Rust code? Should it be illegal for any single internet services company to have more than X number of customers? A lot of the internet also breaks when AWS goes down because many people like to use AWS, so maybe they should be included in this regulatory framework too.
I find myself in the odd position of agreeing with you both.
That we’re even having this discussion is a major step forward. That we’re still having this discussion is a depressing testament to how slow slowly the mainstream has adopted better ideas.
> I can’t wait for the complete meltdown when they discover effect systems in 2040
Zig is undergoing this meltdown. Shame it's not memory safe. You can only get so far in developing programming wisdom before Eternal September kicks in and we're back to re-learning all the lessons of history as punishment for the youthful hubris that plagues this profession.
It's not about whether you should ban unwrap() in production. You shouldn't. Some errors are logic bugs beyond which a program can't reasonably continue. The problem is that the language makes it too easy for junior developers (and AI!) to ignore non-logic-bug problems with unwrap().
Programmers early in their careers will do practically anything to avoid having to think about errors and they get angry when you tell them about it.
> In production code an unwrap or expect should be reviewed exactly like a panic.
An unwrap should never make it to production IMHO. It's fine while prototyping, but once the project gets closer to production it's necessary to just grep `uncheck` in your code and replace those that can happen with a proper error management and replace those that cannot happen with `expect`, with a clear justification of why they cannot happen unless there's a bug somewhere else.
I would say, sure, if you feel the same way about panic calls making to production. In other words, review all of them the same way. Because writing unwrap/expect is exactly the same as writing “if error, panic”.
I don't understand your point: panic! is akin to expect: you think about it consciously, use it explicitly and you write down a panic message explaining its rational.
Isn't the point of this article that pieces of infrastructure don't go down to root causes, but due to bad combinations of components that are correct individually? After reading "engineering a safer world", I find root cause analysis rather reductionistic, because it wasn't just an unwrap, it was that the payload was larger than normal, because of a query that didn't select by database, because a clickhouse made more databases visible. Hard to say "it was just due to an unwrap" imo. Especially in terms of how to fix an issue going forwards. I think the article lists a lot of good ideas, that aren't just "don't unwrap", like enabling more global kill switches for features, or eliminating the ability for core dumps or other error reports to overwhelm system resources.
You're right. A good postmortem/root cause analysis would START from "unwrap" and continue from there.
You might start with a basic timeline of what happened, then you'd start exploring: why did this change affect so many customers (this would be a line of questioning to find a potential root cause), why did it take so long to discover or recover (this might be multiple lines of questioning), etc.
> This is the multi-million dollar .unwrap() story.
That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.
> If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.
It wouldn't, not meaningfully. The outage was caused by change in how they processed the queries. They had no way to observe the changes, nor canaries to see that change is killing them. Plus, they would still need to manually feed and restart services that ingested bad configs.
`expect` would shave a few minutes; you would still spend hours figuring out and fixing it.
Granted, using expect is better, but it's not a silver bullet.
A billion alerts in DD/Sentry/whatever saying the exact problem that coincide with the exact graph of failures would probably be helpful if someone looked at them.
In general for unexpected errors like these the internal function should log the error, and I assume it was either logged or they can quickly deduce reason based on the line number.
I'm not sure if this is serious or not, but to take it at face value: the value of this sort of thing in Rust is not that it prevents crashes altogether but rather that it prevents _implicit_ failures. It forces a programmer to make the explicit choice of whether to crash.
There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this). But when you can't, we leave a comment explaining why it's safe.
The language semantics do not surface `unwrap` usage or make any guarantees. It should be limited to use in `unsafe` blocks.
> There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this). But when you can't, we leave a comment explaining why it's safe.
I would prefer the boiler plate of a match / if-else / if let, etc. to call attention to it. If you absolutely must explicitly panic. Or better - just return an error Result.
It doesn't matter how smart your engineers are. A bad unwrap can sneak in through refactors, changing business logic, changing preconditions, new data, etc.
>> This is the multi-million dollar .unwrap() story.
> That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.
Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assuming `features.append_with_names` would never fail. Instead, the routine likely should have worked within `Result`.
> Instead, the routine likely should have worked within `Result`.
But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same.
Maybe you're saying "it's better to be explicit", as a broad generalization I don't disagree with that.
But that has nothing to do with the actual bug here, which was that the invariant failed. How they choose to implement checking and failing the invariant in the semantics of the chosen language is irrelevant.
>> Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assuming `features.append_with_names` would never fail. Instead, the routine likely should have worked within `Result`.
> But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same.
I agree it is an error, but disagree that it should be a fatal error at that location. The reason being is the method defining the offending `unwrap` construct produces a `Result`, which is fully capable of representing any error `features.append_with_names` could produce.
> But that has nothing to do with the actual bug here, which was that the invariant failed.
The bug is by invoking `unwrap` the process crashed. To the degree that Cloudfare had a massive outage.
Had the logic been such that a `Result` representing this error condition activated an alternate workflow to handle the error (perhaps by logging it, emitting a notification event alerting SRE's, transitioning into a failure mode, or all of these options), then a global outage might have been averted.
Which makes:
> How they choose to implement checking and failing the invariant in the semantics of the chosen language is irrelevant.
I'm not completely sure I agree. I mean, I do agree about the .unwrap() culture being a bug trap. But I don't think this example qualifies.
The root cause here was that a file was mildly corrupt (with duplicate entries, I guess). And there was a validation check elsewhere that said "THIS FILE IS TOO BIG".
But if that's a validation failure, well, failing is correct? What wasn't correct was that the failure reached production. What should have happened is that the validation should have been a unified thing and whatever generated the file should have flagged it before it entered production.
And that's not an issue with function return value API management. The software that should have bailed was somewhere else entirely, and even there an unwrap explosion (in a smoke test or pre-release pass or whatever) would have been fine.
It sounds to me like there was validation, but the system wasn't designed for the validation to ever fail - at which point crashing is the only remaining option. You've essentially turned it into an assertion error rather than a parsing/validation error.
Ideally every validation should have a well-defined failure path. In the case of a config file rotation, validation failure of the new config could mean keeping the old config and logging a high-priority error message. In the case of malformed user-provided data, it might mean dropping the request and maybe logging it for security analysis reasons. In the case of "pi suddenly equals 4" checks the most logical approach might be to intentionally crash, as there's obviously something seriously wrong and application state has corrupted in such a way that any attempt to continue is only going to make things worse.
But in all cases there's a reason behind the post-validation-failure behavior. At a certain point leaving it up to "whatever happens on .unwrap() failure" isn't good enough anymore.
This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a normal thing to do. There's no graceful handling (beyond a nice error message) a program like Nginx could do on a syntax error in its config.
The real issue is further up the chain where the malformed feature file got created and deployed without better checks.
I do not think that if the bot detection model inside your big web proxy has a configuration error it should panic and kill the entire proxy and take 20% of the internet with it. This is a system that should fail gracefully and it didn't.
> The real issue
Are there single "real issues" with systems this large? There are issues being created constantly (say, unwraps where there shouldn't be, assumptions about the consumers of the database schema) that only become apparent when they line up.
I don't know too much about how the feature file distribution works but in the event of failure to read a new file, wouldn't logging the failure and sticking with the previous version of the file be preferable?
That's exactly the point (ie just prior to distribution) where a simple sanity check should have been run and the config replacement/update pipeline stopped on failure. When they introduced the 200 entry limit memory optimised feature loader it should have been a no-brainer to insert that sanity check in the config production pipeline.
Or even truncating the features to their limit and alerting through logs that there is likely performance degradation in their Bot Management.
I'm really confused how so many people are finding it acceptable to bring down your entire reverse-proxy because the length of feature sets for the ML model in one of your components was longer than expected.
One feature failing like this should probably log the error and fail closed. It shouldn't take down everything else in your big proxy that sits in front of your entire business.
Yea, Rust is safe but it’s not magic. However Nginx doesn’t panic on malformed config. It exits with hopefully a helpful error code and message. The question is then could the cloudflare code have exited cleanly in a way that made recovery easier instead of just straight panicking.
Would expect with a message meet that criteria of exiting with a more helpful error message? From the postmortem it seems to me like they just didn’t know it even was panicing
> However Nginx doesn’t panic on malformed config. It exits with hopefully a helpful error code and message.
The thing I dislike most about Nginx is that if you are using it as a reverse proxy for like 20 containers and one of them is up, the whole web server will refuse to start up:
nginx: [emerg] host not found in upstream "my-app"
Obviously making 19 sites also unavailable just because one of them is caught in a crash loop isn't ideal. There is a workaround involving specifying variables, like so (non-Kubernetes example, regular Nginx web server running in a container, talking to other containers over an internal network, like Docker Compose or Docker Swarm):
location / {
resolver 127.0.0.11 valid=30s; # Docker DNS
set $proxy_server my-app;
proxy_pass http://$proxy_server:8080/;
proxy_redirect default;
}
Sadly, if you try to use that approach, then you just get:
nginx: [emerg] "proxy_redirect default" cannot be used with "proxy_pass" directive with variables
Sadly, switching the redirect configuration away from the default makes some apps go into a redirect loop and fail to load: mostly legacy ones, where Firefox shows something along the lines of "The page isn't redirecting properly". It sucks especially badly if you can't change the software that you just need to run and suddenly your whole Nginx setup is brittle. Apache2 and Caddy don't have such an issue.
That's to say that all software out there has some really annoying failure modes, even is Nginx is pretty cool otherwise.
To be fair, this failed in the non-rust path too because the bot management returned that all traffic was a bot. But yes, FL2 needs to catch panics from individual components but I’m not sure if failing open is necessarily that much better (it was in this case but the next incident could easily be the result of failing open).
But more generally you could catch the panic at the FL2 layer to make that decision intentional - missing logic at that layer IMHO.
Catching panic probably isn’t a great idea if there’s any unsafe code in the system. (Do the unsafe blocks really maintain heap invariants if across panics?)
Unsafe blocks have nothing to do with it. Yes - they maintain all the same invariants as safe blocks or those unsafe blocks are unsound regardless of panics. But there’s millions of way to architect this (eg a supervisor process that notices which layer in FL2 is crashing and just completely disables that layer when it starts up the proxy again. There’s challenges here because then you have to figure out what constitutes a perma crashing (eg what if it’s just 20% of all sites? Do you disable?). And in the general case you have the fail open/fail close decision anyway which you should just annotate individual layers with.
But the bigger change is to make sure that config changes roll out gradually instead of all at once. That’s the source of 99% of all widespread outages
I think the parent is implying that the panic should be "caught" via a supervisor process, Erlang-style, rather than implying the literal use of `catch_unwind` to resume within the same process.
Supervisor is the brutalist way. But catch_unwind may be needed for perf and other reasons.
But ultimately it’s not the panic that’s the problem but a failure to specify how panics within FL2 layers should be handled; each layer is at least one team and FL2’s job is providing a safe playground for everyone to safely coexist regardless of the misbehavior of any single component
But as always such failures are emblematic of multiple things going wrong at once. You probably want to end up using both catch_unwind for the typical case and the supervisor for the case where there’s a segfault in some unsafe code you call or native library you invoke.
I also mention the fundamental tension of do you want to fail open or closed. Most layers should probably fail open. Some layers (eg auth) it’s safer to fail closed.
The unwrap should be replaced by code that creates enough alerting to make a P0 incidident from their canary deployment immediately.
OR even, the bot code crashing should itself be generating alerts.
Canary deployment would be automatically rolled back until P0 incident resolved.
All of this could probably have happened and contained at their scale in less than a minute as they would likely generate enough "omg the proxy cannot handle its config" alerts off of a deployment of 0.001% near immediately.
Agreed - a big question why the file wasn’t test driven in staging and progressively rolled out. And also what alerting was missing within FL2 that they couldn’t pinpoint the unwrap instantly.
I'm not a fan of rust, but I don't think that is the only takeaway. All systems have assumptions about their input and if the assumption is violated, it has to be caught somewhere. It seems like it was caught too deep in the system.
Maybe the validation code should've handled the larger size, but also the db query produced something invalid. That shouldn't have ever happened in the first place.
> It seems like it was caught too deep in the system.
Agreed, that's also my takeaway.
I don't see the problem being "lazy programmers shouldn't have called .unwrap()". That's reductive. This is a complex system and complex system failures aren't monocausal.
The function in question could have returned a smarter error rather than panicking, but what then? An invariant was violated, and maybe this system, at this layer, isn't equipped to take any reasonable action in response to that invariant violation and dying _is_ the correct thing to do.
But maybe it could take smarter action. Maybe it could be restarted into a known good state. Maybe this service could be supervised by another system that would have propagated its failure back to the source of the problem, alerting operators that a file was being generated in such a way that violated consumer invariants. Basically, I'm describing a more Erlang model of failure.
Regardless, a system like this should be able to tolerate (or at least correctly propagate) a panic in response to an invariant violation.
The takeaway here isn’t about Rust itself, but that the Rust marketing crew’s claims that we constantly read on HN and elsewhere about the Result type magically saving you from making mistakes is not a good message to send.
They would also tell you that .unwrap() has no place in production code, and should receive as much scrutiny as an `unsafe` block in code review :)
The point of option is the crash path is more verbose and explicit than the crash-free path. It takes more code to check for NULL in C or nil in Go; it takes more code in Rust to not check for Err.
1. They don’t. There is presumably some hypothetical world where they would tell you if you start asking questions, but nobody buying into the sales pitch ever asks questions.
2. You’re getting confused by technology again. This isn’t about technology.
> Today, many friends pinged me saying Cloudflare was down. As a core developer of the first generation of Cloudflare FL, I'd like to share some thoughts.
> This wasn't an attack, but a classic chain reaction triggered by “hidden assumptions + configuration chains” — permission changes exposed underlying tables, doubling the number of lines in the generated feature file. This exceeded FL2's memory preset, ultimately pushing the core proxy into panic.
> Rust mitigates certain errors, but the complexity in boundary layers, data flows, and configuration pipelines remains beyond the language's scope. The real challenge lies in designing robust system contracts, isolation layers, and fail-safe mechanisms.
> Hats off to Cloudflare's engineers—those on the front lines putting out fires bear the brunt of such incidents.
> Technical details: Even handling the unwrap correctly, an OOM would still occur. The primary issue was the lack of contract validation in feature ingest. The configuration system requires “bad → reject, keep last-known-good” logic.
> Why did it persist so long? The global kill switch was inadequate, preventing rapid circuit-breaking. Early suspicion of an attack also caused delays.
> Why not roll back software versions or restart?
> Rollback isn't feasible because this isn't a code issue—it's a continuously propagating bad configuration. Without version control or a kill switch, restarting would only cause all nodes to load the bad config faster and accelerate crashes.
> Why not roll back the configuration?
> Configuration lacks versioning and functions more like a continuously updated feed. As long as the ClickHouse pipeline remains active, manually rolling back would result in new corrupted files being regenerated within minutes, overwriting any fixes.
This tweet thread invokes genuine despair in me. Do we really have to outsource even our tweets to LLMs? Really? I mean, I get spambots and the like tweeting mass-produced slop. But what compels a former engineer of the company in question to offer LLM-generated "insight" to the outage? Why? For what purpose?
* For clarity, I am aware that the original tweets are written in Chinese, and they still have the stench of LLM writing all over them; it's not just the translation provided in the above comment.
This particular excerpt is reeking of it with pretty much every line. I'll point out the patterns in the English translation, but all of these patterns apply cross-language.
"Classic/typical "x + y"", particularly when diagnosing an issue. This one is a really easy tell because humans, on aggregate, do not use quotation marks like this. There is absolutely no reason to quote these words here, and yet LLMs will do a combined quoted "x + y" where a human would simply write something natural like "hidden assumptions and configuration chains" without extraneous quotes.
> The configuration system requires “bad → reject, keep last-known-good” logic.
Another pattern with overeager usage of quotes is this ""x → y, z"" construct with very terse wording.
> This wasn't an attack, but a classic chain reaction
LLMs aggressively use "Not X, but Y". This is also a construct commonly used by humans, of course, but aside from often being paired with an em-dash, another tell is whether it actually contributes anything to the sentence. "Not X, but Y" is strongly contrasting and can add a dramatic flair to the thing being constrasted, but LLMs overuse it on things that really really don't need to be dramatised or contrasted.
> Rust mitigates certain errors, but the complexity in boundary layers, data flows, and configuration pipelines remains beyond the language's scope. The real challenge lies in designing robust system contracts, isolation layers, and fail-safe mechanisms.
Two lists of three concepts back-to-back. LLMs enjoy, love, and adore this construct.
> Hats off to Cloudflare's engineers—those on the front lines putting out fires bear the brunt of such incidents.
This kind of completely vapid, feel-good word soup utilising a heroic analogy for something relatively mundane is another tell.
And more broadly speaking, there's a sort of verbosity and emptiness of actual meaning that permeates through most LLM writing. This reads absolutely nothing like what an engineer breaking down an outage looks like. Like, the aforementioned line of... "Rust mitigates certain errors, but the complexity in boundary layers, data flows, and configuration pipelines remains beyond the language's scope. The real challenge lies in designing robust system contracts, isolation layers, and fail-safe mechanisms.". What is that actually communicating to you? It piles on technical lingo and high-level concepts in a way that is grammatically correct but contains no useful information for the reader.
Bad writing exists, of course. There's plenty of bad writing out there on the internet, and some of it will suffer from flaws like these even when written by a human, and some humans do like their em-dashes. But it's generally pretty obvious when the writing is taken on aggregate and you see recognisable pattern after pattern combined with em-dashes combined with shallowness of meaning combined with unnecessary overdramatisations.
Swift has implicit unwrap (!), and explicit unwrap (?).
I don't like to use implicit unwrap. Even things that are guaranteed to be there, I treat as explicit (For example, (self.view?.isEnabled ?? false), in a view controller, instead of self.view.isEnabled).
So what happens if it ends up being nil? How does your app react?
In this particular case, I would rather crash. It’s easier to spot in a crash report and you get a nice stack trace.
Silent failure is ultimately terrible for users.
Note: for the things I control I try to very explicitly model state in such a way as I never need to force unwrap at all. But for things beyond my control like this situation, I would rather end the program than continue with a state of the world I don’t understand.
Yeah @IBOutlets are generally the one thing that are allowed to be implicitly-unwrapped optionals. They go along with using storyboards & xibs files with Interface Builder. I agree that you really should just crash if you are attempting to access one and it is nil. Either you have done something completely incorrect with regards to initializing and accessing parts of your UI and want to catch that in development, or something has gone horribly, horribly, horribly with UIKit/AppKit and storyboard/xib files are not being loaded properly by the system.
A good tool for catching stuff during development, is the humble assert()[0]. We can use precondition()[1], to do the same thing, in ship code.
The main thing is, is to remain in control, as much as possible. Rather than let the PC leave the stack frame, throw the error immediately when it happens.
> Silent failure is ultimately terrible for users.
Agreed.
Unfortunately, crashes in iOS are “silent failures,” and are a loss of control.
What this practice does, is give me the option to handle the failure “noisily,” and in a controlled manner; even if just emitting a log entry, before calling a system failure. That can be quite helpful, in threading. Also, it gives me the option to have a valid value applied, if there’s a structural failure.
But the main reason that I do that with @IBOutlets, is that it forces me to acknowledge, throughout the rest of the code, that it’s an optional. I could always treat implicit optionals as if they were explicit, anyway. This just forces me to.
I have a bunch of practices that folks can laugh at, but my stuff works pretty effectively, and I sleep well.
Crashes are silent failures but as I mentioned: you can get a lot of your crashes reported via the App Store. This is why I prefer crashes in this situation: it gives me something actionable over silent failures on the client.
This is terrible. The whole reason they introduced this is because IBOutlets would get silently disconnected and then in the field a user would complain that a feature stopped working.
Crash early, crash often. Find the bugs and bad assumptions.
> without taking the time to understand why they do things, the way they do.
Oh I am aware. They do it because
A) they don’t have a mental model of correct execution. Events just happen to them with a feeling of powerlessness. So rather than trying to form one they just litter the code with cases things that might happen
> As I've gotten older, the sharp edges have been sanded off.
B) they have grown in bad organizations with bad incentives that penalize the appearance of making mistakes. So they learn to hide them.
For example there might be an initiative that rewards removing crashes in favor of silent error.
> This is textbook "parse, don't validate" anti-pattern.
How so? “Parse, don’t validate” implies converting input into typed values that prevent representation of invalid state. But the parsing still needs to be done correctly. An unchecked unwrap really has nothing to do with this.
In addition, it looks like this system wasn't on any kind of 1%/10%/50%/100% rollout gating. Such a rollout would trivially have shown the poison input killing tasks.
To me it reads like there was a gradual rollout of the faulty software responsible for generating the config files, but those files are generated on approximately one machine, then propogated across the whole network every 5 minutes.
> Bad data was only generated if the query ran on a part of the cluster which had been updated. As a result, every five minutes there was a chance of either a good or a bad set of configuration files being generated and rapidly propagated across the network.
It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems.
The file should be versioned and rollout of new versions should be staged.
(There is definitely a trade-off; often times in the security critical path, you want to go as fast as possible because changes may be blocking a malicious actor. But if you move too fast, you break things. Here, they had a potential poison input in the pathway for synchronizing this state and Murphy's Law suggests it was going to break eventually, so the question becomes "How much damage can we tolerate when it does?")
> It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems.
That feature file is generated every 5 minutes at all times; the change to permissions was rolled out gradually over the clickhouse cluster, and whether a bad version of that file was generated depended on whether the part of the cluster that had the bad permissions generated the file.
While this is true, I wish that Rust had more of a first-class support for `no_panic`. Every solution we do have is hacky. I wish that I could guarantee that there were no panic calls anywhere in a code path.
But I could screw it up in Go, if I made the same assumptions
fvs, err := features.AppendWithNames(..)
if err != nil {
// this will NEVER break
panic(err)
}
Ultimately I don't think language design can be the sole line of defence against system failures; it can only guide developers to think about error cases
Right, but the point isn't to make errors impossible; the point is to have them be 1) less likely to write, and 2) easier to spot on review.
People's biggest complaints about golang's errors:
1. You have to _TYPE_OUT_ what to do on EVERY.SINGLE.ERROR. SOO BOORING!
2. They clutter up the code and make it look ugly.
Rust is so much cleaner and more convenient (they say)! Just add ?, or .unwrap()!
Well, with ".unwrap()", you can type it fast enough that you're on to the next problem before it occurs to your brain to think about what to do if there is an error. Whereas, in golang, by the time you type in, "if err != nil {", you've broken the flow enough that now you're much more likely to be thinking, "Hmm, could this ever fail? What should we do if it does?" That break in flow is annoying, but necessary.
And ".unwrap()" looks so unassuming, it's easy to overlook on review; that "panic()" looks a lot more dangerous, and again, would be more likely to trigger a reviewer into thinking, "Wait, is it OK if this thing panics? Is this really so unlikely to happen?"
Renaming it `.unwrap_or_panic()` would probably help with both.
That may be me, but `.unwrap()` is much more obvious than `_`:
- it's literally written out that you're assuming it to be Ok
- there are no indications that the `_` is an error: it could very well be some other return value from the function. in your example, it could be the number of appended features, etc
That's why Go's error handling is indeed noisy: it's noise and you reduce noise by not handling errors. Rust's is terse yet verbose: if you add stuff it's because you're doing something wrong. You explicitly spelled out the error is being ignored.
> And it would be far more obvious that an error message is being ignored.
Haven't used Go so maybe I'm missing some consideration, but I don't see how ", _" is more obvious than ".unwrap()". If anything it seems less clear, since you need to check/know the function's signature to see that it's an error being ignored (wouldn't be the case for a function like https://pkg.go.dev/math#Modf).
I haven't been writing Rust for that long (about 2 years) but every time I see .unwrap() I read it as 'panic in production'. Clippy needs to have harder checks on unwrap.
If the error had been an exception instead of a result, could have bubbled up
I have been saying for years that Rust botched error handling in unfixable ways. I will go to the grave believing Rust fumbled.
The design of the Rust language encourages people to use unwrap() to turn foreseeable runtime problems into fatal errors. It's the path of least resistance, so people will take it.
Rust encourages developers to consider only the happy path. No wonder it's popular among people who've never had to deal with failure.
All of the concomitant complexity--- Result, ?, the test thing, anyhow, the inability for stdlib to report allocation failure --- is downstream of a fashion statement against exceptions Rust cargo-culted from Go.
The funniest part is that Rust does have exceptions. It just calls them panics. So Rust code has to deal with the ergonomic footgun of Result but pays anyway for the possibility of exceptions. (Sure, you can compile with panic=abort. You can't count on it.)
I could not be more certain that Rust should have been a language with exceptions, not Result, and that error objects are a gross antipattern we'll regret for decades.
Errors work just like exceptions especially if you use the ? operator and let the error bubble up the chain. This is the Rust equivalent of an unhandled exception and the ripcord being pulled.
In C++, functions are error-colored by default. You write "noexcept" if you want your function to be infallible-colored instead.
(You usually want to make a function infallible if you're using your noexcept function as part of a cleanup path or part of a container interface that allows for more optimizations of it knows certain container operations are infallible.)
Rust makes infallibility the syntactic default and makes you write Result to indicate fallibility. People often don't want to color their functions this way. Guess what happens when a programmer is six levels deep in infallible-colored function calls and does something that can fail.
.unwrap()
Guess what, in Rust, is fallible?
Mutex acquire.
Guess what you need to do often on infallible cleanup paths?
Not panicking code is tedious to write. It is not realistic to expect everything to be non panic. There is a reason that panicking exists in the first place.
Them calling unwrap on a limit check is the real issue imo. Everything that takes in external input should assume it is bad input and should be fuzz tested imo.
In the end, what is the point of having a limit check if you are just unwrapping on it
Using the question mark operator [1] and even adding in some anyhow::context goes a long way to being able to fail fast and return an Err rather then panicking.
Sure you need to handle Results all the way up the stack but it forces you to think about how those nested parts of your app will fail as you travel back up the stack.
By the way - does this discussion matter and were they wrong to use unwrap()?
The way they wrote the code means that having more than 200 features is a hard non-transient error - even if they recovered from it, it meant they'd have had the same error when the code got to the same place.
I'm sure when the process crashed, k8s restarted the pod or something - then it reran the same piece of code and crashed in the same place.
While I don't necessarily agree with crashing as business strategy, I don't think that doing anything other than either dropping the extra rules or allocating more memory - neither of which the original code was built to do (probably by design).
The code made the local hard assumption that there won't ever be more than 200 rules and its okay to crash if that count is exceeded.
If you design your code around an invariant never being violated (which is fine), you have to make it clear on a higher level that they did.
This isn't a Rust problem (though Rust does make it easy to do the wrong thing here imo)
Instead of crashing when applying the new config, it's more common to simply ignore the new config if it cannot be applied. You keep running in the last known good state. Operators then get alerts about the failures and can diagnose and resolve the underlying issue.
That's not always foolproof, e.g. a freshly (re)started process doesn't have any prior state it can fall back to, so it just hard crashes. But restarts are going to be rate limited anyways, so even then there is time to mitigate the issue before it becomes a large scale outage
Interesting to see Rust error handling flunk out in practice.
It may be that forcing handling at every call tends to makes code verbose, and devs insensitized to bad practice. And the diagnostic Rust provided seems pretty garbage.
There is bad practice here too -- config failure manifesting as request failure, lack of failing to safe, unsafe rollout, lack of observability.
Back to language design & error handling. My informed view is that robustness is best when only major reliability boundaries need to be coded.
This the "throw, don't catch" principle with the addition of catches on key reliability boundaries -- typically high-level interactions where you can meaningfully answer a failure.
For example, this system could have a total of three catch clauses "Error Loading Config" which fails to safe, "Error Handling Request" which answers 5xx, and "Socket Error" which closes the HTTP connection.
> It may be that forcing handling at every call tends to makes code verbose
Rust has a lot of helpers to make it less verbose, even that error they demonstrate could've been written in some form `...code()?` with `?` helper that would have propagated the error forwards.
However I do acknowledge that writing Error types is boring sometimes so people don't bother to change their error types and just unwrap. But even my dinghy little apps for my personal use I do simple serach `unwrap` and make sure I have as few as possible.
I don't understand how your takeaway is that this is a language flaw other than to assume that you have some underlying disdain for Rust. That's fine, but state it clearly please.
The end result would've been the exact same if they "handled" the error: a bunch of 500s. The language being used doesn't matter if an invariant in your system is broken.
Some languages and style guides simply forbid throwing exceptions without catching / proper recovery. Google C++ bans exceptions and the main mechanism for propogating errors is `absl::Status` which the caller has to check. Not familiar with Rust but it seems unwrap is such a thing that would be banned.
> Not familiar with Rust but it seems unwrap is such a thing that would be banned.
Panics aren't exceptions, any "panic" in Rust can be thought of as an abort of the process (Rust binaries have the explicit option to implement panics as aborts). Companies like Dropbox do exactly this in their similar Rust-based systems, so it wouldn't surprise me if Cloudflare does the same.
"Banning exceptions" wouldn't have done anything here, what you're looking for is "banning partial functions" (in the Haskell sense).
Yeah I know but isn't unwrap basically a trivial way to: (1) give up catching the exception/error (the E part in `Result<T, E>`) that the callee throws; and also (2) escalate it to the point that nothing can catch it. It has such a benign name.
Unwrap is used in places where in C++ you would just have undefined behavior. It wouldn't make any more sense to blanket ban it than it would ban ever dereferencing a pointer just in case its null - even if you just checked that it wasn't null.
Rust's foo: Option<&T> is rust's rough equivalent to C++'s const T* foo. The C++ *foo is equivalent to the rust unsafe{ *foo.unwrap_unchecked() }, or in safe code *foo.unwrap() (which changes the undefined behavior to a panic).
Rust's unwrap isn't the same as std::expected::value. The former panics - i.e. either aborts the program or unwinds depending on context and is generally not meant to be handled. The latter just throws an exception that is generally expected to be handled. Panics and exceptions use similar machinery (at least they can depending on compiler options) but they are not equivalent - for example nested panics in destructors always abort the program.
In code that isn't meant to crash `unwind` should be treated as a sign saying that "I'm promising that this will never happen", but just like in C++ where you promise that pointers you deference are valid and signed integers you add don't overflow making promises like that is a necessary part of productive programming.
Linting is not good enough. The compiler should refuse to compile the code without it marked with an explicit annotation. Too much Rust code is panic happy since using casual use of `unwrap` is perma-etched into everyone's minds by the amount of demo code out there using unwrap.
> same like chess, engine is better than human grandmaster because its solvable math field
Might be worth noting that your description of chess is slightly incorrect. Chess technically isn't solved in the sense that the optimal move is known for any arbitrary position is known; it's just that chess engines are using what amounts to a fancy brute force for most of the game and the combination of hardware and search algorithm produces a better result than the human brain does. As such, chess engines are still capable of making mistakes, even if actually exploiting them is a challenge.
This is why the Erlang/Elixir methodology of having supervision and letting things crash gracefully is so useful. You can either handle every single error gracefully or handle crashing gracefully - it's much easier and more realistic in large codebases to do the later.
This would not have helped: the code would crash before doing anything useful at all.
If anything, the "crash early" mentality may even be nefarious: instead of handling the error and keeping the old config, you would spin on trying to load a broken config on startup.
Continuing only makes sense for cases you know you can handle.
_In theory_ they could have used the old config, but maybe there are reasons that’s not possible in Cloudflare’s setup. Whether or not that’s an invariant violation or just an error that can be handled and recovered from is a matter of opinion in system design.
And crashing on an invariant violation is exactly the right thing to do rather than proceed in an undefined state.
Given the context and what the configuration file contains, I'd argue it's mission-critical for the software to keep running with the previous configuration. Otherwise you might shutdown the internet. Honestly, I'm pretty sure their pre-rewrite version had such logic, and it was forgotten or still on the TODO pile for the rewrite version.
At a previous job (cloud provider), we've had exactly this kind of issue, with exactly the same root cause. The entrypoint for the whole network had a set of rules (think a NAT gateway) that were reloaded periodically from the database. Someone rewrote that bit of plumbing from Python to Go. Someone else performed a database migration. Suddenly, the plumbing could not find the data, and pushed an empty file to prod. The rewrite lacked "if empty, do nothing and raise an alert", that the previous one had. I'll let you imagine what happened next :)
Say what you want exception haters, but at least in exceptions-as-default languages, the decision of a particular issues is fatal to the whole program can be decided centrally at a high level, and not every choice is forced to be up to individual discretion.
Safe things should be easy, dangerous things should be hard.
This .unwrap() sounds too easy for what it does, certainly much easier than having an entire try..catch block with an explicit panic. Full disclosure: I don't actually know Rust.
Any project has to reason about what sort of errors can be tolerated gracefully and which cannot. Unwrap is reasonable in scenarios you expect to never be reached, because otherwise your code will be full of all sorts of possible permutations and paths that are harder to reason about and may cascade into extremely nuanced or subtle errors.
Rust also has a version of unwrap called "expect" where you provide a string that logs why the unwrap occurred. It's similar, but for pieces of code that are crucial it could be a good idea to require all 'unwraps' to instead be 'expects' so that people at least are forced to write down a reason why they believe the unwrap can never be reached.
I wonder if similar to infrastructure resilience, code resilience is also required for critical services that can never go down? Instead of relying on a single implementation for a critical service, have multiple independent implementations in different languages.
Back when I was running my own DNS servers, I did always ensure that primary and secondary were running on different platforms and different software.
That's such a bad take after reading the article. If you're going to write a system that preallocates and is based on hard assumptions about max size - the panic/unwrap approach is reasonable.
The config bug reaching prod without this being caught and pinpointed immediately is the strange part.
It's reasonable when testing protocols exercise the panic scenario. This is the problem with punting on error recovery. Nobody checks faults that propagate across domains of responsibility.
This is a bummer. The unwrap()'ing function already returned a result and should have just propagated the error. Presumably the caller could have handled more sensibly than just panic'ing.
tokio default behavior within a task is to ignore panics, such as an Err/None unwrap, and only crash that task, so it's impact limited so that's nice, maybe that's where the snowblindness came from.
it'd be kinda hard to amend the clippy lints to ignore coroutine unwraps but still pipe up on system ones. i guess.
edit: i think they'd have to be "solely-task-color-flavored" so definitely probably not trivial to infer
At Facebook they name certain "escape hatch" functions in a way that inescapably make them look like a GIANT EYESORE. Stuff like DANGEROUSLY_CAST_THIS_TO_THAT, or INVOKE_SUPER_EXPENSIVE_ACTION_SEE_YOU_ON_CODE_REVIEW. This really drives home the point that such things must not be used except in rare extraordinary cases.
If unwrap() were named UNWRAP_OR_PANIC(), it would be used much less glibly. Even more, I wish there existed a super strict mode when all places that can panic are treated as compile-time errors, except those specifically wrapped in some may_panic_intentionally!() or similar.
React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED comes to mind. I did have to reach to this before, but it certainly works for keeping this out of example code and other things like reading other implementations without the danger being very apparent.
At some point it was renamed to __CLIENT_INTERNALS_DO_NOT_USE_OR_WARN_USERS_THEY_CANNOT_UPGRADE which is much less fun.
right and if the language designers named it UNWRAP_OR_PANIC() then people would rightfully be asking why on earth we can't just use a try-catch around code and have an easier life
But a panic can be caught and handled safely (e.g. via std:: panic tools). I'd say that this is the correct use case for exceptions (ask Martin Fowler, of all people).
There is already a try/catch around that code, which produces the Result type, which you can presumptuously .unwrap() without checking if it contains an error.
Instead, one should use the question mark operator, that immediately returns the error from the current function if a Result is an error. This is exactly similar to rethrowing an exception, but only requires typing one character, the "?".
...and you can? try-catch is usually less ergonomic than the various ways you can inspect a Result.
try {
data = some_sketchy_function();
} catch (e) {
handle the error;
}
vs
result = some_sketchy_function();
if let Err(e) = result {
handle the error;
}
Or better yet, compare the problematic cases where the error isn't handled:
data = some_sketchy_function();
vs
data = some_sketchy_function().UNWRAP_OR_PANIC();
In the former (the try-catch version that doesn't try or catch), the lack of handling is silent. It might be fine! You might just depend on your caller using `try`. In the latter, the compiler forces you to use UNWRAP_OR_PANIC (or, in reality, just unwrap) or `data` won't be the expected type and you will quickly get a compile failure.
What I suspect you mean, because it's a better argument, is:
which is fair, although how often is it really the right thing to let all the errors from 4 independent sources flow together and then get picked apart after the fact by inspecting `e`? It's an easier life, but it's also one where subtle problems constantly creep in without the compiler having any visibility into them at all.
Unwrap isn't a synonym for laziness, it's just like an assertion, when you do unwrap() you're saying the Result should NEVER fail, and if does, it should abort the whole process. What was wrong was the developer assumption, not the use of unwrap.
It also makes it very obvious in the code, something very dangerous is happening here. As a code reviewer you should see an unwrap() and have alarm bells going off. While in other languages, critical errors are a lot more hidden.
> What was wrong was the developer assumption, not the use of unwrap.
How many times can you truly prove that an `unwrap()` is correct and that you also need that performance edge?
Ignoring the performance aspect that often comes from a hat-trick, to prove such a thing you need to be wary of the inner workings of a call giving you a `Return`. That knowledge is only valid at the time of writing your `unwrap()`, but won't necessarily hold later.
Also, aren't you implicitly forcing whoever changes the function to check for every smartass dev that decided to `unwrap` at their callsite? That's bonkers.
I doubt that this unwrap was added for performance reasons; I suspect it was rather added because the developer temporarily didn't want to deal with what they thought was an unlikely error case while they were working on something else; and no other system recognized that the unwrap was left in and flagged it before it was deployed on production servers.
If I were Cloudflare I would immediately audit the codebase for all uses of unwrap (or similar rust panic idioms like expect), ensure that they are either removed or clearly documented as to why it's worth crashing the program there, and then add a linter to their CI system that will fire if anyone tries to check in a new commit with unwrap in it.
Panics are for unexpected error conditions, like your caller passed you garbage. Results are for expected errors, like your caller passed you something but it's your job to tell if it's garbage.
So the point of unwrap() is not to prove anything. Like an assertion it indicates a precondition of the function that the implementer cannot uphold. That's not to say unwrap() can't be used incorrectly. Just that it's a valid thing to do in your code.
Yeah, I think I expressed wrongly here. A more correct version would be: "when you do unwrap() you're saying that an error on this particular path shouldn't be recoverable and we should fail-safe."
It's a little subtler than this. You want it to be easy to not handle an error while developing, so you can focus on getting the core logic correct before error-handling; but you want it to be hard to deploy or release the software without fully handling these checks. Some kind of debug vs release mode with different lints seems like a reasonable approach.
What's the point of this sarcastic comment? Do you think that some people claim that Rust's memory safety guarantees mean that a Rust program is incapable of crashing or having a bug? This is a dumb thing to claim certainly, but I'm not aware of anyone actually making this claim.
I'm also not sure what you're getting at with the comment about exception handling being lame. I think the ML/Haskell inspired model that Rust uses of having a parameterized Result type for fallible operations is generally better than exceptions for a variety of reasons (although maybe better Exception semantics could help with some of this), but what does this have to do with match blocks?
The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen".
The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they weren't.
The slowness to root cause: sheer bad luck, with the status page down and Azure's DDoS yesterday all over the news.
The broken SQL: this is the one that I'd be up in arms about if I worked for Cloudflare. For a system with the power to roll out config to ~all of prod at once while bypassing a lot of the usual change tracking, having this escape testing and review is a major miss.
IMO: there should be explicit error path for invalid configuration, so the program would abort with specific exit code and/or message. And there should be a superviser which would detect this behaviour, rollback old working config and wait for few minutes before trying to apply new config again (of course with corresponding alerts).
So basically bad config should be explicitly processed and handled by rolling back to known working config.
You don’t even need all the ceremony. If the config gets updated every 5 minutes, it surely is being hot-reloaded. If that’s the case, the old config is already in memory when the new config is being parsed. If that’s the case, parsing shouldn’t have panicked, but logged a warning, and carried on with the old config that must already be in memory.
> If that’s the case, the old config is already in memory when the new config is being parsed
I think that's explicitly a non-goal. My understanding is that Cloudflare prefers fail safe (blocking legitimate traffic) over fail open (allowing harmful traffic).
System outputting the configuration file failed (it could check the size and/or content and stop right away), but also a system importing the file failed. These usually sound simple/stupid in a hindsight. I am not a fan of everything centralising to a few hands. As in bad situation, they can also be weaponised or attacked. And in good situation their blast radius is just too big and a bit random, in this case global.
The query is surely faulty: Even if this wasn’t a huge distributed database with who-knows-what schemas and use cases, looking up a specific table by its unqualified name is sloppy.
But the architectural assumption that the bot file build logic can safely obtain this operationally critical list of features from derivative database metadata vs. a SSOT seems like a bigger problem to me.
It's probably not ok to silently run with a partial config, which could have undefined semantics. An old but complete config is probably ok (or, the system should be designed to be safe to run in this state).
Even if you want it to crash, you almost never unwrap. At the very least you would use .expect() so you get a reasonable error message -- or even better you handle the potential error.
It reads a lot like the Crowdstrike SNAFU. Machine-generated configuration file b0rks-up the software that consumes it.
The "...was then propagated to all the machines that make up our network..." followed by "....caused the software to fail." screams for a phased rollout / rollback methodology. I get that "...it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly" but today's outage highlights that rapid deployment isn't all upside.
The remediation section doesn't give me any sense that phased deployment, acceptance testing, and rapid rollback are part of the planned remediation strategy.
At my employer, we have a small script that automatically checks such generated config files. It does a diff between the old and the new version, and if the diff size exceeds a threshold (either total or relative to the size of the old file), it refuses to do the update, and opens a ticket for a human to look over it.
It has somewhat regularly saved us from disaster in the past.
I don't think this system is best thought of as "deployment" in the sense of CI/CD; it's a control channel for a distributed bot detection system that (apparently) happens to be actuated by published config files (it has a consul-template vibe to it, though I don't know if that's what it is).
That's why I likened it Crowdstrike. It's a signature database that blew up the consumer of said database. (You probably caught my post mid-edit, too. You may be replying to the snarky paragraph I felt better of and removed.)
Edit: Similar to Crowdstrike, the bot detector should have fallen-back to its last-known-good signature database after panicking, instead of just continuing to panic.
Code and Config should be treated similarly. If you would use a ring based rollout, canaries, etc for safely changing your code, then any config that can have the same impact must also use safe rollout techniques.
You're the nth person on this thread to say that and it doesn't make sense. Events that happen multiple times per second change data that you would call "configuration" in systems like these. This isn't `sendmail.cf`.
If you want to say that systems that light up hundreds of customers, or propagate new reactive bot rules, or notify a routing system that a service has gone down are intrinsically too complicated, that's one thing. By all means: "don't build modern systems! computers are garbage!". I have that sticker on my laptop already.
But like: handling these problems is basically the premise of large-scale cloud services. You can't just define it away.
I'm sorry to belabor this but I'm genuinely not understanding what you're saying in this reply. I haven't operated large scale systems. I'm just an IT generalist and casual coder. I acknowledge I'm too inexperienced to even know what I don't know re: running large systems.
I read the parent poster as broadly suggesting configuration updates should have fitness tests applied and be deployed to minimize the blast radius when an update causes a malfunction. That makes intuitive sense to me. It seems like software should be subject to health checks after configuration updates, even if it's just to stop a deployment before it's widely distributed (let alone rolling-back to last-working configurations, etc).
Am I being thick-headed in thinking defensive strategies like those are a good idea? I'm reading your reply as arguing against those types of strategies. I'm also not understanding what you're suggesting as an alternative.
Again, I'm sorry to belabor this. I've replied once, deleted it, tried writing this a couple more times and given up, and now I'm finally pulling the trigger. It's really eating at me. I feel as though I must be deep down the Dunning-Kruger rabbit hole and really thinking "outside my lane".
Everyone is hating on unwrap, but to me the odd and more interesting part is that it took 3 hours to figure this out? Even with a DDoS red herring, shouldn’t there have been a crash log or telemetry anomaly correlated? Also, shouldn’t the next steps and resolution focus more on this aspect, since it’s a high leverage tool for identifying any outage caused by a panic rather than just preventing a recurrence of random weird edge case #9999999?
I have nowhere near the experience managing such complex systems, but I can empathize with this. In a high-pressure situations the most obvious things get missed. If someone is convinced System X is at fault, your mind can make leaps to justify every other degraded system is a downstream effect of that. Cause and effect can get switched.
Sometimes you have smart people in the room who dig deeper and fish it out, but you cannot always rely on that.
I have plenty of empathy, having been in plenty of similar situations. It's not a matter of "I can't BELIEVE it took that long" (although it is a bit surprising) so much as that I disagree with the key takeaways here in the HN comments section and in the blog itself, which focus strongly on fixing rare edge case issues (the bad ClickHouse query and a bad config file causing a panic via unwrap), rather than reducing MTTR for all issues by improving the debug and monitoring experience.
I'm also suspicious that
> Eliminating the ability for core dumps or other error reports to overwhelm system resources
from the blog had a lot more to do with the issue than perhaps the narrative is letting on.
Yes this is the weird part for me. With good monitoring, the panic at unwrap should have been detected immediately. I assume they weren't looking at the right place, but still. If you use Sentry for example, a brand new panic should be pretty visible.
Indeed, nothing about the root issues are particular surprising but why they missed a critical service panicing across their fleet is not bubbling up.
My best guess is too many alerts firing without a clear hierarchy and possibilities to seprate cause from effect. It's a typical challenge but I wish they would shed some light on that. And its a bit concerning that improving observability is not part of their follow up steps.
This took one of the three hours; it seems to have taken from 11:28 to 13:37 to recognize that the configuration file panic was the cause of the issue.
> That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
> The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.
I'm no FAANG 10x engineer, and I appreciate things can be obvious in hindsight, but I'm somewhat surprised that engineering at the level of Cloudflare does not:
1. Push out files A/B to ensure the old file is not removed.
2. Handle the failure of loading the file (for whatever reason) by automatically reloading the old file instead and logging the error.
Yep, a decent canary mechanism should have caught this. There's a trade off between canarying and rollout speed, though. If this was a system for fighting bots, I'd expect it to be optimized for the latter.
Presumably optimal rollout speed entails something like or as close to ”push it everywhere all at once and activate immediately” that you can get — that’s fine if you want to risk short downtime rather than delays in rollout, what I don’t understand is why the nodes don’t have any independent verification and rollback mechanism. I might be underestimating the complexity but it really doesn’t sound much more involved than a process launching another process, concluding that it crashed and restarting it with different parameters.
I think they need to strongly evaluate if they need this level of rollout speed. Even spending a few minutes with an automated canary gives you a ton of safety.
Even if the servers weren't crashing it is possible that a bet set of parameters results in far too many false positives which may as well be complete failure.
"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our status page."
Unfortunately they do not share, what caused the status page to went down as well. (Does this happen often? Otherwise a big coincidence it seems)
The status page is hosted on AWS Cloudfront, right? It sure looks like Cloudfront was overwhelmed by the traffic spike, which is a bit concerning. Hope we'll see a post from their side.
CloudFront has quotas[0] and they likely just hit those quota limits. To request higher quotas requires a service ticket. If they have access logs enabled in CloudFront they could see what the exact error was.
And since it seems this is hosted by Atlassian, this would be up to Atlassian.
It looks a lot like a CloudFront error we randomly saw today from one of our engineers in South America. I suspect there was a small outage in AWS but can't prove it.
Probably non zero number of companies use cloudfront and other cdns as fallback for cloudflare or running a blended cdn so not surprising to see other cdns hit with a thundering herd when cloudflare went down
This situation reminds me of risk assessment, where you sometimes assume two rare events are independent, but later learn they are actually highly correlated.
it seems like a good chance that despite thinking their status page was completely independent of cloudfront, enough of the internet is dependent on cloudfront now that they're simply wrong about the status page's independence.
> work has already begun on how we will harden them against failures like this in the future. In particular we are:
> Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input
> Enabling more global kill switches for features
> Eliminating the ability for core dumps or other error reports to overwhelm system resources
> Reviewing failure modes for error conditions across all core proxy modules
Absent from this list are canary deployments and incremental or wave-based deployment of configuration files (which are often as dangerous as code changes) across fault isolation boundaries -- assuming CloudFlare has such boundaries at all. How are they going to contain the blast radius in the future?
This is something the industry was supposed to learn from the CrowdStrike incident last year, but it's clear that we still have a long way to go.
Also, enabling global anything (i.e., "enabling global kill switches for features") sounds like an incredibly risky idea. One can imagine a bug in a global switch that transforms disabling a feature into disabling an entire system.
They require the bot management config to update and propagate quickly in order to respond to attacks - but this seems like a case where updating a since instance first would have seen the panic and stopped the deploy.
I wonder why clickhouse is used to store the feature flags here, as it has it's own duplication footguns[0] which could have also easily lead to a query blowing up 2/3x in size. oltp/sqlite seems more suited, but i'm sure they have their reasons
I don't think sqlite would come close to their requirements for permissions or resilience, to name a couple. It's not the solution for every database issue.
Also, the link you provided is for eventual deduplication at the storage layer, not deduplication at query time.
I think you're oversimplifying the problem they had, and I would encourage you to dive in to the details in the article. There wasn't a problem with the database, it was with the query used to generate the configs. So if an analogous issue arose with a query against one of many ad-hoc replicated sqlite databases, you'd still have the failure.
I love sqlite for some things, but it's not The One True Database Solution.
Global configuration is useful for low response times to attacks, but you need to have very good ways to know when a global config push is bad and to be able to rollback quickly.
In this case, the older proxy's "fail-closed" categorization of bot activity was obviously better than the "fail-crash", but every global change needs to be carefully validated to have good characteristics here.
Having a mapping of which services are downstream of which other service configs and versions would make detecting global incidents much easier too, by making the causative threads of changes more apparent to the investigators.
It seems they had this continous rollout for the config service, but the services consuming this were affected even by small percentage of these config providers being faulty, since they were auto updating every few minutes their configs. And it seems there is a reason for these updating so fast, presumably having to react to threat actors quickly.
It's in everyone's interest to mitigate threats as quickly as possible. But it's of even greater interest that a core global network infrastructure service provider not DOS a significant proportion of the Internet by propagating a bad configuration too quickly. The key here is to balance responsiveness against safety, and I'm not sure they struck the right balance here. I'm just glad that the impact wasn't as long and as severe as it could have been.
In my 30 years of reliability engineering, I've come to learn that this is a distinction without a difference.
People think of configuration updates (or state updates, call them what you will) as inherently safer than code updates, but history (and today!) demonstrates that they are not. Yet even experienced engineers will allow changes like these into production unattended -- even ones who wouldn't dare let a single line of code go live without being subject to the full CI/CD process.
I think global kill switches are just an last resort machanism, to bypass identified faulty subsystems. Even if there is a risk with it, in this instance the risk was zero, because CF was dead already. This wont change the blast radius, but it's duration and proliferation.
In reference to fault isolation boundaries: I am not familiar with their CI/CD, in theory the error could have been caught/prevented there, but that comes with a lot of depends or it's tricky. But it looks like they didn't go the extra mile to care about safety sensitive areas. So euphemistic speaking, they are now recalibrating balance of safety measures.
it's always a config push. people rollout code slowly but don't have the same mechanisms for configs. But configs are code, and this is a blind spot that causes an outsized percentage of these big outages.
Lots of people here are (perhaps rightfully) pointing to the unwrap() call being an issue. That might be true, but to me the fact that a reasonably "clean" panic at a defined line of code was not quickly picked up in any error monitoring system sounds just as important to investigate.
Assuming something similar to Sentry would be in use, it should clearly pick up the many process crashes that start occurring right as the downtime starts. And the well defined clean crashes should in theory also stand out against all the random errors that start occuring all over the system as it begins to go down, precisely because it's always failing at the exact same point.
In the early 2000s when Google explained how they achieved their (already back then) awesome reliability, ie assuming that any software and hardware will eventually fail, and that they designed everything with the idea that everything was faulty, there were some people who couldn't get it, who would still bring the argument that "yeah but today with modern raid..."
People here chatting about unwrap remind me of them :)
Assuming software and people will fail is exactly what not using unwrap is about.
If you depend on engineers not fucking up, you will fail. Using unwrap is assuming humans won’t get human-enforced invariants wrong. They will. They did here.
As someone that works in formal verification of crypto systems, watching people like yourself advocate for hope-and-prayer development methodology is astonishing.
However, I understand why we’re still having this debate. It’s the same debate that’s been occurring for the same reasons for decades.
Doing things correctly is mentally more difficult, and so people jump through ridiculous rhetorical hoops to justify why they will not — or quite often, mentally cannot — perform that intellectual labor.
It’s a disheartening lack of craftsmanship and industry accountability, but it’s nothing new.
Why does cloudflare allow unwraps in their code? I would've assumed they'd have clippy lints stopping that sort of thing. Why not just match with { ok(value) => {}, Err(error) => {} } the function already has a Result type.
At the bare minimum they could've used an expect("this should never happen, if it does database schema is incorrect").
The whole point of errors as values is preventing this kind of thing.... It wouldn't have stopped the outage but it would've made it easy to diagnose.
If anyone at cloudflare is here please let me in that codebase :)
Not a cloudflare employee but I do write a lot of Rust. The amount of things that can go wrong with any code that needs to make a network call is staggeringly high. unwrap() is normal during development phase but there are a number of times I leave an expect() for production because sometimes there's no way to move forward.
Yeah it seems likely that even if there wasn't an unwrap, there would have been some error handling that wouldn't have panicked the process, but would have still left it inoperable if every request was instead going through an error path.
I'm in a similar boat, at the very leas an expect can give hits to what happened. However this can also be problematic if your a library developer. Sometimes rust is expected to never panic especially in situations like WASM. This is a major problem for companies like Amazon Prime Video since they run in a WASM context for their TV APP. Any panic crashes everything. Personally I usually just either create a custom error type (preferred) or erase it away with Dyn Box Error (no other option). Random unwraps and expects haunt my dreams.
At risk of sounding harsh, that’s a huge failure in your modeling of invariants that should not be permitted in development.
Permitting it in development is why one ends up in the position of having to use an `expect()` in production code, because your API surfaces are wrong and can’t model your actual invariants.
unwrap() is only the most superficial part of the problem. Merely replacing `unwrap()` with `return Err(code)` wouldn't have changed the behavior. Instead of "error 500 due to panic" the proxy would fail with "error 500 due to $code".
Unwrap gives you a stack trace, while retuned Err doesn't, so simply using a Result for that line of code could have been even harder to diagnose.
`unwrap_or_default()` or other ways of silently eating the error would be less catastrophic immediately, but could still end up breaking the system down the line, and likely make it harder to trace the problem to the root cause.
The problem is deeper than an unwrap(), related to handling rollouts of invalid configurations, but that's not a 1-line change.
We don't know what the surrounding code looks like, but I'd expect it handles the error case that's expressed in the type signature (unless they `.unwrap()` there too).
The problem is that they didn't surface a failure case, which means they couldn't handle rollouts of invalid configurations correctly.
The use of `.unwrap()` isn't superficial at all -- it hid an invariant that should have been handled above this code. The failure to correctly account for and handle those true invariants is exactly what caused this failure mode.
> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value
I don't use Rust, but a lot of Rust people say if it compiles it runs.
Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out.
end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually wanna try it for industrial robot stuff or small controllers etc.
there's no bad language - just occassional hiccups from us users who use those tools.
You misunderstand what Rust’s guarantees are. Rust has never promised to solve or protect programmers from logical or poor programming. In fact, no such language can do that, not even Haskell.
Unwrapping is a very powerful and important assertion to make in Rust whereby the programmer explicitly states that the value within will not be an error, otherwise panic. This is a contract between the author and the runtime. As you mentioned, this is a human failure, not a language failure.
Pause for a moment and think about what a C++ implementation of a globally distributed network ingress proxy service would look like - and how many memory vulnerabilities there would be… I shudder at the thought… (n.b. nginx)
This is the classic example of when something fails, the failure cause over indexes on - while under indexing on the quadrillions of memory accesses that went off without a single hitch thanks to the borrow checker.
I postulate that whatever the cost in millions or hundreds of millions of dollars by this Cloudflare outage, it has paid for more than by the savings of safe memory access.
> Pause for a moment and think about what a C++ implementation of a globally distributed network ingress proxy service would look like - and how many memory vulnerabilities there would be… I shudder at the thought
I mean thats an unfalsifiable statement, not really fair. C is used to successfully launch spaceships.
Whereas we have a real Rust bug that crashed a good portion of the internet for a significant amount of time. If this was a C++ service everyone would be blaming the language, but somehow Rust evangelicals are quick to blame it on "unidiomatic Rust code".
A language that lets this easily happen is a poorly designed language. Saying you need to ban a commonly used method in all production code is broken.
Only formal proof languages are immune to such properties. Therefore all languages are poorly designed by your metric.
Consider that the set of possible failures enabled by language design should be as small as possible.
Rust's set is small enough while also being productive. Until another breakthrough in language design as impactful as the borrow checker is invented, I don't imagine more programmers will be able to write such a large amount of safe code.
Well, no, most Rust programmers misunderstand what the guarantees are because they keep parroting this quote. Obviously the language does not protect you from logic errors, so saying "if it compiles, it works" is disingenuous, when really what they mean is "if it compiles, it's probably free of memory errors".
No, the "if it compiles, it works" is genuinely about the program being correct rather than just free of memory errors, but it's more of a hyperbolic statement than a statement of fact.
It's a common thing I've experienced and seen a lot of others say that the stricter the language is in what it accepts the more likely it is to be correct by the time you get it to run. It's not just a Rust thing (although I think Rust is _stricter_ and therefore this does hold true more of the time), it's something I've also experienced with C++ and Haskell.
So no, it's not a guarantee, but that quote was never about Rust's guarantees.
> Rust won't save you from the usual programming mistake.
Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like.
No language will prevent you from writing broken code if you're determined to do so, and no language is impossible to write correct code in if you make a superhuman effort. But most of life happens in the middle, and tools like Rust make a huge difference to how often a small mistake snowballs into a big one.
> Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like.
No one treats it like that and nearly every Rust project is filled with unwraps all over the place even in production system like Cloudflare's.
It's literally not, Rust tutorials are littered with `.unwrap()` calls. It might be Rust 102, but the first impression given is that the language is surprisingly happy with it.
Yep, unwrap() and unsafe are escape hatches that need very good justifications. It's fine for casual scripts where you don't care if it crashes. For serious production software they should be either banned, or require immense scrutiny.
> you can have a linter forbid it entirely if you like.
It would be better if that would be the other way round "linter forbids it unless you ask it not to". Never wrong to allow users to shoot themself in the foot, but it should be explicit.
> Well Rust won't save you from the usual programming mistake
This is not a Rust problem. Someone consciously chose to NOT handle an error, possibly thinking "this will never happen". Then someone else conconciouly reviewed (I hope so) a PR with an unwrap() and let it slide.
And people doing testing failed to ignore their excuse of this never happening and still testing it. With this kind of systems you need the separate group that just ignores any "this will never happen" and still checks what happens if it does.
Now it might be that it was tested, but then ignored or deprioritised by management...
What people are saying is that idiomatic prod rust doesn't use unwrap/expect (both of which panic on the "exceptional" arm of the value) --- instead you "match" on the value and kick the can up a layer on the call chain.
What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than bot mitigation you'd still have the same symptom of getting 500'd.
as they say in the post, these files get generated every 5 minutes and rolled out across their fleet.
so in this case, the thing farther up the callstack is a "watch for updated files and ingest them" component.
that component, when it receives the error, can simply continue using the existing file it loaded 5 minutes earlier.
and then it can increment a Prometheus metric (or similar) representing "count of errors from attempting to load the definition file". that metric should be zero in normal conditions, so it's easy to write an alert rule to notify the appropriate team that the definitions are broken in some way.
that's not a complete solution - in particular it doesn't necessarily solve the problem of needing to scale up the fleet, because freshly-started instances won't have a "previous good" definition file loaded. but it does allow for the existing instances to fail gracefully into a degraded state.
in my experience, on a large enough system, "this could never happen, so if it does it's fine to just crash" is almost always better served by a metric for "count of how many times a thing that could never happen has happened" and a corresponding "that should happen zero times" alert rule.
Given that the bug was elsewhere in the system (the config file parser spuriously failed), it’s hard to justify much of what you suggested.
Panics should be logged, and probably grouped by stack trace for things like prometheus (outside of process). That handles all sorts of panic scenarios, including kernel bugs and hardware errors, which are common at cloudflare scale.
Similarly, mitigating by having rapid restart with backoff outside the process covers far more failure scenarios with far less complexity.
One important scenario your approach misses is “the watch config file endpoint fell over”, which probably would have happened in this outage if 100% of servers went back to watching all of a sudden.
Sure, you could add an error handler for that too, and for prometheus is being slow, and an infinite other things. Or, you could just move process management and reporting out of process.
The way I’ve seen this on a few older systems was that they always keep the previous configuration around so it can switch back. The logic is something like this:
1. At startup, load the last known good config.
2. When signaled, load the new config.
3. When that passes validation, update the last-known-good pointer to the new version.
That way something like this makes the crash recoverable on the theory that stale config is better than the service staying down. One variant also recorded the last tried config version so it wouldn’t even attempt to parse the latest one until it was changed again.
For Cloudflare, it’d be tempting to have step #3 be after 5 minutes or so to catch stuff which crashes soon but not instantly.
The config file subsystem was where the bug lived, not the code with the unwrap, so this sort of change is a special case of “make the unwrap never fail and then fix the API so it is not needed”.
"if it compiles it runs" - this is indeed an inaccurate marketing slogan. A more precise formulation would be "if it compiles then the static type system, pattern matching, explicit errors, Send bounds, etc. will have caught a lot of bugs that in other languages would have manifested as runtime errors".
Anecdotally I can write code for several hours, deploy it to a test sandbox without review or running tests and it will run well enough to use it, without silly errors like null pointer exceptions, type mismatches, OOBs etc. That doesn't mean it's bug-free. But it doesn't immediately crash and burn either.
Recently I even introduced a bug that I didn't immediately notice because careful error handling in another place recovered from it.
> I don't use Rust, but a lot of Rust people say if it compiles it runs.
Do you grok what the issue was with the unwrap, though...?
Idiomatic Rust code does not use that. The fact that it's allowed in a codebase says more about the engineering practices of that particular project/module/whatever. Whoever put the `unwrap` call there had to contend with the notion that it could panic and they still chose to do it.
It's a programmer error, but Rust at least forces you to recognize "okay, I'm going to be an idiot here". There is real value in that.
While I agree that Rust got it right by being more explicit, a lot of bugs in C/C++ can also easily avoided with good engineering practices. The Rust argument that it is mainly the fault of the programming language with C/C++ was always a huge and unfair exaggeration. Now with this entirely predictable ".unwrap" desaster (in general, not necessarily this exact scenarious), the "no true Rustacean would have put unwrap in production" fallacy is sad and funny at the same time.
Unwrap is controversial. The problem is that if you remove it, it makes the bar even higher for newcomers to Rust. One solution is to make it unsafe (along with panic).
Having the feature table pivoted (with 200 feature1, feature2, etc columns) meant they had to do meta queries to system.columns to get all the feature columns which made the query sensitive to permissioning changes (especially duplicate databases).
A Crowdstrike style config update that affects all nodes but obviously isn't tested in any QA or staged rollout strategy beforehand (the application panicking straight away with this new file basically proves this).
Finally an error with bot management config files should probably disable bot management vs crash the core proxy.
I'm interested here why they even decided to name Clickhouse as this error could have been caused by any other database. I can see though the replicas updating causing flip / flopping of results would have been really frustrating for incident responders.
Right but also this is a pretty common pattern in distributed systems that publish from databases (really any large central source of truth); it might be like the problem in systems like this. When you're lucky the corner cases are obvious; in the big one we experienced last year, a new row in our database tripped an if-let/mutex deadlock, which our system dutifully (and very quickly) propagated across our entire network.
The solution to that problem wasn't better testing of database permutations or a better staging environment (though in time we did do those things). It was (1) a watchdog system in our proxies to catch arbitrary deadlocks (which caught other stuff later), (2) segmenting our global broadcast domain for changes into regional broadcast domains so prod rollouts are implicitly staged, and (3) a process for operators to quickly restore that system to a known good state in the early stages of an outage.
(Cloudflare's responses will be different than ours, really I'm just sticking up for the idea that the changes you need don't follow obviously from the immediate facts of an outage.)
We shouldn't be having critical internet-wide outages on a monthly basis. Something is systematically wrong with the way we're architecting our systems.
Cloudflare, Azure, and other single points of failure are solving issues inherent to webhosting, and those problems have become incredibly hard due to the massive scale of bad actors and the massive complexity of managing hardware and software.
What would you propose to fix it? The fixed cost of being DDoS-proof is in the hundreds of millions of dollars.
> Costs to architect systems that serve millions of request daily have gone down. Not up.
I never said serving millions of requests is more expensive. Protecting your servers is more expensive.
> Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client.
HN uses Cloudflare. You're making my point for me. If you included the fixed costs that Cloudflare's CDN/proxy is giving to HN incredibly cheaply, then running HN at the edge with good performance (and protecting it from botnets) would costs hundreds of millions of dollars.
> People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?)
Again, attacking your own straw men here.
Writing high-throughput web applications is easier than ever. Hosting them on the open web is harder than ever.
From the ping output, I can see HN is using m5hosting.com. This is why HN was up yesterday, even though everything on CF was down.
> Writing high-throughput web applications is easier than ever. Hosting them on the open web is harder than ever.
Writing proper high-throughput applications was never easy and will never be. It is a little bit easier because we have highly optimized tools like nginx or nodejs so we can offset critical parts. And hosting is "harder than ever" if you complicate the matter, which is a quite common pattern these days. I saw people running monstrosities to serve some html & js in the name of redundancy. You'd be surprised how much a single bare-metal (hell, even a proper VM from DigitalOcean or Vultr) can handle.
The most surprising thing to me here is that it took 3 hours to root cause, and points to a glaring hole in the platform observability. Even taking into account the fact that the service was failing intermittently at first, it still took 1.5 hours after it started failing consistently to root cause. But the service was crashing on startup. If a core service is throwing a panic at startup like that, it should be raising alerts or at least easily findable via log aggregation. It seems like maybe there was some significant time lost in assuming it was an attack, but it also seems strange to me that nobody was asking "what just changed?", which is usually the first question I ask during an incident.
That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing with (a bot net that had escalated DDoS attacks from 3Tbps to 30Tbps against us and others like Microsoft over the last 3 months). We worked multiple theories in parallel. After an hour we ruled out the DDoS theory. We had other theories also running in parallel, but at that point the dominant theory was that the feature file was somehow corrupt. One thing that made us initially question the theory was nothing in our changelogs seemed like it would have caused the feature file to grow in size. It was only after the incident that we realized the database permissions change had caused it, but that was far from obvious. Even after we identified the problem with the feature file, we did not have an automated process to role the feature file back to a known-safe previous version. So we had to shut down the reissuance and manually insert a file into the queue. Figuring out how to do that took time and waking people up as there are lots of security safeguards in place to prevent an individual from easily doing that. We also needed to double check we wouldn’t make things worse. The propagation then takes some time especially because there are tiers of caching of the file that we had to clear. Finally we chose to restart the FL2 processes on all the machines that make up our fleet to ensure they all loaded the corrected file as quickly as possible. That’s a lot of processes on a lot of machines. So I think best description was it took us an hour for the team to coalesce on the feature file being the cause and then another two to get the fix rolled out.
Thank you for the clarification and insight, with that context it does make more sense to me. Is there anything you think can be done to improve the ability to identify issues like this more quickly in the future?
Any "limits" on system should be alerted... like at 70% or 80% threshold.. it might be worth it for a SRE to revisit the system limits and ensuring threshold based alerting around it..
If one actually looks at the current pingora API, it has limited ability to initialize async components at startup - the current pattern seems to be to lazily initialize on first call. An obvious downside of this is that a service can startup in a broken state. e.g. https://github.com/cloudflare/pingora/issues/169
I can imagine that this could easily lead to less visibility into issues.
Long time ago Google had a very similar incident where ddos protection system ingested a bad config and took everything down. Except it was auto resolved in like four minutes by an automatic rollback system before oncall was even able to do anything. Perhaps Cloudflare should invest in a system like that
absolutely not normal, this is why in my opinion it took them so long to understand the core issue. instead of a nice error message and a backtrace saying something like "failed to parse config feature names" they thought they were under attack because the service was just crashing instead.
People really like to hate on Rust for some reason. This wasn’t a Rust problem, no language would have saved them from this kind of issue. In fact, the compiler would have warned that this was a possible issue.
I get it, don’t pick languages just because they are trendy, but if any company’s use case is a perfect fit for Rust it’s cloudflare.
Yeah even if you handled this situation without unwrap() if you just went down an error path that didn't panic, the service would likely still be inoperable if every single request went down the error path.
The reason why people are criticizing is because Rust evangelicals say stuff like "if it compiles it works" or talk about how Rust's type system is so much better than other languages that it catches logic errors like this. You won't see Go or Java developers making such strong claims about their preferred languages.
Come on now. You can't blame the compiler when the programmer explicitly told the compiler to not worry about it. There is nothing in existence that can protect against something like that.
I integrated Turnstile with a fail-open strategy that proved itself today. Basically, if the Turnstile JS fails to load in the browser (or in a few specific frontend error conditions), we allow the user to submit the web form with a dummy challenge token. On the backend, we process the dummy token like normal, and if there is an error or timeout checking Turnstile's siteverify endpoint, we fail open.
Of course, some users were still blocked, because the Turnstile JS failed to load in their browser but the subsequent siteverify check succeeded on the backend. But overall the fail-open implementation lessened impact to our customers nonetheless.
Fail-open with Turnstile works for us because we have other bot mitigations that are sufficient to fall back on in the event of a Cloudflare outage.
Only if they are able to block the siteverify check performed by our backend server. That's not the kind of attack we are trying to mitigate with Turnstile.
Why call .unwrap() in a function which returns Result<_,_>?
For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.
Fly writes a lot of Rust, do you allow `unwrap()` in your production environment? At Modal we only allow `expect("...")` and the message should follow the recommended message style[1].
I'm pretty surprised that Cloudflare let an unwrap into prod that caused their worst outage in 6 years.
After The Great If-Let Outage Of 2024, we audited all our code for that if-let/rwlock problem, changed a bunch of code, and immediately added a watchdog for deadlocks. The audit had ~no payoff; the watchdog very definitely did.
I don't know enough about Cloudflare's situation to confidently recommend anything (and I certainly don't know enough to dunk on them, unlike the many Rust experts of this thread) but if I was in their shoes, I'd be a lot less interested in eradicating `unwrap` everywhere and more in making sure than an errant `unwrap` wouldn't produce stable failure modes.
But like, the `unwrap` thing is all programmers here have to latch on to, and there's a psychological self-soothing instinct we all have to seize onto some root cause with a clear fix (or, better yet for dopaminergia, an opportunity to dunk).
A thing I really feel in threads like this is that I'd instinctively have avoided including the detail about an `unwrap` call --- I'd have worded that part more ambiguously --- knowing (because I have a pathological affinity for this community) that this is exactly how HN would react. Maybe ironically, Prince's writing is a little better for not having dodged that bullet.
Sounds like if nothing else, additional attention around (their?) use of unwrap() is still warranted from where you're sitting then though, no? I don't think there's anything wrong with flagging that.
It's one thing to not want to be the one to armchair it, but that doesn't mean that one has to suppress their normal and obvious reactions. You're allowed to think things even if they're kitsch, you too are human, and what's kitsch depends and changes. Applies to everyone else here by extension too.
Fair. I agree that saying "it's the unwrap" and calling it a day is wrong. Recently actually we've done an exercise on our Worker which is "assume the worst kind of panic happens. make the Worker be ok with it".
But I do feel strongly that the expect pattern is a highly useful control and that naked unwraps almost always indicate a failure to reason about the reliability of a change. An unwrap in their core proxy system indicates a problem in their change management process (review, linting, whatever).
Rust has debug asserts for that. Using expect with a comment about why the condition should not/can't ever happen is idiomatic for cases where you never expect an Err.
This reads to me more like the error type returned by append with names is not (ErrorFlags, i32) and wasn't trivially convertible into that type so someone left an unwrap in place on an "I'll fix it later" basis, but who knows.
Oh absolutely, that's how it would have been treated.
Surely a unwrap_or_default() would have been a much better fit--if fetching features fails, continue processing with an empty set of rules vs stop world.
I am one of those old grey beards (or at least, I got started shipping C code in the 1990s), and I'd leave asserts in prod serverside code given the choice; better that than a totally unpredictable error path.
Yes it's meant to be used in test code. If you're sure it can't fail do then use .expect() that way it shows you made a choice and it wasn't just a dev oversight.
Limits in systems like these are generally good. They mention the reasoning around it explicitly. It just seems like the handling of that limit is what failed and was missed in review.
"Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero."
This simply means, the exception handling quality of your new FL2 is non-existent and is not at par / code logic wise similar to FL.
I hope it was not because of AI driven efficiency gains.
In most domains, silently returning 0 in a case where your logic didn't actually calculate the thing you were trying to calculate is far worse than giving a clear error.
As an IT person, I wonder what it's like to work for a company like this. Where presumably IT stuff has a priority. Unlike the companies I've worked for where IT takes a backseat to everything until something goes wrong. Company I work had a huge new office built, with the plan it would be big enough for future growth, yet despite repeated attempts to reserve a larger space, our server room and infrastructure is actually smaller than our old building and has no room to grow.
As a former CF employee, I'd say it's a mixed bag.
There are plenty of resources , yet it's somehow never enough. You do tons of pretty amazing things with pretty amazing tools that also have notable shortcomings.
You're surround by smart people who do lots of great work, but you also end up in incident reviews where you find facepalm-y stuff. Sometimes you even find out it was a known corner case that was deemed too unlikely to prioritize.
The last incident for my team that I remember dealing with there ended up with my coworker and I realizing the staging environment we'd taken down hours earlier was actually the source of data for a production dashboard, so we'd lost some visibility and monitoring for a bit.
I've also worked at Facebook (pre-Meta days) and at Datadog, and I'd say it was about the same. Most things are done quite well, but so much stuff is happening that you still end up with occasional incidents that feel like they shouldn't have happened.
While I heavily frown upon using `unwrap` and `expect` in Rust code and make sure to have Clippy tell me about every single usage of them, I also understand that without them Rust might have been seen as an academic curiosity language.
They are escape hatches. Without those your language would never take off.
But here's the thing. Escape hatches are like emergency exits. They are not to be used by your team to go to lunch in a nearby restaurant.
---
Cloudflare should likely invest in better linting and CI/CD alerts. Not to mention isolated testing i.e. deploy this change only to a small subset and monitor, and only then do a wider deployment.
Hindsight is 20/20 and we can all be smartasses after the fact of course. But I am really surprised because lately I am only using Rust for hobby projects and even I know I should not use `unwrap` and `expect` beyond the first iteration phases.
---
I have advocated for this before but IMO Rust at this point will benefit greatly from disallowing those unsafe APIs by default in release mode. Though I understand why they don't want to do it -- likely millions of CI/CD pipelines will break overnight. But in the interim, maybe a rustc flag we can put in our `Cargo.toml` that enables such a stricter mode? Or have that flag just remove all the panicky API _at compile time_ though I believe this might be a Gargantuan effort and is likely never happening (sadly).
In any case, I would expect many other failures from Cloudflare but not _this_ one in particular.
Say you panic, now you need to have an external system that catches this panic and reports back; and does something meanwhile to recover your system.
If you think about it, it’s not really different from handling the bubbled up error inside of Rust. You don’t (?) your results and your errors go away, they just move up the chain.
I agree the failure is in testing but what you can and should do is raise in alert in your APM system before the runtime panic, in the code path that is deemed impossible to hit.
I am not trashing on them, I've made such mistakes in the past, but I do expect more from them is all.
And you will not believe how many alerts I got for the "impossible" errors.
I do agree there was not too much that could have been done, yes. But they should have invested in more visibility and be more thorough. I mean, hobbyist Rust devs seem to do that better.
It was just a bit disappointing for me. As mentioned above, I'd understand and sympathise with many other mistakes but this one stung a bit.
There's certainly a discipline involved here, but it's usually something like guaranteeing all threads are unwind safe (via AssertUnwindSafe) and logging stack traces when your process keeps dying/can't be started after a fixed number of retries. Which would lead you to the culprit immediately.
I'm just pushing back a bit on the idea that unwrap() is unsafe - it's not, and I wouldn't even call it a foot gun. The code did what it was written to do, when it saw the input was garbage it crashed because it couldn't make sense of what to do next. That's a desirable property in reliable systems (of course monitoring that and testing it is what makes it reliable/fixable in the first place).
We don't disagree, my main point was a bit broader and admittedly hijacked the original topic a bit, namely: `unwrap` and `expect` make many Rust devs too comfortable and these two are very tempting mistresses.
Using those should be done in an extremely disciplined manner. I agree that there are many legitimate uses but in the production Rust code I've seen this has rarely been the case. People just want to move on and then forget to circle back and add proper error handling. But yes, in this case that's not quite true. Still, my point that an APM alert should have been raised on the "impossible" code path before panicking, stands.
On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures
As of 17:06 all systems at Cloudflare were functioning as normal
The real take away is that so much functionality depends on a few players. This is a fundamental flaw in design that is getting worse by the year as the winner takes all winners win. Not saying they didn’t earn their wins. But the fact remains. The system is not robust. Then again, so what. It went down for a while. Maybe we shouldn’t depend on the internet being “up” all the time.
> Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system.
And here is the query they used ** (OK, so it's not exactly):
SELECT * from feature JOIN permissions on feature.feature_type_id = permissions.feature_type_id
someone added a new row to permissions and the JOIN started returning two dupe feature rows for each distinct feature.
** "here is the query" is used for dramatic effect. I have no knowledge of what kind of database they are even using much less queries (but i do have an idea).
more edits: OK apparently it's described later in the post as a query against clickhouse's table metadata table, and because users were granted access to an additional database that was actually the backing store to the one they normally worked with, some row level security type of thing doubled up the rows. Not sure why querying system.columns is part of a production level query though, seems overly dynamic.
Cloudflare Access is still experiencing weird issues for us (it’s asking users to SSO login to our public website even though our zone rules - set on a completely different zone - haven’t changed).
I don’t think the infrastructure has been as fully recovered as they think yet…
> a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system ... to keep [that] system up to date with ever changing threats
> The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail
A configuration error can cause internet-scale outages. What an era we live in
Edit: also, after finishing my reading, I have to express some surprise that this type of error wasn't caught in a staging environment. If the entire error is that "during migration of ClickHouse nodes, the migration -> query -> configuration file pipeline caused configuration files to become illegally large", it seems intuitive to me that doing this same migration in staging would have identified this exact error, no?
I'm not big on distributed systems by any means, so maybe I'm overly naive, but frankly posting a faulty Rust code snippet that was unwrapping an error value without checking for the error didn't inspire confidence for me!
It would have been caught only in stage if there was similar amount of data in the database. If stage has 2x less data it would have never occurred there. Not super clear how easy it would have been to keep stage database exactly as production database in terms of quantity and similarity of data etc.
I think it's quite rare for any company to have exact similar scale and size of storage in stage as in prod.
But now consider how much extra data Cloudflare at its size would have to have just for staging, doubling or more their costs to have stage exactly as production. They would have to simulate similar amount of requests on top of themselves constantly since presumably they have 100s or 1000s of deployments per day.
In this case it seems the database table in question seemed modest in size (the features for ML) so naively thinking they could have kept stage features always in sync with prod at the very least, but could be they didn't consider that 55 rows vs 60 rows or similar could be a breaking point given a certain specific bug.
It is much easier to test with 20x data if you don't have the amount of data cloudflare probably handles.
That just means it takes longer to test. It may not be possible to do it in a reasonable timeframe with the volumes involved, but if you already have 100k servers running to serve 25M requests per second, maybe briefly booting up another 100k isn’t going to be the end of the world?
Either way, you don’t need to do it on every commit, just often enough that you catch these kinds of issues before they go to prod.
The speed and transparency of Cloudflare publishing this port mortem is excellent.
I also found the "remediation and follow up" section a bit lacking, not mentioning how, in general, regressions in query results caused by DB changes could be caught in future before they get widely rolled out.
Even if a staging env didn't have a production-like volume of data to trigger the same failure mode of a bot management system crash, there's also an opportunity to detect that something has gone awry if there were tests that the queries were returning functionally equivalent results after the proposed permission change. A dummy dataset containing a single http_requests_features column would suffice to trigger the dupe results behaviour.
In theory there's a few general ways this kind of issue could be detected, e.g. someone or something doing a before/after comparison to test that the DB permission change did not regress query results for common DB queries, for changes that are expected to not cause functional changes in behaviour.
Maybe it could have been detected with an automated test suite of the form "spin up a new DB, populate it with some curated toy dataset, then run a suite of important queries we must support and check the results are still equivalent (after normalising row order etc) to known good golden outputs". This style of regression testing is brittle, burdensome to maintain and error prone when you need to make functional changes and update what then "golden" outputs are - but it can give a pretty high probability of detecting that a DB change has caused unplanned functional regressions in query output, and you can find out about this in a dev environment or CI before a proposed DB change goes anywhere near production.
They only recently rewrote their core in Rust (https://blog.cloudflare.com/20-percent-internet-upgrade/) -- given the newness of the system and things like "Over 100 engineers have worked on FL2, and we have over 130 modules" I won't be surprised for further similar incidents.
May I just say that Matthew Prince is the CEO of Cloudflare and a lawyer by training (and a very nice guy overall). The quality of this postmortem is great but the fact that it is from him makes one respect the company even more.
There's (obviously) a lot of discussion around the use of `unwrap` in production code. I feel like I'm watching comments speak past each other right now.
I'd agree that the use of `unwrap` could possibly make sense in a place where you do want the system to fail hard. There's lot of good reasons to make the system fail hard. I'd lean towards an `expect` here, but whatever.
That said, the function already returns a `Result` and we don't know what the calling code looks like. Maybe it does do an `unwrap` there too, or maybe there is a save way for this to log and continue that we're not aware of because we don't have enough info.
Should a system as critical as the CF proxy fail hard? I don't know. I'd say yes if it was the kind of situation that could revert itself (like an incremental rollout), but this is such an interesting situation since it's a config being rolled out. Hindsight is 20:20 obviously, but it feels like there should've been better logging, deployment, rollback, and parsing/validation capabilities, no matter what the `unwrap`/`Result` option is.
Also, it seems like the initial Clickhouse changes could've been testing much better, but I'm sure the CF team realizes that.
On the bright side, this is a very solid write up so quickly after the outage. Much better than those times we get it two weeks later.
A lot of outages off late seem to be related to automated config management.
Companies seem to place a lot of trust is configs being pushed automatically without human review into running systems. Considering how important these configs are, shouldn't they perhaps first be deployed to a staging/isolated network for a monitoring window before pushing to production systems?
Not trying to pontificate here, these systems are more complicated than anything I have maintained. Just trying to think of best practices perhaps everyone can adopt.
kudos to getting this blog post out so fast, it’s well written and is appreciated.
i’m a little confused on how this was initially confused for an attack though?
is there no internal visibility into where 5xx’s are being thrown? i’m surprised there isn’t some kind of "this request terminated at the <bot checking logic>" error mapping that could have initially pointed you guys towards that over an attack.
also a bit taken aback that .unwrap()’s are ever allowed within such an important context.
1. Cloudflare is in the business of being a lightning rod for large and targeted DoS attacks. A lot of cases are attacks.
2. Attacks that make it through the usual defences make servers run at rates beyond their breaking point, causing all kinds of novel and unexpected errors.
Additionally, attackers try to hit endpoints/features that amplify severity of their attack by being computationally expensive, holding a lock, or trigger an error path that restarts a service — like this one.
this was in the middle of a scheduled maintenance, with all requests failing at a singular point - that being a .unwrap().
there should be internal visibility into the fact a large number of requests are failing all at the same LOC - and attention should be focused there instantly imo.
or at the very least, it shouldn't take 4 hours for anyone to even consider it wasn't an attack.
in situations such as this, where your entire infra is fucked, you should have multiple crisis teams working in parallel, under different assumptions.
if even one additional team was created that worked under the assumption it was an infra issue rather than an attack, this situation could have been resolved many hours earlier.
for a product as vital to the internet as cloudflare, it is unacceptable to not have this kind of crisis management.
There were two things I think went extremely poorly here:
1) Lack of validation of the configuration file.
Rolling out a config file across the global network every 5 minutes is extremely high risk. Even without hindsight, surely one would see then need for very careful validation of this file before taking on that risk?
There were several things "obviously" wrong with the file that validation should have caught:
- It was much bigger than expected.
- It had duplicate entries.
- Most importantly, when loaded into the FL2 proxy, the proxy would panic on every request. At the very least, part of the validation should involve loading the file into the proxy and serving a request?
2) Very long time to identify and then fix such a critical issue.
I can't understand the complete lack of monitoring or reporting? A panic in Rust code, especially from an unwrap, is the application screaming that there's a logic error! I don't understand how that can be conflated with a DDoS attack. How are your logs not filled with backtraces pointing to the exact "unwrap" in question?
Then, once identified, why was it so hard to revert to a known good version of the configuration file? How did noone foresee the need to roll back this file when designing a feature that deploys a new one globally every 5 minutes?
If the software has a limit on the size of the feature file then the process that propagates the file should probably validate the size before propagating ..
>Currently that limit is set to 200, well above our current use of ~60 features. Again, the limit exists because for performance reasons we preallocate memory for the features.
So they basically hardcoded something, didn't bother to cover the overflow case with unit tests, didn't have basic error catching that would fallback and send logs/alerts to their internal monitoring system and this is why half of the internet went down?
> Eliminating the ability for core dumps or other error reports to overwhelm system resources
but this is not mentioned at all in the timeline above. My best guess would be that the process got stuck in a tight restart loop and filled available disk space with logs, but I'm happy to hear other guesses for people more familiar with Rust.
> As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period. This was due to large amounts of CPU being consumed by our debugging and observability systems, which automatically enhance uncaught errors with additional debugging information.
When I first read about it I assumed it would have been a "poison pill" - a bad config where the ingestion of the config leads the process to crash/restart. And due to that crash on startup, there is no automated possibility to revert to a good config. These things are the worst issues that all global control planes have to deal with.
The report actually seems to confirm this - it was indeed a crash on ingesting the bad config.
However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad".
The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.
Cloudflare tried to build their own feature store, and get a grade F.
I wrote a book on feature stores by O'Reilly. The bad query they wrote in Clickhouse could have been caused by another more error - duplicate rows in materialized feature data. For example, in Hopsworks it prevents duplicate rows by building on primary key uniqueness enforcement in Apache Hudi. In contrast, Delta lake and Iceberg do not enforce primary key constraints, and neither does Clickhouse. So they could have the same bug again due to a bug in feature ingestion - and given they hacked together their feature store, it is not beyond the bounds of possibility.
If you knew the expected number of features , any input file with >100 should be discarded as bad input and you failback to the last good feature file received. This would have protected your service even though you are unable to get the newly populated features. I believe these features are not updated that frequently. Even if they were, you would have biased your system towards availability vs 'correctness'.
What were the teams doing between 11 to 1300 hrs , no explanation of what investigations were going on to not being able to figure the root cause.
Interesting technical insight, but I would be curious to hear firsthand accounts from the teams on the ground, particularly regarding how the engineers felt the increasing pressure, frantically refreshing their dashboards, searching for phantom DDoS, scrolling codes updates...
I’ll be honest, I only understand about 30% of what is being said in this thread and that is probably generous. But it is very interesting seeing so many people respond to each other “it’s so simple! what went wrong was…” as they all disagree on what exactly went wrong.
I don't get why that SQL query was even used in the first place. It seems it fetches feature names at runtime instead of using a static hardcoded schema. Considering this decides the schema of a global config, I don't think the dynamicity is a good idea.
Honestly... everyone shit themselves that internet doesn't work, but next week this outage will be forgotten by 99% of population. I was doing something on my PC when I saw clear information that Cloudflare is down, so I decided to just go take a nap, then read a book, then go for a walk. Once I was done, the internet was working again. Panic was not necessary on my side.
What I'm trying to say is that things would be much better if everyone took a chill pill and accepted the possibility that in rare instances, the internet doesn't work and that's fine. You don't need to keep scrolling TikTok 24/7.
It's funny how everyone seems to be having a meltdown over this. I didn't even notice anything was wrong until I read about it on Reddit 5 hours later, even though I was working all day. Sounds to me like people are too reliant on random websites.
Makes me wonder which team is responsible for that feature generating query, and if they follow full engineering level QA. It might be deferred to an MLE team that is better than the data scientists but less rigorous than software needs to be.
It feels like their list of after actions is lacking a bit to me.
How about
1. The permissions change project is paused or rolled back until
2. All impacted database interactions (SQL queries) are evaluated for improper assumptions or better
3. Their design that depends on database metainfo and schema is replaced with ones that use specific tables and rows in tables instead of using the meta info as part of their application.
4. All hard coded limits are centralized in a single global module and referenced from their users and then back propagated to any separate generator processes that validate against the limit before pushing generated changes
Question: customer having issues also couldn't switch their dns to bypass the service, why is the control plane updated along the data plane here it seem a lot of use could save business continuity if they could change their dns entry temporarily
Attempt to reproduce the Cloudflare 2025-11-18 outage.
Cloudflare's incident report is written clearly and explicitly, so based on my own understanding, I’m going to try reproducing this outage.
Already completed:
CK cluster
Permission change triggering data doubling
Cache propagation
Unaffected proxy services
Proxy services with bot score errors
TODO:
unwrap panic during pre-allocation of cache
Full demonstration of the entire outage process
The internet hasn't been the internet in years. It was originally built to withstand wars. The whole idea of our IP based internet was to reroute packages should networks go down. Decentralisation was the mantra and how it differed from early centralised systems such as AOL et al.
This is all gone. The internet is a centralised system in the hand of just a few companies. If AWS goes down half the internet does. If Azure, Google Cloud, Oracle Cloud, Tencent Cloud or Alibaba Cloud goes down a large part of the internet does.
Yesterday with Cloudflare down half the sites I tried gave me nothing but errors.
Thanks for the detailed writeup and explaining the root cause in details
However, I have a question from a release deployment process perspective. Why was this issue not detected during internal testing ? I didn't find the RCA analysis covering this aspect. Doesn't cloudflare have an internal test stage as part of its CICD pipeline. Looking the description of the issue, it should have been immediately detected in internal stage test environment.
Thanks for the detailed RCA. After going through the blog post as well, there’s a curiosity about whether there are opportunities to detect such scenarios earlier in the testing phase. It would be helpful to understand if there are any differences between the test environment and production that might have contributed to this. This could provide insights into strengthening the process going forward.
Wondering why they didn’t disable the bot management temporarily to recover. Websites could have survived temporarily without it compared to the outage itself.
As we know that "With great power comes great responsibility" the team should understand this because Cloudflare is used worldwide, and for many countries it was the peak working time when Cloudflare went down so this affected massively. We always want perfect results but it's not possible, I hope the team is not overworking to get the changes on prod.
Hold up ,- when I used a C or similar language for accessing a database and wanted to clamp down on memory usage to deterministically control how much I want to allocated, I would explicitly limit the number of rows in the query.
There never was an unbound "select all rows from some table" without a "fetch first N rows only" or "limit N"
If you knew that this design is rigid, why not leverage the query to actually do it ?
Because nothing forced them to and they didn't think of it. Maybe the people writing the code that did the query knew that the tables they were working with never had more than 60 rows and figured "that's small" so they didn't bother with a limit. Maybe the people who wrote the file size limit thought "60 rows isn't that much data" and made a very small file size limit and didn't coordinate with the first people.
Anyway regardless of which language you use to construct a SQL query, you're not obligated to put in a max rows
I imagine there's numerous ways to protect against it and protection should've been added by whoever decided on this optimization. In data layer, create some kind of view which never returns more than 200 rows from base table(s). In code, use some kind of iterator. I'm not a Rust guy, just a C defensive practices type of dude, but maybe they just missed a biggie during a code review.
That's interesting. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. It's like the issues with HOSTS.TXT needing to be copied among the network of the early internet to allow routing (taking days to download etc) and DNS having to be created to make that propagation less unwieldy.
Speaking of resiliency, the entire Bot Management module doesn't seems to be a critical part of the system, so for example, what happens if that module goes down for an hour? the other parts of the system should work. So I would rank every module and it's role in the system, and would design it in a way that when a non-critical module fails, other parts still can function.
The outage sucked for everyone. The root cause also feels like something they could have caught much earlier in a canary rollout from my reading of this.
All that said, to have an outage reported turned around practically the same day, that is this detailed, is quite impressive. Here's to hoping they make their changes from this learning, and we don't see this exact failure mode again.
I can never get used to the error happening at call site rather than within the function where the early return of Err happened. It is not "much cleaner", you have no idea which line and file caused it at call site. By default Returning should have a way of setting a marker which can then be used to map back to the line() and file(). 10+ years and still no ergonomics.
Based on this writeup it seems that Cloudflare defaults to a 0 score for bot prevention if there's a failure. Could instead it default to a passing score? Default open instead of default closed? This would have been a non-event to a lot of websites if that change was made.
Would be nice if their Turnstile could be turned off on their login page when something like this happens, so we can attempt to route traffic away from Cloudflare during the outage. Or at least have a simple app where this can be modified from.
Dear Matthew Prince, don't you think we (the ones affected by your staff's mistake) should get some sort of compensation??? Yours truly, a Cloudflare client who lost money during the November 18th outage.
How many changes to production systems does Cloudflare make throughout a day? Are they a part of any change management process? That would be the first place I would check after a random outage, recent changes.
So they made a newbie mistake in SQL that would not even pass an AI review. They did not verify the change in a test environment. And I guess the logs are so full of errors it is hard to pinpoint which matters. Yikes.
Is dual sourcing CDNs feasible these days? Seems like having the capability to swap between CDN providers is good both from a negotiating perspective and a resiliency one.
Just a moment to reflect on how much freaking leverage computers give us today - a single permission change took down half the internet. Truly crazy times.
If you deploy a change to your system, and things start to go wrong that same day, the prime suspect (no matter how unlikely it might seem) should be the change you made.
Why have a limit on the file size if the thing that happens when you hit the limit is the entire network goes down? Surely not having a limit can't be worse?
Excellent write up. Cybersecurity professionals read the story and learn. It’s textbook lesson in post-mortem incident analysis - a mvp for what is expected from us all in a similar situation.
Reputationally this is extremely embarrassing for Cloudflare, but imo they seem to get their feet back on the ground. I was surprised to see not just one, but two apologies to the internet. This just cements how professional and dedicated the Cloudflare team is to ensure stable resilient internet and how embarrassed they must have been.
A reputational hit for sure, but outcome is lessons learned and hopefully stronger resilience.
> Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare.
also cloudflare:
> The Cloudflare Dashboard was also impacted due to both Workers KV being used internally and Cloudflare Turnstile being deployed as part of our login flow.
thanks for clarifying! i guess then they never explained why the status page went down, even though it's supposed to be running on independent infrastructure.
Yes, that was missing (along with the London WARP thing). Other comments mentioned that their status page is an Atlassian Statuspage solution, hosted on AWS CloudFront.
Unclear to me if it's an Atlassian-managed deployment they have, or if it's self-managed, I'm not familiar with Statuspage and their website isn't helping. Though if it's managed, I'm not sure how they can know for sure there's no interdependence. (Though I guess we could technically keep that rabbit hole going indefinitely.)
Wow. What a post mortem.
Rather than Monday morning quarterbacking how many ways this could have been prevented, I'd love to hear people sound-off on things that unexpectedly broke. I, for one, did not realize logging in to porkbun to edit DNS settings would become impossible with a cloudflare meltdown
That's unfortunate. I'll need to investigate whether Porkbun plans on decoupling its auth from being reliant on CloudFlare, otherwise I will need to migrate a few domains off of that registrar.
this is where change management really shines because in a change management environment this would have been prevented by a backout procedure and it would never have been rolled out to production before going into QA, with peer review happening before that... I don't know if they lack change management but it's definitely something to think about
i think that is data rather than code which is where it falls short, in a way you need stringent code and more safeguarded code; it's like if everyone sends you 64k posts as that's all your proxy layer lets in, someone checked sending 128kb and it gave an error before reaching your app - and then someone sends 128kb and the proxy layer has changed - and your app crashes as it was more than 64kb and your app had an assert against that. to actually track issues with erraneous data that overflows well and stuff isn't so much code test but more like fuzz testing, brute force testing etc. which i think people should do; but that's more like we need strong test networks, and also those test networks may need to be more internet like to reflect real issues too, so the whole testing infrastructure in itself becomes difficult to get right - like they have their own tunneling system etc, they could segregate some of their servers and make a test system with better error diagnosis etc potentially. but to my mind, if they had better error propogation back that really identified what was happening and where then that would be a lot better in general. sure, start doing that on a test network. this is something i've beeen tihnking about in general - i made a simple rpc system for being able to send real time rust tracing logs (it allows to just use the normal tracing framework and use a thin rpc layer) back from multiple end servers but that's mostly for granular debugging. i've never quite understood why systems like systemd-journald aren't more network centric when they're going to be big and complex kitchensink approaches - apparently there's dbus support, but to my mind something inbetween debugging level of code and warning/info. like even if it's doing things like 1/20 of log info it's too much volume if things like large files getting close to limits is increasing etc and we can see this as things run, and can see if it's localised or common etc it'd help have more resilient systems. something may already exist in this line but i didn't come across anything in a reasonably passive way - i mean there's debugging tools like dtrace etc that have been around for ages.
Best post mortem I've read in a while, this thing will be studied for years.
A bit ironic that their internal FL2 tool is supposed to make Cloudflare "faster and more secure" but brought a lot of things down. And yeah, as other have already pointed out, that's a very unsafe use of Rust, should've never made it to production.
While it's certainly worthwhile to discuss the Technical and Procedural elements that contributed to this Service Outage, the far more important (and mutually-exclusive aspect) to discuss should be:
Why have we built / permitted the building of / Subscribed to such a Failure-intolerant "Network"?
Who's "we"? This is not a trick question, what specific people do you think acted wrongly here? I don't use Cloudflare personally. I don't run any of the sites that do use it. The people who did make the decision to put thier websites behind Cloudflare could stop, and maybe some will, but presumably they're paying for it because they think, perhaps accurately, that they get value out of it. Should some power compel them not to use Cloudflare?
Here's a random post from their blog by the same author from 2017 with an em dash:
> As we wrote before, we believe Blackbird Tech's dangerous new model of patent trolling — where they buy patents and then act their own attorneys in cases — may be a violation of the rules of professional ethics.
our site realtyblocks.com went down but we redirected the traffic by bypassing the cloudflare DNS routes with few clicks. Thank you for resolving the issue Cloudflare.
our site realtyblocks.com went down but we redirected the traffic by bypassing DNS entries with few clicks. Thank you for resolving the issue Cloudflare.
I am just going by the outage post-mortem report. I could not read the article after I read the first few lines - "feature file" expansion/limits. I am stuck at consuming the design idea here, where you allow multiple inserts for one feature [assuming you have some uniqueness constraint].
Even a simple key-value map per feature should have allowed for insertions as simple as a put/replace of the value and not appending to the file. That was not the case here, where Cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here that Cloudflare should rethink. If someone can educate me on the design, I can continue reading the next few lines.
I am just going by the outage post mortem report. I could not read the article after I read the first few lines - "feature file". I am stuck at consuming the design idea here where you allow multiple inserts for one feature [assuming you have some uniqueness constraint]. Even a simple key-value map per feature should have made the insertions as just a put/replace the value. I think that was not the case here, where cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here cloudflare should rethink. If someone can educate me on the design, I can continue reading the next lines.
tl;dr
A permissions change in a ClickHouse database caused a query to return duplicate rows for a “feature file” used by Cloudflares Bot Management system, which doubled the file size. That oversized file was propagated to their core proxy machines, triggered an unhandled error in the proxy’s bot-module (it exceeded its pre-allocated limit), and as a result the network started returning 5xx errors. The issue wasn’t a cyber-attack — it was a configuration/automation failure.
Did your incident response team look at the last few changes that were executed? If they had , they could have just rolleback the change. or just looking at the changes executed, in the vicinity of the start of the outage could have pointed to the problem.
Didn't the services that were crashing due to OOM raise any alerts?
unwraps are so very easy to use and they have bit me so many times because you can nearly never run into a problem and suddenly crashes from an unwrap that almost always was fine
> Given Cloudflare's importance in the Internet ecosystem any outage of any of our systems is unacceptable.
Excuse me, what you've just said? Who decided on “Cloudflare's importance in the Internet ecosystem”? Some see it differently, you know, there's no need for that self-assured arrogance of an inseminating alpha male.
28M 500 errors/sec for several hours from a single provider. Must be a new record.
No other time in history has one single company been responsible for so much commerce and traffic. I wonder what some outage analogs to the pre-internet ages would be.
Something like a major telco going out, for example the AT&T 1990 outage of long distance calling:
> The standard procedures the managers tried first failed to bring the network back up to speed and for nine hours, while engineers raced to stabilize the network, almost 50% of the calls placed through AT&T failed to go through.
> Until 11:30pm, when network loads were low enough to allow the system to stabilize, AT&T alone lost more than $60 million in unconnected calls.
> Still unknown is the amount of business lost by airline reservations systems, hotels, rental car agencies and other businesses that relied on the telephone network.
Yes, all(most) eggs should not be in one basket. Perfect opportunity to setup a service that checks cloudflare then switches a site's DNS to akami as a backup.
Absolute volume maybe[1], as relative % of global digital communication traffic, the era of early telegraph probably has it beat.
In the pre digital era, East India Company dwarfs every other company in any metric like commerce controlled, global shipping, communication traffic, private army size, %GDP , % of workforce employed by considerable margins.
The default was large consolidated organization throughout history, like say Bell Labs, or Standard Oil before that and so on, only for a brief periods we have enjoyed benefits of true capitalism.
[1] Although I suspect either AWS or MS/Azure recent down-times in the last couple of years are likely higher
So an unhandled error condition after an configuration update similar to Crowdstrike - if they had just used a programming language where this can't happen due to the superior type system such as Rust. Oh wait.
Cloudflare’s write-up is clear and to the point. A small change spread wider than expected, and they explained where the process failed. It’s a good reminder that reliability depends on strong workflows as much as infrastructure.
Did some $300k chief of IT blame it all on some overworked secretary clicking a link in an email they should have run through a filter? Because that’s the MO.
This is the first significant outage that has involved Rust code, and as you can see the .unwrap is known to carry the risk of a panic and should never be used on production code.
I think you should give me a credit for all the income I lost due to this outage. Who authorized a change to the core infrastructure during the period of the year when your customers make the most income? Seriously, this is a management failure at the highest levels of decision-making. We don't make any changes to our server infrastructure/stack during the busiest time of the year, and neither should you. If there were an alternative to Cloudflare, I'd leave your service and move my systems elsewhere.
I think you should get exactly what the contract you signed said you'd get. Outages happen in all infrasturture. Planned and unplanned ones both. The SLA and SLO are literal acknowledgements of the fact that and part of the contract for that reason.
Its fair to be upset at their decision making - use that to renegotiate your contract.
> The change explained above resulted in all users accessing accurate metadata about tables they have access to. Unfortunately, there were assumptions made in the past, that the list of columns returned by a query like this would only include the “default” database:
SELECT
name,
type
FROM system.columns
WHERE
table = 'http_requests_features'
order by name;
Note how the query does not filter for the database name. With us gradually rolling out the explicit grants to users of a given ClickHouse cluster, after the change at 11:05 the query above started returning “duplicates” of columns because those were for underlying tables stored in the r0 database.
Here is a bit more context in addition to the quote above. A ClickHouse permissions change made a metadata query start returning duplicate column metadata from an extra schema, which more than doubled the size and feature count of a Bot Management configuration file. When this oversized feature file was deployed to edge proxies, it exceeded a 200-feature limit in the bot module, causing that module to panic and the core proxy to return 5xx errors globally
- Their database permissions changed unexpectedly (??)
- This caused a 'feature file' to be changed in an unusual way (?!)
- Their SQL query made assumptions about the database; their permissions change thus resulted in queries getting additional results, permitted by the query
- Changes were propagated to production servers which then crashed those servers (meaning they weren't tested correctly)
- They hit an internal application memory limit and that just... crashed the app
- The crashing did not result in an automatic backout of the change, meaning their deployments aren't blue/green or progressive
- After fixing it, they were vulnerable to a thundering herd problem
- Customers who were not using bot rules were not affected; CloudFlare's bot-scorer generated a constant bot score of 0, meaning all traffic is bots
In terms of preventing this from a software engineering perspective, they made assumptions about how their database queries work (and didn't validate the results), and they ignored their own application limits and didn't program in either a test for whether an input would hit a limit, or some kind of alarm to notify the engineers of the source of the problem.
From an operations perspective, it would appear they didn't test this on a non-production system mimicing production; they then didn't have a progressive deployment; and they didn't have a circuit breaker to stop the deployment or roll-back when a newly deployed app started crashing.
People jump to say things like "where's the rollback" and, like, probably yeah, but keep in mind that speculative rollback features (that is: rollbacks built before you've experienced the real error modes of the system) are themselves sources of sometimes-metastable distributed system failures. None of this is easy.
How about where's the most basic test to check if your config file will actually run at all in your application? It was a hard-coded memory limit; a git-hook test suite run a MacBook would have caught this. But nooo, let's not run the app for 0.01 seconds with this config before sending it out to determine the fate of the internet?
This is literally the CrowdStrike bug, in a CDN. This is the most basic, elementary, day 0 test you could possibly invent. Forget the other things they fucked up. Their app just crashes with a config file, and nobody evaluates it?! Not every bug is preventable, but an egregious lack of testing is preventable.
This is what a software building code (like the electrical code's UL listings that prevent your house from burning down from untested electrical components) is intended to prevent. No critical infrastructure should be legal without testing, period.
just before this outage i was exploring bunnycdn as the idea of cloudflare taking over dns still irks me slightly. there are competitors. but there's a certain amount of scale that cloudflare offers which i think can help performance in general. that said in the past i found cloudflare performance terrible when i was doings lots of testing. they are predominantly a pull based system not a push, so if content isn't current the cache miss performance can be kind of blah. i think their general backhaul paths have improved, but at least from new zealand they used to seem to do worse than hitting a los angeles proxy that then hits origin. (although google was in a similar position before, where both 8.8.8.8 and www.google.co.nz/.com were both faster via los angeles than via normal paths - i think google were doing asia parent, like if testing 8.8.8.8 misses it was super far away). i think now that we have http/3 etc though that performance is a bit simpler to achieve, and that ddos, bot protection is kind of the differentiator, and i think that cloudflare's bot protection may work reasonably well in general?
> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network.
As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message.
> The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems' permissions
Also appreciate the honesty here.
> On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures to deliver core network traffic. […]
> Core traffic was largely flowing as normal by 14:30. We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online. As of 17:06 all systems at Cloudflare were functioning as normal.
Why did this take so long to resolve? I read through the entire article, and I understand why the outage happened, but when most of the network goes down, why wasn't the first step to revert any recent configuration changes, even ones that seem unrelated to the outage? (Or did I just misread something and this was explained somewhere?)
Of course, the correct solution is always obvious in retrospect, and it's impressive that it only took 7 minutes between the start of the outage and the incident being investigated, but it taking a further 4 hours to resolve the problem and 8 hours total for everything to be back to normal isn't great.
Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.
Thanks for the explanation! This definitely reminds me of CrowdStrike outages last year:
- A product depends on frequent configuration updates to defend against attackers.
- A bad data file is pushed into production.
- The system is unable to easily/automatically recover from bad data files.
(The CrowdStrike outages were quite a bit worse though, since it took down the entire computer and remediation required manual intervention on thousands of desktops, whereas parts of Cloudflare were still usable throughout the outage and the issue was 100% resolved in a few hours)
It might remind you of Crowdstrike because of the scale.
Outages are in a large majority of cases caused by change, either deployments of new versions or configuration changes.
zone your deployments first -blue/green. Have a small blue zone, and test it out. If it works, then expand to green deployments.
A configuration file should not grow! design failure here, I want to understand
Richard Cook #18 (and #10) strikes again!
https://how.complexsystems.fail/#18
It'd be fun to read more about how you all procedurally respond to this (but maybe this is just a fixation of mine lately). Like are you tabletopping this scenario, are teams building out runbooks for how to quickly resolve this, what's the balancing test for "this needs a functional change to how our distributed systems work" vs. "instead of layering additional complexity on, we should just have a process for quickly and maybe even speculatively restoring this part of the system to a known good state in an outage".
This document by Dr. Cook remains _the standard_ for systems failure. Thank you for bringing it into the discussion.
Why was Warp in London disabled temporarily. No mention of that change was discussed in the RCA despite it being called out in an update.
For London customers this made the impact more severe temporarily.
We incorrectly thought at the time it was attack traffic coming in via WARP into LHR. In reality it was just that the failures started showing up there first because of how the bad file propagated and where it was working hours in the world.
Probably because it was the London team that was actively investigating the incident and initially came to the conclusion that it may be a DDoS while being unable to authenticate to their own systems.
Given the time of the outage that makes sense, they’d mostly be within their work day time (if such a thing apples to us anymore).
Question from a casual bystander, why not have a virtual/staging mini node that receives these feature file changes first and catches errors to veto full production push?
Or you do have something like this but the specific db permission change in this context only failed in production
I think the reasoning behind this is because of the nature of the file being pushed - from the post mortem:
"This feature file is refreshed every few minutes and published to our entire network and allows us to react to variations in traffic flows across the Internet. It allows us to react to new types of bots and new bot attacks. So it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly."
In this case, the file fails quickly. A pretest that consists of just attempting to load the file would have caught it. Minutes is more than enough time to perform such a check.
Just asking out of curiosity, but roughly how many staff would've been involved in some way in sorting out the issue? Either outside regular hours or redirected from their planned work?
Is there some way to check the sanity of the configuration change, monitor it and then revert back to an earlier working configuration if things don't work out?
Yeah, I can imagine that this insertion was some high-pressure job.
The computer science equivalent of choosing between the red, green and blue wires when disarming a nuke with 15 seconds left on the clock
Is it though? Or is it, oh, this is such a simple change that we really don't need to test it attitude? I'm not saying this applies to TFA, but some people are so confident that no pressure is felt.
However, you forgot that the lighting conditions are where only red lights from the klaxons are showing so you really can't differentiate the colors of the wires
Thx for the explanation!
Side thought as we're working on 100% onchain systems (for digital assets security, different goals):
Public chains (e.g. EVMs) can be a tamper‑evident gate that only promotes a new config artifact if (a) a delay or multi‑sig review has elapsed, and (b) a succinct proof shows the artifact satisfies safety invariants like ≤200 features, deduped, schema X, etc.
That could have blocked propagation of the oversized file long before it reached the edge :)
> much better than their completely false “checking the security of your connection” message
The exact wording (which I can easily find, because a good chunk of the internet gives it to me, because I’m on Indian broadband):
> example.com needs to review the security of your connection before proceeding.
It bothers me how this bald-faced lie of a wording has persisted.
(The “Verify you are human by completing the action below.” / “Verify you are human” checkbox is also pretty false, as ticking the box in no way verifies you are human, but that feels slightly less disingenuous.)
Next time open your dev console in your window and look at how much is going on in the background.
checking the checkbox does verify you are human, for the most part.
As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.
I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency.
Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.
> I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did
Damn corporate karma farming is ruthless, only a couple minute SLA before taking ownership of the karma. I guess I'm not built for this big business SLA.
We're in a Live Fast Die Young karma world. If you can't get a TikTok ready with 2 minutes of the post modem drop, you might as well quit and become a barista instead.
> I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate.
I'm so jealous. I've written postmortems for major incidents at a previous job: a few hours to write, a week of bikeshedding by marketing and communication and tech writers and ... over any single detail in my writing. Sanitizing (hide a part), simplifying (our customers are too dumb to understand), etc, so that the final writing was "true" in the sense that it "was not false", but definitely not what I would call "true and accurate" as an engineer.
You call this transparency, but fail to answer the most important questions: what was in the burrito? Was it good? Would you recommend?
Chicken burrito from Coyo Taco in Lisbon. I am not proud of this. It’s worse than ordering from Chipotle. But there are no Chipotle’s in Lisbon… yet.
I DON'T see this as transparency. There is ZERO mention of the burrito in the post-mortem document itself.
0/10, get it right the first time, folks. (/s)
A very human and authentic response. Love to see it.
Fantastic for recruiting, too.
> He didn’t reply after a few minutes so I did
I'd consider applying based on this alone
Appreciate the extra transparency on the process.
A postmortem postmortem, I love it. Transparency to the power of 2.
I really appreciate this level of transparency. Thank you for being a good person in such a powerful position in the world.
I'm not sure I've ever read something from someone so high up in a company that gave me such a strong feeling for "I'd like to work for these people". If job posts could be so informal and open ended, this post could serve as one in the form of a personality fit litmus test.
How do you guys handle redaction? I'm sure even when trusted individuals are in charge of authoring, there's still a potential of accidental leakage which would probably be best mitigated by a team specifically looking for any slip ups.
Thanks for the insight.
Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column names.” John and I usually catch anything the rest of the team may have missed. For me, pays to have gone to law school, but also pays to have studied Computer Science in college and be technical enough to still understand both the SQL and Rust code here.
that's very cool, thanks
The person who posted both this blog article and the hacker news post, is Matthew Prince, one of highly technical billionaire founders of cloudflare. I'm sure if he wants something to happen, it happens.
I'm sure he wanted traffic to flow for those few hours but it didn't :p
I mean the CEO posted the post-mortem so there aren't that many layers of stakeholders above. For other post-mortems by engineers, Matthew once said that the engineering team is running the blog and that he wouldn't event know how to veto even if he wanted [0]
[0] https://news.ycombinator.com/item?id=45588305
Cloudflare seems to have baked this level of transparency into their culture and incident response process
From what I've observed, it depends on whether you're an "engineering company" or not.
And a well-written one at that. Compared to the AWS port-mortem this could be literature.
Except it fails to document anything about the actions they made to Warp in London during the resolution.
There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.
Thanks!
[flagged]
I feel like your username really brings something extra to the party. Now go home.
Can attest: not a single LLM used. Couldn’t if I tried. Old school. And not entirely proud of that.
* published less than 12 hours from when the incident began. Proud of the team for pulling together everything so quickly and clearly.
That's all well & good, but I'm curious...
> Spent some time after we got things under control talking to customers. Then went home.
What did sama / Fidji say? ;) Turnstile couldn't have been worth that.
It's not just a PR-friendly summary either... they included real technical detail, timestamps, even code snippets
Plenty are capable. Most don't bother.
[flagged]
Why give this sort of content more visibility/reach?
I'm sure that's not your intent, so I hope my comment gives you an opportunity to reflect on the effects of syndicating such stupidity, no matter what platform it comes from.
Mainly to make others aware of what’s happening in the context of this Cloudflare outage. Sure I can avoid giving it visibility/reach but it’s growing and proliferating on its own, and I think ignoring it isn’t going to stop it so I am hoping awareness will help. I’ve noticed a huge rise in open racism against Chinese and Indian and workers of other origin, even when they’re here on a legal visa that we have chosen as a nation to grant for our own benefit.
The legislation that MTG (Marjorie Taylor Green) just proposed a few days ago to ban H1B entirely, and the calls to ban other visa types, is going to have a big negative impact on the tech industry and American innovation in general. The social media stupidity is online but it gives momentum to the actual real life legislation and other actions the administration might take. Many congress people are seeing the online sentiment and changing their positions in response, unfortunately.
Fair points; there's certainly a balance to be struck between raising awareness and amplifying, and I admittedly have no idea where that line is.
This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is textbook "parse, don't validate" anti-pattern.
I know, this is "Monday morning quarterbacking", but that's what you get for an outage this big that had me tied up for half a day.
I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is:
1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually.
2. Despite the elevated risk of system wide rapid config propagation, it took them 2 hours to identify the config as the proximate cause, and another hour to roll it back.
SOP for stuff breaking is you roll back to a known good state. If you roll out gradually and your canaries break, you have a clear signal to roll back. Here was a special case where they needed their system to rapidly propagate changes everywhere, which is a huge risk, but didn’t quite have the visibility and rapid rollback capability in place to match that risk.
While it’s certainly useful to examine the root cause in the code, you’re never going to have defect free code. Reliability isn’t just about avoiding bugs. It’s about understanding how to give yourself clear visibility into the relationship between changes and behavior and the rollback capability to quickly revert to a known good state.
Cloudflare has done an amazing job with availability for many years and their Rust code now powers 20% of internet traffic. Truly a great team.
They failed on so many levels here.
How can you write the proxy without handling the config containing more than the maximum features limit you set yourself?
How can the database export query not have a limit set if there is a hard limit on number of features?
Why do they do non-critical changes in production before testing in a stage environment?
Why did they think this was a cyberattack and only after two hours realize it was the config file?
Why are they that afraid of a botnet? Does not leave me confident that they will handle the next Aisuru attack.
I'm migrating my customers off Cloudflare. I don't think they can swallow the next botnet attacks and everyone on Cloudflare go down with the ship, so it will be safer to not be behind Cloudflare when it hits.
> They failed on so many levels here.
That's often the case with human error as especially aviation safety experts know: https://en.wikipedia.org/wiki/Swiss_cheese_model
Exactly. The only way this could happen in the first place was _because_ they failed at so many levels. And as a result, more layers of Swiss cheese will be added, and holes in existing ones will be patched. This process is the reason flying is so safe, and the reason why Cloudflare will be a little bit more resilient tomorrow than it was yesterday.
In organizations with this level of care, if you fail at fewer levels, customers just never notice the error.
Any big and noticeable incident is one of the "we failed on so many levels here" kind, by definition.
> Why did they think this was a cyberattack
Isn’t getting cyberattacked their core business?
If so, why is their discovery of, non ambiguous?
> Why do they do non-critical changes in production before testing in a stage environment?
I guess the noncritical change here was the change to the database? My experience has been a lot of teams do a poor job having a faithful replica of databases in stage environments to expose this type of issue.
In part because it is somewhere between really hard and impossible. Is your staging DB going to be as big? Seeing the same RPS as prod? Seeing the same scenarios?
Permissions stuff might be caught without a completely faithful replica, but there are always going to be attributes of the system that only exist in prod.
It’s easy to pick on logic that failed and for which you have a very detailed and great post mortem write-up.
Yet you omit to acknowledge that the remaining 99.99999% logic written that powers Cloudflare works flawlessly.
Also, hindsight is 20/20
You are less critical with CF then they are with themselves.
A system that is 99.99999% flawless, can still be unusable.
optimism bias: 100/100
I know its easy to criticize what happened after the fact and having a clear(er) picture of all the moving parts and the timeline of events, but I think that while most of the people in the thread are pointing out either Rust-related or lack of configuration validation, what really grinds my gears is something that - in my opinion - is bad engineering.
Having an unprivileged application querying system.columns to infer the table layout is just bad; Not having a proper, well-defined table structure indicates sloppiness in the overall schema design, specially if it changes quickly. Considering specifically clickhouse, and even if this approach would be a good idea, the unprivileged way of doing it would be "DESCRIBE TABLE <name>", NOT iterating system.columns. The gist of it - sloppy design not even well implemented.
Having a critical application issuing ad-hoc commands to system.* tablespace instead of using a well-tested library is just amateurism, and again - bad engineering; IMO it is good practice to consider all system.* privileged applications and ensure their querying is completely separate from your application logic; Sometimes some system tables change, and fields are added and/or removed - not planning for this will basically make future compatibility a nightmare.
Not only the problematic query itself, but the whole context of this screams "lack of proper application design" and devs not knowing how to use the product and/or read the documentation. Granted, this is a bit "close to home" for me, because I use ClickHouse extensively (at a scale - I'm assuming - several orders of magnitude smaller than CloudFlare) and I have spent a lot of time designing specifically to avoid at least some of these kind of mistakes. But, if I can do it at my scale, why aren't they doing it?
On all the other issues, I thought they wanted to do the right thing at heart, but missed to make it fail safe. I can pass it as a problem of a journey to maturity or simply the fact that you can't get everything perfect. Maybe even a bit of sloppiness here and there.
The database issue screamed at me: lack of expertise. I don't use CH, but seeing someone to mess with a production system and they being surprised "Oh, it does that?", is really bad. And this is obviously not knowledge that is hard to achieve, buried deep in a manual or an edge case only discoverable by source code, it's bread and butter knowledge you should know.
What is confusing, that they didn't add this to their follow-up steps. With some benefit of doubt I'd assume they didn't want to put something very basic as a reason out there, just to protect the people behind it from widespread blame. But if that's not the case, then it's a general problem. Sadly it's not uncommon that components like databases are dealt with, on an low effort basis. Just a thing we plug in and works. But it's obviously not.
I don't think these are realistic requirements for any engineered system to be honest. Realistic is to have contingencies for such cases, which are simply errors.
But the case for Cloudflare here is complicated. Every engineer is very free to make a better system though.
What is not realistic? To do simple input validation on data that has the potential to break 20% of the internet? To not have a system in place to rollback to the latest known state when things crash?
Cloudflare builds a global scale system, not an iphone app. Please act like it.
> Why did they think this was a cyberattack and only after two hours realize it was the config file?
They explain that at some length in TFA.
> I'm migrating my customers off Cloudflare.
Is that an overreaction?
Name me global, redundant systems that have not (yet) failed.
And if you used cloudflare to protect against botnet and now go off cloudflare... you are vulnerable and may experience more downtime if you cannot swallow the traffic.
I mean no service have 100% uptime - just that some have more nines than others.
There are many self-hosted alternatives to protect against botnet. We don't have to use cloudflare. Everthing is under their control!
We had better uptime with AWS WAF in us-east-1 than we've had in the last 1.5 years of Cloudflare.
I do like the flat cost of Cloudflare and feature set better but they have quite a few outages compared to other large vendors--especially with Access (their zero trust product)
I'd lump them into GitHub levels of reliability
We had a comparable but slightly higher quote from an Akamai VAR.
[dead]
Yes, it's probably an overreaction.
But at the same time, what value do they add if they:
* Took down the the customers sites due to their bug.
* Never protected against an attack that our infra could not have handled by itself.
* Don't think that they will be able to handle the "next big ddos" attack.
It's just an extra layer of complexity for us. I'm sure there are attacks that could help our customers with, that's why we're using them in the first place. But until the customers are hit with multiple ddos attacks that we can not handle ourself then it's just not worth it.
I agree. I think the comments about how "it is fine, because so many things had to fail" do not apply in this case.
It's not that many things had to fail, it's that many things that are obvious haven't been done. It would be a valid excuse if many "exotic" scenarios would have to align, not when it's obvious error cases that weren't handled and changes have not been tested.
While having wrong first assumptions is just how things work when you try to analyze the issue[1], not testing changes before production is just stupidity and nothing else.
The story would be different if eg. multiple unlikely, hard to track things happened at once without anyone making a clearly linkable event, something that would also happen in staging. Most of the things mentioned could essentially statically checked. This is the prime example of what you want as any tech person, because it's not hard to prevent compared to a lot of scenarios where you deal with balancing likelihoods of scenarios, timings, etc.
You don't think someone is a great plumber, because they forgot their tools and missed that big hole in the pipe and also rang at the wrong door, because all these things failed. You think someone is a good plumber if they said they would have to go back to fetch a bulky specialized tool, because this is the rare case in which they need it, but they could also do this other thing in this spcific case. They are great plumbers if they tell you how this happened in first place and how to fix it. They are great plumbers if they manage to fix something outside of their usual scope.
Here pretty much all of the things that you pay them for failed. At a large scale.
I am sure this has there are reasons which we don't now about, and I hope that CloudFlare can fix them. Be it management focusing on the wrong things, be it developers not being in the wrong position or annoyed enough to care or something else entirely. However, not doing these things is (likely) a sign that currently they are not in the state of creating reliable systems - at least none reliable enough for what they are doing. It would be perfectly fine if they ran a web shop or something, but if as experienced many other companies rely on you being up or their stuff fails, then maybe you should not run a company with products like "Always Online".
[1] And should make you adapt the process of analyzing issues. Eg. making sure config changes are "very loud" in monitoring. It's one of the most easily tracked thing that can go wrong, and can relatively easily be mapped to a point in time compared to many other things.
So where are you migrating to?
Does their ring based rollout really truly have to be 0->100% in a few seconds?
I don’t really buy this requirement. At least make it configurable with a more reasonable default for “routine” changes. E.g. ramping to 100% over 1 hour.
As long as that ramp rate is configurable, you can retain the ability to respond fast to attacks by setting the ramp time to a few seconds if you truly think it’s needed in that moment.
The configuration file is updated every five minutes, so clearly they have some past experience where they’ve decided an hour is too long. That said, even a roll out over five minutes can be helpful.
I think defence against a DDOS against your network is the best reason for a quick rollout
This was not about DDoS defense but the Bot Management feature, which is a paid Enterprise-only feature not enabled by default to block automated requests regardless of whether an attack is going on.
https://developers.cloudflare.com/bots/get-started/bot-manag...
Maybe, but in that case maybe have some special casing logic to detect that yes indeed we're under a massive DDOS at this very moment, do a rapid rollout of this thing that will mitigate said DDOS. Otherwise use the default slower one?
Of course, this is all so easy to say after the fact..
Classic @devops_borat
"To make error is human. To propagate error to all server in automatic way is #devops"
> "To make error is human. To propagate error to all server in automatic way is #devops"
This saying dates back to 1969: To err is human but to really foul things up requires a computer.
* https://quoteinvestigator.com/2010/12/07/foul-computer/
Also: I know there’s a proverb which says ‘To err is human,’ but a human error is nothing to what a computer can do if it tries.
* https://quoteinvestigator.com/2017/05/26/computer-error/
I miss him. It must be more than 10 years now
> Their bot management system is designed to push a configuration out to their entire network rapidly.
Once every 5m is not "rapidly". It isn't uncommon for configuration systems to do it every few seconds [0].
> While it’s certainly useful to examine the root cause in the code.
Believe the issue is as much an output from a periodic run (clickhouse query) caused by (on the surface, an unrelated change) causing this failure. That is, the system that validated the configuration (FL2) was different to the one that generated it (ML Bot Management DB).
Ideally, it is the system that vends a complex configuration that also vends & tests the library to consume it, or the system that consumes it, does so as if it was "tasting" the configuration first before devouring it unconditionally [1].
Of course, as with all distributed system failures, this is all easier said and done in hindsight.
[0] Avoiding overload in distributed systems by putting the smaller service in control (pg 4), https://d1.awsstatic.com/builderslibrary/pdfs/Avoiding%20ove...
[1] Lessons from CloudFront (2016), https://youtube.com/watch?v=n8qQGLJeUYA&t=1050
>Once every 5m is not "rapidly".
Isn't rapidly more of how long it takes to get from A to Z rather than how often it is performed? You can push out a configuration update every fortnight but if it goes through all of your global servers in three seconds, I'd call it quite rapid.
By rapid I mean a rapid rollout of changes to 100% of the fleet, not how often changes are made.
Thanks for sharing that AWS doc
I don't understand why they didn't validate and sanitize the new config file revision. If bad(whatever that reason is) throw an error and revert back to previous version. You don't need to take down the whole internet for that.
Same as for almost every bug I think: the dev in question hadn't considered that the input could be bad in the way that it turned out to be. Maybe they were new, or maybe they hadn't slept much because of a newborn baby, or maybe they thought it was a reasonable assumption that there would never be more than 200 ML features in the array in question. I don't think this developer will ever make the same mistake again at least.
Let those who have never written a bug before cast the first stone.
> Maybe they were new, or maybe they hadn't slept much because of a newborn baby
Reminds me of House of Dynamite, the movie about nuclear apocalypse that really revolves around these very human factors. This outage is a perfect example of why relying on anything humans have built is risky, which includes the entire nuclear apparatus. “I don’t understand why X wasn’t built in such a way that wouldn’t mean we live in an underground bunker now” is the sentence that comes to mind.
I don't think this is an error originating from a single human. At CF scale I'd expect that multiple humans saw that code and gave it a pass. Rust or not, but an experienced dev could have seen this can lead to issues. Panicking without restoring a healthy state is just not an option in this case. They *know* that.
I guess you are right, likely a social issue, but certainly not a single exhausted parent.
> I don't understand why they didn't validate and sanitize the new config file revision.
The new config file was not (AIUI) invalid (syntax-wise) but rather too big:
> […] That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
> The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.
if the config is too big, then its an invalid config
Exactly the right take. Even when you want to have rapid changes on your infra, do it at least by region. You can start with the region where the least amount of users are impacted and if everything is fine, there is no elevated number of crashes for example, you can move forward. It was a standard practice at $RANDOM_FAANG when we had such deployments.
Thank you. I am sympathetic to CF’s need to deploy these configs globally fast and don’t think slowing down their DDoS mitigation is necessarily a good trade off. What I am saying is this presents a bigger reliability risk and needs correspondingly fine crafted observability around such config changes and a rollback runbook. Greater risk -> greater attention.
But the rapid deployment mechanism for bot features wasn’t where the bug was introduced.
In fact, the root bug (faulty assumption?) was in one or more SQL catalog queries that were presumably written some time ago.
(Interestingly the analysis doesn’t go into how these erroneous queries made it into production OR whether the assumption was “to spec” and it’s the security principal change work that was faulty. Seems more likely to be the former.)
I've also led a team of Incident Commanders at a FAANG.
If this was a routine config change, I could see how it could take 2 hours to start the mediation plan. However they should have dashboards that correlate config setting changes with 500 errors (or equivalent). It gets difficult when you have many of of these going out at the same time and they are slowly rolled out.
The root cause document is mostly for high level and the public. The details on this specific outage will be in a internal document with many action items, some of them maybe quarter long projects including fixing this specific bug and maybe some linter/monitor to prevent it from happening again.
Thanks for this assessment.
In a productive way, this view also shifts the focus to improving the system (visibility etc), empowering the team, rather than focusing on the code which broke (probably strikes fear in the individuals, to do anything!)
The "coding error" is a somewhat deliberate choice to fail eagerly that is usually safe but doesn't align with the need to do something (propagation of the configuration file) without failing.
I'm sure that there are misapplied guidelines to do that instead of being nice to incoming bot management configuration files, and someone might have been scolded (or worse) for proposing or attempting to handle them more safely.
Rolling out new code should be done differently than rolling out new data to fight bots.
If every time there's a new bot someone needs to write code that can blow up their whole service, maybe they need to iterate a bit on this design?
This isn't what they do, though. This is a data/config push - original article says _a “feature file” used by our Bot Management system_
I would say that whilst this is a good top down view, that `.unwrap()` should have been caught at code-review and not allowed. Clippy rule could have saved a lot of money.
That and why the hell wasn't their alerting showing up colossal amount of panics in their bot manager thing?
Yes the lack of observability is really the disturbing bit here. You have panics in a bunch of your core infrastructure, you would expect there to be a big red banner on the dashboard that people look at when they first start troubleshooting an incident.
This is also a pretty good example why having stack traces by default is great. That error could have been immediately understood just from a stack trace and a basic exception message.
You can write the safest code in the world, but if you're shipping config changes globally every few minutes without a robust rollback plan or telemetry that pinpoints when things go sideways, you're flying blind
Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code.
And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.
Exactly. This should be the default for production code at companies like Cloudflare.
https://github.com/search?q=unwrap%28%29+language%3ARust&typ...
This is sobering.
My new fear is some dependency unwrap()ing or expect()ing something where they didn't prove the correctness.
Unwrap() and expect() are an anti-pattern and have no place in idiomatic Rust code. The language should move to deprecate them.
What would the difference be if they had enforced no unwraps/expects/slicing and instead logged the error and returned a 500?
As the user, I can't tell the difference, but it might have sped up their recovery a bit.
You could argue that explicitly writing down the assumption would make it clearer to yourself and your reviewer that it might be overly optimistic.
Pretty much - the time spent ruling out the hypothesis that it was a cyberattack would have been time spent investigating the uptick in deliberately written error logs, since you would expect alerts to be triggered if those exceed a threshold.
I imagine it would also require less time debugging a panic. That kind of breadcrumb trail in your logs is a gift to the future engineer and also customers who see a shorter period of downtime.
I would love to go further and explicitely forbid unwrap and similar calls using a `no_panic` attribute.
I actually have to do this for programs that runs in bare metal. You can't afford to have nondeterministic panic like this. If things really gone wrong you'd have a watchdog and health checker to verify the state of program.
It is just 2 different layers. Of course the code is also a problem, if it is in fact as the GP describes it. You are taking the higher level view, which is the second layer of dealing with not only this specific mistake, but also other mistakes, that can be related to arbitrary code paths.
Both are important, and I am pretty sure, that someone is gonna fix that line of code pretty soon.
The bot is efficient. This is by design. It will push out mistakes just as efficiently as it pushes out good changes. Good or bad... the plane of control is unchanged.
This is the danger of automated control systems. If they get hacked or somehow push out bad things (CloudStrike), they will have complete control and be very efficient.
This guy SREs
Back when it meant Site Reliability Engineer and not Sysadmin Really Expensive
[dead]
It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic.
It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this.
> every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this.
How about indexing into a slice/map/vec? Should every `foo[i]` have an infallibility comment? Because they're essentially `get(i).unwrap()`.
Yes? Funnily enough, I don't often use indexed access in Rust. Either I'm looping over elements of a data structure (in which case I use iterators), or I'm using an untrusted index value (in which case I explicitly handle the error case). In the rare case where I'm using an index value that I can guarantee is never invalid (e.g. graph traversal where the indices are never exposed outside the scope of the traversal), then I create a safe wrapper around the unsafe access and document the invariant.
If that's the case then hats off. What you're describing is definitely not what I've seen in practice. In fact, I don't think I've ever seen a crate or production codebase that documents infallibility of every single slice access. Even security-critical cryptography crates that passed audits don't do that. Personally, I found it quite hard to avoid indexing for graph-heavy code, so I'm always on the lookout for interesting ways to enforce access safety. If you have some code to share that would be very interesting.
Usually you'd want to write almost all your slice or other container iterations with iterators, in a functional style.
For the 5% of cases that are too complex for standard iterators? I never bother justifying why my indexes are correct, but I don't see why not.
You very rarely need SAFETY comments in Rust because almost all the code you write is safe in the first place. The language also gives you the tool to avoid manual iteration (not just for safety, but because it lets the compiler eliminate bounds checks), so it would actually be quite viable to write these comments, since you only need them when you're doing something unusual.
I didn't restate the context from the code we're discussing: it must not panic. If you don't care if the code panics, then go ahead and unwrap/expect/index, because that conforms to your chosen error handling scheme. This is fine for lots of things like CLI tools or isolated subprocesses, and makes review a lot easier.
So: first, identify code that cannot be allowed to panic. Within that code, yes, in the rare case that you use [i], you need to at least try to justify why you think it'll be in bounds. But it would be better not to.
There are a couple of attempts at getting the compiler to prove that code can't panic (e.g., the no-panic crate).
For iteration, yes. But there's other cases, like any time you have to deal with lots of linked data structures. If you need high performance, chances are that you'll have to use an index+arena strategy. They're also common in mathematical codebases.
I mean... yeah, in general. That's what iterators are for.
Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handle the error and try to recover, or if impossible, at least do `expect("damn it, how did this happen")`.
> Yes, I always thought it was wrong to use unwrap in examples.
And because it gets picked up by LLMs. It would be interesting to know if this particular .unwrap() was written by a human.
There is a prevailing mentality that LLMs make it easy to become productive in new languages, if you are already proficient in one. That's perhaps true until you suddenly bump up against the need to go beyond your superficial understanding of the new language and its idiosyncrasies. These little collisions with reality occur until one of them sparks an issue of this magnitude.
In theory, experienced human code reviewers can course correct newer LLM-guided devs work before it blows up. In practice, reviewers are already stretched thin and submitters absolute to now rapidly generate more and more code to review makes that exhaustion effect way worse. It becomes less likely they spot something small but obvious amongst the haystack of LLM generated code bailing there way.
I have removed many 'unwraps' from copilot and cursor generated code, but cursor is a lot better at it than copilot was.
Dunno, I think the alternatives have their own pretty significant downsides. All would require front loading more in-depth understanding of error handling and some would just be quite a bit more verbose.
IMO making unwrap a clippy lint (or perhaps a warning) would be a decent start. Or maybe renaming unwrap.
This strikes me as a culture issue more than one of language.
A tenet of systems code is that every possible error must be handled explicitly and exhaustively close to the point of occurrence. It doesn’t matter if it is Rust, C, etc. Knowing how to write systems code is unrelated to knowing a systems language. Rust is a systems language but most people coming into Rust have no systems code experience and are “holding it wrong”. It has been a recurring theme I’ve seen with Rust development in a systems context.
C is pretty broken as a language but one of the things going for it is that it has a strong systems code culture surrounding it that remembers e.g. why we do all of this extra error handling work. Rust really needs systems code practice to be more strongly visible in the culture around the language.
> at least do `expect("damn it, how did this happen")`
That gives you the same behavior as unwrap with a less useful error message though. In theory you can write useful messages, but in practice (and your example) expect is rarely better than unwrap in modern rust
We shouldn't be using unwrap() or expect() at all.
This is Rust's Null Pointer Exception.
unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features.
The language should grow the ability to mark this code as dangerous, and we should have static tools to exclude this code from our dependency tree.
I don't want some library I use to `unwrap()` and cause my application to crash because I didn't anticipate their stupid panic.
Rust developers have clearly leaned on this crutch far too often:
https://github.com/search?q=unwrap%28%29+language%3ARust&typ...
The Rust team needs to plug this leak.
I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality.
Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled.
If you need the value, you need to handle the case that it’s not available explicitly. You need to define your error path(s)
Anything else leads to, well, this.
This.
This is a failure caused by lazy Rust programming and not relying on the language's design features.
It's a shame this code can even be written. It is surprising and escapes the expected safety of the language.
I'm terrified of some dependency using unwrap() or expect() and crashing for something entirely outside of my control.
We should have an opt-in strict Cargo.toml declaration that forbids compilation of any crate that uses entirely preventable panics. The only panics I'll accept are those relating to memory allocation.
This is one of the sharpest edges in the language, and it needs to be smoothed away.
The blog linked in the GP anticipates this rebuttal and already addresses it.
Your argument also implies that things like `slice[i]` are never okay.
> If you have to use unwrap, your types do not match your problem
The problem starts with Rust stdlib. It panics on allocation failure. You expect Rust programmers to look at stdlib and not imitate it?
Sure, you can try to taboo unwrap(), but 1) it won't work, and 2) it'll contort program design in places where failure really is a logic bug, not a runtime failure, and for which unwrap() is actually appropriate.
The real solution is to go back in time, bonk the Rust designers over the head with a cluebat, and have them ship a language that makes error propagation the default and syntactically marks infallible cleanup paths --- like C++ with noexcept.
It's the same blind spot people have to Java's checked exceptions. People commonly resort to Pokemon exception handling and either blindly ignoring or rethrowing as a runtime exception. When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.
"Checked Exceptions Are Actually Good" gang, rise up! :p
I think adoption would have played out very different if there had only been some more syntactic-sugar. For example, an easy syntax for saying: "In this method, any (checked) DeepException e that bubbles up should immediately be replaced by a new (checked) MylayerException(e) that contains the original one as a cause.
We might still get lazy programmers making systems where every damn thing goes into a generic MylayerException, but that mess would still be way easier to fix later than a hundred scattered RuntimeExceptions.
Exception handling would be better than what we're seeing here.
The problem is that any non-trivial software is composition, and encapsulation means most errors aren't recoverable.
We just need easy ways to propagate exceptions out to the appropriate reliability boundary, ie. the transaction/ request/ config loading, and fail it sensibly, with an easily diagnosable message and without crashing the whole process.
C# or unchecked Java exceptions are actually fairly close to ideal for this.
The correct paradigm is "prefer throw to catch" -- requiring devs to check every ret-val just created thousands of opportunities for mistakes to be made.
By contrast, a reliable C# or Java version might have just 3 catch clauses and handle errors arising below sensibly without any developer effort.
https://literatejava.com/exceptions/ten-practices-for-perfec...
I'm with you! Checked exceptions are actually good and the hate for them is super short sighted. The exact same criticisms levied at checked exceptions apply to static typing in general, but people acknowledge the great value static types have for preventing errors at compile time. Checked exceptions have that same value, but are dunked on for some reason.
There is also the problem that they decided to make all references nullable, so `NullPointerException`s could appear everywhere. This "forced" them to introduce the escape hatch of `RuntimeException`, which of course was way overused immediately, normalizing it.
It's a lot lighter: a stack trace takes a lot of overhead to generate; a result has no overhead for a failure. The overhead (panic) only comes once the failure can't be handled. (Most books on Java/C# don't explain that throwing exceptions has high performance overhead.)
Exceptions force a panic on all errors, which is why they're supposed to be used in "exceptional" situations. To avoid exceptions when an error is expected, (eof, broken socket, file not found,) you either have to use an unnatural return type or accept the performance penalty of the panic that happens when you "throw."
In Rust, the stack trace happens at panic (unwrap), which is when the error isn't handled. IE, it's not when the file isn't found, it's when the error isn't handled.
> Exceptions force a panic on all errors
What do you mean?
Exceptions do not force panic at all. In most practical situations, an exception unhandled close to where it was thrown will eventually get logged. It's kind of a "local" panic, if you will, that will terminate the specific function, but the rest of the program will remain unaffected. For example, a web server might throw an exception while processing a specific HTTP request, but other HTTP requests are unaffected.
Throwing an exception does not necessarily mean that your program is suddenly in an unsupported state, and therefore does not require terminating the entire program.
> a stack trace takes a lot of overhead to generate
Can't Hotspot not generate the stack trace when it knows the exception will be caught and the stack trace ignored?
> it's essentially a checked exception without a stack trace
In theory, theory and practice are the same. In practice...
You can't throw a checked exception in a stream, this fact actually underlines the key difference between an exception and a Result: Result is in return position and exceptions are a sort of side effect that has its own control flow. Because of that, once your method throws an Exception or you are writing code in a try block that catches an exception, you become blind to further exceptions of that type, even if you might be able to or required to fix those errors. Results are required to be handled individually and you get syntactic sugar to easily back propagate.
It is trivial to include a stack trace, but stack traces are really only useful for identifying where something occurred, and generally what is superior is attaching context as you back propagate which trivially occurs with judicious use of custom error types with From impls. Doing this means that the error message uniquely defines the origin and paths it passed through without intermediate unimportant stack noise. With exceptions you would always need to catch each exception and rethrow a new exception containing the old to add contextual information, then to avoid catching to much you need variables that will be initialized inside the try block defined outside of the try block. So stack traces are basically only useful when you are doing Pokemon exception handling.
> When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.
It's not a checked exception without a stack trace.
Rust doesn't have Java's checked or unchecked exception semantics at the moment. Panics are more like Java's Errors (e.g. OOM error). Results are just error codes on steroids.
checked exceptions failed because when used properly they fossilize method signatures. they're fine if your code will never be changed and they're fine when you control 100% of users of the throwing code. if you're distributing a library... no bueno.
That’s just not true. They required that you use hierarchical exception types and define your own library exception type that you declare at the boundary.
The same is required for any principled error handling.
Pet peeve: unwrap() should be deprecated and renamed or_panic(). More consistent with the rest of stdlib methods and appropriately scarier.
That's kind of what I'm saying with the blind spot comment. The words "unwrap" and "expect" should be just as much a scary red flag as the word "panic", but for some reason it seems a lot of people don't see them that way.
Even in lowly Java, they later added to Optional the orElseThrow() method since the name of the get() method did not connote the impact of unwrapping an empty Optional.
I've found both methods very useful. I'm using `get()` when I've checked that the value is present and I don't expect any exceptions. I'm using `orElseThrow()` when I actually expect that value can be absent and throwing is fine. Something like
Idea checks it by default and highlights if I've used `get()` without previous check. It's not forced at compiler level, but it's good enough for me.A lot of stuff should be done about the awful unwrap family of methods.
A few ideas:
- It should not compile in production Rust code
- It should only be usable within unsafe blocks
- It should require explicit "safe" annotation from the engineer. Though this is subject to drift and become erroneous.
- It should be possible to ban the use of unsafe in dependencies and transitive dependencies within Cargo.
The `unsafe` keyword means something specific in Rust, and panicking isn't unsafe by Rust's definition. Sometimes avoiding partial functions just isn't feasible, and an unwrap (or whatever you want to call the method) is a way of providing a (runtime-checked) proof to the compiler that the function is actually total.
Than they are going to write None | Err => yolo() that has the same impact. It is not the syntax or the semantic meaning is the problem here but the fact that there is no monitoring around the elevated error counts after a deployment.
Software engineers tend to get stuck in software problems and thinking that everything should be fixed in code. In reality there are many things outside of the code that you can do to operate unreliable components safely.
This thread warms my heart. Rust has set a new baseline that many and myself now take for granted.
We are now discussing what can be done to improve code correctness beyond memory and thread safety. I am excited for what is to come.
Really not! This is a huge faceplant for writing things in Rust. If they had been writing their code in Java/Kotlin instead of Rust, this outage either wouldn't have happened at all (a failure to load a new config would have been caught by a defensive exception handler), or would have been resolved in minutes instead of hours.
The most useful thing exceptions give you is not static compile time checking, it's the stack trace, error message, causal chain and ability to catch errors at the right level of abstraction. Rust's panics give you none of that.
Look at the error message Cloudflare's engineers were faced with:
That's useless, barely better than "segmentation fault". No wonder it took so long to track down what was happening.A proxy stack written in a managed language with exceptions would have given an error message like this:
and so on. It'd have been immediately apparent what went wrong. The bad configs could have been rolled back in minutes instead of hours.In the past I've been able to diagnose production problems based on stack traces so many times I was been expecting an outage like this ever since the trend away from providing exceptions in new languages in the 2010s. A decade ago I wrote a defense of the feature and I hope we can now have a proper discussion about adding exceptions back to languages that need them (primarily Go and Rust):
https://blog.plan99.net/what-s-wrong-with-exceptions-nothing...
That has nothing to do with exceptions, just the ability to unwind the stack. Rust can certainly give you a backtrace on panics; you don’t even have to write a handler to get it. I would find it hard to believe Cloudflare’s services aren’t configured to do it. I suspect they just didn’t put the entire message in the post.
https://doc.rust-lang.org/std/backtrace/index.html#environme...
tldr: Capturing a backtrace can be a quite expensive runtime operation, so the environment variables allow either forcibly disabling this runtime performance hit or allow selectively enabling it in some programs.
By default it is disabled in release mode.
Alternatively you can look at actually innovative programming languages to peek at the next 20 years of innovation.
I am not sure that watching the trendy forefront successfully reach the 1990s and discuss how unwrapping Option is potentially dangerous really warm my heart. I can’t wait for the complete meltdown when they discover effect systems in 2040.
To be more serious, this kind of incident is yet another reminder that software development remains miles away from proper engineering and even key providers like Cloudfare utterly fail at proper risk management.
Celebrating because there is now one popular language using static analysis for memory safety feels to me like being happy we now teach people to swim before a transatlantic boat crossing while we refuse to actually install life boats.
To me the situation has barely changed. The industry has been refusing to put in place strong reliability practices for decades, keeps significantly under investing in tools mitigating errors outside of a few fields where safety was already taken seriously before software was a thing and keeps hiding behind the excuse that we need to move fast and safety is too complex and costly while regulation remains extremely lenient.
I mean this Cloudfare outage probably cost millions of dollars of damage in aggregate between lost revenue and lost productivity. How much of that will they actually have to pay?
Let's try to make effect systems happen quicker than that.
> I mean this Cloudfare outage probably cost millions of dollars of damage in aggregate between lost revenue and lost productivity. How much of that will they actually have to pay?
Probably nothing, because most paying customers of cloudflare are probably signing away their rights to sue Cloudflare for damages by being down for a while when they purchase Cloudflare's services (maybe some customers have SLAs with monetary values attached, I dunno). I honestly have a hard time suggesting that those customers are individually wrong to do so - Cloudflare isn't down that often, and whatever amount it cost any individual customer by being down today might be more than offset by the DDOS protection they're buying.
Anyway if you want Cloudflare regulated to prevent this, name the specific regulations you want to see. Should it be illegal under US law to use `unwrap` in Rust code? Should it be illegal for any single internet services company to have more than X number of customers? A lot of the internet also breaks when AWS goes down because many people like to use AWS, so maybe they should be included in this regulatory framework too.
I find myself in the odd position of agreeing with you both.
That we’re even having this discussion is a major step forward. That we’re still having this discussion is a depressing testament to how slow slowly the mainstream has adopted better ideas.
I agree with you. But onsidering nobody learns any real engineering in software; myself solidly included; this is still an improvement.
But yes, I wish I had learned more, and somehow stumbled upon all the good stuff, or be taught at university about at least what Rust achieves today.
I think it has to be noted Rust still allows performance with the safety it provides. So that's something maybe.
> I can’t wait for the complete meltdown when they discover effect systems in 2040
Zig is undergoing this meltdown. Shame it's not memory safe. You can only get so far in developing programming wisdom before Eternal September kicks in and we're back to re-learning all the lessons of history as punishment for the youthful hubris that plagues this profession.
> people have a blind spot for unwrap
It's not about whether you should ban unwrap() in production. You shouldn't. Some errors are logic bugs beyond which a program can't reasonably continue. The problem is that the language makes it too easy for junior developers (and AI!) to ignore non-logic-bug problems with unwrap().
Programmers early in their careers will do practically anything to avoid having to think about errors and they get angry when you tell them about it.
> In production code an unwrap or expect should be reviewed exactly like a panic.
An unwrap should never make it to production IMHO. It's fine while prototyping, but once the project gets closer to production it's necessary to just grep `uncheck` in your code and replace those that can happen with a proper error management and replace those that cannot happen with `expect`, with a clear justification of why they cannot happen unless there's a bug somewhere else.
I would say, sure, if you feel the same way about panic calls making to production. In other words, review all of them the same way. Because writing unwrap/expect is exactly the same as writing “if error, panic”.
I don't understand your point: panic! is akin to expect: you think about it consciously, use it explicitly and you write down a panic message explaining its rational.
unwrap isn't like that.
The problem here is that for a service "fail fast" is mostly the best default.
> It seems people have a blind spot for unwrap
Not unlike people having a blind spot for Rust in general, no?
Isn't the point of this article that pieces of infrastructure don't go down to root causes, but due to bad combinations of components that are correct individually? After reading "engineering a safer world", I find root cause analysis rather reductionistic, because it wasn't just an unwrap, it was that the payload was larger than normal, because of a query that didn't select by database, because a clickhouse made more databases visible. Hard to say "it was just due to an unwrap" imo. Especially in terms of how to fix an issue going forwards. I think the article lists a lot of good ideas, that aren't just "don't unwrap", like enabling more global kill switches for features, or eliminating the ability for core dumps or other error reports to overwhelm system resources.
You're right. A good postmortem/root cause analysis would START from "unwrap" and continue from there.
You might start with a basic timeline of what happened, then you'd start exploring: why did this change affect so many customers (this would be a line of questioning to find a potential root cause), why did it take so long to discover or recover (this might be multiple lines of questioning), etc.
> This is the multi-million dollar .unwrap() story.
That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.
>If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.
If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.
> If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.
It wouldn't, not meaningfully. The outage was caused by change in how they processed the queries. They had no way to observe the changes, nor canaries to see that change is killing them. Plus, they would still need to manually feed and restart services that ingested bad configs.
`expect` would shave a few minutes; you would still spend hours figuring out and fixing it.
Granted, using expect is better, but it's not a silver bullet.
A billion alerts in DD/Sentry/whatever saying the exact problem that coincide with the exact graph of failures would probably be helpful if someone looked at them.
If the `.unwrap()` was replaced with
1:
2: 3: 4: Then it would have been idiomatic Rust code and wouldn't have failed at all.The function signature returned a `Result<(), (ErrorFlags, i32)>`
Seems like it should have returned an Err((ErrorFlags, i32)) here. Case 2 or 3 above would have done nicely.
Removing unwrap() from Rust would have forced the proper handling of the function call and would have prevented this.
Unwrap() is Rust's original sin.
In general for unexpected errors like these the internal function should log the error, and I assume it was either logged or they can quickly deduce reason based on the line number.
And dare I say, an exhibition of hindsight bias.
semantic? or pedantic?
[flagged]
I'm not sure if this is serious or not, but to take it at face value: the value of this sort of thing in Rust is not that it prevents crashes altogether but rather that it prevents _implicit_ failures. It forces a programmer to make the explicit choice of whether to crash.
There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this). But when you can't, we leave a comment explaining why it's safe.
The language semantics do not surface `unwrap` usage or make any guarantees. It should be limited to use in `unsafe` blocks.
> There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this). But when you can't, we leave a comment explaining why it's safe.
I would prefer the boiler plate of a match / if-else / if let, etc. to call attention to it. If you absolutely must explicitly panic. Or better - just return an error Result.
It doesn't matter how smart your engineers are. A bad unwrap can sneak in through refactors, changing business logic, changing preconditions, new data, etc.
blaming the language is not the way to approach this. if an engineer writes bad code that’s the engineers fault, not the languages.
this was bad code that should have never hit production, it is not a rust language issue.
No. Don't say "you're holding it wrong". The language says "safe" on the tin. It advertises safety. This shouldn't be possible.
This is a null pointer. In Rust.
Unwrap needs to die. We should all fight to remove it.
[flagged]
>> This is the multi-million dollar .unwrap() story.
> That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.
Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assuming `features.append_with_names` would never fail. Instead, the routine likely should have worked within `Result`.
> Instead, the routine likely should have worked within `Result`.
But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same.
Maybe you're saying "it's better to be explicit", as a broad generalization I don't disagree with that.
But that has nothing to do with the actual bug here, which was that the invariant failed. How they choose to implement checking and failing the invariant in the semantics of the chosen language is irrelevant.
>> Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assuming `features.append_with_names` would never fail. Instead, the routine likely should have worked within `Result`.
> But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same.
I agree it is an error, but disagree that it should be a fatal error at that location. The reason being is the method defining the offending `unwrap` construct produces a `Result`, which is fully capable of representing any error `features.append_with_names` could produce.
> But that has nothing to do with the actual bug here, which was that the invariant failed.
The bug is by invoking `unwrap` the process crashed. To the degree that Cloudfare had a massive outage.
Had the logic been such that a `Result` representing this error condition activated an alternate workflow to handle the error (perhaps by logging it, emitting a notification event alerting SRE's, transitioning into a failure mode, or all of these options), then a global outage might have been averted.
Which makes:
> How they choose to implement checking and failing the invariant in the semantics of the chosen language is irrelevant.
Very relevant indeed.
A failed config load probably shouldn't be a fatal error if a valid config is already loaded?
I'm not completely sure I agree. I mean, I do agree about the .unwrap() culture being a bug trap. But I don't think this example qualifies.
The root cause here was that a file was mildly corrupt (with duplicate entries, I guess). And there was a validation check elsewhere that said "THIS FILE IS TOO BIG".
But if that's a validation failure, well, failing is correct? What wasn't correct was that the failure reached production. What should have happened is that the validation should have been a unified thing and whatever generated the file should have flagged it before it entered production.
And that's not an issue with function return value API management. The software that should have bailed was somewhere else entirely, and even there an unwrap explosion (in a smoke test or pre-release pass or whatever) would have been fine.
It sounds to me like there was validation, but the system wasn't designed for the validation to ever fail - at which point crashing is the only remaining option. You've essentially turned it into an assertion error rather than a parsing/validation error.
Ideally every validation should have a well-defined failure path. In the case of a config file rotation, validation failure of the new config could mean keeping the old config and logging a high-priority error message. In the case of malformed user-provided data, it might mean dropping the request and maybe logging it for security analysis reasons. In the case of "pi suddenly equals 4" checks the most logical approach might be to intentionally crash, as there's obviously something seriously wrong and application state has corrupted in such a way that any attempt to continue is only going to make things worse.
But in all cases there's a reason behind the post-validation-failure behavior. At a certain point leaving it up to "whatever happens on .unwrap() failure" isn't good enough anymore.
This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a normal thing to do. There's no graceful handling (beyond a nice error message) a program like Nginx could do on a syntax error in its config.
The real issue is further up the chain where the malformed feature file got created and deployed without better checks.
> panicking is a normal thing to do
I do not think that if the bot detection model inside your big web proxy has a configuration error it should panic and kill the entire proxy and take 20% of the internet with it. This is a system that should fail gracefully and it didn't.
> The real issue
Are there single "real issues" with systems this large? There are issues being created constantly (say, unwraps where there shouldn't be, assumptions about the consumers of the database schema) that only become apparent when they line up.
I don't know too much about how the feature file distribution works but in the event of failure to read a new file, wouldn't logging the failure and sticking with the previous version of the file be preferable?
That's exactly the point (ie just prior to distribution) where a simple sanity check should have been run and the config replacement/update pipeline stopped on failure. When they introduced the 200 entry limit memory optimised feature loader it should have been a no-brainer to insert that sanity check in the config production pipeline.
Or even truncating the features to their limit and alerting through logs that there is likely performance degradation in their Bot Management.
I'm really confused how so many people are finding it acceptable to bring down your entire reverse-proxy because the length of feature sets for the ML model in one of your components was longer than expected.
One feature failing like this should probably log the error and fail closed. It shouldn't take down everything else in your big proxy that sits in front of your entire business.
Yea, Rust is safe but it’s not magic. However Nginx doesn’t panic on malformed config. It exits with hopefully a helpful error code and message. The question is then could the cloudflare code have exited cleanly in a way that made recovery easier instead of just straight panicking.
Would expect with a message meet that criteria of exiting with a more helpful error message? From the postmortem it seems to me like they just didn’t know it even was panicing
> However Nginx doesn’t panic on malformed config. It exits with hopefully a helpful error code and message.
The thing I dislike most about Nginx is that if you are using it as a reverse proxy for like 20 containers and one of them is up, the whole web server will refuse to start up:
Obviously making 19 sites also unavailable just because one of them is caught in a crash loop isn't ideal. There is a workaround involving specifying variables, like so (non-Kubernetes example, regular Nginx web server running in a container, talking to other containers over an internal network, like Docker Compose or Docker Swarm): Sadly, if you try to use that approach, then you just get: Sadly, switching the redirect configuration away from the default makes some apps go into a redirect loop and fail to load: mostly legacy ones, where Firefox shows something along the lines of "The page isn't redirecting properly". It sucks especially badly if you can't change the software that you just need to run and suddenly your whole Nginx setup is brittle. Apache2 and Caddy don't have such an issue.That's to say that all software out there has some really annoying failure modes, even is Nginx is pretty cool otherwise.
Exactly! Sometimes exploding is simply the least bad option, and is an entirely sensible approach.
In this case it definitely wasn’t the least bad option though.
Falling back to a generic base configuration in the presence of an incoming invalid config file would probably be a sensible thing to do.
To be fair, this failed in the non-rust path too because the bot management returned that all traffic was a bot. But yes, FL2 needs to catch panics from individual components but I’m not sure if failing open is necessarily that much better (it was in this case but the next incident could easily be the result of failing open).
But more generally you could catch the panic at the FL2 layer to make that decision intentional - missing logic at that layer IMHO.
Catching panic probably isn’t a great idea if there’s any unsafe code in the system. (Do the unsafe blocks really maintain heap invariants if across panics?)
Unsafe blocks have nothing to do with it. Yes - they maintain all the same invariants as safe blocks or those unsafe blocks are unsound regardless of panics. But there’s millions of way to architect this (eg a supervisor process that notices which layer in FL2 is crashing and just completely disables that layer when it starts up the proxy again. There’s challenges here because then you have to figure out what constitutes a perma crashing (eg what if it’s just 20% of all sites? Do you disable?). And in the general case you have the fail open/fail close decision anyway which you should just annotate individual layers with.
But the bigger change is to make sure that config changes roll out gradually instead of all at once. That’s the source of 99% of all widespread outages
Incremental config changes sounds like it could lead to a LOT of bugs
I think the parent is implying that the panic should be "caught" via a supervisor process, Erlang-style, rather than implying the literal use of `catch_unwind` to resume within the same process.
Supervisor is the brutalist way. But catch_unwind may be needed for perf and other reasons.
But ultimately it’s not the panic that’s the problem but a failure to specify how panics within FL2 layers should be handled; each layer is at least one team and FL2’s job is providing a safe playground for everyone to safely coexist regardless of the misbehavior of any single component
But as always such failures are emblematic of multiple things going wrong at once. You probably want to end up using both catch_unwind for the typical case and the supervisor for the case where there’s a segfault in some unsafe code you call or native library you invoke.
I also mention the fundamental tension of do you want to fail open or closed. Most layers should probably fail open. Some layers (eg auth) it’s safer to fail closed.
The unwrap should be replaced by code that creates enough alerting to make a P0 incidident from their canary deployment immediately.
OR even, the bot code crashing should itself be generating alerts.
Canary deployment would be automatically rolled back until P0 incident resolved.
All of this could probably have happened and contained at their scale in less than a minute as they would likely generate enough "omg the proxy cannot handle its config" alerts off of a deployment of 0.001% near immediately.
Agreed - a big question why the file wasn’t test driven in staging and progressively rolled out. And also what alerting was missing within FL2 that they couldn’t pinpoint the unwrap instantly.
I'm not a fan of rust, but I don't think that is the only takeaway. All systems have assumptions about their input and if the assumption is violated, it has to be caught somewhere. It seems like it was caught too deep in the system.
Maybe the validation code should've handled the larger size, but also the db query produced something invalid. That shouldn't have ever happened in the first place.
> It seems like it was caught too deep in the system.
Agreed, that's also my takeaway.
I don't see the problem being "lazy programmers shouldn't have called .unwrap()". That's reductive. This is a complex system and complex system failures aren't monocausal.
The function in question could have returned a smarter error rather than panicking, but what then? An invariant was violated, and maybe this system, at this layer, isn't equipped to take any reasonable action in response to that invariant violation and dying _is_ the correct thing to do.
But maybe it could take smarter action. Maybe it could be restarted into a known good state. Maybe this service could be supervised by another system that would have propagated its failure back to the source of the problem, alerting operators that a file was being generated in such a way that violated consumer invariants. Basically, I'm describing a more Erlang model of failure.
Regardless, a system like this should be able to tolerate (or at least correctly propagate) a panic in response to an invariant violation.
The takeaway here isn’t about Rust itself, but that the Rust marketing crew’s claims that we constantly read on HN and elsewhere about the Result type magically saving you from making mistakes is not a good message to send.
They would also tell you that .unwrap() has no place in production code, and should receive as much scrutiny as an `unsafe` block in code review :)
The point of option is the crash path is more verbose and explicit than the crash-free path. It takes more code to check for NULL in C or nil in Go; it takes more code in Rust to not check for Err.
1. They don’t. There is presumably some hypothetical world where they would tell you if you start asking questions, but nobody buying into the sales pitch ever asks questions.
2. You’re getting confused by technology again. This isn’t about technology.
> Today, many friends pinged me saying Cloudflare was down. As a core developer of the first generation of Cloudflare FL, I'd like to share some thoughts.
> This wasn't an attack, but a classic chain reaction triggered by “hidden assumptions + configuration chains” — permission changes exposed underlying tables, doubling the number of lines in the generated feature file. This exceeded FL2's memory preset, ultimately pushing the core proxy into panic.
> Rust mitigates certain errors, but the complexity in boundary layers, data flows, and configuration pipelines remains beyond the language's scope. The real challenge lies in designing robust system contracts, isolation layers, and fail-safe mechanisms.
> Hats off to Cloudflare's engineers—those on the front lines putting out fires bear the brunt of such incidents.
> Technical details: Even handling the unwrap correctly, an OOM would still occur. The primary issue was the lack of contract validation in feature ingest. The configuration system requires “bad → reject, keep last-known-good” logic.
> Why did it persist so long? The global kill switch was inadequate, preventing rapid circuit-breaking. Early suspicion of an attack also caused delays.
> Why not roll back software versions or restart?
> Rollback isn't feasible because this isn't a code issue—it's a continuously propagating bad configuration. Without version control or a kill switch, restarting would only cause all nodes to load the bad config faster and accelerate crashes.
> Why not roll back the configuration?
> Configuration lacks versioning and functions more like a continuously updated feed. As long as the ClickHouse pipeline remains active, manually rolling back would result in new corrupted files being regenerated within minutes, overwriting any fixes.
https://x.com/guanlandai/status/1990967570011468071
This tweet thread invokes genuine despair in me. Do we really have to outsource even our tweets to LLMs? Really? I mean, I get spambots and the like tweeting mass-produced slop. But what compels a former engineer of the company in question to offer LLM-generated "insight" to the outage? Why? For what purpose?
* For clarity, I am aware that the original tweets are written in Chinese, and they still have the stench of LLM writing all over them; it's not just the translation provided in the above comment.
Out of interest... apart from the em dash, how else can you tell it's an LLM response? What are the telltales signs?
This particular excerpt is reeking of it with pretty much every line. I'll point out the patterns in the English translation, but all of these patterns apply cross-language.
> classic chain reaction triggered by “hidden assumptions + configuration chains”
"Classic/typical "x + y"", particularly when diagnosing an issue. This one is a really easy tell because humans, on aggregate, do not use quotation marks like this. There is absolutely no reason to quote these words here, and yet LLMs will do a combined quoted "x + y" where a human would simply write something natural like "hidden assumptions and configuration chains" without extraneous quotes.
> The configuration system requires “bad → reject, keep last-known-good” logic.
Another pattern with overeager usage of quotes is this ""x → y, z"" construct with very terse wording.
> This wasn't an attack, but a classic chain reaction
LLMs aggressively use "Not X, but Y". This is also a construct commonly used by humans, of course, but aside from often being paired with an em-dash, another tell is whether it actually contributes anything to the sentence. "Not X, but Y" is strongly contrasting and can add a dramatic flair to the thing being constrasted, but LLMs overuse it on things that really really don't need to be dramatised or contrasted.
> Rust mitigates certain errors, but the complexity in boundary layers, data flows, and configuration pipelines remains beyond the language's scope. The real challenge lies in designing robust system contracts, isolation layers, and fail-safe mechanisms.
Two lists of three concepts back-to-back. LLMs enjoy, love, and adore this construct.
> Hats off to Cloudflare's engineers—those on the front lines putting out fires bear the brunt of such incidents.
This kind of completely vapid, feel-good word soup utilising a heroic analogy for something relatively mundane is another tell.
And more broadly speaking, there's a sort of verbosity and emptiness of actual meaning that permeates through most LLM writing. This reads absolutely nothing like what an engineer breaking down an outage looks like. Like, the aforementioned line of... "Rust mitigates certain errors, but the complexity in boundary layers, data flows, and configuration pipelines remains beyond the language's scope. The real challenge lies in designing robust system contracts, isolation layers, and fail-safe mechanisms.". What is that actually communicating to you? It piles on technical lingo and high-level concepts in a way that is grammatically correct but contains no useful information for the reader.
Bad writing exists, of course. There's plenty of bad writing out there on the internet, and some of it will suffer from flaws like these even when written by a human, and some humans do like their em-dashes. But it's generally pretty obvious when the writing is taken on aggregate and you see recognisable pattern after pattern combined with em-dashes combined with shallowness of meaning combined with unnecessary overdramatisations.
It’s kinda funny that the “not …, but…” + em dash slop signifiers translate directly to Chinese “不是。。。而是。。。” and double-width em dash
Swift has implicit unwrap (!), and explicit unwrap (?).
I don't like to use implicit unwrap. Even things that are guaranteed to be there, I treat as explicit (For example, (self.view?.isEnabled ?? false), in a view controller, instead of self.view.isEnabled).
I always redefine @IBOutlets from:
to: I'm kind of a "belt & suspenders" type of guy.So what happens if it ends up being nil? How does your app react?
In this particular case, I would rather crash. It’s easier to spot in a crash report and you get a nice stack trace.
Silent failure is ultimately terrible for users.
Note: for the things I control I try to very explicitly model state in such a way as I never need to force unwrap at all. But for things beyond my control like this situation, I would rather end the program than continue with a state of the world I don’t understand.
Yeah @IBOutlets are generally the one thing that are allowed to be implicitly-unwrapped optionals. They go along with using storyboards & xibs files with Interface Builder. I agree that you really should just crash if you are attempting to access one and it is nil. Either you have done something completely incorrect with regards to initializing and accessing parts of your UI and want to catch that in development, or something has gone horribly, horribly, horribly with UIKit/AppKit and storyboard/xib files are not being loaded properly by the system.
> … you really should just crash if …
See my above/below comment.
A good tool for catching stuff during development, is the humble assert()[0]. We can use precondition()[1], to do the same thing, in ship code.
The main thing is, is to remain in control, as much as possible. Rather than let the PC leave the stack frame, throw the error immediately when it happens.
[0] https://docs.swift.org/swift-book/documentation/the-swift-pr...
[1] https://docs.swift.org/swift-book/documentation/the-swift-pr...
> Silent failure is ultimately terrible for users.
Agreed.
Unfortunately, crashes in iOS are “silent failures,” and are a loss of control.
What this practice does, is give me the option to handle the failure “noisily,” and in a controlled manner; even if just emitting a log entry, before calling a system failure. That can be quite helpful, in threading. Also, it gives me the option to have a valid value applied, if there’s a structural failure.
But the main reason that I do that with @IBOutlets, is that it forces me to acknowledge, throughout the rest of the code, that it’s an optional. I could always treat implicit optionals as if they were explicit, anyway. This just forces me to.
I have a bunch of practices that folks can laugh at, but my stuff works pretty effectively, and I sleep well.
Crashes are silent failures but as I mentioned: you can get a lot of your crashes reported via the App Store. This is why I prefer crashes in this situation: it gives me something actionable over silent failures on the client.
This is terrible. The whole reason they introduced this is because IBOutlets would get silently disconnected and then in the field a user would complain that a feature stopped working.
Crash early, crash often. Find the bugs and bad assumptions.
> This is terrible.
No it's not. Read my other comments.
ToMAYto, ToMAHto.
I have learned that it's a bad idea to trash other folks' methodologies without taking the time to understand why they do things, the way they do.
I have found dogma to be an impediment, in my own work. As I've gotten older, the sharp edges have been sanded off.
Have a great day!
> without taking the time to understand why they do things, the way they do.
Oh I am aware. They do it because
A) they don’t have a mental model of correct execution. Events just happen to them with a feeling of powerlessness. So rather than trying to form one they just litter the code with cases things that might happen
> As I've gotten older, the sharp edges have been sanded off.
B) they have grown in bad organizations with bad incentives that penalize the appearance of making mistakes. So they learn to hide them.
For example there might be an initiative that rewards removing crashes in favor of silent error.
> This is textbook "parse, don't validate" anti-pattern.
How so? “Parse, don’t validate” implies converting input into typed values that prevent representation of invalid state. But the parsing still needs to be done correctly. An unchecked unwrap really has nothing to do with this.
GP completely misunderstands “parse, don’t validate” and also calls it an anti-pattern. GP clearly has no idea what this is.
It rang more as "A/B deployments are pointless if you can't tell if a downstream failure is related." To me.
In addition, it looks like this system wasn't on any kind of 1%/10%/50%/100% rollout gating. Such a rollout would trivially have shown the poison input killing tasks.
To me it reads like there was a gradual rollout of the faulty software responsible for generating the config files, but those files are generated on approximately one machine, then propogated across the whole network every 5 minutes.
> Bad data was only generated if the query ran on a part of the cluster which had been updated. As a result, every five minutes there was a chance of either a good or a bad set of configuration files being generated and rapidly propagated across the network.
Not a DBA, how do you do DB permission rollout gating?
It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems.
The file should be versioned and rollout of new versions should be staged.
(There is definitely a trade-off; often times in the security critical path, you want to go as fast as possible because changes may be blocking a malicious actor. But if you move too fast, you break things. Here, they had a potential poison input in the pathway for synchronizing this state and Murphy's Law suggests it was going to break eventually, so the question becomes "How much damage can we tolerate when it does?")
> It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems.
That feature file is generated every 5 minutes at all times; the change to permissions was rolled out gradually over the clickhouse cluster, and whether a bad version of that file was generated depended on whether the part of the cluster that had the bad permissions generated the file.
While this is true, I wish that Rust had more of a first-class support for `no_panic`. Every solution we do have is hacky. I wish that I could guarantee that there were no panic calls anywhere in a code path.
> This is the multi-million dollar .unwrap() story.
While there are certainly many things to admire about Rust, this is why I prefer Golang's "noisy" error handling. In golang that would be either:
And the compiler would have complained that this value of `err` was unused; or you'd write: And it would be far more obvious that an error message is being ignored.(Renaming `unwrap` to `unwrapOrPanic` would probably help too.)
But I could screw it up in Go, if I made the same assumptions
Ultimately I don't think language design can be the sole line of defence against system failures; it can only guide developers to think about error casesRight, but the point isn't to make errors impossible; the point is to have them be 1) less likely to write, and 2) easier to spot on review.
People's biggest complaints about golang's errors:
1. You have to _TYPE_OUT_ what to do on EVERY.SINGLE.ERROR. SOO BOORING!
2. They clutter up the code and make it look ugly.
Rust is so much cleaner and more convenient (they say)! Just add ?, or .unwrap()!
Well, with ".unwrap()", you can type it fast enough that you're on to the next problem before it occurs to your brain to think about what to do if there is an error. Whereas, in golang, by the time you type in, "if err != nil {", you've broken the flow enough that now you're much more likely to be thinking, "Hmm, could this ever fail? What should we do if it does?" That break in flow is annoying, but necessary.
And ".unwrap()" looks so unassuming, it's easy to overlook on review; that "panic()" looks a lot more dangerous, and again, would be more likely to trigger a reviewer into thinking, "Wait, is it OK if this thing panics? Is this really so unlikely to happen?"
Renaming it `.unwrap_or_panic()` would probably help with both.
Eh, I'm not convinced.
1. Culturally, using `unwrap` is an omerta to Rust developers in the same way `panic` is an omerta to Go devs;
2. In the Rust projects I've seen there is usually a linter rule forbidding `unwrap` so you can't use it in production
That may be me, but `.unwrap()` is much more obvious than `_`:
- it's literally written out that you're assuming it to be Ok
- there are no indications that the `_` is an error: it could very well be some other return value from the function. in your example, it could be the number of appended features, etc
That's why Go's error handling is indeed noisy: it's noise and you reduce noise by not handling errors. Rust's is terse yet verbose: if you add stuff it's because you're doing something wrong. You explicitly spelled out the error is being ignored.
> And it would be far more obvious that an error message is being ignored.
Haven't used Go so maybe I'm missing some consideration, but I don't see how ", _" is more obvious than ".unwrap()". If anything it seems less clear, since you need to check/know the function's signature to see that it's an error being ignored (wouldn't be the case for a function like https://pkg.go.dev/math#Modf).
I haven't been writing Rust for that long (about 2 years) but every time I see .unwrap() I read it as 'panic in production'. Clippy needs to have harder checks on unwrap.
If the error had been an exception instead of a result, could have bubbled up
I have been saying for years that Rust botched error handling in unfixable ways. I will go to the grave believing Rust fumbled.
The design of the Rust language encourages people to use unwrap() to turn foreseeable runtime problems into fatal errors. It's the path of least resistance, so people will take it.
Rust encourages developers to consider only the happy path. No wonder it's popular among people who've never had to deal with failure.
All of the concomitant complexity--- Result, ?, the test thing, anyhow, the inability for stdlib to report allocation failure --- is downstream of a fashion statement against exceptions Rust cargo-culted from Go.
The funniest part is that Rust does have exceptions. It just calls them panics. So Rust code has to deal with the ergonomic footgun of Result but pays anyway for the possibility of exceptions. (Sure, you can compile with panic=abort. You can't count on it.)
I could not be more certain that Rust should have been a language with exceptions, not Result, and that error objects are a gross antipattern we'll regret for decades.
Errors work just like exceptions especially if you use the ? operator and let the error bubble up the chain. This is the Rust equivalent of an unhandled exception and the ripcord being pulled.
In C++, functions are error-colored by default. You write "noexcept" if you want your function to be infallible-colored instead.
(You usually want to make a function infallible if you're using your noexcept function as part of a cleanup path or part of a container interface that allows for more optimizations of it knows certain container operations are infallible.)
Rust makes infallibility the syntactic default and makes you write Result to indicate fallibility. People often don't want to color their functions this way. Guess what happens when a programmer is six levels deep in infallible-colored function calls and does something that can fail.
.unwrap()
Guess what, in Rust, is fallible?
Mutex acquire.
Guess what you need to do often on infallible cleanup paths?
Mutex acquire.
Not panicking code is tedious to write. It is not realistic to expect everything to be non panic. There is a reason that panicking exists in the first place.
Them calling unwrap on a limit check is the real issue imo. Everything that takes in external input should assume it is bad input and should be fuzz tested imo.
In the end, what is the point of having a limit check if you are just unwrapping on it
> Not panicking code is tedious to write.
Using the question mark operator [1] and even adding in some anyhow::context goes a long way to being able to fail fast and return an Err rather then panicking.
Sure you need to handle Results all the way up the stack but it forces you to think about how those nested parts of your app will fail as you travel back up the stack.
[1]: https://doc.rust-lang.org/rust-by-example/std/result/questio...
By the way - does this discussion matter and were they wrong to use unwrap()?
The way they wrote the code means that having more than 200 features is a hard non-transient error - even if they recovered from it, it meant they'd have had the same error when the code got to the same place.
I'm sure when the process crashed, k8s restarted the pod or something - then it reran the same piece of code and crashed in the same place.
While I don't necessarily agree with crashing as business strategy, I don't think that doing anything other than either dropping the extra rules or allocating more memory - neither of which the original code was built to do (probably by design).
The code made the local hard assumption that there won't ever be more than 200 rules and its okay to crash if that count is exceeded.
If you design your code around an invariant never being violated (which is fine), you have to make it clear on a higher level that they did.
This isn't a Rust problem (though Rust does make it easy to do the wrong thing here imo)
Instead of crashing when applying the new config, it's more common to simply ignore the new config if it cannot be applied. You keep running in the last known good state. Operators then get alerts about the failures and can diagnose and resolve the underlying issue.
That's not always foolproof, e.g. a freshly (re)started process doesn't have any prior state it can fall back to, so it just hard crashes. But restarts are going to be rate limited anyways, so even then there is time to mitigate the issue before it becomes a large scale outage
Interesting to see Rust error handling flunk out in practice.
It may be that forcing handling at every call tends to makes code verbose, and devs insensitized to bad practice. And the diagnostic Rust provided seems pretty garbage.
There is bad practice here too -- config failure manifesting as request failure, lack of failing to safe, unsafe rollout, lack of observability.
Back to language design & error handling. My informed view is that robustness is best when only major reliability boundaries need to be coded.
This the "throw, don't catch" principle with the addition of catches on key reliability boundaries -- typically high-level interactions where you can meaningfully answer a failure.
For example, this system could have a total of three catch clauses "Error Loading Config" which fails to safe, "Error Handling Request" which answers 5xx, and "Socket Error" which closes the HTTP connection.
> It may be that forcing handling at every call tends to makes code verbose
Rust has a lot of helpers to make it less verbose, even that error they demonstrate could've been written in some form `...code()?` with `?` helper that would have propagated the error forwards.
However I do acknowledge that writing Error types is boring sometimes so people don't bother to change their error types and just unwrap. But even my dinghy little apps for my personal use I do simple serach `unwrap` and make sure I have as few as possible.
I don't understand how your takeaway is that this is a language flaw other than to assume that you have some underlying disdain for Rust. That's fine, but state it clearly please.
The end result would've been the exact same if they "handled" the error: a bunch of 500s. The language being used doesn't matter if an invariant in your system is broken.
Some languages and style guides simply forbid throwing exceptions without catching / proper recovery. Google C++ bans exceptions and the main mechanism for propogating errors is `absl::Status` which the caller has to check. Not familiar with Rust but it seems unwrap is such a thing that would be banned.
> Not familiar with Rust but it seems unwrap is such a thing that would be banned.
Panics aren't exceptions, any "panic" in Rust can be thought of as an abort of the process (Rust binaries have the explicit option to implement panics as aborts). Companies like Dropbox do exactly this in their similar Rust-based systems, so it wouldn't surprise me if Cloudflare does the same.
"Banning exceptions" wouldn't have done anything here, what you're looking for is "banning partial functions" (in the Haskell sense).
Yeah I know but isn't unwrap basically a trivial way to: (1) give up catching the exception/error (the E part in `Result<T, E>`) that the callee throws; and also (2) escalate it to the point that nothing can catch it. It has such a benign name.
Unwrap is used in places where in C++ you would just have undefined behavior. It wouldn't make any more sense to blanket ban it than it would ban ever dereferencing a pointer just in case its null - even if you just checked that it wasn't null.
Rust's Result is the same thing as C++'s std::expected. How is calling std::expected::value undefined behaviour?
Rust's foo: Option<&T> is rust's rough equivalent to C++'s const T* foo. The C++ *foo is equivalent to the rust unsafe{ *foo.unwrap_unchecked() }, or in safe code *foo.unwrap() (which changes the undefined behavior to a panic).
Rust's unwrap isn't the same as std::expected::value. The former panics - i.e. either aborts the program or unwinds depending on context and is generally not meant to be handled. The latter just throws an exception that is generally expected to be handled. Panics and exceptions use similar machinery (at least they can depending on compiler options) but they are not equivalent - for example nested panics in destructors always abort the program.
In code that isn't meant to crash `unwind` should be treated as a sign saying that "I'm promising that this will never happen", but just like in C++ where you promise that pointers you deference are valid and signed integers you add don't overflow making promises like that is a necessary part of productive programming.
Tangential but funnily enough calling std::expected::error is ub if there is no error :D
There are even lints for this but people get impatient and just override them or fight for them to no longer be the default.
As usual: people problem, not a tech problem. In the last years a lot of strides have been made. But people will be people.
Linting is not good enough. The compiler should refuse to compile the code without it marked with an explicit annotation. Too much Rust code is panic happy since using casual use of `unwrap` is perma-etched into everyone's minds by the amount of demo code out there using unwrap.
I completely agree. But IMO Rust would have not gained traction if it was as strict. It would be branded as an academic toy language.
But now after we are past that and it has a lot of mind share, I'd say it's time to start tightening the bolts.
!#[deny(unwrap_used)] at the top of the file would take care of it.
and people make mistake
at some point machine would be better in coding because well machine code is machine instruction task
same like chess, engine is better than human grandmaster because its solvable math field
coding is no different
> same like chess, engine is better than human grandmaster because its solvable math field
Might be worth noting that your description of chess is slightly incorrect. Chess technically isn't solved in the sense that the optimal move is known for any arbitrary position is known; it's just that chess engines are using what amounts to a fancy brute force for most of the game and the combination of hardware and search algorithm produces a better result than the human brain does. As such, chess engines are still capable of making mistakes, even if actually exploiting them is a challenge.
This is why the Erlang/Elixir methodology of having supervision and letting things crash gracefully is so useful. You can either handle every single error gracefully or handle crashing gracefully - it's much easier and more realistic in large codebases to do the later.
This would not have helped: the code would crash before doing anything useful at all.
If anything, the "crash early" mentality may even be nefarious: instead of handling the error and keeping the old config, you would spin on trying to load a broken config on startup.
Continuing only makes sense for cases you know you can handle.
_In theory_ they could have used the old config, but maybe there are reasons that’s not possible in Cloudflare’s setup. Whether or not that’s an invariant violation or just an error that can be handled and recovered from is a matter of opinion in system design.
And crashing on an invariant violation is exactly the right thing to do rather than proceed in an undefined state.
Given the context and what the configuration file contains, I'd argue it's mission-critical for the software to keep running with the previous configuration. Otherwise you might shutdown the internet. Honestly, I'm pretty sure their pre-rewrite version had such logic, and it was forgotten or still on the TODO pile for the rewrite version.
At a previous job (cloud provider), we've had exactly this kind of issue, with exactly the same root cause. The entrypoint for the whole network had a set of rules (think a NAT gateway) that were reloaded periodically from the database. Someone rewrote that bit of plumbing from Python to Go. Someone else performed a database migration. Suddenly, the plumbing could not find the data, and pushed an empty file to prod. The rewrite lacked "if empty, do nothing and raise an alert", that the previous one had. I'll let you imagine what happened next :)
Say what you want exception haters, but at least in exceptions-as-default languages, the decision of a particular issues is fatal to the whole program can be decided centrally at a high level, and not every choice is forced to be up to individual discretion.
But you can do the same thing with Rust, by piping up Results.
Safe things should be easy, dangerous things should be hard.
This .unwrap() sounds too easy for what it does, certainly much easier than having an entire try..catch block with an explicit panic. Full disclosure: I don't actually know Rust.
I don't think 'unwrap' is inherently the problem.
Any project has to reason about what sort of errors can be tolerated gracefully and which cannot. Unwrap is reasonable in scenarios you expect to never be reached, because otherwise your code will be full of all sorts of possible permutations and paths that are harder to reason about and may cascade into extremely nuanced or subtle errors.
Rust also has a version of unwrap called "expect" where you provide a string that logs why the unwrap occurred. It's similar, but for pieces of code that are crucial it could be a good idea to require all 'unwraps' to instead be 'expects' so that people at least are forced to write down a reason why they believe the unwrap can never be reached.
I wonder if similar to infrastructure resilience, code resilience is also required for critical services that can never go down? Instead of relying on a single implementation for a critical service, have multiple independent implementations in different languages. Back when I was running my own DNS servers, I did always ensure that primary and secondary were running on different platforms and different software.
Handling the error still would've returned a 5xx in this case, since the config file was still over the limit of features the service could handle.
That's such a bad take after reading the article. If you're going to write a system that preallocates and is based on hard assumptions about max size - the panic/unwrap approach is reasonable.
The config bug reaching prod without this being caught and pinpointed immediately is the strange part.
It's reasonable when testing protocols exercise the panic scenario. This is the problem with punting on error recovery. Nobody checks faults that propagate across domains of responsibility.
Exactly. The newbie mistake in SQL is also way worse than this. But the whole design is also bad. Clearly implementing things at the wrong place.
And, it took like over an hour between the problem started til my sites went down. That is just crazy.
I agree there's no way to soft-error this, though "truncate and raise an alert" is arguably the better pattern.
This is a bummer. The unwrap()'ing function already returned a result and should have just propagated the error. Presumably the caller could have handled more sensibly than just panic'ing.
I wonder what happens if they handle it gracefully? sounds like performance degradation (better than reliability degradation!).
Also wonder with a sharded system why are they not slow rolling out changes and monitoring?
As a gopher I never understand why is there so many unwraps in an average rust code.
Average Go code has much less panics than Rust has unwraps, which are functionally equivalent.
The average golang code segfaults by design.
I love Go and write a ton of it. I've had real segfaults quite a lot.
Because Go silently gives you zero/null instead
Idiomatically, it gives you `err` and you do `if err != nil {return err}`. While in rust you mostly do `.unwrap` and panic.
It's not in the type system, but it's idiomatic
Get a key from a map and forget to check the error.
which mean an unexpected behavior could go unnoticed for a long time.
I'd prefer a loud crash over that.
Well look at the failure modes in the original article.
In the original PHP code, all worked, only it didn't properly check for bots.
The new Rust code did a loud crash and took off half of the internet.
tokio default behavior within a task is to ignore panics, such as an Err/None unwrap, and only crash that task, so it's impact limited so that's nice, maybe that's where the snowblindness came from.
it'd be kinda hard to amend the clippy lints to ignore coroutine unwraps but still pipe up on system ones. i guess.
edit: i think they'd have to be "solely-task-color-flavored" so definitely probably not trivial to infer
Which is something I will bookmark for the usual Rust doesn't do exceptions discussions, except it kind of does even if called differently.
Feels like a case where safety guarantees of Rust lulled them into thinking the edge cases were covered
> This is the multi-million dollar .unwrap() story.
First multi-million dollar .unwrap() story.
They should link this article in the docs for `unwrap()`.
it's usually because of fail fast and fail hard, in theory critical bugs will be caught in dev/test
if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language
At Facebook they name certain "escape hatch" functions in a way that inescapably make them look like a GIANT EYESORE. Stuff like DANGEROUSLY_CAST_THIS_TO_THAT, or INVOKE_SUPER_EXPENSIVE_ACTION_SEE_YOU_ON_CODE_REVIEW. This really drives home the point that such things must not be used except in rare extraordinary cases.
If unwrap() were named UNWRAP_OR_PANIC(), it would be used much less glibly. Even more, I wish there existed a super strict mode when all places that can panic are treated as compile-time errors, except those specifically wrapped in some may_panic_intentionally!() or similar.
> make them look like a GIANT EYESORE
React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED comes to mind. I did have to reach to this before, but it certainly works for keeping this out of example code and other things like reading other implementations without the danger being very apparent.
At some point it was renamed to __CLIENT_INTERNALS_DO_NOT_USE_OR_WARN_USERS_THEY_CANNOT_UPGRADE which is much less fun.
> it certainly works
Not for this guy:
https://github.com/reactjs/react.dev/issues/3896
right and if the language designers named it UNWRAP_OR_PANIC() then people would rightfully be asking why on earth we can't just use a try-catch around code and have an easier life
But a panic can be caught and handled safely (e.g. via std:: panic tools). I'd say that this is the correct use case for exceptions (ask Martin Fowler, of all people).
There is already a try/catch around that code, which produces the Result type, which you can presumptuously .unwrap() without checking if it contains an error.
Instead, one should use the question mark operator, that immediately returns the error from the current function if a Result is an error. This is exactly similar to rethrowing an exception, but only requires typing one character, the "?".
Probably not, since errors as values are way better than exceptions.
...and you can? try-catch is usually less ergonomic than the various ways you can inspect a Result.
vs Or better yet, compare the problematic cases where the error isn't handled: vs In the former (the try-catch version that doesn't try or catch), the lack of handling is silent. It might be fine! You might just depend on your caller using `try`. In the latter, the compiler forces you to use UNWRAP_OR_PANIC (or, in reality, just unwrap) or `data` won't be the expected type and you will quickly get a compile failure.What I suspect you mean, because it's a better argument, is:
which is fair, although how often is it really the right thing to let all the errors from 4 independent sources flow together and then get picked apart after the fact by inspecting `e`? It's an easier life, but it's also one where subtle problems constantly creep in without the compiler having any visibility into them at all.Unwrap isn't a synonym for laziness, it's just like an assertion, when you do unwrap() you're saying the Result should NEVER fail, and if does, it should abort the whole process. What was wrong was the developer assumption, not the use of unwrap.
It also makes it very obvious in the code, something very dangerous is happening here. As a code reviewer you should see an unwrap() and have alarm bells going off. While in other languages, critical errors are a lot more hidden.
I hate that it's a method. That can get lost in a method chain easily enough during a code review.
A function or a keyword would interrupt that and make it less tempting
> What was wrong was the developer assumption, not the use of unwrap.
How many times can you truly prove that an `unwrap()` is correct and that you also need that performance edge?
Ignoring the performance aspect that often comes from a hat-trick, to prove such a thing you need to be wary of the inner workings of a call giving you a `Return`. That knowledge is only valid at the time of writing your `unwrap()`, but won't necessarily hold later.
Also, aren't you implicitly forcing whoever changes the function to check for every smartass dev that decided to `unwrap` at their callsite? That's bonkers.
I doubt that this unwrap was added for performance reasons; I suspect it was rather added because the developer temporarily didn't want to deal with what they thought was an unlikely error case while they were working on something else; and no other system recognized that the unwrap was left in and flagged it before it was deployed on production servers.
If I were Cloudflare I would immediately audit the codebase for all uses of unwrap (or similar rust panic idioms like expect), ensure that they are either removed or clearly documented as to why it's worth crashing the program there, and then add a linter to their CI system that will fire if anyone tries to check in a new commit with unwrap in it.
Panics are for unexpected error conditions, like your caller passed you garbage. Results are for expected errors, like your caller passed you something but it's your job to tell if it's garbage.
So the point of unwrap() is not to prove anything. Like an assertion it indicates a precondition of the function that the implementer cannot uphold. That's not to say unwrap() can't be used incorrectly. Just that it's a valid thing to do in your code.
Note that none of this is about performance.
> when you do unwrap() you're saying the Result should NEVER fail
Returning a Result by definition means the method can fail.
> Returning a Result by definition means the method can fail.
No more than returning an int by definition means the method can return -2.
Yeah, I think I expressed wrongly here. A more correct version would be: "when you do unwrap() you're saying that an error on this particular path shouldn't be recoverable and we should fail-safe."
Division can fail, but I can write a program where I'm sure division will not fail.
It's a little subtler than this. You want it to be easy to not handle an error while developing, so you can focus on getting the core logic correct before error-handling; but you want it to be hard to deploy or release the software without fully handling these checks. Some kind of debug vs release mode with different lints seems like a reasonable approach.
So… basically every language ever?
Except maybe Haskell.
It's easy to cause this kind of failure in Haskell also.
And Gleam
In Rust, `.unwrap()` is nine characters, whereas propagating the Result via `?` is one.
All languages with few exceptions have these kinds of escape hatches like unwrap
https://en.wikipedia.org/wiki/Crash-only_software
Works when you have the Erlang system that does graceful handing for you: reporting, restarting.
This is untrue. A `?` operator would have done just fine here. I agree with you though that it should be possible to explicitely forbid unwraps.
git blame on .unwrap() line
Oh come on, stop spreading FUD. Rust programs are 100% immune to crashes and bugs, they have memory safety (c).
Also, exception handling is hard and lame. We don't need exceptions, just add a "match" block after every line in your program.
What's the point of this sarcastic comment? Do you think that some people claim that Rust's memory safety guarantees mean that a Rust program is incapable of crashing or having a bug? This is a dumb thing to claim certainly, but I'm not aware of anyone actually making this claim.
I'm also not sure what you're getting at with the comment about exception handling being lame. I think the ML/Haskell inspired model that Rust uses of having a parameterized Result type for fallible operations is generally better than exceptions for a variety of reasons (although maybe better Exception semantics could help with some of this), but what does this have to do with match blocks?
> Do you think that some people claim that Rust's memory safety guarantees mean that a Rust program is incapable of crashing or having a bug?
Undoubtedly yes.
> ...but what does this have to do with match blocks?
You tell me. You're the one advocating for placing one after every single function call.
You write so much rust you causally apply unwrap now to everything?
Rust compiler is a god of sorts, or at least a law of nature haha
Way to comment and go instantly off topic
The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen".
The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they weren't.
The slowness to root cause: sheer bad luck, with the status page down and Azure's DDoS yesterday all over the news.
The broken SQL: this is the one that I'd be up in arms about if I worked for Cloudflare. For a system with the power to roll out config to ~all of prod at once while bypassing a lot of the usual change tracking, having this escape testing and review is a major miss.
IMO: there should be explicit error path for invalid configuration, so the program would abort with specific exit code and/or message. And there should be a superviser which would detect this behaviour, rollback old working config and wait for few minutes before trying to apply new config again (of course with corresponding alerts).
So basically bad config should be explicitly processed and handled by rolling back to known working config.
You don’t even need all the ceremony. If the config gets updated every 5 minutes, it surely is being hot-reloaded. If that’s the case, the old config is already in memory when the new config is being parsed. If that’s the case, parsing shouldn’t have panicked, but logged a warning, and carried on with the old config that must already be in memory.
> If that’s the case, the old config is already in memory when the new config is being parsed
I think that's explicitly a non-goal. My understanding is that Cloudflare prefers fail safe (blocking legitimate traffic) over fail open (allowing harmful traffic).
Well, they should then add some reliability goals into the mix too to balance it out a bit.
System outputting the configuration file failed (it could check the size and/or content and stop right away), but also a system importing the file failed. These usually sound simple/stupid in a hindsight. I am not a fan of everything centralising to a few hands. As in bad situation, they can also be weaponised or attacked. And in good situation their blast radius is just too big and a bit random, in this case global.
The query is surely faulty: Even if this wasn’t a huge distributed database with who-knows-what schemas and use cases, looking up a specific table by its unqualified name is sloppy.
But the architectural assumption that the bot file build logic can safely obtain this operationally critical list of features from derivative database metadata vs. a SSOT seems like a bigger problem to me.
It's probably not ok to silently run with a partial config, which could have undefined semantics. An old but complete config is probably ok (or, the system should be designed to be safe to run in this state).
For unwrap, Cloudflare should consider adding lint tooling that prevents unwrap being added to production code.
It’s a feature, not a bug. Assert assumptions and crash on bad one.
Crashing is not an outage. It’s a restart and a stack trace for you to fix.
Even if you want it to crash, you almost never unwrap. At the very least you would use .expect() so you get a reasonable error message -- or even better you handle the potential error.
> Crashing is not an outage.
Are you in the right thread?
Did you skip all the other context about the other systems that failed?
The problem was a query producing incorrect data. The crash helped them find it.
What do you think happens when a program crashes?
The type system is for asserting assumptions like "this cannot fail". You don't crash at all.
Most properties of programs cannot be validated at compile time and must be checked at runtime.
But you’re still missing it. Crashing is not bad. It’s good. It’s how you leverage OS level security and reliability.
Quite surprising a single bad config file brought down their entire global network across multiple products
Share the same opinion, as others pointed out, the status page down probably caused by bots checking it.
It reads a lot like the Crowdstrike SNAFU. Machine-generated configuration file b0rks-up the software that consumes it.
The "...was then propagated to all the machines that make up our network..." followed by "....caused the software to fail." screams for a phased rollout / rollback methodology. I get that "...it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly" but today's outage highlights that rapid deployment isn't all upside.
The remediation section doesn't give me any sense that phased deployment, acceptance testing, and rapid rollback are part of the planned remediation strategy.
At my employer, we have a small script that automatically checks such generated config files. It does a diff between the old and the new version, and if the diff size exceeds a threshold (either total or relative to the size of the old file), it refuses to do the update, and opens a ticket for a human to look over it.
It has somewhat regularly saved us from disaster in the past.
I don't think this system is best thought of as "deployment" in the sense of CI/CD; it's a control channel for a distributed bot detection system that (apparently) happens to be actuated by published config files (it has a consul-template vibe to it, though I don't know if that's what it is).
That's why I likened it Crowdstrike. It's a signature database that blew up the consumer of said database. (You probably caught my post mid-edit, too. You may be replying to the snarky paragraph I felt better of and removed.)
Edit: Similar to Crowdstrike, the bot detector should have fallen-back to its last-known-good signature database after panicking, instead of just continuing to panic.
Code and Config should be treated similarly. If you would use a ring based rollout, canaries, etc for safely changing your code, then any config that can have the same impact must also use safe rollout techniques.
You're the nth person on this thread to say that and it doesn't make sense. Events that happen multiple times per second change data that you would call "configuration" in systems like these. This isn't `sendmail.cf`.
If you want to say that systems that light up hundreds of customers, or propagate new reactive bot rules, or notify a routing system that a service has gone down are intrinsically too complicated, that's one thing. By all means: "don't build modern systems! computers are garbage!". I have that sticker on my laptop already.
But like: handling these problems is basically the premise of large-scale cloud services. You can't just define it away.
I'm sorry to belabor this but I'm genuinely not understanding what you're saying in this reply. I haven't operated large scale systems. I'm just an IT generalist and casual coder. I acknowledge I'm too inexperienced to even know what I don't know re: running large systems.
I read the parent poster as broadly suggesting configuration updates should have fitness tests applied and be deployed to minimize the blast radius when an update causes a malfunction. That makes intuitive sense to me. It seems like software should be subject to health checks after configuration updates, even if it's just to stop a deployment before it's widely distributed (let alone rolling-back to last-working configurations, etc).
Am I being thick-headed in thinking defensive strategies like those are a good idea? I'm reading your reply as arguing against those types of strategies. I'm also not understanding what you're suggesting as an alternative.
Again, I'm sorry to belabor this. I've replied once, deleted it, tried writing this a couple more times and given up, and now I'm finally pulling the trigger. It's really eating at me. I feel as though I must be deep down the Dunning-Kruger rabbit hole and really thinking "outside my lane".
That’s correct.
Is it actually consul-template? (I have post-consul-template stress disorder).
I'd love to hear any commentary on Consul if anyone else has it.
Did you know: PCTSD affects more than 2 in 5 engineers.
I'm amazed that they are not using any simulator of some sort and pushing changes directly to production.
I’m fairly certain it will be after they read this thread. It doesn’t feel like they don’t want, or are incapable of improving?
Everyone is hating on unwrap, but to me the odd and more interesting part is that it took 3 hours to figure this out? Even with a DDoS red herring, shouldn’t there have been a crash log or telemetry anomaly correlated? Also, shouldn’t the next steps and resolution focus more on this aspect, since it’s a high leverage tool for identifying any outage caused by a panic rather than just preventing a recurrence of random weird edge case #9999999?
I have nowhere near the experience managing such complex systems, but I can empathize with this. In a high-pressure situations the most obvious things get missed. If someone is convinced System X is at fault, your mind can make leaps to justify every other degraded system is a downstream effect of that. Cause and effect can get switched.
Sometimes you have smart people in the room who dig deeper and fish it out, but you cannot always rely on that.
I have plenty of empathy, having been in plenty of similar situations. It's not a matter of "I can't BELIEVE it took that long" (although it is a bit surprising) so much as that I disagree with the key takeaways here in the HN comments section and in the blog itself, which focus strongly on fixing rare edge case issues (the bad ClickHouse query and a bad config file causing a panic via unwrap), rather than reducing MTTR for all issues by improving the debug and monitoring experience.
I'm also suspicious that
> Eliminating the ability for core dumps or other error reports to overwhelm system resources
from the blog had a lot more to do with the issue than perhaps the narrative is letting on.
Yes this is the weird part for me. With good monitoring, the panic at unwrap should have been detected immediately. I assume they weren't looking at the right place, but still. If you use Sentry for example, a brand new panic should be pretty visible.
Indeed, nothing about the root issues are particular surprising but why they missed a critical service panicing across their fleet is not bubbling up.
My best guess is too many alerts firing without a clear hierarchy and possibilities to seprate cause from effect. It's a typical challenge but I wish they would shed some light on that. And its a bit concerning that improving observability is not part of their follow up steps.
Once they figured it out they didn't have a way to load in a new feature file, had to figure that out, and then restart every machine.
This took one of the three hours; it seems to have taken from 11:28 to 13:37 to recognize that the configuration file panic was the cause of the issue.
> That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
> The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.
I'm no FAANG 10x engineer, and I appreciate things can be obvious in hindsight, but I'm somewhat surprised that engineering at the level of Cloudflare does not:
1. Push out files A/B to ensure the old file is not removed.
2. Handle the failure of loading the file (for whatever reason) by automatically reloading the old file instead and logging the error.
This seems like pretty basic SRE stuff.
Yep, a decent canary mechanism should have caught this. There's a trade off between canarying and rollout speed, though. If this was a system for fighting bots, I'd expect it to be optimized for the latter.
I'm shocked that an automatic canary rollout wasn't an action item. Pushing anything out globally is a guaranteed failure again in the future.
Even if you want this data to be very fresh you can probably afford to do something like:
1. Push out data to a single location or some subset of servers.
2. Confirm that the data is loaded.
3. Wait to observe any issues. (Even a minute is probably enough to catch the most severe issues.)
4. Roll out globally.
Presumably optimal rollout speed entails something like or as close to ”push it everywhere all at once and activate immediately” that you can get — that’s fine if you want to risk short downtime rather than delays in rollout, what I don’t understand is why the nodes don’t have any independent verification and rollback mechanism. I might be underestimating the complexity but it really doesn’t sound much more involved than a process launching another process, concluding that it crashed and restarting it with different parameters.
I think they need to strongly evaluate if they need this level of rollout speed. Even spending a few minutes with an automated canary gives you a ton of safety.
Even if the servers weren't crashing it is possible that a bet set of parameters results in far too many false positives which may as well be complete failure.
"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our status page."
Unfortunately they do not share, what caused the status page to went down as well. (Does this happen often? Otherwise a big coincidence it seems)
We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.
The status page is hosted on AWS Cloudfront, right? It sure looks like Cloudfront was overwhelmed by the traffic spike, which is a bit concerning. Hope we'll see a post from their side.
CloudFront has quotas[0] and they likely just hit those quota limits. To request higher quotas requires a service ticket. If they have access logs enabled in CloudFront they could see what the exact error was.
And since it seems this is hosted by Atlassian, this would be up to Atlassian.
[0] https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
Yes, probably a bunch of automated bots decided to check the status page when they saw failures in production.
It looks a lot like a CloudFront error we randomly saw today from one of our engineers in South America. I suspect there was a small outage in AWS but can't prove it.
Probably non zero number of companies use cloudfront and other cdns as fallback for cloudflare or running a blended cdn so not surprising to see other cdns hit with a thundering herd when cloudflare went down
This situation reminds me of risk assessment, where you sometimes assume two rare events are independent, but later learn they are actually highly correlated.
Quite possibly it was due to high traffic.
IDK Atlassian Statuspage clientele, but it's possible Cloudflare is much larger than usual.
it seems like a good chance that despite thinking their status page was completely independent of cloudfront, enough of the internet is dependent on cloudfront now that they're simply wrong about the status page's independence.
i think you've got cloudflare and cloudfront mixed up.
ahah oops. yeah, it's a problem. i've got two projects ongoing that each rely on one of them, and i can never keep it straight.
I mean, that would require a postmortem from statuspage.io right? Is that a service operated by cloudflare?
Atlasaian
[dead]
> work has already begun on how we will harden them against failures like this in the future. In particular we are:
> Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input
> Enabling more global kill switches for features
> Eliminating the ability for core dumps or other error reports to overwhelm system resources
> Reviewing failure modes for error conditions across all core proxy modules
Absent from this list are canary deployments and incremental or wave-based deployment of configuration files (which are often as dangerous as code changes) across fault isolation boundaries -- assuming CloudFlare has such boundaries at all. How are they going to contain the blast radius in the future?
This is something the industry was supposed to learn from the CrowdStrike incident last year, but it's clear that we still have a long way to go.
Also, enabling global anything (i.e., "enabling global kill switches for features") sounds like an incredibly risky idea. One can imagine a bug in a global switch that transforms disabling a feature into disabling an entire system.
They require the bot management config to update and propagate quickly in order to respond to attacks - but this seems like a case where updating a since instance first would have seen the panic and stopped the deploy.
I wonder why clickhouse is used to store the feature flags here, as it has it's own duplication footguns[0] which could have also easily lead to a query blowing up 2/3x in size. oltp/sqlite seems more suited, but i'm sure they have their reasons
[0] https://clickhouse.com/docs/guides/developer/deduplication
I don't think sqlite would come close to their requirements for permissions or resilience, to name a couple. It's not the solution for every database issue.
Also, the link you provided is for eventual deduplication at the storage layer, not deduplication at query time.
I think the idea is to ship the sqlite database around.
It’s not a terrible idea, in that you can test the exact database engine binary in CI, and it’s (by definition) not a single point of failure.
I think you're oversimplifying the problem they had, and I would encourage you to dive in to the details in the article. There wasn't a problem with the database, it was with the query used to generate the configs. So if an analogous issue arose with a query against one of many ad-hoc replicated sqlite databases, you'd still have the failure.
I love sqlite for some things, but it's not The One True Database Solution.
Global configuration is useful for low response times to attacks, but you need to have very good ways to know when a global config push is bad and to be able to rollback quickly.
In this case, the older proxy's "fail-closed" categorization of bot activity was obviously better than the "fail-crash", but every global change needs to be carefully validated to have good characteristics here.
Having a mapping of which services are downstream of which other service configs and versions would make detecting global incidents much easier too, by making the causative threads of changes more apparent to the investigators.
It seems they had this continous rollout for the config service, but the services consuming this were affected even by small percentage of these config providers being faulty, since they were auto updating every few minutes their configs. And it seems there is a reason for these updating so fast, presumably having to react to threat actors quickly.
It's in everyone's interest to mitigate threats as quickly as possible. But it's of even greater interest that a core global network infrastructure service provider not DOS a significant proportion of the Internet by propagating a bad configuration too quickly. The key here is to balance responsiveness against safety, and I'm not sure they struck the right balance here. I'm just glad that the impact wasn't as long and as severe as it could have been.
This isn't really "configuration" so much as it is "durable state" within the context of this system.
In my 30 years of reliability engineering, I've come to learn that this is a distinction without a difference.
People think of configuration updates (or state updates, call them what you will) as inherently safer than code updates, but history (and today!) demonstrates that they are not. Yet even experienced engineers will allow changes like these into production unattended -- even ones who wouldn't dare let a single line of code go live without being subject to the full CI/CD process.
I think global kill switches are just an last resort machanism, to bypass identified faulty subsystems. Even if there is a risk with it, in this instance the risk was zero, because CF was dead already. This wont change the blast radius, but it's duration and proliferation.
In reference to fault isolation boundaries: I am not familiar with their CI/CD, in theory the error could have been caught/prevented there, but that comes with a lot of depends or it's tricky. But it looks like they didn't go the extra mile to care about safety sensitive areas. So euphemistic speaking, they are now recalibrating balance of safety measures.
it's always a config push. people rollout code slowly but don't have the same mechanisms for configs. But configs are code, and this is a blind spot that causes an outsized percentage of these big outages.
When a failsafe system fails, it fails by failing to fail safely.
Lots of people here are (perhaps rightfully) pointing to the unwrap() call being an issue. That might be true, but to me the fact that a reasonably "clean" panic at a defined line of code was not quickly picked up in any error monitoring system sounds just as important to investigate.
Assuming something similar to Sentry would be in use, it should clearly pick up the many process crashes that start occurring right as the downtime starts. And the well defined clean crashes should in theory also stand out against all the random errors that start occuring all over the system as it begins to go down, precisely because it's always failing at the exact same point.
Exactly! You could have `rand() > 0.5 && panic!()` in the code of your bot module, and that should not put the internet on fire.
The issue here is about the system as a whole not any line of code.
> The issue here is about the system as a whole not any line of code.
Unsoundness in the type system that leads to a systemic failure is about the system as a whole.
Not everything can be recovered from restarting a process, and process correctness and recovery is something that also derives from your type system.
In the early 2000s when Google explained how they achieved their (already back then) awesome reliability, ie assuming that any software and hardware will eventually fail, and that they designed everything with the idea that everything was faulty, there were some people who couldn't get it, who would still bring the argument that "yeah but today with modern raid..."
People here chatting about unwrap remind me of them :)
Assuming software and people will fail is exactly what not using unwrap is about.
If you depend on engineers not fucking up, you will fail. Using unwrap is assuming humans won’t get human-enforced invariants wrong. They will. They did here.
As someone that works in formal verification of crypto systems, watching people like yourself advocate for hope-and-prayer development methodology is astonishing.
However, I understand why we’re still having this debate. It’s the same debate that’s been occurring for the same reasons for decades.
Doing things correctly is mentally more difficult, and so people jump through ridiculous rhetorical hoops to justify why they will not — or quite often, mentally cannot — perform that intellectual labor.
It’s a disheartening lack of craftsmanship and industry accountability, but it’s nothing new.
Why does cloudflare allow unwraps in their code? I would've assumed they'd have clippy lints stopping that sort of thing. Why not just match with { ok(value) => {}, Err(error) => {} } the function already has a Result type.
At the bare minimum they could've used an expect("this should never happen, if it does database schema is incorrect").
The whole point of errors as values is preventing this kind of thing.... It wouldn't have stopped the outage but it would've made it easy to diagnose.
If anyone at cloudflare is here please let me in that codebase :)
Not a cloudflare employee but I do write a lot of Rust. The amount of things that can go wrong with any code that needs to make a network call is staggeringly high. unwrap() is normal during development phase but there are a number of times I leave an expect() for production because sometimes there's no way to move forward.
Yeah it seems likely that even if there wasn't an unwrap, there would have been some error handling that wouldn't have panicked the process, but would have still left it inoperable if every request was instead going through an error path.
I'm in a similar boat, at the very leas an expect can give hits to what happened. However this can also be problematic if your a library developer. Sometimes rust is expected to never panic especially in situations like WASM. This is a major problem for companies like Amazon Prime Video since they run in a WASM context for their TV APP. Any panic crashes everything. Personally I usually just either create a custom error type (preferred) or erase it away with Dyn Box Error (no other option). Random unwraps and expects haunt my dreams.
At risk of sounding harsh, that’s a huge failure in your modeling of invariants that should not be permitted in development.
Permitting it in development is why one ends up in the position of having to use an `expect()` in production code, because your API surfaces are wrong and can’t model your actual invariants.
unwrap() is only the most superficial part of the problem. Merely replacing `unwrap()` with `return Err(code)` wouldn't have changed the behavior. Instead of "error 500 due to panic" the proxy would fail with "error 500 due to $code".
Unwrap gives you a stack trace, while retuned Err doesn't, so simply using a Result for that line of code could have been even harder to diagnose.
`unwrap_or_default()` or other ways of silently eating the error would be less catastrophic immediately, but could still end up breaking the system down the line, and likely make it harder to trace the problem to the root cause.
The problem is deeper than an unwrap(), related to handling rollouts of invalid configurations, but that's not a 1-line change.
We don't know what the surrounding code looks like, but I'd expect it handles the error case that's expressed in the type signature (unless they `.unwrap()` there too).
The problem is that they didn't surface a failure case, which means they couldn't handle rollouts of invalid configurations correctly.
The use of `.unwrap()` isn't superficial at all -- it hid an invariant that should have been handled above this code. The failure to correctly account for and handle those true invariants is exactly what caused this failure mode.
And the error magically disappears when the function returns it?
It doesn’t disappear, it forces you to handle it.
Propagating upwards a valid way of handling it and often the correct answer.
There needs to be something at the top level that can handle a crashing process.
You mean like kubernetes that restarts your program when it crashes?
can Rust handle global panics?
Or can a unwrap be stopped?
This is just a normal Tuesday for languages with Exception and try/catch.
Blog post from less than a week ago on how Cloudflare avoids outages on configuration changes: https://blog.cloudflare.com/finding-the-grain-of-sand-in-a-h...
This has to sting a bit after that post.
> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value
I don't use Rust, but a lot of Rust people say if it compiles it runs.
Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out.
end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually wanna try it for industrial robot stuff or small controllers etc.
there's no bad language - just occassional hiccups from us users who use those tools.
You misunderstand what Rust’s guarantees are. Rust has never promised to solve or protect programmers from logical or poor programming. In fact, no such language can do that, not even Haskell.
Unwrapping is a very powerful and important assertion to make in Rust whereby the programmer explicitly states that the value within will not be an error, otherwise panic. This is a contract between the author and the runtime. As you mentioned, this is a human failure, not a language failure.
Pause for a moment and think about what a C++ implementation of a globally distributed network ingress proxy service would look like - and how many memory vulnerabilities there would be… I shudder at the thought… (n.b. nginx)
This is the classic example of when something fails, the failure cause over indexes on - while under indexing on the quadrillions of memory accesses that went off without a single hitch thanks to the borrow checker.
I postulate that whatever the cost in millions or hundreds of millions of dollars by this Cloudflare outage, it has paid for more than by the savings of safe memory access.
See: https://en.wikipedia.org/wiki/Survivorship_bias
> Pause for a moment and think about what a C++ implementation of a globally distributed network ingress proxy service would look like - and how many memory vulnerabilities there would be… I shudder at the thought
I mean thats an unfalsifiable statement, not really fair. C is used to successfully launch spaceships.
Whereas we have a real Rust bug that crashed a good portion of the internet for a significant amount of time. If this was a C++ service everyone would be blaming the language, but somehow Rust evangelicals are quick to blame it on "unidiomatic Rust code".
A language that lets this easily happen is a poorly designed language. Saying you need to ban a commonly used method in all production code is broken.
Only formal proof languages are immune to such properties. Therefore all languages are poorly designed by your metric.
Consider that the set of possible failures enabled by language design should be as small as possible.
Rust's set is small enough while also being productive. Until another breakthrough in language design as impactful as the borrow checker is invented, I don't imagine more programmers will be able to write such a large amount of safe code.
I would say the impact of the borrow checker is exaggerated.
> You misunderstand what Rust’s guarantees are.
Well, no, most Rust programmers misunderstand what the guarantees are because they keep parroting this quote. Obviously the language does not protect you from logic errors, so saying "if it compiles, it works" is disingenuous, when really what they mean is "if it compiles, it's probably free of memory errors".
No, the "if it compiles, it works" is genuinely about the program being correct rather than just free of memory errors, but it's more of a hyperbolic statement than a statement of fact.
It's a common thing I've experienced and seen a lot of others say that the stricter the language is in what it accepts the more likely it is to be correct by the time you get it to run. It's not just a Rust thing (although I think Rust is _stricter_ and therefore this does hold true more of the time), it's something I've also experienced with C++ and Haskell.
So no, it's not a guarantee, but that quote was never about Rust's guarantees.
Everyone understands Rust doesn't offer such guarantees.
Even more now after this outage.
But it's a fact that "if it compiles it runs" is often associated with Rust, in HN at least. A quick Algolia search tells me that.
> Rust won't save you from the usual programming mistake.
Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like.
No language will prevent you from writing broken code if you're determined to do so, and no language is impossible to write correct code in if you make a superhuman effort. But most of life happens in the middle, and tools like Rust make a huge difference to how often a small mistake snowballs into a big one.
> Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like.
No one treats it like that and nearly every Rust project is filled with unwraps all over the place even in production system like Cloudflare's.
Well let me avoid those that don’t understand it. It’s literally Rust 101.
It's literally not, Rust tutorials are littered with `.unwrap()` calls. It might be Rust 102, but the first impression given is that the language is surprisingly happy with it.
I've worked on commercial codebases that did better, shrug.
Yep, unwrap() and unsafe are escape hatches that need very good justifications. It's fine for casual scripts where you don't care if it crashes. For serious production software they should be either banned, or require immense scrutiny.
> you can have a linter forbid it entirely if you like.
It would be better if that would be the other way round "linter forbids it unless you ask it not to". Never wrong to allow users to shoot themself in the foot, but it should be explicit.
> Well Rust won't save you from the usual programming mistake
This is not a Rust problem. Someone consciously chose to NOT handle an error, possibly thinking "this will never happen". Then someone else conconciouly reviewed (I hope so) a PR with an unwrap() and let it slide.
And people doing testing failed to ignore their excuse of this never happening and still testing it. With this kind of systems you need the separate group that just ignores any "this will never happen" and still checks what happens if it does.
Now it might be that it was tested, but then ignored or deprioritised by management...
What people are saying is that idiomatic prod rust doesn't use unwrap/expect (both of which panic on the "exceptional" arm of the value) --- instead you "match" on the value and kick the can up a layer on the call chain.
What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than bot mitigation you'd still have the same symptom of getting 500'd.
> What happens to it up the callstack?
as they say in the post, these files get generated every 5 minutes and rolled out across their fleet.
so in this case, the thing farther up the callstack is a "watch for updated files and ingest them" component.
that component, when it receives the error, can simply continue using the existing file it loaded 5 minutes earlier.
and then it can increment a Prometheus metric (or similar) representing "count of errors from attempting to load the definition file". that metric should be zero in normal conditions, so it's easy to write an alert rule to notify the appropriate team that the definitions are broken in some way.
that's not a complete solution - in particular it doesn't necessarily solve the problem of needing to scale up the fleet, because freshly-started instances won't have a "previous good" definition file loaded. but it does allow for the existing instances to fail gracefully into a degraded state.
in my experience, on a large enough system, "this could never happen, so if it does it's fine to just crash" is almost always better served by a metric for "count of how many times a thing that could never happen has happened" and a corresponding "that should happen zero times" alert rule.
Given that the bug was elsewhere in the system (the config file parser spuriously failed), it’s hard to justify much of what you suggested.
Panics should be logged, and probably grouped by stack trace for things like prometheus (outside of process). That handles all sorts of panic scenarios, including kernel bugs and hardware errors, which are common at cloudflare scale.
Similarly, mitigating by having rapid restart with backoff outside the process covers far more failure scenarios with far less complexity.
One important scenario your approach misses is “the watch config file endpoint fell over”, which probably would have happened in this outage if 100% of servers went back to watching all of a sudden.
Sure, you could add an error handler for that too, and for prometheus is being slow, and an infinite other things. Or, you could just move process management and reporting out of process.
The way I’ve seen this on a few older systems was that they always keep the previous configuration around so it can switch back. The logic is something like this:
1. At startup, load the last known good config.
2. When signaled, load the new config.
3. When that passes validation, update the last-known-good pointer to the new version.
That way something like this makes the crash recoverable on the theory that stale config is better than the service staying down. One variant also recorded the last tried config version so it wouldn’t even attempt to parse the latest one until it was changed again.
For Cloudflare, it’d be tempting to have step #3 be after 5 minutes or so to catch stuff which crashes soon but not instantly.
Presumably you kick up the error to a level that says “if parsing new config fails, keep the old config”
The config file subsystem was where the bug lived, not the code with the unwrap, so this sort of change is a special case of “make the unwrap never fail and then fix the API so it is not needed”.
Yeah, see, that's what I mean.
"if it compiles it runs" - this is indeed an inaccurate marketing slogan. A more precise formulation would be "if it compiles then the static type system, pattern matching, explicit errors, Send bounds, etc. will have caught a lot of bugs that in other languages would have manifested as runtime errors".
Anecdotally I can write code for several hours, deploy it to a test sandbox without review or running tests and it will run well enough to use it, without silly errors like null pointer exceptions, type mismatches, OOBs etc. That doesn't mean it's bug-free. But it doesn't immediately crash and burn either. Recently I even introduced a bug that I didn't immediately notice because careful error handling in another place recovered from it.
> I don't use Rust, but a lot of Rust people say if it compiles it runs.
Do you grok what the issue was with the unwrap, though...?
Idiomatic Rust code does not use that. The fact that it's allowed in a codebase says more about the engineering practices of that particular project/module/whatever. Whoever put the `unwrap` call there had to contend with the notion that it could panic and they still chose to do it.
It's a programmer error, but Rust at least forces you to recognize "okay, I'm going to be an idiot here". There is real value in that.
While I agree that Rust got it right by being more explicit, a lot of bugs in C/C++ can also easily avoided with good engineering practices. The Rust argument that it is mainly the fault of the programming language with C/C++ was always a huge and unfair exaggeration. Now with this entirely predictable ".unwrap" desaster (in general, not necessarily this exact scenarious), the "no true Rustacean would have put unwrap in production" fallacy is sad and funny at the same time.
Unwrap is controversial. The problem is that if you remove it, it makes the bar even higher for newcomers to Rust. One solution is to make it unsafe (along with panic).
> the "no true Rustacean would have put unwrap in production"
The "no unwrap" rule is common in most production codebases. Chill.
Could you point one that is Open source?
other people might say - why use unsafe rust - but we don't know the conditions of what the original code shipped under. why the pr was approved.
could have been tight deadline, managerial pressure or just the occasional slip up.
[dead]
Classic combination of errors:
Having the feature table pivoted (with 200 feature1, feature2, etc columns) meant they had to do meta queries to system.columns to get all the feature columns which made the query sensitive to permissioning changes (especially duplicate databases).
A Crowdstrike style config update that affects all nodes but obviously isn't tested in any QA or staged rollout strategy beforehand (the application panicking straight away with this new file basically proves this).
Finally an error with bot management config files should probably disable bot management vs crash the core proxy.
I'm interested here why they even decided to name Clickhouse as this error could have been caused by any other database. I can see though the replicas updating causing flip / flopping of results would have been really frustrating for incident responders.
Right but also this is a pretty common pattern in distributed systems that publish from databases (really any large central source of truth); it might be like the problem in systems like this. When you're lucky the corner cases are obvious; in the big one we experienced last year, a new row in our database tripped an if-let/mutex deadlock, which our system dutifully (and very quickly) propagated across our entire network.
The solution to that problem wasn't better testing of database permutations or a better staging environment (though in time we did do those things). It was (1) a watchdog system in our proxies to catch arbitrary deadlocks (which caught other stuff later), (2) segmenting our global broadcast domain for changes into regional broadcast domains so prod rollouts are implicitly staged, and (3) a process for operators to quickly restore that system to a known good state in the early stages of an outage.
(Cloudflare's responses will be different than ours, really I'm just sticking up for the idea that the changes you need don't follow obviously from the immediate facts of an outage.)
We shouldn't be having critical internet-wide outages on a monthly basis. Something is systematically wrong with the way we're architecting our systems.
Cloudflare, Azure, and other single points of failure are solving issues inherent to webhosting, and those problems have become incredibly hard due to the massive scale of bad actors and the massive complexity of managing hardware and software.
What would you propose to fix it? The fixed cost of being DDoS-proof is in the hundreds of millions of dollars.
Costs to architect systems that serve millions of request daily have gone down. Not up.
Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client.
People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?)
> Costs to architect systems that serve millions of request daily have gone down. Not up.
I never said serving millions of requests is more expensive. Protecting your servers is more expensive.
> Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client.
HN uses Cloudflare. You're making my point for me. If you included the fixed costs that Cloudflare's CDN/proxy is giving to HN incredibly cheaply, then running HN at the edge with good performance (and protecting it from botnets) would costs hundreds of millions of dollars.
> People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?)
Again, attacking your own straw men here.
Writing high-throughput web applications is easier than ever. Hosting them on the open web is harder than ever.
> HN uses Cloudflare
From the ping output, I can see HN is using m5hosting.com. This is why HN was up yesterday, even though everything on CF was down.
> Writing high-throughput web applications is easier than ever. Hosting them on the open web is harder than ever.
Writing proper high-throughput applications was never easy and will never be. It is a little bit easier because we have highly optimized tools like nginx or nodejs so we can offset critical parts. And hosting is "harder than ever" if you complicate the matter, which is a quite common pattern these days. I saw people running monstrosities to serve some html & js in the name of redundancy. You'd be surprised how much a single bare-metal (hell, even a proper VM from DigitalOcean or Vultr) can handle.
Saying "other single points of failure" makes no sense whatsoever.
Cloudflare and us-east-1 are single points of failures for millions of their customers, many of which overlap.
"Single" means "you only need one," not that there is only one.
The most surprising thing to me here is that it took 3 hours to root cause, and points to a glaring hole in the platform observability. Even taking into account the fact that the service was failing intermittently at first, it still took 1.5 hours after it started failing consistently to root cause. But the service was crashing on startup. If a core service is throwing a panic at startup like that, it should be raising alerts or at least easily findable via log aggregation. It seems like maybe there was some significant time lost in assuming it was an attack, but it also seems strange to me that nobody was asking "what just changed?", which is usually the first question I ask during an incident.
That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing with (a bot net that had escalated DDoS attacks from 3Tbps to 30Tbps against us and others like Microsoft over the last 3 months). We worked multiple theories in parallel. After an hour we ruled out the DDoS theory. We had other theories also running in parallel, but at that point the dominant theory was that the feature file was somehow corrupt. One thing that made us initially question the theory was nothing in our changelogs seemed like it would have caused the feature file to grow in size. It was only after the incident that we realized the database permissions change had caused it, but that was far from obvious. Even after we identified the problem with the feature file, we did not have an automated process to role the feature file back to a known-safe previous version. So we had to shut down the reissuance and manually insert a file into the queue. Figuring out how to do that took time and waking people up as there are lots of security safeguards in place to prevent an individual from easily doing that. We also needed to double check we wouldn’t make things worse. The propagation then takes some time especially because there are tiers of caching of the file that we had to clear. Finally we chose to restart the FL2 processes on all the machines that make up our fleet to ensure they all loaded the corrected file as quickly as possible. That’s a lot of processes on a lot of machines. So I think best description was it took us an hour for the team to coalesce on the feature file being the cause and then another two to get the fix rolled out.
Thank you for the clarification and insight, with that context it does make more sense to me. Is there anything you think can be done to improve the ability to identify issues like this more quickly in the future?
Any "limits" on system should be alerted... like at 70% or 80% threshold.. it might be worth it for a SRE to revisit the system limits and ensuring threshold based alerting around it..
If one actually looks at the current pingora API, it has limited ability to initialize async components at startup - the current pattern seems to be to lazily initialize on first call. An obvious downside of this is that a service can startup in a broken state. e.g. https://github.com/cloudflare/pingora/issues/169
I can imagine that this could easily lead to less visibility into issues.
Long time ago Google had a very similar incident where ddos protection system ingested a bad config and took everything down. Except it was auto resolved in like four minutes by an automatic rollback system before oncall was even able to do anything. Perhaps Cloudflare should invest in a system like that
An unwrap like that in production code on the critical path is very surprising to me.
I haven’t worked in Rust codebases, but I have never worked in a Go codebase where a `panic` in such a location would make it through code review.
Is this normal in Rust?
absolutely not normal, this is why in my opinion it took them so long to understand the core issue. instead of a nice error message and a backtrace saying something like "failed to parse config feature names" they thought they were under attack because the service was just crashing instead.
People really like to hate on Rust for some reason. This wasn’t a Rust problem, no language would have saved them from this kind of issue. In fact, the compiler would have warned that this was a possible issue.
I get it, don’t pick languages just because they are trendy, but if any company’s use case is a perfect fit for Rust it’s cloudflare.
Yeah even if you handled this situation without unwrap() if you just went down an error path that didn't panic, the service would likely still be inoperable if every single request went down the error path.
Okay, but if you returned a wrapped error it’d at least be easier to debug.
not necessarily, by default `Result` does not even carry a stack trace
The reason why people are criticizing is because Rust evangelicals say stuff like "if it compiles it works" or talk about how Rust's type system is so much better than other languages that it catches logic errors like this. You won't see Go or Java developers making such strong claims about their preferred languages.
> [...] it catches logic errors like this
but Rust's type system did catch this error - and then author decided it's fine to panic if this error happens
> You won't see Go or Java developers making such strong claims about their preferred languages.
yess no Java developer ever said that OOP will solve world hunger
> but Rust's type system did catch this error - and then author decided it's fine to panic if this error happens
The issue is that it wasn't fine to panic, thus Rust did not catch this error.
Come on now. You can't blame the compiler when the programmer explicitly told the compiler to not worry about it. There is nothing in existence that can protect against something like that.
I integrated Turnstile with a fail-open strategy that proved itself today. Basically, if the Turnstile JS fails to load in the browser (or in a few specific frontend error conditions), we allow the user to submit the web form with a dummy challenge token. On the backend, we process the dummy token like normal, and if there is an error or timeout checking Turnstile's siteverify endpoint, we fail open.
Of course, some users were still blocked, because the Turnstile JS failed to load in their browser but the subsequent siteverify check succeeded on the backend. But overall the fail-open implementation lessened impact to our customers nonetheless.
Fail-open with Turnstile works for us because we have other bot mitigations that are sufficient to fall back on in the event of a Cloudflare outage.
So to bypass captcha all a user has to do is block the script from loading? I can see that working but only for attacks that aren’t targeted?
Only if they are able to block the siteverify check performed by our backend server. That's not the kind of attack we are trying to mitigate with Turnstile.
Wow. 26M/s 5xx error HTTP status codes over a span of roughly two hours. That's roughly 187 billion HTTP errors that interrupted people (and systems)!
Some of these would be retries that wouldn't have happened if not for earlier errors.
Why call .unwrap() in a function which returns Result<_,_>?
For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.
Probably because this case was something more akin to an assert than an error check.
Fly writes a lot of Rust, do you allow `unwrap()` in your production environment? At Modal we only allow `expect("...")` and the message should follow the recommended message style[1].
I'm pretty surprised that Cloudflare let an unwrap into prod that caused their worst outage in 6 years.
1. https://doc.rust-lang.org/std/option/enum.Option.html#recomm...
After The Great If-Let Outage Of 2024, we audited all our code for that if-let/rwlock problem, changed a bunch of code, and immediately added a watchdog for deadlocks. The audit had ~no payoff; the watchdog very definitely did.
I don't know enough about Cloudflare's situation to confidently recommend anything (and I certainly don't know enough to dunk on them, unlike the many Rust experts of this thread) but if I was in their shoes, I'd be a lot less interested in eradicating `unwrap` everywhere and more in making sure than an errant `unwrap` wouldn't produce stable failure modes.
But like, the `unwrap` thing is all programmers here have to latch on to, and there's a psychological self-soothing instinct we all have to seize onto some root cause with a clear fix (or, better yet for dopaminergia, an opportunity to dunk).
A thing I really feel in threads like this is that I'd instinctively have avoided including the detail about an `unwrap` call --- I'd have worded that part more ambiguously --- knowing (because I have a pathological affinity for this community) that this is exactly how HN would react. Maybe ironically, Prince's writing is a little better for not having dodged that bullet.
Sounds like if nothing else, additional attention around (their?) use of unwrap() is still warranted from where you're sitting then though, no? I don't think there's anything wrong with flagging that.
It's one thing to not want to be the one to armchair it, but that doesn't mean that one has to suppress their normal and obvious reactions. You're allowed to think things even if they're kitsch, you too are human, and what's kitsch depends and changes. Applies to everyone else here by extension too.
Fair. I agree that saying "it's the unwrap" and calling it a day is wrong. Recently actually we've done an exercise on our Worker which is "assume the worst kind of panic happens. make the Worker be ok with it".
But I do feel strongly that the expect pattern is a highly useful control and that naked unwraps almost always indicate a failure to reason about the reliability of a change. An unwrap in their core proxy system indicates a problem in their change management process (review, linting, whatever).
Rust has debug asserts for that. Using expect with a comment about why the condition should not/can't ever happen is idiomatic for cases where you never expect an Err.
This reads to me more like the error type returned by append with names is not (ErrorFlags, i32) and wasn't trivially convertible into that type so someone left an unwrap in place on an "I'll fix it later" basis, but who knows.
Oh absolutely, that's how it would have been treated.
Surely a unwrap_or_default() would have been a much better fit--if fetching features fails, continue processing with an empty set of rules vs stop world.
You are saying this would not have happened in a C release build where asserts define to nothing?
Wonder why these old grey beards chose to go with that.
I am one of those old grey beards (or at least, I got started shipping C code in the 1990s), and I'd leave asserts in prod serverside code given the choice; better that than a totally unpredictable error path.
> You are saying this would not have happened in a C release build where asserts define to nothing?
Afaik, Go and Java are the only languages that make you pause and explicitly deal with these exceptions.
And rust, but they chose to panic on the error condition. Wild.
In that case, it should probably be expect rather than unwrap, to document why the assertion should never fail.
Yes, can't have .unwrap() in production code (it's ok in tests)
Like goto, unwrap is just a tool that has its use cases. No need to make a boogeyman out of it.
To be fair, if you’re not “this tall” you really shouldn’t consider using goto in a c program. Most people aren’t that tall.
Nonsense. Linux kernel for one example, uses goto everywhere for error handling.
Yes it's meant to be used in test code. If you're sure it can't fail do then use .expect() that way it shows you made a choice and it wasn't just a dev oversight.
panicans should be using .expect() in production
unwrap itself isn't the problem...
Why is there a 200 limit on appending names?
Everything has a limit. You can define it, or be surprised when you find out what it is.
Limits in systems like these are generally good. They mention the reasoning around it explicitly. It just seems like the handling of that limit is what failed and was missed in review.
"Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero."
This simply means, the exception handling quality of your new FL2 is non-existent and is not at par / code logic wise similar to FL.
I hope it was not because of AI driven efficiency gains.
In most domains, silently returning 0 in a case where your logic didn't actually calculate the thing you were trying to calculate is far worse than giving a clear error.
As an IT person, I wonder what it's like to work for a company like this. Where presumably IT stuff has a priority. Unlike the companies I've worked for where IT takes a backseat to everything until something goes wrong. Company I work had a huge new office built, with the plan it would be big enough for future growth, yet despite repeated attempts to reserve a larger space, our server room and infrastructure is actually smaller than our old building and has no room to grow.
As a former CF employee, I'd say it's a mixed bag.
There are plenty of resources , yet it's somehow never enough. You do tons of pretty amazing things with pretty amazing tools that also have notable shortcomings.
You're surround by smart people who do lots of great work, but you also end up in incident reviews where you find facepalm-y stuff. Sometimes you even find out it was a known corner case that was deemed too unlikely to prioritize.
The last incident for my team that I remember dealing with there ended up with my coworker and I realizing the staging environment we'd taken down hours earlier was actually the source of data for a production dashboard, so we'd lost some visibility and monitoring for a bit.
I've also worked at Facebook (pre-Meta days) and at Datadog, and I'd say it was about the same. Most things are done quite well, but so much stuff is happening that you still end up with occasional incidents that feel like they shouldn't have happened.
While I heavily frown upon using `unwrap` and `expect` in Rust code and make sure to have Clippy tell me about every single usage of them, I also understand that without them Rust might have been seen as an academic curiosity language.
They are escape hatches. Without those your language would never take off.
But here's the thing. Escape hatches are like emergency exits. They are not to be used by your team to go to lunch in a nearby restaurant.
---
Cloudflare should likely invest in better linting and CI/CD alerts. Not to mention isolated testing i.e. deploy this change only to a small subset and monitor, and only then do a wider deployment.
Hindsight is 20/20 and we can all be smartasses after the fact of course. But I am really surprised because lately I am only using Rust for hobby projects and even I know I should not use `unwrap` and `expect` beyond the first iteration phases.
---
I have advocated for this before but IMO Rust at this point will benefit greatly from disallowing those unsafe APIs by default in release mode. Though I understand why they don't want to do it -- likely millions of CI/CD pipelines will break overnight. But in the interim, maybe a rustc flag we can put in our `Cargo.toml` that enables such a stricter mode? Or have that flag just remove all the panicky API _at compile time_ though I believe this might be a Gargantuan effort and is likely never happening (sadly).
In any case, I would expect many other failures from Cloudflare but not _this_ one in particular.
This is not a reasonable take to me. unwrap/expect are the idiomatic way to express code paths returning Option/Result as unreachable.
Bubbling up the error or None does not make the program correct. Panicking may be the only reasonable thing to do.
If panicking is guaranteed because of some input mistake to the system your failure is in testing.
Say you panic, now you need to have an external system that catches this panic and reports back; and does something meanwhile to recover your system.
If you think about it, it’s not really different from handling the bubbled up error inside of Rust. You don’t (?) your results and your errors go away, they just move up the chain.
I agree the failure is in testing but what you can and should do is raise in alert in your APM system before the runtime panic, in the code path that is deemed impossible to hit.
I am not trashing on them, I've made such mistakes in the past, but I do expect more from them is all.
And you will not believe how many alerts I got for the "impossible" errors.
I do agree there was not too much that could have been done, yes. But they should have invested in more visibility and be more thorough. I mean, hobbyist Rust devs seem to do that better.
It was just a bit disappointing for me. As mentioned above, I'd understand and sympathise with many other mistakes but this one stung a bit.
There's certainly a discipline involved here, but it's usually something like guaranteeing all threads are unwind safe (via AssertUnwindSafe) and logging stack traces when your process keeps dying/can't be started after a fixed number of retries. Which would lead you to the culprit immediately.
I'm just pushing back a bit on the idea that unwrap() is unsafe - it's not, and I wouldn't even call it a foot gun. The code did what it was written to do, when it saw the input was garbage it crashed because it couldn't make sense of what to do next. That's a desirable property in reliable systems (of course monitoring that and testing it is what makes it reliable/fixable in the first place).
We don't disagree, my main point was a bit broader and admittedly hijacked the original topic a bit, namely: `unwrap` and `expect` make many Rust devs too comfortable and these two are very tempting mistresses.
Using those should be done in an extremely disciplined manner. I agree that there are many legitimate uses but in the production Rust code I've seen this has rarely been the case. People just want to move on and then forget to circle back and add proper error handling. But yes, in this case that's not quite true. Still, my point that an APM alert should have been raised on the "impossible" code path before panicking, stands.
I'm feeling generous tonight, I'm willing to consider 0.99986 to round to 99.99%
The real take away is that so much functionality depends on a few players. This is a fundamental flaw in design that is getting worse by the year as the winner takes all winners win. Not saying they didn’t earn their wins. But the fact remains. The system is not robust. Then again, so what. It went down for a while. Maybe we shouldn’t depend on the internet being “up” all the time.
> Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system.
And here is the query they used ** (OK, so it's not exactly):
someone added a new row to permissions and the JOIN started returning two dupe feature rows for each distinct feature.** "here is the query" is used for dramatic effect. I have no knowledge of what kind of database they are even using much less queries (but i do have an idea).
more edits: OK apparently it's described later in the post as a query against clickhouse's table metadata table, and because users were granted access to an additional database that was actually the backing store to the one they normally worked with, some row level security type of thing doubled up the rows. Not sure why querying system.columns is part of a production level query though, seems overly dynamic.
I believe they mentioned ClickHouse
Cloudflare Access is still experiencing weird issues for us (it’s asking users to SSO login to our public website even though our zone rules - set on a completely different zone - haven’t changed).
I don’t think the infrastructure has been as fully recovered as they think yet…
> a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system ... to keep [that] system up to date with ever changing threats
> The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail
A configuration error can cause internet-scale outages. What an era we live in
Edit: also, after finishing my reading, I have to express some surprise that this type of error wasn't caught in a staging environment. If the entire error is that "during migration of ClickHouse nodes, the migration -> query -> configuration file pipeline caused configuration files to become illegally large", it seems intuitive to me that doing this same migration in staging would have identified this exact error, no?
I'm not big on distributed systems by any means, so maybe I'm overly naive, but frankly posting a faulty Rust code snippet that was unwrapping an error value without checking for the error didn't inspire confidence for me!
It would have been caught only in stage if there was similar amount of data in the database. If stage has 2x less data it would have never occurred there. Not super clear how easy it would have been to keep stage database exactly as production database in terms of quantity and similarity of data etc.
I think it's quite rare for any company to have exact similar scale and size of storage in stage as in prod.
> I think it's quite rare for any company to have exact similar scale and size of storage in stage as in prod.
We’re like a millionth the size of cloudflare and we have automated tests for all (sort of) queries to see what would happen with 20x more data.
Mostly to catch performance regressions, but it would work to catch these issues too.
I guess that doesn’t say anything about how rare it is, because this is also the first company at which I get the time to go to such lengths.
But now consider how much extra data Cloudflare at its size would have to have just for staging, doubling or more their costs to have stage exactly as production. They would have to simulate similar amount of requests on top of themselves constantly since presumably they have 100s or 1000s of deployments per day.
In this case it seems the database table in question seemed modest in size (the features for ML) so naively thinking they could have kept stage features always in sync with prod at the very least, but could be they didn't consider that 55 rows vs 60 rows or similar could be a breaking point given a certain specific bug.
It is much easier to test with 20x data if you don't have the amount of data cloudflare probably handles.
That just means it takes longer to test. It may not be possible to do it in a reasonable timeframe with the volumes involved, but if you already have 100k servers running to serve 25M requests per second, maybe briefly booting up another 100k isn’t going to be the end of the world?
Either way, you don’t need to do it on every commit, just often enough that you catch these kinds of issues before they go to prod.
The speed and transparency of Cloudflare publishing this port mortem is excellent.
I also found the "remediation and follow up" section a bit lacking, not mentioning how, in general, regressions in query results caused by DB changes could be caught in future before they get widely rolled out.
Even if a staging env didn't have a production-like volume of data to trigger the same failure mode of a bot management system crash, there's also an opportunity to detect that something has gone awry if there were tests that the queries were returning functionally equivalent results after the proposed permission change. A dummy dataset containing a single http_requests_features column would suffice to trigger the dupe results behaviour.
In theory there's a few general ways this kind of issue could be detected, e.g. someone or something doing a before/after comparison to test that the DB permission change did not regress query results for common DB queries, for changes that are expected to not cause functional changes in behaviour.
Maybe it could have been detected with an automated test suite of the form "spin up a new DB, populate it with some curated toy dataset, then run a suite of important queries we must support and check the results are still equivalent (after normalising row order etc) to known good golden outputs". This style of regression testing is brittle, burdensome to maintain and error prone when you need to make functional changes and update what then "golden" outputs are - but it can give a pretty high probability of detecting that a DB change has caused unplanned functional regressions in query output, and you can find out about this in a dev environment or CI before a proposed DB change goes anywhere near production.
This wild `unwrap()` kinda took me aback as well. Someone really believed in themselves writing this. :)
They only recently rewrote their core in Rust (https://blog.cloudflare.com/20-percent-internet-upgrade/) -- given the newness of the system and things like "Over 100 engineers have worked on FL2, and we have over 130 modules" I won't be surprised for further similar incidents.
The irony of a rust rewrite taking down the internet is not lost on me.
I have to wonder if AI was involved with the change.
I don't think this is the case with CloudFlare, but for every recent GitHub outage or performance issue... oh boy, I blame the clankers!
May I just say that Matthew Prince is the CEO of Cloudflare and a lawyer by training (and a very nice guy overall). The quality of this postmortem is great but the fact that it is from him makes one respect the company even more.
There's (obviously) a lot of discussion around the use of `unwrap` in production code. I feel like I'm watching comments speak past each other right now.
I'd agree that the use of `unwrap` could possibly make sense in a place where you do want the system to fail hard. There's lot of good reasons to make the system fail hard. I'd lean towards an `expect` here, but whatever.
That said, the function already returns a `Result` and we don't know what the calling code looks like. Maybe it does do an `unwrap` there too, or maybe there is a save way for this to log and continue that we're not aware of because we don't have enough info.
Should a system as critical as the CF proxy fail hard? I don't know. I'd say yes if it was the kind of situation that could revert itself (like an incremental rollout), but this is such an interesting situation since it's a config being rolled out. Hindsight is 20:20 obviously, but it feels like there should've been better logging, deployment, rollback, and parsing/validation capabilities, no matter what the `unwrap`/`Result` option is.
Also, it seems like the initial Clickhouse changes could've been testing much better, but I'm sure the CF team realizes that.
On the bright side, this is a very solid write up so quickly after the outage. Much better than those times we get it two weeks later.
A lot of outages off late seem to be related to automated config management.
Companies seem to place a lot of trust is configs being pushed automatically without human review into running systems. Considering how important these configs are, shouldn't they perhaps first be deployed to a staging/isolated network for a monitoring window before pushing to production systems?
Not trying to pontificate here, these systems are more complicated than anything I have maintained. Just trying to think of best practices perhaps everyone can adopt.
kudos to getting this blog post out so fast, it’s well written and is appreciated.
i’m a little confused on how this was initially confused for an attack though?
is there no internal visibility into where 5xx’s are being thrown? i’m surprised there isn’t some kind of "this request terminated at the <bot checking logic>" error mapping that could have initially pointed you guys towards that over an attack.
also a bit taken aback that .unwrap()’s are ever allowed within such an important context.
would appreciate some insight!
1. Cloudflare is in the business of being a lightning rod for large and targeted DoS attacks. A lot of cases are attacks.
2. Attacks that make it through the usual defences make servers run at rates beyond their breaking point, causing all kinds of novel and unexpected errors.
Additionally, attackers try to hit endpoints/features that amplify severity of their attack by being computationally expensive, holding a lock, or trigger an error path that restarts a service — like this one.
eh, i'm not convinced.
this was in the middle of a scheduled maintenance, with all requests failing at a singular point - that being a .unwrap().
there should be internal visibility into the fact a large number of requests are failing all at the same LOC - and attention should be focused there instantly imo.
or at the very least, it shouldn't take 4 hours for anyone to even consider it wasn't an attack.
in situations such as this, where your entire infra is fucked, you should have multiple crisis teams working in parallel, under different assumptions.
if even one additional team was created that worked under the assumption it was an infra issue rather than an attack, this situation could have been resolved many hours earlier.
for a product as vital to the internet as cloudflare, it is unacceptable to not have this kind of crisis management.
There were two things I think went extremely poorly here:
1) Lack of validation of the configuration file.
Rolling out a config file across the global network every 5 minutes is extremely high risk. Even without hindsight, surely one would see then need for very careful validation of this file before taking on that risk?
There were several things "obviously" wrong with the file that validation should have caught:
- It was much bigger than expected.
- It had duplicate entries.
- Most importantly, when loaded into the FL2 proxy, the proxy would panic on every request. At the very least, part of the validation should involve loading the file into the proxy and serving a request?
2) Very long time to identify and then fix such a critical issue.
I can't understand the complete lack of monitoring or reporting? A panic in Rust code, especially from an unwrap, is the application screaming that there's a logic error! I don't understand how that can be conflated with a DDoS attack. How are your logs not filled with backtraces pointing to the exact "unwrap" in question?
Then, once identified, why was it so hard to revert to a known good version of the configuration file? How did noone foresee the need to roll back this file when designing a feature that deploys a new one globally every 5 minutes?
If the software has a limit on the size of the feature file then the process that propagates the file should probably validate the size before propagating ..
>Currently that limit is set to 200, well above our current use of ~60 features. Again, the limit exists because for performance reasons we preallocate memory for the features.
So they basically hardcoded something, didn't bother to cover the overflow case with unit tests, didn't have basic error catching that would fallback and send logs/alerts to their internal monitoring system and this is why half of the internet went down?
One of the remediations listed is
> Eliminating the ability for core dumps or other error reports to overwhelm system resources
but this is not mentioned at all in the timeline above. My best guess would be that the process got stuck in a tight restart loop and filled available disk space with logs, but I'm happy to hear other guesses for people more familiar with Rust.
I understood this to be related to this section:
> As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period. This was due to large amounts of CPU being consumed by our debugging and observability systems, which automatically enhance uncaught errors with additional debugging information.
(Just above https://blog.cloudflare.com/18-november-2025-outage/#how-clo...)
Ah, yes, of course. I must have missed that. Thanks!
Having a system which automatically deploys configuration files across a million servers every 5 minutes without testing it seems stupid to me.
When I first read about it I assumed it would have been a "poison pill" - a bad config where the ingestion of the config leads the process to crash/restart. And due to that crash on startup, there is no automated possibility to revert to a good config. These things are the worst issues that all global control planes have to deal with.
The report actually seems to confirm this - it was indeed a crash on ingesting the bad config. However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad".
The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.
I don't understand what's the business of cloudflare.
They just sell proxies, to whoever.
Why are they the only company doing ddos protection?
I just don't get it.
Momentum, I guess, pretty much like you wouldn’t get fired for using AWS or IBM (if that’s still the case now).
[dead]
Cloudflare tried to build their own feature store, and get a grade F.
I wrote a book on feature stores by O'Reilly. The bad query they wrote in Clickhouse could have been caused by another more error - duplicate rows in materialized feature data. For example, in Hopsworks it prevents duplicate rows by building on primary key uniqueness enforcement in Apache Hudi. In contrast, Delta lake and Iceberg do not enforce primary key constraints, and neither does Clickhouse. So they could have the same bug again due to a bug in feature ingestion - and given they hacked together their feature store, it is not beyond the bounds of possibility.
Reference: https://www.oreilly.com/library/view/building-machine-learni...
If you knew the expected number of features , any input file with >100 should be discarded as bad input and you failback to the last good feature file received. This would have protected your service even though you are unable to get the newly populated features. I believe these features are not updated that frequently. Even if they were, you would have biased your system towards availability vs 'correctness'.
What were the teams doing between 11 to 1300 hrs , no explanation of what investigations were going on to not being able to figure the root cause.
Interesting technical insight, but I would be curious to hear firsthand accounts from the teams on the ground, particularly regarding how the engineers felt the increasing pressure, frantically refreshing their dashboards, searching for phantom DDoS, scrolling codes updates...
I’ll be honest, I only understand about 30% of what is being said in this thread and that is probably generous. But it is very interesting seeing so many people respond to each other “it’s so simple! what went wrong was…” as they all disagree on what exactly went wrong.
I don't get why that SQL query was even used in the first place. It seems it fetches feature names at runtime instead of using a static hardcoded schema. Considering this decides the schema of a global config, I don't think the dynamicity is a good idea.
Honestly... everyone shit themselves that internet doesn't work, but next week this outage will be forgotten by 99% of population. I was doing something on my PC when I saw clear information that Cloudflare is down, so I decided to just go take a nap, then read a book, then go for a walk. Once I was done, the internet was working again. Panic was not necessary on my side.
What I'm trying to say is that things would be much better if everyone took a chill pill and accepted the possibility that in rare instances, the internet doesn't work and that's fine. You don't need to keep scrolling TikTok 24/7.
> but my use case is especially important
Take a chill pill. Probably it isn't.
It's funny how everyone seems to be having a meltdown over this. I didn't even notice anything was wrong until I read about it on Reddit 5 hours later, even though I was working all day. Sounds to me like people are too reliant on random websites.
Makes me wonder which team is responsible for that feature generating query, and if they follow full engineering level QA. It might be deferred to an MLE team that is better than the data scientists but less rigorous than software needs to be.
Exactly. The post screams about all the issues I've seen in multiple companies between DS/MLE and SE/DevOps.
It feels like their list of after actions is lacking a bit to me.
How about 1. The permissions change project is paused or rolled back until 2. All impacted database interactions (SQL queries) are evaluated for improper assumptions or better 3. Their design that depends on database metainfo and schema is replaced with ones that use specific tables and rows in tables instead of using the meta info as part of their application. 4. All hard coded limits are centralized in a single global module and referenced from their users and then back propagated to any separate generator processes that validate against the limit before pushing generated changes
Question: customer having issues also couldn't switch their dns to bypass the service, why is the control plane updated along the data plane here it seem a lot of use could save business continuity if they could change their dns entry temporarily
Attempt to reproduce the Cloudflare 2025-11-18 outage.
Cloudflare's incident report is written clearly and explicitly, so based on my own understanding, I’m going to try reproducing this outage. Already completed:
CK cluster Permission change triggering data doubling Cache propagation Unaffected proxy services Proxy services with bot score errors
TODO:
unwrap panic during pre-allocation of cache Full demonstration of the entire outage process
https://github.com/Laotree/reproduce_cf20251118
The internet hasn't been the internet in years. It was originally built to withstand wars. The whole idea of our IP based internet was to reroute packages should networks go down. Decentralisation was the mantra and how it differed from early centralised systems such as AOL et al.
This is all gone. The internet is a centralised system in the hand of just a few companies. If AWS goes down half the internet does. If Azure, Google Cloud, Oracle Cloud, Tencent Cloud or Alibaba Cloud goes down a large part of the internet does.
Yesterday with Cloudflare down half the sites I tried gave me nothing but errors.
The internet is dead.
It's not that deep, if AWS or Cloudflare suddenly disappeared sites would move to different hosts, it wouldn't mean the internet would die.
I mean you still depend on authoritative dns servers no?
What I read is that a non-critical feature (blocking / managing bot - access) was able to impact a critical feature (routing traffic).
Shouldn't the architecture setup in such a way that subcomponents can fail without impacting the critical function of the component?
https://x.com/guanlandai/status/1990967570011468071
Thanks for the detailed writeup and explaining the root cause in details
However, I have a question from a release deployment process perspective. Why was this issue not detected during internal testing ? I didn't find the RCA analysis covering this aspect. Doesn't cloudflare have an internal test stage as part of its CICD pipeline. Looking the description of the issue, it should have been immediately detected in internal stage test environment.
Thanks for the detailed RCA. After going through the blog post as well, there’s a curiosity about whether there are opportunities to detect such scenarios earlier in the testing phase. It would be helpful to understand if there are any differences between the test environment and production that might have contributed to this. This could provide insights into strengthening the process going forward.
Wondering why they didn’t disable the bot management temporarily to recover. Websites could have survived temporarily without it compared to the outage itself.
As we know that "With great power comes great responsibility" the team should understand this because Cloudflare is used worldwide, and for many countries it was the peak working time when Cloudflare went down so this affected massively. We always want perfect results but it's not possible, I hope the team is not overworking to get the changes on prod.
Hold up ,- when I used a C or similar language for accessing a database and wanted to clamp down on memory usage to deterministically control how much I want to allocated, I would explicitly limit the number of rows in the query.
There never was an unbound "select all rows from some table" without a "fetch first N rows only" or "limit N"
If you knew that this design is rigid, why not leverage the query to actually do it ?
What am I missing ?
Because nothing forced them to and they didn't think of it. Maybe the people writing the code that did the query knew that the tables they were working with never had more than 60 rows and figured "that's small" so they didn't bother with a limit. Maybe the people who wrote the file size limit thought "60 rows isn't that much data" and made a very small file size limit and didn't coordinate with the first people.
Anyway regardless of which language you use to construct a SQL query, you're not obligated to put in a max rows
I imagine there's numerous ways to protect against it and protection should've been added by whoever decided on this optimization. In data layer, create some kind of view which never returns more than 200 rows from base table(s). In code, use some kind of iterator. I'm not a Rust guy, just a C defensive practices type of dude, but maybe they just missed a biggie during a code review.
That's interesting. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. It's like the issues with HOSTS.TXT needing to be copied among the network of the early internet to allow routing (taking days to download etc) and DNS having to be created to make that propagation less unwieldy.
Speaking of resiliency, the entire Bot Management module doesn't seems to be a critical part of the system, so for example, what happens if that module goes down for an hour? the other parts of the system should work. So I would rank every module and it's role in the system, and would design it in a way that when a non-critical module fails, other parts still can function.
The outage sucked for everyone. The root cause also feels like something they could have caught much earlier in a canary rollout from my reading of this.
All that said, to have an outage reported turned around practically the same day, that is this detailed, is quite impressive. Here's to hoping they make their changes from this learning, and we don't see this exact failure mode again.
how would you build redundancy around cloudflare failing?
i think this is happening way too frequently
meanwhile VPS, dedicated servers hum along without any issues
i dont want to use kubernetes but if we have to build mission critical systems doesn't seem like building on cloudflare is going to cut it
I can never get used to the error happening at call site rather than within the function where the early return of Err happened. It is not "much cleaner", you have no idea which line and file caused it at call site. By default Returning should have a way of setting a marker which can then be used to map back to the line() and file(). 10+ years and still no ergonomics.
This is an excellent lesson learned: Harden loading of internally generated config files as though they were untrusted content.
Gonna use that one at $WORK.
It actually wasn't resolved quite that fast. My site continued to have issues for hours afterward before things finally resolved completely.
> The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.
What could have prevented this failure?
Cloudflare's software could have included a check that refused to generate the feature file if it's size was higher than the limit.
A testcase could have caught this.
Great post-mortem. Very clear. Surprised that num(panicking threads) didn't show up somewhere in telemetry.
Based on this writeup it seems that Cloudflare defaults to a 0 score for bot prevention if there's a failure. Could instead it default to a passing score? Default open instead of default closed? This would have been a non-event to a lot of websites if that change was made.
Bit funny. "Memory management" bug in Rust. (writing that as kind of a fan of Rust)
This incident feels like a strong argument for stricter guardrails around internal config propagation
Given this was triggered by an old school configuration change across multiple servers, there's too little discussion of that particular process.
It sounds like the change could've been rolled out more slowly, halted when the incident started and perhaps rolled back just in case.
Would be nice if their Turnstile could be turned off on their login page when something like this happens, so we can attempt to route traffic away from Cloudflare during the outage. Or at least have a simple app where this can be modified from.
Is this true: from that core proxy diagram, I didn't realize cloudflare sees the full unencrypted packet between you and the server.
If that's true, is there a way to tell (easily) whether a site is using cloudflare or not?
It is pretty easy to see if cloudflare is proxying a site.
Just ping the host and see if the ip belongs to CF.
https://www.cloudflare.com/en-ca/ips/
The NSA has to see the data somehow, right?
Dear Matthew Prince, don't you think we (the ones affected by your staff's mistake) should get some sort of compensation??? Yours truly, a Cloudflare client who lost money during the November 18th outage.
What does your contract with Cloudflare say?
How many changes to production systems does Cloudflare make throughout a day? Are they a part of any change management process? That would be the first place I would check after a random outage, recent changes.
So they made a newbie mistake in SQL that would not even pass an AI review. They did not verify the change in a test environment. And I guess the logs are so full of errors it is hard to pinpoint which matters. Yikes.
ClickHouse db was mentioned. Does this incident raise any critiques about it?
I would have been a bit cheeky and opened with 'It wasn't DNS.'
Unbelievable. I guess it's time to grep for every .unwrap in our code.
theres a clippy lint for that
> The first automated test detected the issue at 11:31 and manual investigation started at 11:32. The incident call was created at 11:35.
I'm impressed they were able to corral people this quickly.
Reason for the failure: switched to Chad IDE to ship new features.
Is dual sourcing CDNs feasible these days? Seems like having the capability to swap between CDN providers is good both from a negotiating perspective and a resiliency one.
It's unbelievable that the end of this postmortem is an advertisement for Cloudflare.
The last thing we need here is for more of the internet to sign up for Cloudflare.
It's in Cloudflare's interest and they wrote the blog post.
Just a moment to reflect on how much freaking leverage computers give us today - a single permission change took down half the internet. Truly crazy times.
If you deploy a change to your system, and things start to go wrong that same day, the prime suspect (no matter how unlikely it might seem) should be the change you made.
My first question when faced with an unknown error is "What was the last change and when was it promoted?"
This is an area where they are allowed to think yet another record setting DDoS attack first and bad config second.
Why have a limit on the file size if the thing that happens when you hit the limit is the entire network goes down? Surely not having a limit can't be worse?
Excellent write up. Cybersecurity professionals read the story and learn. It’s textbook lesson in post-mortem incident analysis - a mvp for what is expected from us all in a similar situation.
Reputationally this is extremely embarrassing for Cloudflare, but imo they seem to get their feet back on the ground. I was surprised to see not just one, but two apologies to the internet. This just cements how professional and dedicated the Cloudflare team is to ensure stable resilient internet and how embarrassed they must have been.
A reputational hit for sure, but outcome is lessons learned and hopefully stronger resilience.
This is a small issue in the writeup (everything else made sense), but -- why does doubling 60 features exceed the 200 limit? Missing something.
Time to rewrite with golang, explicit error handling ;-)
cloudflare:
> Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare.
also cloudflare:
> The Cloudflare Dashboard was also impacted due to both Workers KV being used internally and Cloudflare Turnstile being deployed as part of our login flow.
I believe you're mistakenly equating Cloudflare's status page with the Cloudflare Dashboard? They're not the same thing.
Cloudflare's status page: https://www.cloudflarestatus.com/
Cloudflare Dashboard: https://dash.cloudflare.com/
thanks for clarifying! i guess then they never explained why the status page went down, even though it's supposed to be running on independent infrastructure.
Yes, that was missing (along with the London WARP thing). Other comments mentioned that their status page is an Atlassian Statuspage solution, hosted on AWS CloudFront.
Unclear to me if it's an Atlassian-managed deployment they have, or if it's self-managed, I'm not familiar with Statuspage and their website isn't helping. Though if it's managed, I'm not sure how they can know for sure there's no interdependence. (Though I guess we could technically keep that rabbit hole going indefinitely.)
Sure this has been said but the issue here is not code is the ability to canary and rollback quickly from any arbitrary (config) change.
Wow. What a post mortem. Rather than Monday morning quarterbacking how many ways this could have been prevented, I'd love to hear people sound-off on things that unexpectedly broke. I, for one, did not realize logging in to porkbun to edit DNS settings would become impossible with a cloudflare meltdown
That's unfortunate. I'll need to investigate whether Porkbun plans on decoupling its auth from being reliant on CloudFlare, otherwise I will need to migrate a few domains off of that registrar.
Perils of a centralized chokepoint. Substantial amounts of the internet shouldn't go down because Cloudflare had a hiccup.
this is where change management really shines because in a change management environment this would have been prevented by a backout procedure and it would never have been rolled out to production before going into QA, with peer review happening before that... I don't know if they lack change management but it's definitely something to think about
i think that is data rather than code which is where it falls short, in a way you need stringent code and more safeguarded code; it's like if everyone sends you 64k posts as that's all your proxy layer lets in, someone checked sending 128kb and it gave an error before reaching your app - and then someone sends 128kb and the proxy layer has changed - and your app crashes as it was more than 64kb and your app had an assert against that. to actually track issues with erraneous data that overflows well and stuff isn't so much code test but more like fuzz testing, brute force testing etc. which i think people should do; but that's more like we need strong test networks, and also those test networks may need to be more internet like to reflect real issues too, so the whole testing infrastructure in itself becomes difficult to get right - like they have their own tunneling system etc, they could segregate some of their servers and make a test system with better error diagnosis etc potentially. but to my mind, if they had better error propogation back that really identified what was happening and where then that would be a lot better in general. sure, start doing that on a test network. this is something i've beeen tihnking about in general - i made a simple rpc system for being able to send real time rust tracing logs (it allows to just use the normal tracing framework and use a thin rpc layer) back from multiple end servers but that's mostly for granular debugging. i've never quite understood why systems like systemd-journald aren't more network centric when they're going to be big and complex kitchensink approaches - apparently there's dbus support, but to my mind something inbetween debugging level of code and warning/info. like even if it's doing things like 1/20 of log info it's too much volume if things like large files getting close to limits is increasing etc and we can see this as things run, and can see if it's localised or common etc it'd help have more resilient systems. something may already exist in this line but i didn't come across anything in a reasonably passive way - i mean there's debugging tools like dtrace etc that have been around for ages.
It is a bit tough to do all that in five minutes.
Any feature failing should still allow the traffic to continue. This should be the first bullet in the future actions list.
It is staggering to see that even large companies like CF have zero monitoring, so they would know what happened in t=0.
Timely post mortem. Sucks to have this happened, but at least they are quite transparent and detailed in the writeup.
Interesting. Although principle of least privilege is great, it should not be applied as a feature to filter data.
But Rust was supposed to cure cancer and solve world hunger. Is this the end of the hello world but in Rust saga?
Catastrophique failure for failing to read a file bigger that expected? Wow. This is really embarrassing.
Ironically just now I got a Cloudflare "Error code 524" page because blog.cloudflare.com was down
No publicity is bad publicity.
Best post mortem I've read in a while, this thing will be studied for years.
A bit ironic that their internal FL2 tool is supposed to make Cloudflare "faster and more secure" but brought a lot of things down. And yeah, as other have already pointed out, that's a very unsafe use of Rust, should've never made it to production.
“…and the fluctuation stabilized in the failing state.”
Sounds like the ops team had one hell of a day.
While it's certainly worthwhile to discuss the Technical and Procedural elements that contributed to this Service Outage, the far more important (and mutually-exclusive aspect) to discuss should be:
Why have we built / permitted the building of / Subscribed to such a Failure-intolerant "Network"?
Who's "we"? This is not a trick question, what specific people do you think acted wrongly here? I don't use Cloudflare personally. I don't run any of the sites that do use it. The people who did make the decision to put thier websites behind Cloudflare could stop, and maybe some will, but presumably they're paying for it because they think, perhaps accurately, that they get value out of it. Should some power compel them not to use Cloudflare?
> I worry this is the big botnet flexing.
Even worse - the small botnet that controls everything.
This post was written by chatgpt??
https://blog.cloudflare.com/18-november-2025-outage/#:~:text...
Here's a random post from their blog by the same author from 2017 with an em dash:
> As we wrote before, we believe Blackbird Tech's dangerous new model of patent trolling — where they buy patents and then act their own attorneys in cases — may be a violation of the rules of professional ethics.
https://blog.cloudflare.com/patent-troll-battle-update-doubl...
ChatGPT didn't invent the em dash, some people were always using it. But yeah, it's often one of the signs of AI.
Too many think AI instead of horses.
It was ASSUREDLY a cyber attack.
Go to jeffblearning on LinkedIn. I took it down with 253 copies of a text file delivered through a vulnerability in Novo’s systems.
I’ve documented all of it.
It’s not done yet…
our site realtyblocks.com went down but we redirected the traffic by bypassing the cloudflare DNS routes with few clicks. Thank you for resolving the issue Cloudflare.
Thank you for being honest, all must learn from the mistakes.
My website was down too, because a tree fell on my power line
our site realtyblocks.com went down but we redirected the traffic by bypassing DNS entries with few clicks. Thank you for resolving the issue Cloudflare.
I am just going by the outage post-mortem report. I could not read the article after I read the first few lines - "feature file" expansion/limits. I am stuck at consuming the design idea here, where you allow multiple inserts for one feature [assuming you have some uniqueness constraint].
Even a simple key-value map per feature should have allowed for insertions as simple as a put/replace of the value and not appending to the file. That was not the case here, where Cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here that Cloudflare should rethink. If someone can educate me on the design, I can continue reading the next few lines.
our site realtyblocks.com went down but we redirected the traffic with few clicks. Thank you for resolving the issue Cloudflare.
I am just going by the outage post mortem report. I could not read the article after I read the first few lines - "feature file". I am stuck at consuming the design idea here where you allow multiple inserts for one feature [assuming you have some uniqueness constraint]. Even a simple key-value map per feature should have made the insertions as just a put/replace the value. I think that was not the case here, where cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here cloudflare should rethink. If someone can educate me on the design, I can continue reading the next lines.
tl;dr A permissions change in a ClickHouse database caused a query to return duplicate rows for a “feature file” used by Cloudflares Bot Management system, which doubled the file size. That oversized file was propagated to their core proxy machines, triggered an unhandled error in the proxy’s bot-module (it exceeded its pre-allocated limit), and as a result the network started returning 5xx errors. The issue wasn’t a cyber-attack — it was a configuration/automation failure.
Configuration can be validated. https://cuelang.org
Did your incident response team look at the last few changes that were executed? If they had , they could have just rolleback the change. or just looking at the changes executed, in the vicinity of the start of the outage could have pointed to the problem.
Didn't the services that were crashing due to OOM raise any alerts?
This is shitty at so many levels.
Wow, crazy disproportional drop in the stock price… good buying opportunity for $NET.
Agree.
Cloudflare is very cheap at these prices.
Matt, Looking forward in regaining Elon's and his team trust to use CF again.
I wish Elon would regain my trust!
Git blame disabled on the line which crashed it - Cowards!
unwraps are so very easy to use and they have bit me so many times because you can nearly never run into a problem and suddenly crashes from an unwrap that almost always was fine
How was AI involved?
I'm honestly curious what culturally is going on inside Cloudflare given they've had a few outages this year.
Hack Free fire 8000 5487 3565 644664 464664644
464646449
F
deny (clippy:: unwrap_used)
A fucking unhandled exception brought down a majority of the internet? Why do we continue to let these clowns run a large portion of the internet?
Big tech is a fucking joke.
20% sure seems to grow with every commenter piling on. It’s like an Internet game of telephone.
Glad it's fixed, keep going!
hope no one was fired
Cloudflare rewrites Rust services to <next-cool-language> /joke
...
(I'd pick Haskell, cause I'm having fun with it recently :P)
> Given Cloudflare's importance in the Internet ecosystem any outage of any of our systems is unacceptable.
Excuse me, what you've just said? Who decided on “Cloudflare's importance in the Internet ecosystem”? Some see it differently, you know, there's no need for that self-assured arrogance of an inseminating alpha male.
Was it DNS this time?
28M 500 errors/sec for several hours from a single provider. Must be a new record.
No other time in history has one single company been responsible for so much commerce and traffic. I wonder what some outage analogs to the pre-internet ages would be.
Something like a major telco going out, for example the AT&T 1990 outage of long distance calling:
> The standard procedures the managers tried first failed to bring the network back up to speed and for nine hours, while engineers raced to stabilize the network, almost 50% of the calls placed through AT&T failed to go through.
> Until 11:30pm, when network loads were low enough to allow the system to stabilize, AT&T alone lost more than $60 million in unconnected calls.
> Still unknown is the amount of business lost by airline reservations systems, hotels, rental car agencies and other businesses that relied on the telephone network.
https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collap...
Yes, all(most) eggs should not be in one basket. Perfect opportunity to setup a service that checks cloudflare then switches a site's DNS to akami as a backup.
> I wonder what some outage analogs to the pre-internet ages would be.
Lots of things have the sky in common. Maybe comet-induced ice ages...
> No other time in history has one single company been responsible for so much commerce and traffic.
AWS very likely has Cloudflare beat in commerce responsibility. Amazon is equal to ~2.3% of US GDP by itself.
Absolute volume maybe[1], as relative % of global digital communication traffic, the era of early telegraph probably has it beat.
In the pre digital era, East India Company dwarfs every other company in any metric like commerce controlled, global shipping, communication traffic, private army size, %GDP , % of workforce employed by considerable margins.
The default was large consolidated organization throughout history, like say Bell Labs, or Standard Oil before that and so on, only for a brief periods we have enjoyed benefits of true capitalism.
[1] Although I suspect either AWS or MS/Azure recent down-times in the last couple of years are likely higher
So an unhandled error condition after an configuration update similar to Crowdstrike - if they had just used a programming language where this can't happen due to the superior type system such as Rust. Oh wait.
Hack
Free fire
But but but muh rust makes EVERYTHING safer!!!!
My dude, everything is a footgun if you hold it wrong enough
Another long day...
[dead]
[dead]
[flagged]
...ai
me af when there's a postmortem rubbing hands, impish smile on my face
I give them a pass on lots of things, but this is inexcusable
I thought it was an internal mess-up. I thought an employee screwed a file up. Old methods are sometimes better than new. AI fails us again!
Cloudflare’s write-up is clear and to the point. A small change spread wider than expected, and they explained where the process failed. It’s a good reminder that reliability depends on strong workflows as much as infrastructure.
Seems like a substantial fraction of the web was brought down because of a coding error that should have been caught in CI by a linter.
These folks weren't operating for charity. They were highly paid so-called professionals.
Who will be held accountable for this?
Did some $300k chief of IT blame it all on some overworked secretary clicking a link in an email they should have run through a filter? Because that’s the MO.
Hopefully that's the case.
This post was written by chatgpt????
https://blog.cloudflare.com/18-november-2025-outage/#:~:text...
Not all em dashes are ChatGPT. Good writers use it wherever required.
Great write up.
This is the first significant outage that has involved Rust code, and as you can see the .unwrap is known to carry the risk of a panic and should never be used on production code.
I think you should give me a credit for all the income I lost due to this outage. Who authorized a change to the core infrastructure during the period of the year when your customers make the most income? Seriously, this is a management failure at the highest levels of decision-making. We don't make any changes to our server infrastructure/stack during the busiest time of the year, and neither should you. If there were an alternative to Cloudflare, I'd leave your service and move my systems elsewhere.
I think you should get exactly what the contract you signed said you'd get. Outages happen in all infrasturture. Planned and unplanned ones both. The SLA and SLO are literal acknowledgements of the fact that and part of the contract for that reason.
Its fair to be upset at their decision making - use that to renegotiate your contract.
Here is a bit more context in addition to the quote above. A ClickHouse permissions change made a metadata query start returning duplicate column metadata from an extra schema, which more than doubled the size and feature count of a Bot Management configuration file. When this oversized feature file was deployed to edge proxies, it exceeded a 200-feature limit in the bot module, causing that module to panic and the core proxy to return 5xx errors globally
So, to recap:
In terms of preventing this from a software engineering perspective, they made assumptions about how their database queries work (and didn't validate the results), and they ignored their own application limits and didn't program in either a test for whether an input would hit a limit, or some kind of alarm to notify the engineers of the source of the problem.From an operations perspective, it would appear they didn't test this on a non-production system mimicing production; they then didn't have a progressive deployment; and they didn't have a circuit breaker to stop the deployment or roll-back when a newly deployed app started crashing.
People jump to say things like "where's the rollback" and, like, probably yeah, but keep in mind that speculative rollback features (that is: rollbacks built before you've experienced the real error modes of the system) are themselves sources of sometimes-metastable distributed system failures. None of this is easy.
How about where's the most basic test to check if your config file will actually run at all in your application? It was a hard-coded memory limit; a git-hook test suite run a MacBook would have caught this. But nooo, let's not run the app for 0.01 seconds with this config before sending it out to determine the fate of the internet?
This is literally the CrowdStrike bug, in a CDN. This is the most basic, elementary, day 0 test you could possibly invent. Forget the other things they fucked up. Their app just crashes with a config file, and nobody evaluates it?! Not every bug is preventable, but an egregious lack of testing is preventable.
This is what a software building code (like the electrical code's UL listings that prevent your house from burning down from untested electrical components) is intended to prevent. No critical infrastructure should be legal without testing, period.
Looks like you have the perfect window to disrupt them with a superior product.
just before this outage i was exploring bunnycdn as the idea of cloudflare taking over dns still irks me slightly. there are competitors. but there's a certain amount of scale that cloudflare offers which i think can help performance in general. that said in the past i found cloudflare performance terrible when i was doings lots of testing. they are predominantly a pull based system not a push, so if content isn't current the cache miss performance can be kind of blah. i think their general backhaul paths have improved, but at least from new zealand they used to seem to do worse than hitting a los angeles proxy that then hits origin. (although google was in a similar position before, where both 8.8.8.8 and www.google.co.nz/.com were both faster via los angeles than via normal paths - i think google were doing asia parent, like if testing 8.8.8.8 misses it was super far away). i think now that we have http/3 etc though that performance is a bit simpler to achieve, and that ddos, bot protection is kind of the differentiator, and i think that cloudflare's bot protection may work reasonably well in general?