This is a super interesting analysis, I wonder what scale do you need to reach the point where the ease of adoption of AWS/GCloud is not worth it anymore. The article compares compute in each scenario, which makes sense, but I would take into account the speed to live as a secondary metric. I'm pretty sure cloud is faster, at least initially
Really appreciate how this isn't just another "cloud vs bare metal" holy war.The "own the core, rent the periphery" approach is exactly the kind of nuanced thinking we need more of in these discussions.
That said,important caveat for anyone reading this: Veridion is processing petabytes of data with predictable workloads. If you're still figuring out product-market fit or your usage patterns are all over the place, premature infrastructure optimization will absolutely kill you.The cloud's "boring reliability" has real value when you need to focus on building the actual product.
The 30x multiplier is real, but it only kicks in at serious scale with stable, known workloads. Most startups aren't there yet, and that's fine.
This is maybe the first “ditch the cloud” post that isn’t totally ideological. Makes sense for their scale + workload. I wish more companies ran the numbers instead of just accepting AWS bills as fate.
the irony is cloud providers would love to sell a customer like this giant blocks of reserved compute. wonder if they actually negotiated or just said “lol too expensive” and bailed.
not sure i buy the “$2M/mo on cloud” number. did they actually run it or just plug public on-demand pricing into a spreadsheet? nobody pays sticker price at that scale.
That's a fair point, the scale discounts certainly matter. That said:
We did in fact run our prototype on AWS, and early on they flagged our account because our VMs were under heavy CPU load (we were told we'd need to cut below ~50 % or risk suspension).
When you switch to dedicated hardware, you're no longer paying for "noisy neighbours" or hypervisor over-provisioning, so the effective per-unit cost often cleaner fully justifies the shift.
Well, we do have some servers that we have in the office(the gpus servers, those have been a pain in the... mostly because intel cpus suck). But other then these the entire hardware part is managed by hetzner (thank you hetzner for being awesome)
the part about “predictable heavy workloads” should be bolded in neon. 90% of startups THINK they have predictable workloads. they do not. these guys actually do.
i would argue 90% of startups think they are going to need autoscaling, invest in that, and then never reach the scale up point, while having spent the investment money already.
besides the point of the article, tho. even if at the end of the day the numbers don't perfectly add up, i don't believe they are WAY OFF. we are paying for cables and HDDs man. "ancient" tech.
This is a super interesting analysis, I wonder what scale do you need to reach the point where the ease of adoption of AWS/GCloud is not worth it anymore. The article compares compute in each scenario, which makes sense, but I would take into account the speed to live as a secondary metric. I'm pretty sure cloud is faster, at least initially
Really appreciate how this isn't just another "cloud vs bare metal" holy war.The "own the core, rent the periphery" approach is exactly the kind of nuanced thinking we need more of in these discussions. That said,important caveat for anyone reading this: Veridion is processing petabytes of data with predictable workloads. If you're still figuring out product-market fit or your usage patterns are all over the place, premature infrastructure optimization will absolutely kill you.The cloud's "boring reliability" has real value when you need to focus on building the actual product. The 30x multiplier is real, but it only kicks in at serious scale with stable, known workloads. Most startups aren't there yet, and that's fine.
so… they built a 4PB cluster, 14k cores, 50 GPUs, and it costs them less than my prod Kubernetes cluster on GCP. feels bad.
dont worry, their real cost was their sanity
[dead]
This is maybe the first “ditch the cloud” post that isn’t totally ideological. Makes sense for their scale + workload. I wish more companies ran the numbers instead of just accepting AWS bills as fate.
not saying they’re wrong but $2m → $60k is like… a 97% cost drop. that is either cloud pricing insanity or some very optimistic bare-metal accounting.
If they are going from AWS, that would be the cloud pricing insanity.
good story, but "works for us, so it should work for everyone". most startups can't touch this.
and what about the team of 1000 engineers you need to do the work of AWS support?
the irony is cloud providers would love to sell a customer like this giant blocks of reserved compute. wonder if they actually negotiated or just said “lol too expensive” and bailed.
not sure i buy the “$2M/mo on cloud” number. did they actually run it or just plug public on-demand pricing into a spreadsheet? nobody pays sticker price at that scale.
That's a fair point, the scale discounts certainly matter. That said: We did in fact run our prototype on AWS, and early on they flagged our account because our VMs were under heavy CPU load (we were told we'd need to cut below ~50 % or risk suspension). When you switch to dedicated hardware, you're no longer paying for "noisy neighbours" or hypervisor over-provisioning, so the effective per-unit cost often cleaner fully justifies the shift.
can’t tell if they’re geniuses or just suffering a massive case of “we built a data center so now everything looks like a hardware nail."
idk man, running your own hardware sounds awesome until you have to replace a dead RAID controller at 3am. article doesn’t really mention that part.
Well, we do have some servers that we have in the office(the gpus servers, those have been a pain in the... mostly because intel cpus suck). But other then these the entire hardware part is managed by hetzner (thank you hetzner for being awesome)
ngl the cloud evangelists aren’t gonna like this one. this is basically “we did the math and cloud wasn’t worth it
the part about “predictable heavy workloads” should be bolded in neon. 90% of startups THINK they have predictable workloads. they do not. these guys actually do.
i would argue 90% of startups think they are going to need autoscaling, invest in that, and then never reach the scale up point, while having spent the investment money already.
besides the point of the article, tho. even if at the end of the day the numbers don't perfectly add up, i don't believe they are WAY OFF. we are paying for cables and HDDs man. "ancient" tech.
[dead]