Congrats on finding it in within three days. I recently discovered something similar on our infra that had been going on for months (just an INFO log in a tight loop).
This is the type of thing that major cloud providers should absolutely have sensible defaults and alerts on, but don't because it's how they make all their money.
It's one thing when it's company cash but I would never ever use a big cloud provider for a personal project (with my credit card). There are way too many ways to accidentally run up an infinite bill. DigitalOcean has plenty of services and more predictable costs, or your local data center will be happy to make a deal on some bare metal servers if you need more horsepower.
I hope you're able to get it reversed - try finding your GCP rep. Either way let this be a valuable lesson on using cloud.
Implementing rate limiting in a scalable manner is harder than just making logging faster, and it makes the cloud provider money whereas rate limiting causes them to make less money. The incentive makes it pretty clear what side of the tradeoff cloud providers are going to come down on.
Congrats on finding it in within three days. I recently discovered something similar on our infra that had been going on for months (just an INFO log in a tight loop). This is the type of thing that major cloud providers should absolutely have sensible defaults and alerts on, but don't because it's how they make all their money.
It's one thing when it's company cash but I would never ever use a big cloud provider for a personal project (with my credit card). There are way too many ways to accidentally run up an infinite bill. DigitalOcean has plenty of services and more predictable costs, or your local data center will be happy to make a deal on some bare metal servers if you need more horsepower.
I hope you're able to get it reversed - try finding your GCP rep. Either way let this be a valuable lesson on using cloud.
Yes, it's my 4th email to Billing Support and getting "No" as answers. Moving to Azure..
That's my point though. Azure or AWS are just as bad and that won't solve anything. What are you running that requires a major cloud provider?
Implementing rate limiting in a scalable manner is harder than just making logging faster, and it makes the cloud provider money whereas rate limiting causes them to make less money. The incentive makes it pretty clear what side of the tradeoff cloud providers are going to come down on.