The press release is rather light on details. They found that it is every 60seconds (duh!) and generators (or such). So what exactly is it? (Make and model?)
Seems like they did not find the exact manufacturer or the exact devices and were not able to change firmware then?
"following our communication with the manufacturer, a software update was issued that removed the strict periodicity of the current peaks in one battery block of the investigated UPS system"
You see the same thing with e.g. cron jobs, where everyone schedules on xx:00 or xx:30. No communication needed, it's just natural. More senior people recognize this and add randomness to their start times or skew a bit.
Doesn't seem that complicated from the article though. They say that they immediately knew it was clock based due to the strict interval. With REA etc. it might become easier to proof, but I guess it's a measurement problem that only appears in large networks?
Can someone with more understanding explain why this was apparently so difficult to find out?
If it were not a substantial research effort spanning years, it would not get funded. This motivates academic researchers to ensure that every topic is explored thoroughly and at great length.
i'm confused, why would a system simply checking its own power state be taking so much electricity as to show up on grid measurements? unless they mean that the systems are choosing to start charging the batteries, but then it's a continuous load and not a spike at the start of the minute, right?
It can be that part of battery is disconnected, to estimate cells voltage. You can not measure voltage while another source is chsrging battery. Or capacitors are charging.
I think you can. RC toys do that all the time. Maybe the measurement is less precise, but you usually don't need super precise voltage, especially if you know your capacitance and measure current.
Current doesn't tell you if the battery is leaking charge, or otherwise failing to store it though. Vastly increased losses could be a sign of failure, for example. In any case it's important to be sure.
Oh interesting that got reformed, too! Strange choice to split it in two instead of three words (the latter which would have mimicked English: “all too”).
Well, then it seems we don’t have any single German word for all too regular anymore :(
Not a scholar of languages, but maybe something like "mit unheimlicher Regelmäßigkeit" (with un-home-ly regular-like-ness, read "home" like in "family" -> "familiar").
The water-hammer effect, or in this case electron-hammer effect, is why you don't reboot every Windows server in the data center at exactly 2am, to stagger the load shift over time. Surprised that the power guys didn't realize that switching a presumably multi-hundred-MW load in and out at the top of each minute would cause a problem.
Sounds like they've been set up as cron tasks. I've always been unhappy with the 1 minute granularity of cron tasks. Only thing I could come up with was putting a sleep(random()) type of line to avoid a thundering herd type of issue.
Yes, using prime numbers are a good way to reduce the risk of multiple timers coinciding, for example a 10s, 30s, 60s period on a freshly restarted node / process.
That's interesting as it is also visible with a crontab -l so that new scripts can avoid using the same second. However, if you have a farm of machines with the same hardcoded value, they will all still stampede at the same time just not at the top of the minute. By allowing a random value, their start times are slightly offset with in the minute. Is the precise 60s consistent interval really that critical or just that each machine checks once per minute-ish?
The Google internal equivalent to cron has a ? placeholder in the scheduling spec syntax letting the scheduler deterministically pick an arbitrary valid value within the field range. So ? 2 ... will pick any minute between 02:00 and 02:59 (and I suspect any second within that minute). The value is derived from a hash of the job definition, so overall this spreads the cron load within the hour.
Obviously cron jobs are not all equal in terms of load and duration, and hours are not all equal in terms of usage, but that still helps prevent trivial spikes similar to the one mentioned in the article (disclaimer: I'm not aware of a recent counterfactual experiment proving this, but that seems a fair assumption).
I suppose one would end up with checks happening on a random second instead of a random minute - reducing the size of the spike by a factor of 60, but still spiky.
It seems more like one or multiple suppliers of battery systems not taking into account that the decentralized deployments of their system could have that effect. If it's not regulated and an open market, this was bound to happen sooner or later.
In computer/systems/distributed engineering this is called request smoothing or traffic shaping, and it's implemented by adding jitter, which is a random time offset/pad added to the interval.
The funny thing is, modern electrical codes already account for this phenomenon. In order to implement Net Metering (selling your power to the grid), modern standards implement two kinds of jitter:
- Sandia Frequency Shift: a minute shift in the power frequency generated by inverters. If the grid is connected, this is effectively eliminated; if the inverter is disconnected from the grid, it detects the frequency has shifted and disconnects the inverter from the grid.
- Slip-Mode Frequency Shift: basically the same thing, but phase shift (which I don't understand).
Both of these are used for anti-islanding (don't create an island of power). When the inverter detects it's isolated and stops outputting current, it then has to check every 5 minutes, with jitter, for whether the grid is restored or not. It has to add random jitter, because otherwise all the mini battery/solar installs will come online at the same time and swamp the grid, causing another outage.
power guys got surprised as rest of the grid load is human randomised and even huge factories will come on one chonk at a time, vs miilisecond co ordinated load switching at scale, with batteries as the load of the initial surge into even fully charged batteries can be almost infinite, and the whole calculation around grid scale turbines is that they run at very precise rpm's for stability and longevity, but worse, and a big fail, as battery storage is supposed to help with that, not be the problem.
bet it gets fixed realy fast as those grid guys definitly get listened to.
> grid scale turbines is that they run at very precise rpm's for stability and longevity
The turbines/generators set the grid frequency itself.
Having the batteries disconnect/connect would cause the generators to slow down and then speed up (or vice versa) which makes the rest of the grid frequency fluctuate as well.
It may also trigger the FCAS part of the grid that is paid to keep the frequency within the required range which is +/- 10mHz-200mHz.
> as the load of the initial surge into even fully charged batteries can be almost infinite
There was an post a couple of days ago about a telco debrief of an interesting outage that happened. Several things went wrong like the audible bell was manually silence from being heard as a step in routine working but was forgotten to put back into normal mode once work was complete. The actual loss in power triggered the automatic switch to the batteries, but something else seemingly benign happened at the same time so that when power was restored, the batteries had been depleted to the point the charging process could not start because of something similar to this. I wish I could remember the write up, as it was really well written about the cascades of failures that occurred.
The grid frequency is set directly by 3 phase generators. So if the grid frequency drops, there is indeed a very strong torque trying to decelerate the turbines.
To some extent, this is a good thing. Because it means the spinning mass of all turbines in the grid actually provide a buffer for sub-second mismatches in demand and supply. But it does mean that big demand spikes (or drops) put some wear on the systems. They are made for this, but if it gets very periodic it might be more than designed for.
Note that this means that all turbines in a power network are perfectly synchronized. And if they are not, they are subject to massive torques. This is why a cold start is so hard. You need to bring on generators from one central point to keep everything in sync. And as you bring on new generation capacity you also need to bring on new consumption, that electricity has to go somewhere.
The reason for this synchronization is because the generators are spinning a magnet past 3 coils, and those coils are directly wired into the power network. If your magnets are out of phase then effectively the entire power network will use those coils to generate a field to get that magnet back in sync. All of those coils on all those generators really are all hard-wired together, it still kind of blows my mind.
When the battery storage systems all temporarily pause providing grid inertia due to the state of charge check they’re performing, grid synchronized spinning mass has to compensate, which slows it down (transferring the kinetic energy to electrical energy), and then is sped back up as battery storage resumes inertia support.
Broadly speaking, it’s a collective momentum problem. If part of the system inertia sags, another part needs to step in to compensate for the inertia loss, otherwise you have brownouts and eventually, a blackout.
The press release is rather light on details. They found that it is every 60seconds (duh!) and generators (or such). So what exactly is it? (Make and model?)
Seems like they did not find the exact manufacturer or the exact devices and were not able to change firmware then?
The paper has a little more details:
https://www.nature.com/articles/s41560-026-02144-0Given the issue is so widespread (international, affecting pretty much every grid), it's likely that multiple manufacturers use similar approaches.
You see the same thing with e.g. cron jobs, where everyone schedules on xx:00 or xx:30. No communication needed, it's just natural. More senior people recognize this and add randomness to their start times or skew a bit.
You ought to use RandomizedDelaySec=xxx
The paper has more details: https://www.nature.com/articles/s41560-026-02144-0
https://news.ycombinator.com/item?id=50314617
It had both ended up in the second chance pool.
> It really was years of detective work
Doesn't seem that complicated from the article though. They say that they immediately knew it was clock based due to the strict interval. With REA etc. it might become easier to proof, but I guess it's a measurement problem that only appears in large networks?
Can someone with more understanding explain why this was apparently so difficult to find out?
There are many millions of devices connected that draw power, and increasingly so, supply power.
The effect is apparently small enough that they had to use Fourier analysis over many minutes to see it in the noise.
If it were not a substantial research effort spanning years, it would not get funded. This motivates academic researchers to ensure that every topic is explored thoroughly and at great length.
i'm confused, why would a system simply checking its own power state be taking so much electricity as to show up on grid measurements? unless they mean that the systems are choosing to start charging the batteries, but then it's a continuous load and not a spike at the start of the minute, right?
It can be that part of battery is disconnected, to estimate cells voltage. You can not measure voltage while another source is chsrging battery. Or capacitors are charging.
I think you can. RC toys do that all the time. Maybe the measurement is less precise, but you usually don't need super precise voltage, especially if you know your capacitance and measure current.
Current doesn't tell you if the battery is leaking charge, or otherwise failing to store it though. Vastly increased losses could be a sign of failure, for example. In any case it's important to be sure.
temperature is also a problem
>unusually regular
Oh I’m sure there is a German phrase for this.
I would go with “allzuregelmäßig”.
Similar to “allzuschön um wahr zu sein”, “allzugroß” etc.
However this is not a single word, but would be split as "allzu regelmäßig" cf. https://www.korrekturen.de/wortliste/allzuoft.shtml
Oh interesting that got reformed, too! Strange choice to split it in two instead of three words (the latter which would have mimicked English: “all too”).
Well, then it seems we don’t have any single German word for all too regular anymore :(
Not a scholar of languages, but maybe something like "mit unheimlicher Regelmäßigkeit" (with un-home-ly regular-like-ness, read "home" like in "family" -> "familiar").
Thinking about it tactically, I’ll go with taktisch. :) Takt-isch.
rather tak-tisch if we are talking hyphenation
Surely you mean Tick-Tacktisch?
verdächtig !
uhrwerksmäßig
That doesn’t capture the “unusually” in “unusually regular”
Ungewöhnlichregelmäßig or Hyperregelmäßig works for me, but then there is Regelmäßigkeitsmonsterhaft.
Still, I think we can push the limits to Bürokratieapokalyptischhyperregelmäßigkeitsgötterungeheuerhaft and feel pretty comfortable with it.
The water-hammer effect, or in this case electron-hammer effect, is why you don't reboot every Windows server in the data center at exactly 2am, to stagger the load shift over time. Surprised that the power guys didn't realize that switching a presumably multi-hundred-MW load in and out at the top of each minute would cause a problem.
Sounds like they've been set up as cron tasks. I've always been unhappy with the 1 minute granularity of cron tasks. Only thing I could come up with was putting a sleep(random()) type of line to avoid a thundering herd type of issue.
Yes it’s a real effect. I’ve just hardcoded offsets on various jobs:
etc.That way I still maintain a per-job 60-second cadence consistently (which random would not), but don’t do it right on the top of the minute.
Yes, using prime numbers are a good way to reduce the risk of multiple timers coinciding, for example a 10s, 30s, 60s period on a freshly restarted node / process.
That's interesting as it is also visible with a crontab -l so that new scripts can avoid using the same second. However, if you have a farm of machines with the same hardcoded value, they will all still stampede at the same time just not at the top of the minute. By allowing a random value, their start times are slightly offset with in the minute. Is the precise 60s consistent interval really that critical or just that each machine checks once per minute-ish?
Systemd timers do this natively btw, no need to hardcode And imho are also friendlier.
But systemd bad :[
People slowly are getting used to it. (Less so with journalctl)
BSD cron has supported jitter for decades. This is strictly a linux problem.
The Google internal equivalent to cron has a ? placeholder in the scheduling spec syntax letting the scheduler deterministically pick an arbitrary valid value within the field range. So ? 2 ... will pick any minute between 02:00 and 02:59 (and I suspect any second within that minute). The value is derived from a hash of the job definition, so overall this spreads the cron load within the hour.
Obviously cron jobs are not all equal in terms of load and duration, and hours are not all equal in terms of usage, but that still helps prevent trivial spikes similar to the one mentioned in the article (disclaimer: I'm not aware of a recent counterfactual experiment proving this, but that seems a fair assumption).
I suppose one would end up with checks happening on a random second instead of a random minute - reducing the size of the spike by a factor of 60, but still spiky.
It seems more like one or multiple suppliers of battery systems not taking into account that the decentralized deployments of their system could have that effect. If it's not regulated and an open market, this was bound to happen sooner or later.
In computer/systems/distributed engineering this is called request smoothing or traffic shaping, and it's implemented by adding jitter, which is a random time offset/pad added to the interval.
The funny thing is, modern electrical codes already account for this phenomenon. In order to implement Net Metering (selling your power to the grid), modern standards implement two kinds of jitter:
- Sandia Frequency Shift: a minute shift in the power frequency generated by inverters. If the grid is connected, this is effectively eliminated; if the inverter is disconnected from the grid, it detects the frequency has shifted and disconnects the inverter from the grid.
- Slip-Mode Frequency Shift: basically the same thing, but phase shift (which I don't understand).
Both of these are used for anti-islanding (don't create an island of power). When the inverter detects it's isolated and stops outputting current, it then has to check every 5 minutes, with jitter, for whether the grid is restored or not. It has to add random jitter, because otherwise all the mini battery/solar installs will come online at the same time and swamp the grid, causing another outage.
power guys got surprised as rest of the grid load is human randomised and even huge factories will come on one chonk at a time, vs miilisecond co ordinated load switching at scale, with batteries as the load of the initial surge into even fully charged batteries can be almost infinite, and the whole calculation around grid scale turbines is that they run at very precise rpm's for stability and longevity, but worse, and a big fail, as battery storage is supposed to help with that, not be the problem. bet it gets fixed realy fast as those grid guys definitly get listened to.
> grid scale turbines is that they run at very precise rpm's for stability and longevity
The turbines/generators set the grid frequency itself.
Having the batteries disconnect/connect would cause the generators to slow down and then speed up (or vice versa) which makes the rest of the grid frequency fluctuate as well.
It may also trigger the FCAS part of the grid that is paid to keep the frequency within the required range which is +/- 10mHz-200mHz.
https://mainsfrequency.com/frequ_info_en.php
> as the load of the initial surge into even fully charged batteries can be almost infinite
There was an post a couple of days ago about a telco debrief of an interesting outage that happened. Several things went wrong like the audible bell was manually silence from being heard as a step in routine working but was forgotten to put back into normal mode once work was complete. The actual loss in power triggered the automatic switch to the batteries, but something else seemingly benign happened at the same time so that when power was restored, the batteries had been depleted to the point the charging process could not start because of something similar to this. I wish I could remember the write up, as it was really well written about the cascades of failures that occurred.
I believe this is what you're referring to.
https://news.ycombinator.com/item?id=50153229&hl=en-US
Yes, thanks for allowing me to offload my RAM to yours!
You'd think they'd randomise offsets to avoid a thundering herd effect, like anyone scheduling multiple processes does?
To anyone herding a herd this is obvious. To people crafting animals less so.
Apparently not everyone does ;)
Thanks for sharing, very cool!
> turbines slow down momentarily before accelerating back to their original rotational speed. This increases wear and tear.
Why though? Due to increased torque?
The grid frequency is set directly by 3 phase generators. So if the grid frequency drops, there is indeed a very strong torque trying to decelerate the turbines.
To some extent, this is a good thing. Because it means the spinning mass of all turbines in the grid actually provide a buffer for sub-second mismatches in demand and supply. But it does mean that big demand spikes (or drops) put some wear on the systems. They are made for this, but if it gets very periodic it might be more than designed for.
Note that this means that all turbines in a power network are perfectly synchronized. And if they are not, they are subject to massive torques. This is why a cold start is so hard. You need to bring on generators from one central point to keep everything in sync. And as you bring on new generation capacity you also need to bring on new consumption, that electricity has to go somewhere.
The reason for this synchronization is because the generators are spinning a magnet past 3 coils, and those coils are directly wired into the power network. If your magnets are out of phase then effectively the entire power network will use those coils to generate a field to get that magnet back in sync. All of those coils on all those generators really are all hard-wired together, it still kind of blows my mind.
Stuxnet damaged gas centrifuges by introducing a subtle change in rotational speed over time, eventually the motors failed mechanically.
https://en.wikipedia.org/wiki/Stuxnet
Because a variation in force is a cycle of stress, which is more destructive than constant force. It's called hysteresis, leading to material fatigue.
When the battery storage systems all temporarily pause providing grid inertia due to the state of charge check they’re performing, grid synchronized spinning mass has to compensate, which slows it down (transferring the kinetic energy to electrical energy), and then is sped back up as battery storage resumes inertia support.
Broadly speaking, it’s a collective momentum problem. If part of the system inertia sags, another part needs to step in to compensate for the inertia loss, otherwise you have brownouts and eventually, a blackout.
https://neutrino-energy.com/technology/grid-inertia/
I wish that article wasn't written entirely in Claude speak, it has some useful information, but the obvious AI voice will repel readers.