I think we saw a lot of news that "this is going to be a black start" and so on. Yet they were operating quite normally already the same night (direct personal communication with people there). I think HN in general didn't provide a reliable picture of what was going to happen at all. Same with my Twitter feed.
I agree and I still see people speculating over the cause of the outage in this very thread, showing a worrying degree of confidence that for example renewable energies played a negative role
There’s a subset of the population that has a survivalist fetish and wants to see worst-case scenarios in infrastructure, including black starts, play out in real life.
I think it’s because they get to witness uncommon procedures play out.
The rest of the population just wants to get on with life.
In a more upbeat note, the people I talked to in person were really cool-headed and realistic regarding speculation about the cause, compared to how lots of people seemed to have reacted on the web. Most people were basically "Someone incompetent fucked up, it'll be back soon" in mood during yesterday.
It's nice to think about what is involved in a "black start", but as people working with power grids are engineers too and these scenarios are things they will often consider and train for, it wasn't that big a deal.
I suspect it's a bit of insecurity; what would you do if your application's landscape suddenly went offline? In theory it's all infrastructure-as-code and pressing 'start' will boot everything up, pull up the last backups, etc... but when's the last time you tested that? When's the last time you did a training?
I've been working at an energy company for a few years now; while we have an office fire drill a few times a year, there is never an IT outage training. At best a load test late at night. And they paid the price a few years ago; due to the energy crisis, people checked their energy contracts and prices more often, causing 2-5x more load than normal. And the IT hadn't been updated, even though they were warned for a decade that their IT had bottlenecks (single database etc) and they would run into issues sometime.
Sir it was as black of a start as you could wish for. The France and Morocco interconnects helped only a slight bit. The grid went to 0 (zero) generation according to the grid operator; 10GW number was a data issue (!)
At least HN did better than the BBC that started by reporting and I quote "It is thought to be caused by a Cyberattack".
Also should be noted the Guardian is not doing better as the article does not clarify what the cause was. Most likely explanation at the moment is this:
I'm in Spain and went through the blackout yesterday. It was very interesting experience and I went around to learn the most from it. I used a Starlink, lithium battery and solar panels among other stuff. I hope you might find it useful:
12:00 I was working when around noon my computer display switched off. I tried turning on a light in the room: no light either. I went to the main circuit breaker, switched it off/on again : still no light. I went out to the street. The house is in a small suburban area so each house has it's own meter. The smart meter was not displaying any light or text. I knew that in my village a blinking red LED on the smart meter means electricity flowing. A fixed-constant red LED light means the grid is on but the house is not consuming electricity. No light and no text means there is no electricity reaching the smart meter. I checked all the neighbors' meters and I saw all lights were off. I started walking to the next street and was checking when another man came out. He told me he was also missing electricity. I checked Google, on the News tab there is no article about "{my_city} outage". So I started to walk towards the center. I called my parents which are in another EU country, they answered and told me all was good and they had electricity. So no electricity but the 3G mobile network was on (it remained on for about 4 hours before both 3G mobile network and normal phone signal went off). I started to walk toward the city center checking meters to understand how big is the problem, all streets are off. I realized my car had only 1/3 of the tank full. I decided to try the petrol station. On the way there, there was a train level crossing, the lights were blinking and the warning sound was on. A young man was keeping the train bar up and gesturing for cars to pass by without risk. I quickly heard him shouting "No electricity". I passed under with the car since many cars were doing it and I knew electricity was off so probably the bars were not working properly.
PETROL STATION
After 30-45 minutes of outage there were 4-5 cars parked around the petrol station pumps. No-one was actively using the pumps to fill the tank. I saw 4-5 people arguing, not sure about what. I went in the petrol station shop and there was an usual long queue of 30 people, the employee at the desk was writing on a paper. I asked another employee if it was possible to put petrol and she said no. I decided it was too late and it was not worth it to wait there. Given also some people already seemed agitated.
SUPERMARKET
Supermarket on the way was closed because it was a bank holiday on Monday. Maybe that's also why the outage had been more likely to happen on that day with less people checking things.
TRAIN LEVEL CROSSING
I was driving back home. At the train level crossing the same guy from before and another man were holding up the bars. I asked them if they needed some tools to rig the bar up and they said yes. Rushed back home and got some tools and wood and ropes. Got back there in 15 minutes. They had found a way to remove the bars by unscrewing a bolt. I chat with them. Another man passe by car and said he had notified the local police they had to remove the bar and the police was aware. At the moment I though those men were smart and capable and thinking on their feet. We exchanged numbers. We promised to all check the electricity at home and come to fix the bar at the train crossing. (I did go to check this morning at 8am when I woke and electricity was back.) The bar was still missing and lights were blinking. There was a long queue of 40 cars. I checked no train was passing and I started to let the people pass by. After I noticed a technician on the track 100 meters out in a yellow suit. I went to speak with him. Walking on tracks and pebbles is not super fast since you can't run easily. The technician said the trains were not running today. I double checked he was aware the protection bars were removed and that he would put them back when he knew the train were passing.
BACK HOME
When I came back home from the train level crossing during the outage I decided to do what I could. I filled up all bottles of waters I had in the house. Even if the outage was temporary I can still drink that water and it wasn't a waste.
STARLINK INTERNET SATELLITE
I have a Starlink antenna that I use in the summer during the holidays. I rushed to resubscribe that I had cancelled. As far as I understand if you don't have internet is not possible to subscribe and use the Starlink (which you need for internet). I also noticed there is a new cheaper plan for 9€ for 10GB in Spain. So I will be permanently subscribed to that to avoid the fear internet goes away and I can't subscribe back to Starlink and make it work.
FIBER OPTIC/DSL
Anyway this time the 3G mobile network was working and I had a small UPS that was keeping the modem and router working. The fiber connection with the modem&router powered by external battery (UPS) was giving normal internet. It worked for 4 hours until around 18:00 when even with power to the modem&router the fiber stopped working. It was around the same time the 3G/mobile signal had stopped.
I'm not sure if there 4 hours diesel generators somewhere or there was some priority line for telecoms. Please let me know if you know anything.
ECOFLOW BATTERY AND SOLAR PANELS
Somehow I had been a bit paranoid in the past few weeks and I had bought 1 month ago an Ecoflow Delta 3 Plus with 1024Wh, 2 solar panels of 800W each and the Powerstream which is a microinverter from Ecoflow to provide up to 800W by plugging in a normal socket of the house. I had decided to actually test them this weeks during the Easter holidays. I had left the Ecoflow battery uncharged because I thought it would be more obvious with the solar panels input. I cursed by perfectionism. Luckily the Ecoflow came 25% pre-charged.
The UPS was down in 40 minutes already and I saw the Ecoflow battery could maintain the modem&router for 4 hours. I was not sure I would be able to set them up correctly if internet went down. Also there were only 2-4 hours left of sun to charge the battery. So I decided to rush it and try to get the solar panel work. I did manage to connect all stuff but I realized I had forgot to buy the XT60i connector to connect the solar panels directly to the battery without the Powerstream. The shadow of a building in front obscured the patio where the solar pannels where around 18:00. Somehow the INPUT of the solar panels even in the last minutes with sun was reported as 0Watt. I went online and double check but I could not find anything obvious. Time was up for the day (or better said, Sun was down for the day). I decided to switch off the battery and save power for the day after with sun to have internet.
BIKING AROUND AND CHINESE SHOP
Given the sun was down and I couldn't do more I decided to take my electric bike and go around. My first petrol car had 1/3 gasoline tank full. I had just bought a second van with popup top roof last week. The tank was in reserve from the dealer and I had planned to go fill it like tomorrow. I cursed myself. Luckily I thought I still had the electric bike to charge from solar. Not sure exactly for what. So I went around the suburban neighborhood, and people were having a good time. It's Eastern holidays, one bar was opened selling drinks. I passed in front of a small Chinese shop. The shop was open with old people and kids going in and out. In Europe there are commonly small shops with tons of useful house things. They are usually managed by a whole Chinese family. Some people were buying torching and batteries. Only cash accepted of course. I already was stocked on torches and batteries and tools like that. But I browser around anyway. Some kids were playing hide and seek in the dark halls of the little shops. It was very cute. Out of the outage on the first hours people were very excited and happy. I bought a 4 in 1 key with triangle/square heads for 2.30€ (3$). It was good to have anyway and when I had checked the smart meters of the neighbors some required a triangle key to be opened and see if they were working. I don't plan to do anything nefarious. It seemed I was among the fastest and most active responders of the situation so it feels ethically okay for me to own this technician key. Just in case.
THE NEWS
Checked internet through Starlink one last time before bed. They said the outage would resolve from 21:00 to 1:00am. Some friends in Valencia around 20:00 CET told the power was back there.
GOING TO SLEEP
I wen to bed. I left the Ecoflow Delta 3 Plus plugged in the sockets in case the electricity came back during the night (and would go away before I would wake up).
LIGHT BACK
Around 1am I was woken up by the bed-side table turning on. Half-asleep I switched it off and kept sleeping.
WAKING UP
This morning electricity was back when I woke up around 8am. As I said in the TRAIN LEVEL CROSSING section I went there to check. I'll spend the day properly learning how the Ecoflow battery and solar panels work together with calm.
Thanks for the saga. It was an enjoyable read. You seemed more prepared than most, intentionally or not. Today (I'm in Romania) I went to buy a bunch of nonperishables. It was heartening to hear that people made the best of it.
Curious how random HN commentator knows the cause before our own government does? AFAIK, official answer is still "Not sure, we're trying to figure out".
Note the very first expression employed to answer the question in the title: "No.". If that ain't an exceedingly confident and ignorant stance on a subject we lack a huge amount of context on for us to give such a direct answer, I don't know what is.
Lots of HN users in Spain have transversal knowledge of tons of terms. Bear mind Spain it's highly social and engineers (proper ones, with a Bachelor's at least) often share trivia back and forth from their respective areas. Is was not weird seeing for people ask Biologysts when the pandemics and viceversa on cyberattacks.
By just politely asking you can hit tons of different terms in a day.
> REN said: “Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as ‘induced atmospheric vibration’. These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European network.”
- Redes Energéticas Nacionais (REN) is a Portuguese energy sector company
> What is the role of renewables?
> Spain is on its way to being a green energy leader: it has abundant sun and wind. Last year was a record period for renewable power generation, which accounted for 56% of all electricity used. By 2030 that proportion will rise to 81%.
> That shift will help Spain end its reliance on energy imports, but it also brings its own challenges. Every national grid in the world will need to spend heavily to upgrade distribution systems to connect scattered renewable generation and ensure it is balanced.
That does not answer the interesting (and slightly conspiratory) question. Did renewables play any negative role in this at all?
Someone already answered this, and yes renewables don't help and you need other kinds of plants. That doesn't tell us much about what actually caused it.
But those other kind of power plants are okay to stay off and start only when such situations arise, right? Still good for the environment I guess - you only burn fossils when the network restarts, which will probably only happen every few years in some corner of the continent. If I understood it correctly, that is.
The issue with solar and wind is that when they're being initiated in the project stage, they're advertised as the equivalent of some amount of average power (or even maximum power, if the project initiators want to be disingenuous about it), when the required amount of construction to match baseload capacity would easily be 3-4x that number.
There really is no one-size-fits-all solution, except for having an energy mix of renewables, with existing thermal power on standby.
Which is basically every winter in Northern Europe at least, and is usually when electricity demand is the highest (very cold days tend to have the least wind).
The UK was very very close to a blackout last winter for this exact reason. If Denmark weren't able to suspend maintenance on part of their grid to max out the Viking HVDC (it was running at half capacity) there would have been loss of supply. Potentially contained to one region and temporarily but it's hard to say for sure.
What's worse is the communication from NESO (now government owned, very recently) was downright misleading to how close it came.
People will believe whatever they want to believe, well in advance of any concrete statement. The actual truth won't matter. The right will use this as a call to burn more coal, the left will say "it doesn't matter because the planet is at stake", and we'll move on.
The cause needs to be investigated and appropriate measures put in place. Renewables however can't be the only answer - storage is key too.
Wide scale blackouts aren't exactly unknown, no matter what the generation.
And while you and I may agree, sadly the people who need convincing are those who listen and vote for the parties who say "it doesn't matter what we do here when China builds more coal power plants a week than we've ever had".
When they say "the reason you were without power is solar", and the opposition say "doesn't matter, you can't have gas", they aren't going to vote for the opposition.
> Do we not agree that the reality is that renewables are the only sustainable option for energy production?
No, we do not, unless you use a very narrow definition of sustainable. Nuclear will play a big part, especially in Western Europe, thanks to France which exports to all its neighbours and is a good stabilising provider.
And other than that, who knows. Green hydrogen could be a solution for energy storage.
France is effectively planning to reduce nuclear power though. Since they want to keep nuclear weapons it won’t go to zero, but they’re not planning to build enough new reactors to maintain their current output. Electricity demand however will increase in the future.
Since we are taking about the planet about "the planet at stake" I am not so sure that nuclear in france can play such a big part. Climate change will make it much more difficult, this is from Wikipedia:
As of early September 2022, 32 of France's 56 nuclear reactors were shut down due to maintenance or technical problems. In 2022, Europe's driest summer in 500 years had serious consequences for power plant cooling systems, as the drought reduced the amount of river water available for cooling.
> In 2022, Europe's driest summer in 500 years had serious consequences for power plant cooling systems, as the drought reduced the amount of river water available for cooling.
And rules around this have been adjusted to have less of an impact for the next "once in a 500 years" event.
Also, in the same way that France's nuclear power generation being out was covered by German and British renewables, during low points of renewable generation in those countries, French nuclear covered the gaps. That's the point of an interconnected network.
Thanks non-people-smart 10%, the "right" it's only trying to not close the remaining nuclear plants. I didn't hear anybody call to burn more coal (that will be stupid, specially in Spain). New modern nuclear plants are needed to have a more balanced energy pool, and not depend so much in natural gas. Anyway there is no confirmation yet on what happened, we will see.
Thanks, I know. At which point in this comment chain were we discussing the president of Spain though? Buxato stated they didn't hear "anybody", not "anybody in Spain".
Richard Tice, the Reform party’s deputy leader and energy spokesman, said the events in Spain should be a warning to Britain and showed the risks of net zero.
He said: “We need to know the exact causes but this should be seen as a wake-up call to the eco-zealots.
“Power grids need to operate within tight parameters to remain stable. Wind and solar outputs by contrast, vary hugely over long and short periods so they add risk to the system. The UK’s grid operators and our Government should take heed.”
That's the same Richard Tice that blamed a substation fire on Net Zero and wants to tax renewable power to subsidise non-renewables.
> The core priority of grid services right now is getting power restored, rather than establishing the narrow chain of causal events, which will take weeks or likely months. That will be more than enough time for the narrative to be set in stone; immune to whatever reality emerges from careful investigation.
Wait until you find Spain it's the 2nd most mountainous region in Europe.
Hint: anything non-coast can have dry summers and cold winters, brutal in the mountain regions such as Asturias and Leon, were they burnt coal like crazy, and having temperatures below 0 was and it's still the norm, not the exception. And not just the south of Cantabria, Asturias and so; the Castilles, Aragon and the mountain regions can amaze tourist as if they were phonied expecting a 'sunny' Spain like the beaches.
Mostly humid and foggy panoramas with mountains; and hot and dry summers in the Castilles; but it's like Ohio and the inner USA: you have extremes in both sides. Scorching sun and crazy cold winters, and yet the corn -wheat in Spain- raises like nothing.
So, one valid approach for one region can be void for another one. You can't except to set a global energy policy that works everywhere. Kinda like the US; not everything it's like Texas or Florida.
And, yes, we had tons of dams too near the Tagus and Duero rivers, they can produce tons of power too.
Nowadays, well, I'm pro nuclear, coal was something obsolete since the 80's and 90's.
Coal in Spain was indeed phased out even before wind+solar was a thing. It was not economically viable against imported energy sources such as oil and gas.
Spanish grid saying solar is a likely cause (though the PM seems to be disputing this 'in his view', but may be a poor translation from BBC).
My personal hunch is they were struggling to balance the grid as peak solar approached with too much supply. The last snapshot of data showed less than 1GW of CCGT which is not a lot for a grid of this size (the UK runs 2.5GW CCGT 24/7 for inertia and balancing reasons, even if there is excess renewables available - they will be paid to be turned off). Spain does also have hydro which can respond to the grid in fairness, but I am unsure how well prepared they would be to do this if CCGT is the main balancing source.
I also saw data showing Spain was exporting to France heavily at the time. I suspect if that interconnector went down (as rumored), you'd have potentially 1GW+ more load with nowhere for it to go.
This may have then caused the disconnection events (the spanish grid operator said there were 2), as various plants trip as they saw the grid becoming unstable. They recovered from the first but couldn't recover from the 2nd, there isn't a huge amount of detail on these.
This is basically exactly what happened in the 2019 outage in the UK (and from comments on HN yesterday was also a major factor in the Brazilian outage recently) which affected SE England and London (but didn't cause a full grid collapse, I suspect averted because of more CCGT online to balance).
In that event:
1) 400kV cable goes down. Power is rerouted. Was a lightning strike which is pretty common.
2) This caused a very brief change in the voltage/frequency on the immediate substation that was taking power from Hornsea 1 offshore wind farm to the national grid
3) Hornsea 1's voltage detection equipment was too sensitive and tripped, dropping 700MW of supply. It shouldn't have tripped for such a short drop while they rerouted grid power to bypass the out of service 400kV line.
4) This then caused a cascade of other generating assets to trip (also too sensitive), causing a major drop in grid frequency and this then triggered various distribution points in England to disconnect automatically to try and drop load by 5%. Furthermore all 'variable' generation is requested to immediately raise output to balance the system while load shedding goes on.
5) The load shedding operation did not work as expected. They expected drop was 850MW but only 350MW was achieved. This is likely due to embedded solar which they hadn't taken into account in their calculations (or one of the reasons). This causes frequency to drop further and another level of load shedding to occur.
6) Frequency recovers.
Even worse, there was another potential near miss here - the DNO started reconnecting customers before they had the goahead to connect from the grid. It is lucky that another outage didn't occur because of this as the grid was not fully checked and they should have waited.
Anyone saying that renewables are not affecting grid stability are unfortunately just wrong. I am huge proponent of renewables but we have the ratio of solar/wind/batteries/other inertia (flywheels) really wrong. We should not be running so much variable capacity with so little storage.
I haven't wrote that the government said that. But a lot of online commentators and even some news outlets immediately started to discuss "cyberattacks" (including the linked article) and asking "maybe it was the Russians?".
Most people who are familiar with how electric grids operate are in agreement that high degree of renewable generation has likely made the situation worse not better. We know that right before the blackout the grid had a sudden dip in frequency, which is exactly the condition which leads to unstable work of inverters used by renewable plants. Just watch this Practical Engineering video [1], if you are not familiar with the topic. It discusses exactly the problems I wrote about. So renewables may have not been the direct reason for the blackout, but they are highly likely have helped to tip the gird behind the critical point.
A lot of news outlets were not part of the left leaning side of the goverment; the theory for an attack was set from every media, from the Basque/Catalan left and right to the pro-Spain left and right.
I think we saw a lot of news that "this is going to be a black start" and so on. Yet they were operating quite normally already the same night (direct personal communication with people there). I think HN in general didn't provide a reliable picture of what was going to happen at all. Same with my Twitter feed.
I agree and I still see people speculating over the cause of the outage in this very thread, showing a worrying degree of confidence that for example renewable energies played a negative role
There’s a subset of the population that has a survivalist fetish and wants to see worst-case scenarios in infrastructure, including black starts, play out in real life.
I think it’s because they get to witness uncommon procedures play out.
The rest of the population just wants to get on with life.
In a more upbeat note, the people I talked to in person were really cool-headed and realistic regarding speculation about the cause, compared to how lots of people seemed to have reacted on the web. Most people were basically "Someone incompetent fucked up, it'll be back soon" in mood during yesterday.
It's nice to think about what is involved in a "black start", but as people working with power grids are engineers too and these scenarios are things they will often consider and train for, it wasn't that big a deal.
I suspect it's a bit of insecurity; what would you do if your application's landscape suddenly went offline? In theory it's all infrastructure-as-code and pressing 'start' will boot everything up, pull up the last backups, etc... but when's the last time you tested that? When's the last time you did a training?
I've been working at an energy company for a few years now; while we have an office fire drill a few times a year, there is never an IT outage training. At best a load test late at night. And they paid the price a few years ago; due to the energy crisis, people checked their energy contracts and prices more often, causing 2-5x more load than normal. And the IT hadn't been updated, even though they were warned for a decade that their IT had bottlenecks (single database etc) and they would run into issues sometime.
Sir it was as black of a start as you could wish for. The France and Morocco interconnects helped only a slight bit. The grid went to 0 (zero) generation according to the grid operator; 10GW number was a data issue (!)
> I think we saw a lot of news that "this is going to be a black start" and so on.
Well, it basically was, wasn't it, just a fairly orderly one?
Disagree strongly. They did a black start:
https://news.ycombinator.com/item?id=43820964
https://news.ycombinator.com/item?id=43824032
At least HN did better than the BBC that started by reporting and I quote "It is thought to be caused by a Cyberattack".
Also should be noted the Guardian is not doing better as the article does not clarify what the cause was. Most likely explanation at the moment is this:
"What is "Induced Atmospheric Vibration"?" - https://physics.stackexchange.com/questions/848666/what-is-i...
"Diagnosis and Mitigation of Observed Oscillations in IBR-Dominant Power Systems - A PRACTICAL GUIDE" - https://www.esig.energy/wp-content/uploads/2024/10/esig-rpt-...
I'm in Spain and went through the blackout yesterday. It was very interesting experience and I went around to learn the most from it. I used a Starlink, lithium battery and solar panels among other stuff. I hope you might find it useful:
12:00 I was working when around noon my computer display switched off. I tried turning on a light in the room: no light either. I went to the main circuit breaker, switched it off/on again : still no light. I went out to the street. The house is in a small suburban area so each house has it's own meter. The smart meter was not displaying any light or text. I knew that in my village a blinking red LED on the smart meter means electricity flowing. A fixed-constant red LED light means the grid is on but the house is not consuming electricity. No light and no text means there is no electricity reaching the smart meter. I checked all the neighbors' meters and I saw all lights were off. I started walking to the next street and was checking when another man came out. He told me he was also missing electricity. I checked Google, on the News tab there is no article about "{my_city} outage". So I started to walk towards the center. I called my parents which are in another EU country, they answered and told me all was good and they had electricity. So no electricity but the 3G mobile network was on (it remained on for about 4 hours before both 3G mobile network and normal phone signal went off). I started to walk toward the city center checking meters to understand how big is the problem, all streets are off. I realized my car had only 1/3 of the tank full. I decided to try the petrol station. On the way there, there was a train level crossing, the lights were blinking and the warning sound was on. A young man was keeping the train bar up and gesturing for cars to pass by without risk. I quickly heard him shouting "No electricity". I passed under with the car since many cars were doing it and I knew electricity was off so probably the bars were not working properly.
PETROL STATION
After 30-45 minutes of outage there were 4-5 cars parked around the petrol station pumps. No-one was actively using the pumps to fill the tank. I saw 4-5 people arguing, not sure about what. I went in the petrol station shop and there was an usual long queue of 30 people, the employee at the desk was writing on a paper. I asked another employee if it was possible to put petrol and she said no. I decided it was too late and it was not worth it to wait there. Given also some people already seemed agitated.
SUPERMARKET
Supermarket on the way was closed because it was a bank holiday on Monday. Maybe that's also why the outage had been more likely to happen on that day with less people checking things.
TRAIN LEVEL CROSSING
I was driving back home. At the train level crossing the same guy from before and another man were holding up the bars. I asked them if they needed some tools to rig the bar up and they said yes. Rushed back home and got some tools and wood and ropes. Got back there in 15 minutes. They had found a way to remove the bars by unscrewing a bolt. I chat with them. Another man passe by car and said he had notified the local police they had to remove the bar and the police was aware. At the moment I though those men were smart and capable and thinking on their feet. We exchanged numbers. We promised to all check the electricity at home and come to fix the bar at the train crossing. (I did go to check this morning at 8am when I woke and electricity was back.) The bar was still missing and lights were blinking. There was a long queue of 40 cars. I checked no train was passing and I started to let the people pass by. After I noticed a technician on the track 100 meters out in a yellow suit. I went to speak with him. Walking on tracks and pebbles is not super fast since you can't run easily. The technician said the trains were not running today. I double checked he was aware the protection bars were removed and that he would put them back when he knew the train were passing.
BACK HOME
When I came back home from the train level crossing during the outage I decided to do what I could. I filled up all bottles of waters I had in the house. Even if the outage was temporary I can still drink that water and it wasn't a waste.
STARLINK INTERNET SATELLITE
I have a Starlink antenna that I use in the summer during the holidays. I rushed to resubscribe that I had cancelled. As far as I understand if you don't have internet is not possible to subscribe and use the Starlink (which you need for internet). I also noticed there is a new cheaper plan for 9€ for 10GB in Spain. So I will be permanently subscribed to that to avoid the fear internet goes away and I can't subscribe back to Starlink and make it work.
FIBER OPTIC/DSL
Anyway this time the 3G mobile network was working and I had a small UPS that was keeping the modem and router working. The fiber connection with the modem&router powered by external battery (UPS) was giving normal internet. It worked for 4 hours until around 18:00 when even with power to the modem&router the fiber stopped working. It was around the same time the 3G/mobile signal had stopped.
I'm not sure if there 4 hours diesel generators somewhere or there was some priority line for telecoms. Please let me know if you know anything.
ECOFLOW BATTERY AND SOLAR PANELS
Somehow I had been a bit paranoid in the past few weeks and I had bought 1 month ago an Ecoflow Delta 3 Plus with 1024Wh, 2 solar panels of 800W each and the Powerstream which is a microinverter from Ecoflow to provide up to 800W by plugging in a normal socket of the house. I had decided to actually test them this weeks during the Easter holidays. I had left the Ecoflow battery uncharged because I thought it would be more obvious with the solar panels input. I cursed by perfectionism. Luckily the Ecoflow came 25% pre-charged.
The UPS was down in 40 minutes already and I saw the Ecoflow battery could maintain the modem&router for 4 hours. I was not sure I would be able to set them up correctly if internet went down. Also there were only 2-4 hours left of sun to charge the battery. So I decided to rush it and try to get the solar panel work. I did manage to connect all stuff but I realized I had forgot to buy the XT60i connector to connect the solar panels directly to the battery without the Powerstream. The shadow of a building in front obscured the patio where the solar pannels where around 18:00. Somehow the INPUT of the solar panels even in the last minutes with sun was reported as 0Watt. I went online and double check but I could not find anything obvious. Time was up for the day (or better said, Sun was down for the day). I decided to switch off the battery and save power for the day after with sun to have internet.
BIKING AROUND AND CHINESE SHOP
Given the sun was down and I couldn't do more I decided to take my electric bike and go around. My first petrol car had 1/3 gasoline tank full. I had just bought a second van with popup top roof last week. The tank was in reserve from the dealer and I had planned to go fill it like tomorrow. I cursed myself. Luckily I thought I still had the electric bike to charge from solar. Not sure exactly for what. So I went around the suburban neighborhood, and people were having a good time. It's Eastern holidays, one bar was opened selling drinks. I passed in front of a small Chinese shop. The shop was open with old people and kids going in and out. In Europe there are commonly small shops with tons of useful house things. They are usually managed by a whole Chinese family. Some people were buying torching and batteries. Only cash accepted of course. I already was stocked on torches and batteries and tools like that. But I browser around anyway. Some kids were playing hide and seek in the dark halls of the little shops. It was very cute. Out of the outage on the first hours people were very excited and happy. I bought a 4 in 1 key with triangle/square heads for 2.30€ (3$). It was good to have anyway and when I had checked the smart meters of the neighbors some required a triangle key to be opened and see if they were working. I don't plan to do anything nefarious. It seemed I was among the fastest and most active responders of the situation so it feels ethically okay for me to own this technician key. Just in case.
THE NEWS
Checked internet through Starlink one last time before bed. They said the outage would resolve from 21:00 to 1:00am. Some friends in Valencia around 20:00 CET told the power was back there.
GOING TO SLEEP
I wen to bed. I left the Ecoflow Delta 3 Plus plugged in the sockets in case the electricity came back during the night (and would go away before I would wake up).
LIGHT BACK
Around 1am I was woken up by the bed-side table turning on. Half-asleep I switched it off and kept sleeping.
WAKING UP
This morning electricity was back when I woke up around 8am. As I said in the TRAIN LEVEL CROSSING section I went there to check. I'll spend the day properly learning how the Ecoflow battery and solar panels work together with calm.
Thank you for reading until here!
Thanks for the saga. It was an enjoyable read. You seemed more prepared than most, intentionally or not. Today (I'm in Romania) I went to buy a bunch of nonperishables. It was heartening to hear that people made the best of it.
That article is dated last evening and is outdated, especially since we recently had a press conference from Spanish national grid
Do you have a link/summary for what was said in the press conference?
No. Power grids are inherently unstable, this appears to have been "just" a cascading failure that couldn't really be planned for.
Curious how random HN commentator knows the cause before our own government does? AFAIK, official answer is still "Not sure, we're trying to figure out".
note the keywords: "appears" and "not sure"
Note the very first expression employed to answer the question in the title: "No.". If that ain't an exceedingly confident and ignorant stance on a subject we lack a huge amount of context on for us to give such a direct answer, I don't know what is.
missed that, a valid point, thank you
Lots of HN users in Spain have transversal knowledge of tons of terms. Bear mind Spain it's highly social and engineers (proper ones, with a Bachelor's at least) often share trivia back and forth from their respective areas. Is was not weird seeing for people ask Biologysts when the pandemics and viceversa on cyberattacks.
By just politely asking you can hit tons of different terms in a day.
TL;DR
> REN said: “Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as ‘induced atmospheric vibration’. These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European network.”
- Redes Energéticas Nacionais (REN) is a Portuguese energy sector company
Which is one theory, out of many. Still no concrete answer beyond a couple of theories.
We'll probably never know. Our government has been markedly opaque and corrupt for too long to trust any of the information they put out.
> What is the role of renewables? > Spain is on its way to being a green energy leader: it has abundant sun and wind. Last year was a record period for renewable power generation, which accounted for 56% of all electricity used. By 2030 that proportion will rise to 81%. > That shift will help Spain end its reliance on energy imports, but it also brings its own challenges. Every national grid in the world will need to spend heavily to upgrade distribution systems to connect scattered renewable generation and ensure it is balanced.
That does not answer the interesting (and slightly conspiratory) question. Did renewables play any negative role in this at all?
Someone already answered this, and yes renewables don't help and you need other kinds of plants. That doesn't tell us much about what actually caused it.
But those other kind of power plants are okay to stay off and start only when such situations arise, right? Still good for the environment I guess - you only burn fossils when the network restarts, which will probably only happen every few years in some corner of the continent. If I understood it correctly, that is.
You need something that can provide instantaneous reserve to stabilize the frequency. Traditionally that’s inertia of lots of big spinning generators.
https://en.m.wikipedia.org/wiki/Inertial_response
The issue with solar and wind is that when they're being initiated in the project stage, they're advertised as the equivalent of some amount of average power (or even maximum power, if the project initiators want to be disingenuous about it), when the required amount of construction to match baseload capacity would easily be 3-4x that number.
There really is no one-size-fits-all solution, except for having an energy mix of renewables, with existing thermal power on standby.
With the caveat that the standby is likely only going to be required during winter in case of prolonged dark and windless weather.
Everything else will be covered by renewables + battery.
Which is basically every winter in Northern Europe at least, and is usually when electricity demand is the highest (very cold days tend to have the least wind).
The UK was very very close to a blackout last winter for this exact reason. If Denmark weren't able to suspend maintenance on part of their grid to max out the Viking HVDC (it was running at half capacity) there would have been loss of supply. Potentially contained to one region and temporarily but it's hard to say for sure.
What's worse is the communication from NESO (now government owned, very recently) was downright misleading to how close it came.
> With the caveat that the standby is likely only going to be required during winter in case of prolonged dark and windless weather
Which had already happened a few times across big parts of Europe, with weeks of overcast and low wind weather.
Specifically for Spain, a better backup wil probably be more interconnects with France and their stable nuclear power.
Yup they did. We’ll have to wait for the postmortem to understand how much. Lack of rotating mass and/or battery farms is a known problem.
People will believe whatever they want to believe, well in advance of any concrete statement. The actual truth won't matter. The right will use this as a call to burn more coal, the left will say "it doesn't matter because the planet is at stake", and we'll move on.
The reality doesn't matter to 90% of people.
But the planet is at stake. Do we not agree that the reality is that renewables are the only sustainable option for energy production?
The cause needs to be investigated and appropriate measures put in place. Renewables however can't be the only answer - storage is key too.
Wide scale blackouts aren't exactly unknown, no matter what the generation.
And while you and I may agree, sadly the people who need convincing are those who listen and vote for the parties who say "it doesn't matter what we do here when China builds more coal power plants a week than we've ever had".
When they say "the reason you were without power is solar", and the opposition say "doesn't matter, you can't have gas", they aren't going to vote for the opposition.
> Do we not agree that the reality is that renewables are the only sustainable option for energy production?
No, we do not, unless you use a very narrow definition of sustainable. Nuclear will play a big part, especially in Western Europe, thanks to France which exports to all its neighbours and is a good stabilising provider.
And other than that, who knows. Green hydrogen could be a solution for energy storage.
France is effectively planning to reduce nuclear power though. Since they want to keep nuclear weapons it won’t go to zero, but they’re not planning to build enough new reactors to maintain their current output. Electricity demand however will increase in the future.
Since we are taking about the planet about "the planet at stake" I am not so sure that nuclear in france can play such a big part. Climate change will make it much more difficult, this is from Wikipedia:
As of early September 2022, 32 of France's 56 nuclear reactors were shut down due to maintenance or technical problems. In 2022, Europe's driest summer in 500 years had serious consequences for power plant cooling systems, as the drought reduced the amount of river water available for cooling.
> In 2022, Europe's driest summer in 500 years had serious consequences for power plant cooling systems, as the drought reduced the amount of river water available for cooling.
And rules around this have been adjusted to have less of an impact for the next "once in a 500 years" event.
Also, in the same way that France's nuclear power generation being out was covered by German and British renewables, during low points of renewable generation in those countries, French nuclear covered the gaps. That's the point of an interconnected network.
Thanks non-people-smart 10%, the "right" it's only trying to not close the remaining nuclear plants. I didn't hear anybody call to burn more coal (that will be stupid, specially in Spain). New modern nuclear plants are needed to have a more balanced energy pool, and not depend so much in natural gas. Anyway there is no confirmation yet on what happened, we will see.
Well, here is one person who even has somewhat of an influence calling to burn more coal: https://www.npr.org/2025/04/16/nx-s1-5359013/trump-orders-co...
Trump is not president of Spain.
Thanks, I know. At which point in this comment chain were we discussing the president of Spain though? Buxato stated they didn't hear "anybody", not "anybody in Spain".
Not yet... did lose in Canada though.
https://www.telegraph.co.uk/gift/f4476083e9854ba3
Richard Tice, the Reform party’s deputy leader and energy spokesman, said the events in Spain should be a warning to Britain and showed the risks of net zero.
He said: “We need to know the exact causes but this should be seen as a wake-up call to the eco-zealots.
“Power grids need to operate within tight parameters to remain stable. Wind and solar outputs by contrast, vary hugely over long and short periods so they add risk to the system. The UK’s grid operators and our Government should take heed.”
That's the same Richard Tice that blamed a substation fire on Net Zero and wants to tax renewable power to subsidise non-renewables.
A roundup of people jumping to conclusions is at
https://reneweconomy.com.au/spains-blackout-has-already-trig...
> The core priority of grid services right now is getting power restored, rather than establishing the narrow chain of causal events, which will take weeks or likely months. That will be more than enough time for the narrative to be set in stone; immune to whatever reality emerges from careful investigation.
>Specially in Spain
Wait until you find Spain it's the 2nd most mountainous region in Europe. Hint: anything non-coast can have dry summers and cold winters, brutal in the mountain regions such as Asturias and Leon, were they burnt coal like crazy, and having temperatures below 0 was and it's still the norm, not the exception. And not just the south of Cantabria, Asturias and so; the Castilles, Aragon and the mountain regions can amaze tourist as if they were phonied expecting a 'sunny' Spain like the beaches.
Mostly humid and foggy panoramas with mountains; and hot and dry summers in the Castilles; but it's like Ohio and the inner USA: you have extremes in both sides. Scorching sun and crazy cold winters, and yet the corn -wheat in Spain- raises like nothing.
So, one valid approach for one region can be void for another one. You can't except to set a global energy policy that works everywhere. Kinda like the US; not everything it's like Texas or Florida.
And, yes, we had tons of dams too near the Tagus and Duero rivers, they can produce tons of power too.
Nowadays, well, I'm pro nuclear, coal was something obsolete since the 80's and 90's.
Coal in Spain was indeed phased out even before wind+solar was a thing. It was not economically viable against imported energy sources such as oil and gas.
Spanish grid saying solar is a likely cause (though the PM seems to be disputing this 'in his view', but may be a poor translation from BBC).
My personal hunch is they were struggling to balance the grid as peak solar approached with too much supply. The last snapshot of data showed less than 1GW of CCGT which is not a lot for a grid of this size (the UK runs 2.5GW CCGT 24/7 for inertia and balancing reasons, even if there is excess renewables available - they will be paid to be turned off). Spain does also have hydro which can respond to the grid in fairness, but I am unsure how well prepared they would be to do this if CCGT is the main balancing source.
I also saw data showing Spain was exporting to France heavily at the time. I suspect if that interconnector went down (as rumored), you'd have potentially 1GW+ more load with nowhere for it to go.
This may have then caused the disconnection events (the spanish grid operator said there were 2), as various plants trip as they saw the grid becoming unstable. They recovered from the first but couldn't recover from the 2nd, there isn't a huge amount of detail on these.
This is basically exactly what happened in the 2019 outage in the UK (and from comments on HN yesterday was also a major factor in the Brazilian outage recently) which affected SE England and London (but didn't cause a full grid collapse, I suspect averted because of more CCGT online to balance).
In that event:
1) 400kV cable goes down. Power is rerouted. Was a lightning strike which is pretty common.
2) This caused a very brief change in the voltage/frequency on the immediate substation that was taking power from Hornsea 1 offshore wind farm to the national grid
3) Hornsea 1's voltage detection equipment was too sensitive and tripped, dropping 700MW of supply. It shouldn't have tripped for such a short drop while they rerouted grid power to bypass the out of service 400kV line.
4) This then caused a cascade of other generating assets to trip (also too sensitive), causing a major drop in grid frequency and this then triggered various distribution points in England to disconnect automatically to try and drop load by 5%. Furthermore all 'variable' generation is requested to immediately raise output to balance the system while load shedding goes on.
5) The load shedding operation did not work as expected. They expected drop was 850MW but only 350MW was achieved. This is likely due to embedded solar which they hadn't taken into account in their calculations (or one of the reasons). This causes frequency to drop further and another level of load shedding to occur.
6) Frequency recovers.
Even worse, there was another potential near miss here - the DNO started reconnecting customers before they had the goahead to connect from the grid. It is lucky that another outage didn't occur because of this as the grid was not fully checked and they should have waited.
Anyone saying that renewables are not affecting grid stability are unfortunately just wrong. I am huge proponent of renewables but we have the ratio of solar/wind/batteries/other inertia (flywheels) really wrong. We should not be running so much variable capacity with so little storage.
Spaniard here. If it were a cyberattack, the Baliaric and Canary islands would be equally harmed; yet they went neraly without issues.
Resident of Spain here. Why and how? They're different grids, not sure why an attack would have to "propagate automatically" as you imply?
A cyberattack would try crack up homogeneous systems being geographically spread over, even physically.
Depends on the implementation, no? Another angle is that this theoretical attack might have tried that but failed/got foiled.
No. please see Betteridge's law of headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
[flagged]
Who is blaming hackers? Not the government or network operator at least.
In fact, you are doing exactly the thing they have avoided so far: Jumping to premature conclusions.
I haven't wrote that the government said that. But a lot of online commentators and even some news outlets immediately started to discuss "cyberattacks" (including the linked article) and asking "maybe it was the Russians?".
Most people who are familiar with how electric grids operate are in agreement that high degree of renewable generation has likely made the situation worse not better. We know that right before the blackout the grid had a sudden dip in frequency, which is exactly the condition which leads to unstable work of inverters used by renewable plants. Just watch this Practical Engineering video [1], if you are not familiar with the topic. It discusses exactly the problems I wrote about. So renewables may have not been the direct reason for the blackout, but they are highly likely have helped to tip the gird behind the critical point.
[1]: https://www.youtube.com/watch?v=7G4ipM2qjfw
A lot of news outlets were not part of the left leaning side of the goverment; the theory for an attack was set from every media, from the Basque/Catalan left and right to the pro-Spain left and right.
It's too early to jump to conclusions. We literally didn't know what happened.
Ghost Riders in the Sky causing abnormal atmospheric vibrations.
( https://en.wikipedia.org/wiki/(Ghost)_Riders_in_the_Sky:_A_C... )
Yippie-yi-oh, yippie-yi-aie!