IATC - Living With the Enemy – How to protect yourself (and Energy Systems)

hi there um firstly I'm only 5 fo five so I'm sitting up here instead of down there um also I do not need medical attention so I'm going to stay sitting up here in case I fall over so hi um so to last year I was here um I gave a talk about like where we're at in the power grid how many utilities we have in the country various things to do with sort of current state of how our grid is um I decided to give a little more of an extreme version this year which is uh essentially where we're at with renewable energy what's going on with the grid what our current trends are for

modernization not where we're at right now but where we're going um but I also have a quick question at the beginning I have a Choose Your Own Adventure in here I have two versions of this talk with a scenario at the end one of the scenarios is about an invertor that we're calling sunsat and imagine we uh I was calling it Crow stried all of the invertors um what would that look like I've got scenario to go through the other one is what if we were to just yeah sure is that better there we go um nope how's that I'll stop flailing around thank you um the other one was essentially if we lost all power in the

Eastern Seaboard entirely and we had to Black start our system which is when we restart generators we restart load um what does that look like in a modernized grid so while I'm talking at some point I'm going to give you a choice cuz there's not time for both and people are going to have to put up their hands and tell me which disastrous scenario you're going to have to look at um so while I'm talking also what I'm really talking about here is what I'm thinking about people can do in this scenario is how do we raise the bar on the Technical Solutions we're trying to actually provide to the electric grid so we can

have all of these futures um how do we get companies to operate within reasonable bounds with these Technologies and learn good lessons about how to do this securely and still afford to do it and how do we moderate how people think um or talk um fear uncertainty and doubt in the electric grid is one of the key selling points for most of the Technologies you'll see probably at some of the conferences here um we like to say oh there's this vulnerability it could cause the entirety of New York to lose all power um a few weeks ago someone said that about electric vehicles they said 3,000 electric vehicles in um Manhattan could cause the parag grid to goow

completely I don't know where their parking spaces were to begin with I don't know where these vehicles were but we have got to stop the this kind of fear uncertainty and doubt and come up with moderate scenarios where we can actually operate through all of these things so now for the catastrophe um a few people have mentioned uh climate change over the last few days or last few hours um our planet is dying um we are killing our planet and again doesn't take me being a scientist or a doctor to say what that is but Las Vegas in a few years will be unlivable it will be too hot we won't be able to live here the

scenario where the if we don't have AC we will just die is is there so um if we keep going the direction we're going in the next few years we're going to reach this 1.5 degree limit where um essentially they will be completely unable to recover from this some of the solutions about 28% of that problem comes from how we generate and use electricity so one of the big Solutions that's being pushed is renewable energy and that's primarily what I'm going to talk about today so just current state of our grid I actually grabbed this picture while we were talking looking just to see what the current state of houses were like how many people don't

have power today cuz every day seems to be a new adventure and weather disasters also um around 600,000 people in the country today don't have power um to do with various storms um you can see Florida's fairly affected by I think it was Hurricane Barrel is that the right name don't know de um Debbie was Deb was it Barrel last week or yeah okay was two weeks ago okay um that also talks the number of weather events we're having this I don't know the name anymore um 600,000 people today don't have power that that's a lot that's not a small number um again there's about 3,000 utilities there's lots of things that can go into that but um usually we're

I've been used to seeing these graphs just be all blue in the last few years and it's unusual we see whole states being a different color every day and that's kind of where we're at just now but we are trying to solve that not all of this is caused by weather some of this just an aging system some of this is increased load there's many things that are going into this um we've put about $1.2 billion dollar sorry I should say trillion apologies trillion dollars in the last three years into trying to modernize our Electric System um a large majority of that is going into renewable energy um split in the country some people hate renewable

energy some people love it some people change words like me to try and make people not hate it as much um but there is a split in the country but no matter what we're past the point where this isn't going to happen I've been working in this field for 20 years most most actually in renewable energy it's it's not going away there's no amount of foot stomping that's going to do it um there is misinformation everywhere I've got a running joke here where um in Texas when they had a a ramp down in the evening they referred to it as a surprise Sunset I think which is strange because they were blaming the solar and wanted to say

they wanted their coal back um weather events are still are biggest threat no matter what we say about in this presentation weather events are still going to take our power more often in the next few years the N think cyber security will um we also have insane load growth that I'll talk about in a minute um but we're still pretty reliable like as you look at it in the world we we're still in a room with power lights on we still rely on our power being on we're not um the least reliable country in the world we're actually the most reliable country in the world for power delivery that being said we have challenges so we are trying to clean

this up um referring to all of the presentations today renewable energy is every everywhere just now or clean energy whatever you want to call it um if you look at agriculture from this morning a lot of these um different locations are looking at electric tractors but also looking at Agra voltaics which is where people are putting solar on top of their um agricultural land as well um water plants are looking at using solar and storage for backup generation as well again we're talking about water right here if you don't have power you can't necessarily actually pump water um so they use solar and storage as backup our communication sites are one of the main vendors for our communication site

backup generators is Tesla who are selling batteries um we're pitting it everywhere essentially Walmart is now actually the biggest owner of solar in the country which is kind of insane but it's everywhere um we're not getting away from it it's here but with that comes this is now connected to every sector it's not just our grid it's now even the generators that are meant to back up the grid that are backing up every other sector as well that are related to this problem but why am I worried about it um well firstly why is it good let's go with that I'll go with good things first I'm trying not to be I mean as much as the previous talk was

super interesting I'm trying not to include As Much Death in this one just I can talk about death till the cows come home but you know trying to keep it a little bit on some of the the good things that we're doing with renewable energy um if you live in Texas you've had about 17 different events in the past couple years that have involved batteries and energy storage ramping up super fast to try and save you from whatever was going on that day that may have been a surprise Sunset failure of a natural gas generator failure of wind there's been a number of failures that batteries have saved a lot of people from again I can talk about death here

um the first time aot Texas had a major failure multiple people did die for various reasons because of lack of heating um things to do with them sitting in their garages people do silly things when the power goes out and people do die because of that California's also had many heat events remember I talked about really hot during those heat events um batteries are giving these emergency discharge signals meaning hey by the way we've got way too much load we need you to absolutely discharge all of your generation right now onto the grid and they're doing it and it works um at this point California has reached I believe their 100% clean which includes many

other things but batteries are now becoming the single largest aggregate generator in California so it's not an imaginary scenario anymore that this is happening people are doing it it's useful it's actually really cool I like it I've worked on batteries a long time again there are some challenges but why is electricity and energy delivery a little different to other sectors um when we talk about energy delivery when we talk about um generation assets we're also talking about distributed assets so I don't know how many people have solar in their home or batteries or an electric vehicle you can show hands or just cool cool um you don't necessarily control the water in your house and pump it back into the

system uh where the the nice person who was talking about water earlier um do you pump water back into the system at all to supply your water no I'm hoping you don't I really hope you

don't yeah so we don't necessarily regularly pump water back into our power or to our water system in power generation though we're trying to put solar and storage on people's houses and then have them respond as a grid asset for the system that means they're also responsible for that generator sitting on their house so if there's a vulnerability in their invertor if there's something wrong they're the ones that are meant to fix it so we're putting this on humans and humans are notoriously unreliable um why there is a picture of bridge here I used the I use water since that's been today I like to use the bridge analogy of if you were told a bridge was going to fall down and

you were told you need to drive your car across it to fix it would you do that like would a human go out there and go I'm going to risk my life to to drive across this bridge and make sure it stays standing most humans are not that stupid to begin with um but we're doing that effectively on people's houses we in California last year it might be two years ago I'm getting old um they sent out an emergency alert that was similar to hey we're we've got a major problem from the emergency Al system first time they've done it everyone's phones it was forced into basically said you need to turn down your AC you need to do

something or grid is going to fail it was a huge emergency at the time every other automatic action had failed got this I was living in California got the text message everyone's looking at it like what what why this is usually for either lost children or some kind of massive weather event usually wfire actually in California and uh basically it did actually work the humans did do it that one time because they got that emergency alert next time they didn't because every time you send them this if you send them it 17 times a year they're going to stop responding they want their AC nobody necessarily wants to sit in 87° for 2 days because they're trying to

save the electric grid in particular they don't want to sit there when they're in California where their electric rates are the highest in the country and they're wondering what they're paying for at this point because they're sitting in 87 degrees in their house so it's one of the sectors where we're relying on humans to take an action to save the greater good and as we've seen in recent years humans are not necessarily that great great as on mass at saving the larger population other trends that we're seeing and this relates to my later talk um my later scenario is massive load increases everywhere um just for reference this will be a big impact on cyber security as well because these new

loads are going to be able to impact our system massively and there have been vulnerabilities in these loads and how they operate um I think I was trying to use an estimate of how much it actually is going to be um by the year 2030 and given the accent I was using six times Scotland's load as a whole will be added to our grid which is huge um or 1.7 Virginia's I was trying to use it in terms of States um no one called Virginia is involved in this it's the state of Virginia but that's the size of the load that's going to be added from just data centers from needing data AI all these cool things we're trying to do

they all need a data Center somewhere and so that's a huge load increase it's also a huge beginning of vulnerability in our system it's also kind of a cycle um we're telling people that modernization and cyber security now also need AI that AI needs to go somewhere um that's the data center yes we're using it to create stupid pictures just now but most of those AI locations need those data centers we then need that data center to be served by the grid that we're trying to modernize that doesn't have enough power to serve the load so we're using AI to keep going and we're in this enormous cycle of I'm not sure where it's going to end if we don't

moderate this um some key facts and figures about this whole new renewable energy supply chain I'm going to give the spoiler it's all from China all of it every single piece of it um I was asked a few months ago and I'll talk about this event to do an analysis of where one particular battery was across the country um where is it all I remember looking at them like I'm not going to spend a million dollars of taxpayer money trying to work out everywhere where this system is in the country that's ridiculous I'm going to tell you it's 97% of the systems they're like why are you answering like that like cuz it's probably 97% of the

systems and I'd rather spend the money trying to fix this situation than spend the money trying to come up with a number when I know how many there are it's kind of this look of shock um but even when we look at the number of vendors on this system meaning the number of people whose vulnerabilities we possibly have to account for there's at least 73 invertor manufacturers that can be used in the country that have passed all of the safety regulations all of the requirements we have 73 invertor manufacturers and 2% of those are us-based so that's not very many um if you look at the top graph these are the global invertor manufacturers again we need

invertors for almost every Renewable Energy site renewable energy is usually DC we need to convert it to AC um we need it for almost everything when you hear invertor based resources this is what we're talking about um the number one supplier in the world is still Huawei so that's good um followed by a number of other Chinese vendors who are essentially supplying the majority of our Market currently um we're in a interesting geopolitical climate right now that's not necessarily a good solution for our grid in the US currently and there's also about 80 plus battery vendors um and none of those are made in America so here we are um again the majority of these in veritor vendors are actually

doing decent work on quality of their systems I'll talk about that in a in a minute um but there's various features of the systems that lead to some well perceived and real insecurities to do with remote Communications how we patch but this is my are we ourselves in the foot with all of this um this is a graph from 2020 um it was in Bloomberg I believe and it was announced as the US's numers is second in battery manufact in the world and look how great we're doing we're we're we're second and yep we've been lapped eight times in Olympic terms but you know we're we're second well done um there's this isn't a race we're

going to win at this point not in the near future not tomorrow not if we took everything out we're not going to win right now we have not put the money into this this manufacturing race is extremely far gone at this point so we need to come up with other ways to work on this this is just an unreasonable expectation that this country will somehow make up that much manufacturer in the next 3 years so this was kind of the the I'll call it the Canon event but the set of events that led to me being very very busy over the last few months um around April 2023 a large military base that's named on the screen launched their new battery

project that they were very proud of it's pretty cool actually um this lovely selfie was taken with the command of the of the the site if you see the name behind their heads one one piece of advice if you don't want a lot of attention don't take selfies with the Chinese battery in your military uniform because a few people are going to get upset so they took this picture and there's there began a series of things that were going on um if you haven't seen this it's kind of well known that cattle this battery company are one are the biggest manufacturer in the world they are well known for this in the renewable energy space turns out

they weren't well known in the cyber space and everyone was pretty worked up about this situation um then we get on to I believe December 24 and uh one of a very large utility that happened to be connected to there basically announced that they were removing that battery disconnecting it getting rid of all their batteries from the system they will not have any more Chinese batteries connected to their system cool they put out an RFP um the RFP to find new batteries got no answers because or it got answers none of them were eligible for what the requirements are they'd put on there because people finally realized by the way guess where all of our supply

chains from it's inside it's outside it's everywhere and so this kind of a a moment for everyone of our solution isn't we're going to rip and replace this this is a whole other plan um around that time also in the new National Defense authorization act for FY 24 there was 10 battery companies that were named it was part of in the middle of all of this this has happened and that was part of being removed those 10 battery companies make up our entire supply of batteries at this point so um essentially if we all followed what was in there it was going to be complete stop of the energy transition the stop of grid mization the stop of almost

everything that we're working on um again um DOD gets up and says to various people no we don't have any more of these batteries in our system we're okay like 3 weeks later I think it was Edwards Air Force Base then announced they just got their new large um connected battery energy storage system online they were using byd batteries that was another one on that list well done guys really good at publication here I do not know who works in comms for these people but they need to read like what's going on in history um great so now everyone's screaming for rip and replace they're screaming in Congress they're screaming all over the place and

we've got in my job we have utilities come and talk to us and different vendors to say like can you help us how do we deal with this and we're like well we're not going to help you rip and replace it that's that's ridiculous you'll actually lose reliability so roll on we get to now um a few maybe about a month ago um everyone members we did ban Huawei we did lots of other things the ex did the US invera market still became number one in the world um during that time essentially we DOD were told that you have to remove it from all of their systems and about a month ago they said hey can we get a waiver from that rip

and replace order because uh we can't actually remove them from all of our systems so rip and essentially my summary here is rip and replace is a terrible plan from a cyber security perspective also CU we just can't do it cuz we'll lose power in various places so great great let's build a supply chain here how long is that going to take we have put $30 billion into this already to build new systems build P Electronics here it's going relatively well one of the big things we need is chips um so there's chips for America that is building just now but it's not going to be in time for the next few years or for the next 10 years um the

battery material was a lot of the focus critical material goals act came in um we then changed that I think um SMA solar has an a very large invertor Plant to be built in the US but at the same time nase who's the other this they kind of make up the 2% of things that are made here um and phas actually announced they were going to close a plant here which is the other big inversor manufacturer and also um sunpow who was like the The Golden Child of solar energy and manufacturer for a very long time they fell for bankruptcy I believe yesterday so um we're not gaining traction on this manufacturer again this is not to make everyone sad we're saying

we have options but we need to actually work at how people are going to apply these because building it here is not necessarily the solution so just moving on what are we actually seeing from technical risk on these systems like why do we care that invertors are built in China um well we're seeing Trends and vulnerability in these are not great um and they're also very hard to fix whereas a US invertor company might stand up and say sure we'll fix that vulnerability thank you for disclosing the Chinese companies are not going to do that so we're seeing weak credentials persistent Communications to home home not being a US base um there's lots of web page vulnerabilities with

the applications essentially these things have problems they're getting better but they have problems there have also been ransomware attacks um again it's not necessarily shut down any power generation but again ransomware doesn't need to do that um has caused massive Financial challenges with these um remote monitoring has been lost in a number of cases again we're seeing challenges um there's been about 17 different events that we've seen in the past that are reported in the past um four years I believe um and they're not getting better they're speeding up the FBI just released an announcement about hey by the way people are targeting renewable energy people finally worked out as there so technically though some

of what makes these devices a bit of a risk is also a benefit to the system we can't necessarily update firmware on these um you know remotely you can't go to millions and millions of devices and plug something in unless you're a crowd strike um but sorry but you uh there's there's been problems there have been invertors that have had software challenges that meant they've had a shutdown or something has happened again that wasn't a cyber problem it was just a straightup software problem um a lot of these things were fixed with a hot fix where they push software remotely to the devices so that they wouldn't have the same problem anymore and so some things have been fixed with

software changes that's still a risk um the other interesting issue is the quality of our Bassy sales again remember I just said they're all from China the quality has actually been increasing over the past few years we used to have a problem with fires and you see lots of pictures of battery fires I say we used to because there haven't been as many they're well publicized when they happen but there haven't been as many nationally or globally um and a lot of that is attributed to just improving the maturity of the supply chain funnily enough it's a attributed to a lot of these companies collecting data from all of their battery systems through other Communications devices and improving the

safety of their devices over time so we're kind of stuck if we ban these 10 companies or we ban other companies we end up just going to another 10 that are popping up and they probably have a worse fire risk we do have data that some of them do so uh we're kind of damned if we do damned if we don't at this point cuz you know fire risk will be higher again what do we do do we rip all out and start again or do we operate through this because we need to actually design this better future but also exist with the supply chain we have currently because we need to do both of these

things to both help us generate power in a clean way so we don't end the world through climate change but we also you know have challenges with where things are coming from so again everything has its place one of my messages of this was communication right now if you follow any doe or DHS things in the US there's this kind of battle going on between secure by Design and cyber informed engineering who's going to win both they both need to win at this point there's a solution for both we can't necessarily tell a company another country to do secure by Design we can secure around it so how do we do things that people install these properly um we have

designed programs for this getting people to use them is a bit of a challenge again I i' built this just to explain for another presentation um how do we actually talk about integration of components um how do we scale risk not everything can be done for every component it depends on the criticality of where it is um I mentioned earlier the different sectors that are getting batteries renewable energy as backup generators one of the big ones is actually hospitals rural hospitals in particular are funded about 170 of them got money recently to install batteries as backup generators with a goal that they would uh remove their diesel generators over the next few years and be dependent on those batteries so um

when you look at how they're going to buy those they're going to be somewhere in the middle of this graph be it fully Chinese Supply or fully us Supply um there's no such thing as a fully us supplied device so we're clear that's why there's a unicorn Philly floating around on the right hand side um but we do have Solutions I won't necessarily cover what all of them are but again my message is how do we get these out to people in a reasonable way prioritize the right way so that people aren't just chasing down magical devices all over the place um when people announced like oh hey Vol tyone you need to go check

that you need to do this there's Utilities in the middle of the country that have diverted every single resource to we must solve this problem because we're going to be attacked in the near future every single resource on cyber security is diverted to this which is about three people which is not that many but um well that's true um but there's they're all focused on this and in the end there's these things coming at them from all angles that they're not seeing the whole picture because they got so hyperfocused on one message so scaling solutions by criticality is one of the solutions we look at again what are you actually trying to do with it

what's Mission critical what the consequencies before you install it and how do we actually get there again Living With the Enemy here because this is where we're at um how do we get the best of both worlds enemy also being climate change we do not want the world to end we don't don't want it to end for our children um we are deploying Solutions there are solutions in place this isn't impossible this isn't a Sol a system we cannot fix this point we can secure around most of the most consequential issues we can secure around we can't get people to actually do it though so this is what we're trying to do is how do we message this

to people that they'll do the things that are simplest to get this done we we can deploy them we can do secure by design for the future we can cyber to nearing it we also need policy to use a lot of these um we can't necessarily inspect all the devices that come into the country because the policy doesn't allow for it we need policy but communication matters here this was something that i' mentioned at the beginning um we need people to be prioritized on the right things for the electric grid just now again most cases weather is one of our biggest problems them prioritizing keeping the poll standing is actually kind of a big deal

just now poles fall down all the time they need to keep that up they cannot divert all their resources but it's a systemic problem it's no individual component no one solution no one solution fits all but communication is really a big deal I'm going to implore anyone to just talk to the engineers in the field about how they fix things um I see a number of people talk about oh if this cyber events happens this whole substation is going to blow up I'm like there's fuses like there's things in the substations that tend to protect us from the biggest events and it stops people looking at the other events that would be helpful but so raising the bar in

technical communication is my request to everyone again I wanted to do this at the end I'm not sure how much time I have um sorry just checking he told me to speed up so kidding so I did choose your own slightly upsetting Adventure cuz I don't think I can cover both of them I called it slightly upsetting they're not as upsetting as they could be there's no one doing CPR here um but I gave two so I want a vote here of what if we crou stried millions of invertors what would that look like talking through an exercise that had done before and the second one is what if we uh needed to Black Star the entire e

okay I you know I said the word Black Star so Dem and Gloom okay I'll go to this one so I presented this last year um it was talking about a real event that happened in Livermore in California where we'd had um essentially a massive lightning storm it was 100° in California um California system operator had called this rolling Blackout um we lost a large power transformer hey guess what those are all from China also so it was going to take months for us to get a new one um the Wastewater Plant had also failed Livermore was falling apart at this point Livermore who which also um supplies I believe to a military base and a large National Lab that happened

to work at at the time the fire that had been triggered by the lightning event had also started became the largest fire well it might now not I say largest fire yet since I think this year there had been a larger one than that one it was called the uh lightning complex fires in California it took out a lot of things um but at one point it turned around wind is weird in Northern California and started to actually head straight for a National Lab and a military base and so we were kind of in the middle of how we going to evacuate this system how we actually going to do things something I didn't talk about last year was there's

two large supercomputers sitting there and one of the things that was happening is we' had to ramp down all of the supercomputers because hey by the way we're losing power you need to get rid of your giant large load bank that's sitting on the system also meant we had to co-op a bunch of actual analyses that was going going on that was a national security thing um everything had to be moved because we had to ramp down analyses that was going on for other things cuz the computer was needing to be turned off um so yeah it was impacting other things again and then my running joke was though some point in that summer they also announced we're

going to get rid of all gas vehicles in California um you're going to only have electric vehicles people in this town that were evacuating hadn't had power for 3 days and they're like what what we going to evacuate a town from a fire when we've got all electric vehicles that haven't had power for 3 days like you going to bring us a sko bus like what's happening so it was a really kind of strange event that was happening but it made me think about what I'm talking about here like what if we add cyber um so from a black star scenario it's one of my favorite things to talk about so I'm happy um black star is when we've

everything is black we've lost power in large sections of the grid we've lost large major generators um effectively when we talk about we're usually talking about the whole country but let's consider just one large section of the country to make it less dramatic what happens when that has it has happened we have had to do black start scenarios we do plan for in the Electric System we don't necessarily plan normally for cyber to be part of that scenario so we're all there um we ran exercises in the last few years one's called Liberty Eclipse there's gridex we know do plan for cyber to be there but it's still not necessarily as embedded in the field Crews and people

that will be doing the work um when we ran an exercise in I believe Plum Island um a few years ago for Li it's not called Liberty e clipse anymore but we ran an exercise one of the first things that happened is one of the fuel crew that was out there trying to simulate this black start happening with cyber security we told them the hospital funnily enough was not generating their goal was they had to get power back on for the hospital in this exercise this Fiel crew guy runs out and he's just basically screaming at everyone cuz he was angry um they tend to get that way sometimes but he was determined that his One

mission was that hospital needs to come on so he started to do the things to make sure that would happen cuz he wanted to win the exercise also so he was doing those things and there was a a vulnerability embedded in one of the relays that was basically triggered by him turning it back on to Brick everything that was related to it so essentially he bricked half the system by doing this and we're trying to explain you need to bring a cyber person with you to do this now if we think it was a Cyber attack you should probably bring someone with you okay don't let them touch the wires but like they can come with you it's okay like they they

just they can be trained don't worry about them but that was one of the big Lessons Learned is they need people to come with them to do that if we do suspect there's anything to do with cyber security involves here because people are mean and so when we have a black start normally I say normally but it's a it's a pretty bad event a Fiel crew will have to go out and assess damage is probably caused by some form of damage at this point the people in the bucket trucks are staged um if anyone works with David he does a lot of this on helping stage for large weather events um they'll be staged outside um

outside the weather event they will go out and they will determine how much is damaged what they need to fix what they need to do and do an instant response plan to get things back online they then go back through they kind of they reconnect the big lines and generators they go through any major load blocks and then they start restoring and resynchronizing all of these big things together just imagine resynchronizing lots of things in little Islands until they all talk to each other and match up properly takes quite a while it can take days for them to pull this back up depending on the size um so it's a big deal when they need equipment there's

usually these large yards of Transformers sitting there um we like to think of Transformers is this kind of mysterious thing there's yards of them sitting around you can find them they're just sitting everywhere um you'll find Transformer yards pole yards they have equipment staged they know how to do this this isn't a mystery it's not not a problem we know how to do it now so my scenario I wanted to bring it all together with is our future of all the things I've talked about of everything that's now integrated into the system and what that means at one point we will not be able to go back to this being a manual system there will be no option

for us to manually restart all of these things we'll be stuck I say stuck it will be good but we'll have a more digitized automated system so um I had a picture at the beginning of the robot in the substation one of those robot dogs that is a real thing thing now they're sending robot dogs into substations it's kind of weird um but they it's for safety as well they can go in and be less likely to be hurt by something in that substation so they also can collect more data they can do a lot more things they can assess what's going on quickly automatically so imagine our robot dogs are running around connect collecting

all this data of we've had the power outage assume it's happened we don't know necessarily all what's caused it there have been a number of cyber events linked with weather events so we have to deal with both um so they're running around their robot dogs okay where are they going to send their data to actually do the assessment if they don't have power in that whole area have they worked out a plan that that data is going to go to some other data center somewhere across the country we do have plan with that in different ways what if the whole country wasn't there okay how do they even communicate that data out to do this AI assessment of what's

actually damaged and come up with a solution again I'm not clear right now but imagine also in a very practical way the batteries on that robot need to be charged for at least 3 days does everyone plan their robot dogs are charged for 3 days so that we can actually get the data to restore the system again we then need cyber assessments and root cause analysis um great again still needs data to do that so how do we get those teams together we again need the data sensors for processing and damage we need them for the remediation tools also need communication systems with througho capability that can actually take that data and do something with it um and

those need backup generators on them also which might also be those batteries and those solar plants that we're talking about that may have cyber risk on them so who goes out and checks those are okay um again we need to determine where the load is from a locational priority um depending on the size of this National outage we're talking about did that data center that we need to use have backup Generations that's actually working that wasn't that battery that we're worried about that we're now talking about maybe maybe not um we can start going through other things at this point there's things also like okay we need an additional yard for p electronics and technicians that are

going to work on all of these things because we need digital systems to be installed and that isn't the field crew who are hauling the wires around normally that's a different set of people people that look more like everyone in this room than the normal field crew again field crew can look like anything I have been field crew but normally they're a set set type of people working on big wires so do they have an additional yard for those things again we've got the physical equipment needs and then we can go out to things like actually dispatching these cyber and physical teams together so are those cyber people trained to be going into substations going to work together on

these things um do they know not to lick the wires I hate to say this it's a running joke for me I'm both of these things we don't know I don't trust a lot of people in substations so there's that so can we dispatch them together um and then again can we call in cyber Mutual assistance this is something that exists in the country now but that's when we would start to need that we don't just need storm assistance which is well done we need cyber assistance do we have enough people to even turn up to do this after that we do get to the sort of resyncing the grid imagine we've got through all of these situations trying

not to scare everyone but then we also need to resync all the cloud the data the analyses everything else together so the system stands back up in a way that is actually functional I'm mildly terrified by this scenario I'm not sure I'm excited as a scientist that gets to do research on how these things will work in future because it's a problem I don't think is solved but I'm looking at this kind of big picture scenario is that's a lot of things to deal with when right now it's really kind of turn on the lights so yeah um we've got some interesting challenges that are all linked together um again don't have solutions for all of them looking for

help on those things and ideas and sort of reasonable thoughts of how we're going to put together this future grid as well so the everything we've talked about today can still have power there's another adventure but I'm going to stop there cuz I think I'm getting the look of okay it's 7:00 nearly but uh any questions discussion is everyone terrified I was hoping for

that okay I see some questions and I will run to you carefully without tripping

the crow strike one okay fine um about battery suppliers all coming from China are there uh is it a financial incentive problem is that why like GE doesn't want to do it or or or even like uh Seaman or one of our one of the many amazing companies in Europe so it's partially a material problem to begin with cuz shipping material across the world is expensive so using the material to make the batteries in the country it's being bought from is one thing um again it's incentive as well we we didn't incentivize this 10 years ago we incentivized cheap high volume products being created for the future which is good because it meant we got there um

but G seans big companies are developing things here it's just it's going to take a while and not every component is necessarily the priority if that makes sense I would prioritize inverters before i' prioritize battery sales but that way okay so all right so the chair of the session has requested that we go through our next scenario and after we do that then we'll do some questions okay so this is the one I call what if we crou strike all the invertors so it's the best way for me to describe the scenario of what's Happening um I ran a small tabletop or a beginning of a tabletop at the S4 conference last year that was addressing some of this cuz I

wanted to put together a team of people in a room that included a utility an invertor manufacturer an aggregator someone that owns all of these distributed resources a group of people and cyber teams to say like what if this happened didn't realize something like crowd strike would give me an great example for this later on but here we are um and so the scenario I presented was and I wanted people to think about was this fictional aggregator someone that Aggregates millions of devices in the country all of your behind the meter resources that are on your houses one of ways financially they survived doing that is they aggregate them and sell that all back to the market and you

you get to benefit from that thing on your house generating renewable energy so the scenario I presented was imagine we have that very hot California day um where the grid is strained we're having some challenges um the independent system operator in California is calling for this conservation they're sending you the text messages that you're ignoring and they're asking for emergency discharges um they're saying that there's a virtual power plant there as well which means said well really doesn't mean anything it's a new buzz word that they're using but there's a virtual power plant a cluster of resources operating together in one city and they're they're concerned that there's a this aggregator that has a challenge I called them a

ruro power if you get the joke for me being from Idaho National Lab then haha but um they're a fictional aggregator so when I ran this it wasn't called sunsat I'm really proud of this and somebody said this should be called the Suns spat inverter so we created a picture that is now the sunsat scenario um which was imagine those these millions of invertors across the country have a vulnerability that's been identified they've said it's it's really bad it's level 10 cbss it's published for all of them we know it's there it cause causes a problem through your entire fleet of invertors which is millions of devices great um The Brick of all those devices imagine that

exploit could actually cause a brick of the device or just turn it off so both the market they're participating in and potentially the grid would be impacted by the loss of this generator that's essentially across the whole country but there's millions and millions and millions of devices it's not just one so that brick would cost an aggregate or $26 million to actually fix which in lots of Aggregates or cases could be bankruptcy so you just lose the support of them um and then sunsat the invertor company has said well the latest firmware is possibly the issue funnily enough imagine that company Sunspot is actually a Chinese company they're trying to communicate with a Chinese company about a vulnerability that was

disclosed in the US that somebody wasn't particularly happy about being disclosed but they've identified a fix um the fix is pushing firmware to all of the devices um remotely from the main aggregation point or from the manufacturer um they think comms will be maintained while doing that they need to do that the reboot will take 10 minutes it'll take 5 minutes to reconnect great we've got a solution what do we do then so this was a scenario we presented to all of this group um be a utility an aggregator everyone involved and everyone kind of just looked at us like what do you think would have like what like not a single person in the room had

a version of the story that seemed to make sense for how it would fix this um a bunch of people are kind of looking like the utilities like well it's nothing to do with us I'm like what the aggregators also looking at me like that sounds bad like yeah yeah they were the Cyber team and they're like we don't want to deal with this I was like yeah doesn't sound good so basically no one seemed to know who would be fully responsible for this and I think this is one of the the bigger challenges we have had vulnerabilities in these systems so um so that was the scenario presented and again we ran through this step by

step and I made it slightly worse each time and I'm not sure I included the last slide but I made it slightly worse each time so the scenario we presented at the end of this was okay they pushed out everyone panicked put it that way we got two days into this people were starting to exploit that vulnerability that was being talked about it was causing problems they knew it was happening and there was a panic because somebody said this is going to take down the whole grid you need to fix it now I'll call one of the Cyber companies said by the way we're going to cause nage in the whole system we can fix this

but you need to push it immediately so imagine that company did they pushed out to everything all at the same time that's never happened right no no um yes so that's why I'm calling this the cro T scenario so imagine it then bricked everything every single system needed a manual reboot that was one scenario and they were like could that happen I was like technically it kind of did happen once um a firmware push was done in a to an island we'll say and it did cause a whole bunch of problems because it was pushed in the firmware didn't work right so to every single invertor on one Island so again I'm looking for people

to to volunteer scenarios here but we were trying to work out so so who's locally responsible like physically we're those invertors that who's responsible for this like who needs to go and do the restart of these systems or find out a way to plug in some USB and reboot this system um cuz they need them they can't just replace them all one scenario I did present was there was a cluster of devices that worked together that did cause a power outage because they were working together sitting in one area there was no it was a monoculture of invertors sitting there imagine there was actually a power outage caused by that they blew some fuses very hot day someone died somebody

had medical devices there who's responsible for that like medical devices in your home who's responsible for somebody died because we pushed a firmware out and how do we do this Mass reboot that's the other like how do you go and ask every homeowner to do it do you ask them to restart at 16 times and hope for the best like what's the what is the scenario here cuz I don't know that many home owners that would go and do this so is there cyber teams going to come out to 25 million endpoints to go are this um if anyone did see the Delta staff flying around to different airports last week that's not a great

scenario either so that's again there are ways to do this but we just don't have a plan um instant Communications who actually talks to each other like is it between the aggregator distribution utility the electricians law enforcement who's who's dealing with this um how do we know isn't a bigger issue is the distribution utility now responsible cuz they BW the fuses um what would be expected of this inverter manufacturer like what are they expected to do do they write letters to the news do they turn up to fix it and then what can we actually do what regulatory or legal requirements could actually fix this so these are the kind of questions I mean again I'm going to

ask you all questions if you have ideas or thoughts on this I'd love to hear it cu the group of intelligent people that were in that room all kind of gave us a look of uh and some people made some nice pictures of sunspot invertors for me so that was nice but but thank I was just curious

how so the general can you repeat the question quickly what happens with Communications essentially during a black start do we have all loss of comms how do we restart that if we need it um right now the hope is our communication Stars would still be up because we'd have backup generators attached to all of the com stars there are alternate Solutions as well there's things to do with radios relays there are solutions for communications and again a lot of them depend on those backup generators also working at the same time so um there are solutions that but again it depends is the answer I think is and I will say if I stare at you I am

slightly desp so masks make it really hard for me so do you don't need to take off your mask I just I'm if I stare at you so I my qu oh my question is why isn't this more common I mean I'm I'm thinking for instance let's say you know heat event let's say I'm going to throw out names let's say every Ecco bee or every Nest all of a sudden glitched and turned on the heat pumps or something like that that could cause this kind of thing with it it's not just the inverters that have this problem right there's many other things at the at the load end that have this are we I

mean what's in there to stop that right now and are we seeing that happen and we're just not hearing about it there's various safety controls in place right now that some of them just won't go past a certain temperature level there's manual responses so on inverters for example one thing someone said once was oh could they make the voltage go super high and everywhere and just cause a fire technically we have over voltage protection that should work separately so there are alternate Protections in place for a lot of it but doesn't mean they're always going to work as we keep growing this and S start to lose those manual protections as well that's why when I say it we're not quite there yet

but we have time it's because those manual protections are still in place just now but as we get to massive amounts of renewable energy load control all those things those controls start to become a bit more fuzzy as well so cuz we start to need those to be automated as well like the manual protections need to change based on other things at that point so yeah I'm wondering how much were state agencies involved in this because this is the type of scenario where the National Guard State militias local La en for ment FEMA would be rolled out and were they involved in the exercise and they have any yes any input they were sitting there and they said they wouldn't know

what to do with the inverters and they'd have to have training in how to do those things they it's they weren't sure what they would be touching from an electrical standpoint the training wasn't there yet there also isn't necessarily all of the powers in every state to have that happen and get those controls there um the other thing from small Utilities in particular the person that is now National Guard or state and local responsible is often the single cyber person also sitting in that utility so there's and the very small ones so there's kind of a weird Catch 22 for them in a lot of this as well so yes there are people that know those things

exist and can know how to make them action that they were still we um actually when we ran the Plum Island exercises as well we had a lot of National Guard members there there was kits made that were meant to train people in how do you go into a substation and plug in things in the right place and don't touch the wires basically um that still needs to happen nationally it's not there yet so can get there that's a good solution but we need the training to be there I think so just want to make a comment on um some of the conversation been talking about right now sorry it's right here yeah and and

and there is um vast lack of understanding with some of the process right you know in this room if I bring out like Dr Bia BCP right so business you know impact ass assessment were analysis were you know business continuity plan right all those exist right how many people are actually from a cyber perspective put into uh you know focus on those areas having those organization you know done a Bia having a solid you know Dr Disaster Recovery plan being established and and really put on the focus and and and looking at this change as a non technical problem but also a business problem how can we the other business come in to work together and because it's very difficult

if we're just deploying Solutions and just turn that into a technology problem rather than a process and a policy uh change right that's no I I agree that should be in people's business continu business continuity and in their plans um that's that's one of the I don't see simple but that's one of the solutions I just I I hope to try and communicate to people it should be there for future so so so now that they know that they have a problem do you know if anybody if these companies have started to work on a solution at least two or three have that came to talk to us afterwards um um there are plans there are things that

are changing they're trying but there's ones that will never like 10 of those inverter companies are never going to come to work with a US National Laboratory on how to fix their invertor vulnerabilities so there choices of how we're going to do this I think is the so uh this will be our last question and then we're going to talk a little bit about tomorrow yeah thanks um I really kind of hesitate to bring this up as a potential solution because sounds like a great question but all right so I believe Underwriters Laboratories has requirements for testing you know uh they have requirements on uh inverter safety um so now I'm thinking about things like not just like cyber security

requirements but now like maybe having some testing requirements that show like the reliability of firmware updates because if it's got if it if you need to do a fir more update maybe there needs to be I really I really hate to bring this up but you know some sort of like testing regimen where they actually bring you know they run multiple firmware updates and put in like various like fault scenarios and make sure that they can recover correctly I know this is something that we've kind of like voed on the private sector to you know we want the the Manu device manufacturer to do all that stuff but in the l l of the fact that they're not

I don't know it's just a thought so UL 1741 I hate that I know these numbers I really do um is the safety UL T testing we have for invertors interestingly right now we can certify Chinese companies and other companies to do that testing so from a safety standpoint I that's great testing their things in situ is a good idea like doing this that right way from a cyber perspective given geopolitical concerns we do have have something being developed called UL 2941 that is a um it's the beginning of a requirement whether I believe in it right now or whether I believe it's reasonable that's a whole other personal opinion it's in development it will be something that

could do that um my personal opinion right now is we need to make sure that's tested in the right places where we can certify the testing itself as opposed to just certify the device if that makes sense but there are that is testing these things again how do you persuade some of these companies to test it without we need a regulatory requirement to do that as well which will increase cost which will yeah is a cycle of things so this up because the iot world is kind of like stuck in this yep okay um please join me in thanking Dr Emma thank you doctor