Power Play: AI Dominance Depends on Energy Resilience

Besides, I am the Calvary. My name is David Bots and I am here with uh Mr. Josh Corman and we are delighted to bring you this very carefully curated track. We are going to be talking about a number of topics today. Yesterday, we can't go back in time, but let's talk about today and tomorrow. We're going to be talking about a number of topics, some of which might lead to a level of discomfort. So, but that's okay. We encourage you to sit in your discomfort a little bit and say, "Ah, this conversation is making me think big thoughts." And that's good. We want you to think big thoughts. societal impacts, personal impacts, family, neighborhood, community, all of

these things. We'd like you to think about that because this is intended to help you think about not just the future, but also your future, your family, your neighborhood, your community, the the place where you live. That is the goal is for you to be thinking about all of these things. We have put together, we've assembled a variety of different people talking about different specific issues regarding the overall theme of I am the Calvary. And if you don't know what I am the Calvary is, I will say it very poorly compared to my co-chair, Mr. Josh. Um, no one is coming to save you. Nobody. If you're waiting for somebody to come and save you, you will be sorely

disappointed because they're not coming. Nobody's coming. Do you know who who can save you? You and your neighbor and your neighbor's neighbor and the people you bring along with you to help protect you and your household and your neighborhood. That's who can save you. Okay. Um so we put together uh a series of discussions to have you know some big thoughts. So today and tomorrow we'll wrap it up and uh at the end of the day tomorrow morning so we've got a forget if that's 90 minute two-hour block we'll we'll Mr. Josh is going to bring it all home and talk about Monday and Tuesday and Wednesday and wrap it into a very elegant package. So we encourage you if

you can to hang in for today and tomorrow just hang in. We're going to be talking about some very really interesting things and community things that are important. I am delighted delighted. Please step on stage a delight. Please join me in welcoming Dr. Emma Stewart and Manish Walther Perry and they're going to be talking about their own bios. So, I'm not going to I'm not going to bore you by talking about this twice. But I tell you, both of these people are amazing. And you're going to learn more about the electric grid and AI than possibly maybe even you wanted to know, but you should know this. And one of the things that's going on this morning is within the the

theme of I am the Calvary, we're going to be talking about certain externalities that may affect how each of us respond to the current emerging threat landscape that all of us live in. So, please put your hands together and join me in welcoming these two fabulous presenters. Over to you, Okay, making sure I've turned this on because that's my first problem with electricity apparently. >> Um, hi there. Uh, thank you everyone for having us here today to talk about this. Um, as David introduced, um, this is obviously a talk in I am the cavalry. So, we're talking about what's been going on with the electric grid and how that applies to lessons learned, what the most emerging threats are. When we

started to put this together, we actually had a slightly different theme. And then in talking with Josh and working out where the direction of this was going, we really came up with sort of new direction for this presentation and to talk with you all about how AI is actually impacting what our biggest threats are to the electric grid. Um I've been here a couple years and given where we going in the electric grid talks. Um that's usually my this is where we're going and um you'll see in some of here like the direction has changed drastically in the last year a way that I never even fully understood myself was going to happen. And so

that's really what we're going to talk about and also what we think people could be doing um when they're out working with say small electric entities and others to help improve this and what we need to think about now when we're out there. So, tale of two grids is the idea here. And so, you have me. Um, I am Emma Stewart. I have worked in power grid my whole career. If you want to know how long that is, and I make this joke a lot, that picture on the bottom right is me in my first utility job many, many years ago. It is an actual Polaroid from when Polaroids were really a Polaroid thing and not like a new

fashion item. So, like it's so that's if you want to know how long I've been doing this, that is how long. Um but I've worked in power grid my entire career. Everything from uh working in a few national labs over areas like hydrogen defense, electric grid, cyber security. Again, always focused on how the electric grid works and things that we're adding to it, doing to it, using with it as well. Um I've also done all sorts of things to do with at one point I wanted to be a race car engineer. I just like to throw that in. And then I changed my mind because I decided uh it'd probably be a more lucrative industry for me to work in power than to

work in race car design. So off I went into doing that. But I also have Manish here who >> Hi everybody. Uh my name is Manish Walther Pori. Um and I'm here primarily this is my first time on the stage. So I'm here primarily as Emma's hype man. Um if any of you are fans of rap music, there's always like the primary performer and then there's a person off to the side who just sort of throws in stuff. That's my role. Um I'm a lot of style, a little bit of substance. Um, I've worked in risk pretty much my entire career in some way or another. I started my career in nuclear policy doing now what's now known as OSENT.

I've worked in geopolitical risk, terrorism, fraud, and of course cyber security, critical infrastructure, and supply chain. Uh, one of the best opportunities I had to really understand these kinds of challenges at scale was also one of my um, uh, most meaningful roles where I was the first director of cyber risk for the city of New York at New York City Cyber Command. So, I will give the disclaimer that I'm speaking here even though I no longer employed there. I'm still speaking from my experiences here and not on behalf of that organization or the city of New York. Um, and I've gotten to know Emma because she's along with many people in this community and and I am the cavalry.

Josh and and Dave actually was one of the key people while I was at the city who have helped me understand the complexity around energy and we'll get into that but I hope to share some of that with you as Dr. Stew's hype man. So glad to be here. >> I was going to say you also put my name on the abstract and then told me I was presenting. >> That is true. That is how that happened. That is precisely how that happened. Yes. >> Well, so in the last few years, um, we've talked a lot about the word cyber poverty comes up a lot talking about people that have and have nots in the

cyber community. Interestingly, I called this here we go again. Um, because in the electric grid, we're also seeing this. This has always been a thing. We've always had energy poverty also. But now we're witnessing and this was a quote from the new Emerald AI co recently that I really liked where he's basically saying we're witnessing the birth of two energy systems where essentially we're we have got one that's integrating massive amounts of load. We're able to serve things on big data. We're doing huge amounts of work for that. Then we also have a grid that's basically failing the basic needs of humans at this point. Um we have lots of power outages caused by many things. And

there's also a huge risk that these huge loads that we're adding to the system and prioritizing getting on there for supporting AI are becoming a threat to the average human experience on the electric grid as well. So I thought that's a really great quote to explain sort of why this talk fits into the we're witnessing two grids being birthed at this point and we're not sure which way it's going to go but it's not necessarily going the best direction. So this and I've now lost my microphone so >> hold on I will just hold this. It's much easier. Um so when you walk around Black Hat for example just now or walk around this um any of the talks that you're

seeing about AI a lot of what people are talking around is defending from AI or defending with AI. So that's the the two sorts of worlds we're hearing about in if you walk around the show floor of Black Hat for example that's all you'll see everywhere. Um but what we really wanted to talk about was just defending AI itself in particular looking at the infrastructure on which it relies because um if you look at all the people that are developing these real time threatbased how we're going to respond to all of the threats they're defend they're depending on AI and therefore on the infrastructure on which it is based. So when you have models that are running

in cloud-based infrastructure that's on a data center that funnily enough also needs water and power itself to be running, we need to actually defend that infrastructure for it to be useful as an emergency solution. Um we also are looking at the defending the infrastructure that everyone else is relying on from the AI infrastructure in that all of these things that are the giant large loads um they're essentially there and causing problems already on our electric system. So we need to also help defend the infrastructure that everyone else is relying on to serve their everyday need from the AI infrastructure itself. So that's kind of the theme here is this is a cyclical problem. We're relying on AI for things

that are relying on the electric grid. We're using AI to support the electric grid and then we're also potentially going to destroy the electric grid by integrating lots of things that might hurt it. So it's kind of a cyclical problem that we're in. So if you can the next time you hear AI add on and the infrastructure upon which it relies and see how that sentence ends. See how the story goes. See if they're talking about it. They probably aren't. So just quick overview. We're talking about the new frontier for the cavalry essentially is the AI and load growth that's happening on the system. When we talk about electric, we can no longer separate these two things at this point.

Um AI needs load, load needs AI. That's it. Um but we have two grids. One is focused on the load growth, the other is ignoring the basic needs of every human at this point. Um, our threat models are cyclical. We're dependent like I said, but what can we do? We are going to cover a what do we think we can do about these things. You've seen some talks already on things like cyber informed engineering. There are things that we can be doing to make this more secure. But the urgency and one of the reasons I'm having this talk here is this is now. This isn't five years from now. This isn't even 2027. This is happening

immediately at this point. So we have to look at ways to solve these problems as a as a community now at this point. So so 2023 just to explain where things have gone. I gave this talk hopefully it's showing up okay >> uh just on where the electric grid was going and where it had been. Um this was from my slide. It's now on a dark background because I had to be more hackery. Um but the uh I gave a this talk where I talked about things going from I I mean 20 years ago we had the dumb grid it got smart it got clean like over the last 20 years we've moved through different

problems every few years and come up with solutions interdependency we think we're doing great then we spoke too soon now we have cyber security as a common word then shields up sort of moving on to where we are now when I gave this in 2023 I was actually talking about we have huge infrastructure investments happening in the electric grid and there was I think $1.2 two trillion dollars was going into upgrading our infrastructure at that point and I'd said now what we have lots of money what are people going to do with it so move on um now we're here I have a now what now this is my my slide for now two years later giant large loads everywhere

AI everything and energy abundance and to understand this is a complete shift in understanding of how we plan the electric grid for 20 years if the one thing I should have had in this flowchart We actually planned for things like energy efficiency. One of our ways to help help grow and help the world continue moving forward was to make things more efficient. So we actually used less energy in our system. We're in the opposite space now where basically if you listen to anyone from Department of Energy um or from the US government, we're saying energy abundance, we must grow. We're going to use more. We're going to have lots of energy for everyone. It's going to be affordable.

It's going to be great. But there's just going to be tons of it everywhere. And that's new for me as well. And that's not how we planned the system. We planned it for um a series of uh um what's the word? Margins essentially that we're not this this is not what we planned for. So here we are. This is a fundamental shift in our understanding of the electric grid. So just for anyone that wants to understand why I'm saying these things, I had to do a electric grid primer for just very a couple basic concepts for people that don't deal with this every day because I've been told I sometimes have a habit of just saying

big electric grid words and then people stare at me. So um it most basic level in the long-term planning of the electric grid, it's balancing load and generation. We have the loads, we have the big spinny generators, we balance those out to maintain the frequency of the system. So essentially they must be equal for the system to be stable. At this point you must serve load and generation together. On the short term um we have when we talk about load and generation matching on the big numbers over a long time frame we're talking about the capacity of the system. And so when we say load growth mostly we're talking about capacity growth of how much we serve. Um when we talk about

problems with things tripping off say somebody's talking about their we hacked a system and managed to trip off this big generator. The problem they're talking about that causes a cascading failure is really around frequency, which is a short-term event caused by the loss of generation or the loss of load at the same time. So that's why we're talking about giant loads because these are now giant concentrated things sitting together that can cause that imbalance instantaneously by tripping off or doing something weird. So again, giant spinny interconnected system. That's the basics. >> Technical term. >> It's a technical term. I I have a PhD in this, you know, so it's geez. Would you like to? >> Nope. Nope. Nope. Nope. Can't. Uh the

three things that we want to lay out for you and really is going to lay it out. Is the thinking about the demand here. So there's the type of demand, what that demand is like, its behavior, and who supplies that demand and the nature of that. Those are the three three elements. So while it is all about demand, there's some different aspects of there that are pretty important when you think about AI specifically. >> Okay. And just to explain a little bit more why this is interesting, especially for the I am the cavalry efforts. Um when we again this is the same picture from a few years ago of the the spread of the different types of utilities in

the country. We have 3,000 of them for electric, many more for water. um they are split in ownership between investorowned meaning the sort of big money efforts publicly owned like your munis and then we have cooperatives that are not for profit. Um the while the investorowned cover the most of the customers in the country the publicly owned and cooperatives cover most of the land in the country. when you're trying to build a big data center for example, one thing you actually need is land because it's kind of annoying when they build them on your golf course, but um you need land and that's why a lot of them have been moving out into these areas and why it's critically important

for these smaller utilities to actually be thinking about it and knowing what to do from a cyber security perspective because they are going to have them all over their space. So to explain that um like to give some numbers and I Manish loves this because I apparently am now describing things in terms of the size of Scotland. So >> it's a unit of measure because >> it's a unit of measurement >> because we are in the US and in the US we like to measure things in weird numbers like hands on on horses or something I don't know and don't use the metric system. >> So this is a postimperial um measurement >> colonial. Yeah. Yeah. So, Virginia, I I

happen to live in Virginia. Um, but the entire state of Virginia serves about 20 gawatts of load. Um, they were predicting by 2030, which feels like it should be a really long time away, but it's not. Um, as it turns out, in 5 years they will be growing by about 35 gawatt um of load, meaning they'll be 1.7 the size they are currently being added to their system, which is huge. um which and I did the math is nearly six Scotlands. So, six times the size of Scotland being added to the state of Virginia in load, meaning that they also need six times the size of Scotland in generation to be added to that system to be able to keep

the system stable. They also need water and land. Um I was looking at what the largest data center that's been integrated. Again, data centers are used for training EI. They're used for other things, but this is like data center in Texas. The biggest one I've seen was 1.2 gawatt. That was about 5% of the load in Texas. And that was a single site. So these things are enormous. And 1.2 gawatt is about a quarter of a Scotland. So >> we're we're sort of joking, but not really about measuring it in terms of >> a country or people. So think about this capability displacing or a tradeoff of serving that group of people. You hear it a lot. 85%

is controlled, owned and operated by the private sector. But what the the last slide and this slide show you is that it's not just about who who controls it, but who it's serving and what the trade-offs what the trade-off is if you're serving one group versus the few and the many. And you heard Josh talk about that last yesterday too. So explaining that a little as well, if you have a data center and you're a co-op for example, and do you know how utilities make money? They serve load. So basically a load is a giant income for anyone. And Carl, please stop laughing in the back, please. [laughter] But >> you're allowed to snicker because we're

going to talk about dropping big loads. So now is the time in the talk where you can you can laugh about that. We're gonna keep saying it. So, if you didn't get the chance earlier, you'll get one later. Don't worry. >> We may have rehearsed this and had some problems laughing. So, it's okay. >> Um, >> we do serious work, but we do not take ourselves seriously. >> Yes. >> Thank you. Thank you. So, yeah. Now, I'm now crying laughing. Please keep talking. Um, yes. So, to follow up, so imagine we have a utility, a small utility sitting out there. They have a giant load sitting attached to them. um they are getting the electrons through

their site. So if something varies at that site drastically, say the AI training model steps up its load by 50% at that given point. It's that small utilities whose electrons are impacted. It's not the business model that's the problem. It's that if they have a dynamic event, they're the ones that lose all the power. And that's where their hospital loses the power. The water loses the power. Everyone is screwed at that point because of this one single item that's bigger than the whole utility itself. But they don't own it. They don't control it. They also don't know when somebody has decided they're going to train a model to create pictures of people with three boobs.

Like I they don't know when that's going to happen. So like that's a problem right now for forecasting these things as well. So >> yeah. >> Okay. So overall chip to grid risk everything from down at the chip level that's in these sites up to the electrons that are flowing onto the electric system that are caused by these things ramping up and down. I hope everyone feels guilty now every time they use something weird for AI. You should now all feel very guilty. That's it. I just want everyone to be guilty for [clears throat] all your useless applications that are just generating the pictures that are used in this presentation also. So you know there's

that. Um but yeah, it's interlin risk where small repeated failures can lead to a systemic crisis essentially. Now again, traditional planning is failing because we've gone from this um being an a sort of thing that was happening. I also gave a presentation at Blackhat last year that was around where are we going in the next year talking about data centers. I didn't predict how many there would be. So I predict a lot of things really well. Didn't predict how this was going to turn in the last year at all. Um, and then there's new models for fast-moving risks. We need better models for threat models for this happening. We need to work out both from

a physical standpoint and from a cyber standpoint, we need better threat models for these risks. And again, it's a cycl cyclical threat. It's going in circles. We are in defense of and from AI. So, just to lighten the mood, how many power outages have we had in the US caused by a cyber attack? >> Does anybody know the answer? >> And Megan's not allowed to answer because she knows this. Anyone? >> How many people think more than 10? I don't know if people think five to 10. >> How many people think less than five? >> Ah, good. >> How many people think zero? Okay, zero. We haven't had any. That's that we have had no power outages or

lights out event caused in the US by a cyber attack at this point. That's good. By cyber attack, I mean a cyber attack on the actual electric infrastructure. Um, the only one I would refer to in that way is maybe Colonial didn't really cause the lights to go off the Colonial Pipeline attack. It caused the gas to go away. And again, it wasn't an attack on the OT infrastructure. It was an attack on the IT, which somebody hit the big red button and they lost the the gas. So, you know, >> there's that. >> Squirrels, >> but squirrels are terrible. But again, how many news reports in the past four days have claimed cyber attacks are

leading cause of power outages in the leadup to Black Hat? >> Anyone? 17 I went and Googled. So 17 have said this. So again, we also have challenges with people sort of talking about these in this particular array and not understanding in particular what we're actually dealing with at this point. So I read 17 reports and then I stopped. There's probably more. Walk around the show floor, you'll see lots of things. So you know, so who's afraid of the big Lord? Me. This is I'm a power engineer. Um, usually, and Josh rolls his eyes at me for this quite a bit, but someone will say to me things, I'm it's an okay eye roll. It's fine.

>> Um, David does it too. But when people sometimes talk about like things like vulnerabilities that can cause a major cyber event, they'll go from, hey, we've got this vulnerability in this inverter, that was one I talked about last year, other things. It's going to take down the whole power grid. And it's a massive leap into saying that. And I'll usually be like, eh, there's things that will protect us. You know, the protection systems will action. Something will happen. I'm not usually as concerned. You'll see me kind of be like, eh, you know, it'll be bad, but not that bad. This will be bad. Like this is like this is my like where I'm actually starting

to flag, I think this is a problem because it's concentrated risk. >> Um, >> let me calibrate something here just from my perspective that I want to share with you. People like Dave and Emma are the people that I go to when I generate nightmare fuel. And I go, is this is this real? How real is this? Calibrate this for me. Where should I worry about this? When I worked for the city of New York, had a lot of things to worry about and needed to prioritize very honestly like what should I worry about today? What can we do something about all that? What are the things we can worry about? What are the things we can control? And

within within that environment and even more broadly, of course, energy was super essential. One of the things that was hard to calibrate was the speed. So Emma just talked about it. She got the trajectory right of where things were going, but the speed at which that happened was way accelerated. And all the other things that come with that cognitive understanding are not accelerated. People's preparedness, the partnerships, the policy, the money to support these things is not also accelerated. Well, the money is >> on the other side, [laughter] not on this side. And so again, when you think about those three things, there's a different kind of demand that's coming from AI. Number one, it behaves

differently. You'll hear Emma talk about volatility behaves differently. And then the rural utilities to a certain extent, I mean, keep me honest here, need and and want or want to participate in supporting that because it's revenue for them and they need they need that to be a going concern. And so calibrating these three against what we're actually concerned about is very very concerning. I'll also add that they are also being handied like grants money from these entities that want to install their data center there because they have the land or the water rights or something. Um they have options now for actually doing cyber controls in these locations because they have income that can actually help support it now and they

can grow. But making those right choices is a challenge because who are they going to they still need people. They still need choices in technology. they still need to choose the right things, but they're not catching up in that they don't still don't have the cyber teams to do the work that they're going to need to do to secure these things that are going to impact them. So, >> you could just use AI, right? >> Or they could use AI. Yes, that's a that's that's the solution that is also that's the problem. So, so yeah, or they can use an AI based solution that they don't know where it's actually being trained and if it ends up being the data

center that's connected to them and they have a problem, they're hosed. So, >> interesting. Yeah, that's that's kind of where we're >> that's where we're going. >> Cycle of crap that we're in right now. So, >> yeah. >> And again, I don't despise AI. I like that. Sorry. And patients over here is my AI friend and she is looking at me and I'm saying I don't actually hate it. I like it. I think it's interesting. I think it's a really cool technology that's moving the world forward in a really interesting way. But I think the work that patients and our team does is amazing. So I'm not just blaming you for it. It's you know um

>> I could I do blame you often for that. So um but I don't I think it's great. I just think responsible use of these things is really important for the system and planning for it as part of the threat model. Like if you lose that use of your algorithm or use of something that you were using to control your system because it was on a data center that was based on it, you're in trouble. Yes, you can move that load to somewhere else, but what if they have an outage? So, you know, there's a lot going on moving forward. Okay, so some rural and smaller entities are becoming the epicenter for this growth because

why? We need land efficiencies, modernization. There's not that much guidance for them adopting it safely. But for example, I've got a bigger slide on this later, but south of Virginia, um there's one utility, one co-op that's building 50 new substations currently, but to encourage that growth because they want them to come there. It's part of the data center alley going down Virginia, but they want people to come there because that's a business model for them. So, they're no longer these kind of side players of, oh, these small utilities don't don't have that much of a contribution. It will be okay. our definition of critical is now changing because now they're becoming more critical to the system.

So here's my Virginia example. Um as I think as people know I say Virginia a lot just because I live there. Um but uh 70% of the internet traffic goes through Northern Virginia. Um increasingly the hyperscaler of loads that were that were there are starting to move to the rural areas because they're just losing the ability to build. We're running out of land in these areas just now. Um, Rapahhanic is the the small utility I'm talking about. Their public load prediction in 2023, going back two years, was that they would by 2040 need to add about 3.3 gawatts of generation. Again, let's go back to that being about half of Scotland. Um, and then themselves, they said in 2025, we're

going to had have to add 20. We're like, wait, what? Um, so this is drastically increasing for them as well. remember they're a small rural entity but they came up with a really interesting business model. They're one of the more progressive small co-ops that I know of. They have a really cool business model for cyber security as well that they built a a an offshoot basically a small subsidiary that's their cyber security business so they could make it for profit so they could actually pay people enough to work for it which is you know interesting point. Um but they also decided to do that for their data centers. They're having each one register now as a load serving entity,

meaning it's a sort of style of utility in the country that separates the financial risk from them. So they're their own entity even though they're part of the co-op that's serving it. So they isolated their financial risk of doing this. They're they're good. They that's not their problem if something goes wrong there, but they didn't isolate their cyber physical risk because it's still the electrons that are connected to them. So ownership doesn't really separate electron flow. And you're like, "Oh, why don't they just put a battery on everything?" and some big generators. That's still electricity. Like, so we're on the same page. It's still they need electricity and the generators and the switches and all the same things that we're looking

for for the rest of the grid as well. So, it's not that separate. Um, I just like to add in these questions for fun, but how many priorities have we had in 2025 related to a woman climbing on a transformer? >> Same options as before. Who thinks more than 10? Who thinks five to 10? Less than five. zero. >> Technically, we had one. Um, and it wasn't me, so we're on the same page, [laughter] but in uh earlier in the year, actually, like talking about outages that have happened, a woman decided to climb on a very large power transformer in an area. Um, no idea what was going on with her. It's a really bad idea. I was joking

about it though. I was like, she didn't actually get electrocuted climbing on the transformer, which is kind of wild and impressive, actually. In the same way that people like the FBI wanted to hire the bad hackers so they because they thought they were good at their job, I think they should be hiring her to work on power because I'm kind of impressed she didn't get electrocuted. >> Um, but there was an outage related to that. Not because she was on the transformer and touching things, because they turned off the transformer to not kill her. So, they remotely shut down the transformer so she didn't die. Also, so they could throw things at her and get her to get off the transformer and

not break the transformer. So, there's a picture of this. I didn't put it in, but she's standing there. The police are sort of throwing bean bags at her to try and get her to come down, which I feel bad for the sub. It's not a great idea. But again, 800 houses lost power that day because they had to shut down one side of that substation to not hurt someone. So again, we have physical events that also can affect this whole situation. >> Emma, do you expect this number to go up in 2026? just >> I mean just for women or >> I mean that was the question but I think about you know people protesting right

or um trying to gather attention for it like this something to pay attention to. >> Y but to get back to that how many events have we had related to data center load tripping offline without warning? >> How many people think more than 10? Five to 10. less than five, zero to five, one to five, and zero. >> So, the last number I saw, um, there's reportedly been over 200 near miss events. By near miss, it doesn't mean the lights actually didn't go out. Near miss means there wasn't a reportable bulk system event caused by this. So, there have been around 200 events that have been caused by data centers tripping offline without notice to the

utility. like data centers are also sort of sensitive little snowflakes as well. The power electronic load is weird. Um it's particularly sensitive to fluctuations in voltage in a way that just like your normal house isn't or a normal load hasn't been. So if you your voltage starts shifting around too fast the data center is like I'm out. I'm you're not going to break my stuff. I'm I'm going to trip offline. They're not really tripping themselves. They're switching to their backup generator where they can get cleaner power effectively. Um so this happened. Um, again I mentioned I live in Virginia. I live in Fairfax. Um, sorry just to, you know, you can all come visit. It's fine.

But, um, live in Fairfax. I think it was July 10th last year. It's a really I remember distinctly what happened because I had also lost my AC for like three days to give a personal version of this story as well. Um, I hadn't had AC for three days because I'm not very good at making phone calls to come repair my AC apparently. But it was really really hot and uh I was slightly dying and finally got it back. It was over July 4th. Got it back. Day later my power goes out for four hours and I'm like, "Oh, come on." Like I'm going to die. The puppies were hating it. I was about to move them to a Marriott. It was all

bad. But [laughter] guess but it was a really bad day. There had also been some big storms roll through and as it turns out um in the report a lightning artor had failed. meaning that the thing that's meant to absorb if lightning strikes an electrical line, there's a lightning arrest that's meant to stop it from blowing up other things like a transformer. Um, it failed. There's a whole story about why that failed, but um on a large transmission line in the eastern interconnect that resulted in a a permanent fault which basically took down a major part of the Fairfax area. Um, it also which was included my house. But the thing that also happened during

these lightning strikes, there was this kind of power wiggle. The voltage was varying because it was having an impact from the electric strikes on the grid um of 60 data centers decided to go onto their backup power. And this is just in Fairfax County. They decided to evacuate the building, get off the grid because they were going to be damaged. And that was a drop in power of about uh or of load of about a large load drop of uh 1480 megawatt. So 1.5 gawatt roughly just >> Scotland >> it's about a quarter of Scotland dropped offline at the same time in just that one county and so there was an event caused by that other things started to

go offline because they suddenly lost all of this. They had to get generators to go offline as well because it was so large it dropped offline and suddenly everyone's like what just happened and they were like did we not have a control on this to stop them from doing that during because that's really bad and they were like no. So there's now been reports and I've been going through the reports and about 200 other events related to this over the last few years. So you know it's happening now. >> So that volatility then so let me say this and see if I have this right. So the volatility uh data centers can't handle that kind

of volatility. No >> so they move to their backup power. >> That shift to backup power changes their demand on the grid. >> Yes. And that further exacerbates >> like this. And so in talking of my numbers of Scotland, I gave this as well a couple years ago and this was related to load during sporting events where uh Andy Murray was winning Wimbledon and everyone in Scotland during one of the set breaks went and turned on their tea kettles at the same time to go make a cup of tea because that's what we do. Okay? like went and made a cup of tea at the same time, but it caused a frequency event in the UK because so many people

did exactly the same thing at the same time that basically also everyone sat down at the same time and stopped doing things that involved absorbing load. So basically there was two things that happened. Everyone turned on their tea kettle and everyone sat down and stopped doing things that would mean they needed to generate load and that caused a frequency event in the UK because they dropped about 220 megawatts at the same time. Again, in terms of Scotland, this was Scotland. So, they like this was Scotland. Um, but they they dropped the load and there was a frequency event because it wasn't expected that that would happen. Um, because we have to plan for these things as well. If you

expect that one day in the middle of the day when you forecast this, oh, by the way, there's a sporting event, so there's probably going to be an increase in load at a certain time, the grid operators will be like, we're going to have to turn on those generators or do this at that time. And they will have a plan for it. is built into the dispatch schedule. This was unexpected cuz Scotland's really bad at sport and it wasn't like a [laughter] they were winning was unexpected for the everyone. So, you know, there was that. So, everyone in Scotland stopped and that caused an event. So, imagine that for these data centers unexpectedly doing things because they're training a

different model or doing something else. So again, yep. >> Sorry, basic question. Let me get you a mic. >> I can throw this one. [laughter]

>> Yeah. I'm not a power guy, so this is really interesting for me. Help me understand. If everybody in all of these manufacturing plants turns off their power, there isn't a demand on the grid anymore. And so all of a sudden they would have 1.3 gigawatts to spend for everybody else. Um I would understand that there would be a big demand as they go back into the power grid and turn off their generation, but why would it be a problem when they start going into their own local generation? >> Well, load and generation needs to be balanced. So if they instantaneously take 1.5 gawatt off the system, um that's big enough to push against the

size of the system and the generation that's there. So there's a bias in every system of basically it's like 1.5 gawatt per hertz. Meaning if you drop that amount, you will shift the frequency of the system by that amount as well. If that makes sense. So everything is based on our system being balanced. And so if you if your frequency starts to drop, you start to lose other things because you can't operate that way. So it's similar to what happened in Spain essentially. So they started to lose the balance of the system. They had to start dropping different things offline to try and get it to balance back together and they couldn't. So like 1.5 gawatt is big

in that it's we did this math yesterday because we thought it was funny. I think it was it's around 14 million tea kettles. So unless you have some control in your system that would have you as a customer, we're going to turn on your washing machine all at the same time to help rebalance that. You'd have to drop a generator and have something that can drop that fast. So like coal can't drop that fast. It's a kind of base loaded generation, but diesel for example could drop offline instantly. >> What's being what's being used I had to learn this too. What's being used and what's being produced has to be close to each other if not exactly in balance. As

close as possible. there's only a little bit of tolerance, >> you know, if if it goes off. >> What would happen? I think that's your question. >> Yeah. See the Spain outage like so essentially if you have a big shift in frequency, your protection systems in the in the grid start to action. So they're trying to save the rest of the system. So they'll start to drop off say this load, this generator, pieces of the system that were planned to be taken off to try and bring that back into balance. >> Bring the balance >> and if it keeps going eventually you get a Spain style event where you lose everything on your system and you have

to restart it because at some point that volatility can no longer hold and they they trip offline to protect themselves because damage is worse than the power outage if that makes sense. So >> should we take questions now or do you want to keep going through? >> Just kind of a quick comment. One second. Sorry. There there few folks with questions and I just want to be thoughtful. Do you want to try and >> Do you want to pause? >> One more question and then we'll go on to a couple more slides. Who wants to >> Okay. Are we >> This gentleman back here had his hand up right there. Yeah. >> And then if you can just keep them we're

we're almost want to take questions right here. You you sir. Yeah. >> Yeah. I just I think that one of the things that people are missing is that the tolerances are incredibly tight. I mean, like, we run on 50 hertz. >> If you drop to 49.9, >> then you're starting to trip stuff offline >> and and the damage that can be done is pretty incredible. I mean, you can actually melt lines if you get too far out of >> if we drop like three hertz. We start to lose parts of the system. So, the balance is very tight on our system as well on purpose like so. >> Okay, let's keep going and then we'll go

through questions. All right, >> we'll just that is also relevant to the next slide as well. So, just to bring that up on the tolerance, um, normal non-AI workload on a cloud data center looks like the left where it's it's not that volatile. It's not, you know, moving around too much. I I borrowed this from the internet, but um, the AI workloads are kind of insane. So, when they start to train models or start to do different things, um, they can go through, this example was 15 megawatt, so 10 times as big a spike in the system and a rapid spike. That's why I'm saying volatile AF. Um because >> yeah, like 10 times the size of a spike

basically saying I've decided to train this model at this time or it hits a checkpoint and is pausing. That's that that sort of spike just locally on one data center processing is huge. And that's again a variable. You can't trip things off that line. There's only a couple types of generator that can absorb that fast a spike. Um, batteries and natural gas are two of them actually, but your coal plant can't ramp that fast. Your your nuclear can't actually do this either. So, you need a generator that's spinning to be able to do something to help resolve this. And that's really really hard to plan for when the people training the models are not the grid operators

>> in the in the city on a on a hot day or otherwise. Coned had hotlines to waste management to other places to say take this on or we're going to do this so that they could communicate when there were big things happening. How many how many of you think that the AI data centers or the AI companies are going we're going to retrain our model and give you a heads up that this is happening. >> Yeah. >> Go ahead. >> Y keep moving. >> We to keep moving. So um lastly, the interesting part is one of the the positives of all this is these small rural entities are trying to modernize. So they're also adopting AI into their

operations and into their systems as well at the same time. They're integrating models for things like managing their uh um what customer service. So half the time now when you call a customer service for your utility, you're probably talking to a chatbot initially. It's fine. They're probably pretty good at this point. They don't tell you weird stuff. It's fine. um or they're buying products with it integrated. So you can go buy an outage management system or wildfire management tool and it will have some really cool algorithms for wildfire management in particular has been one of the areas that's had huge benefits using AI for faster detection of you know sparklike events that burn things down. See smoke

in Vegas yesterday and then integrated to devices. Most of the meters that you get for your house now, the new ones that are coming out have integrated edge AI processing for different things as well. That's kind of insane to me actually. But most things you're buying now have something in it that claims to be a I'm not validating that it's real and it's not just marketing material, but they're claiming it. So, I'm going to go with it. Um, and again, they if they don't have cyber teams, a lot of them are saying, "Hey, you know where we can get efficiencies for um cyber security?" again as you mentioned using AI. So here we are. So they're also

adopting it in these entities. But again just to summarize our models don't account for this um for the AI supporting infrastructure. I asked actually a few of the AI related companies not looking at any people in here in particular just like do your models account for the infrastructure you're working on for your systems? No. Okay. Um, we're defending from AIdriven attacks with AI depends on this backup generators and off-site generation is still electric infrastructure. Um, and it's still electric and it's still water. It's still there. Um, and we're becoming dependent on this. Just one other slight worry. Um, when you go work in a utility operations room, there's usually, and I'm not going to not going to lie,

there's usually some old dude that's there who absolutely knows everything about he's the grid whisperer. He knows everything about that system. He's there. He's the the the the book of how you do this. We're now looking at those and people retiring and people are saying, you know what, we probably just need to use AI instead. I think there I jokingly called it I think we're developing old white dude AI, which is unfortunate. Sorry, old white dudes. Um but you know, >> I resemble that remark. >> I know. [laughter] Um so I think that's being developed and essentially we're going to have a brain drain. No one will actually know what it was meant to do on the system. So

there's this other issue coming in as well. And again, concentrated large load, we talked about this. It's enough to push against the system frequency and cause a resonate. So that's absolutely new to our system as well. So we need to win. You can say this. You wanted decathlon in here. So >> I wanted decathlon because I think people think about it as a race and it's not it's not one event. It's a bunch of events. And in in this case, China's planning it that way. think about supply chain, they think about cyber, they think about critical infrastructure, they think about people, they think about economic security, they think about ownership down the supply chain. You want to dominate a technology, you

dominate the infrastructure. You defend a technology, you defend the infrastructure. I'm sitting next to one of I think one of the countries maybe the world's foremost experts on battery energy storage and ownership of that battery energy storage. So when you hear this competition, geostrate strategic competition with China, it is not just about the frontier models or who's going to get to aentic AI or even who can buy chips. It's about the infrastructure underneath that. Remember what we said up front. If you want to talk about AI, talk about the infrastructure upon which it depends. >> So I think we've mentioned this one already, but traditional planning is failing. We've these load forecasts aren't predictable. We've got

transformer shortages. Guess where everyone's getting their transformers for like from in this country. Got a surprise for you. >> Yep. Yep. Yeah. Um we've got data centers without models. When I say not models for AI, we've got data centers without models for how they operate on the electric system. So predicting them is quite hard to actually do at the moment. Um and again, we've got these sort of static approaches to planning that were just never built for this. Well, so that all sounds terrible. Have a great day. >> Sorry. Um, seems pretty bad. There's things we can do. Um, we kind of need to do it as well. We It's not We can't stop this. There isn't a We There's no going

to a utility and saying no, don't do this. That they are not going to follow those approaches. So there's no no here. There's a just a how we make this better. Um, so what can we really do? Understanding the priorities is important. understand that why they are integrating these large loads to their business models and what benefits that could give us to cyber security being that they have money to hire people and do it now. So helping people with those approaches is really important. Understanding it's there and why they're doing it. Um again encouraging good citizenship like of the customers. Again we're talking about consumer and commercial load here are two different things. Um the world has shifted. We

need to understand that in 12 months things have have shifted really fast. It's not what can we do. We do need to just know that this has happened. There's not just the small electric anymore. There's a small electric and giant load. Um our highest consequences are now dynamic and we actually need to understand the use cases they have for this. Like if somebody is buying a tool that says we're using AI, helping them understand how to responsibly use that in their own environment in a way that they can actually depend on it as well. So protecting AI's foundation is also important because AI attacks are a thing. Um but don't tell people don't because tell them how to do it right

because if you tell them don't you immediately stop them from wanting to even talk to you. So when you're a cyber defense type person saying well you shouldn't do that. Okay it's not the same as plugging in your phone to a different USB. It's something else. It's a bigger problem for their business model that they need to do this. Resilience planning needs to consider the supply chains for this. Um, and again defending the foundation of AI, helping small utilities understand their AI creep and demand transparency is really important. You love the demand transparency part. So that's >> I I think this is kind of cool. The demand transparency. So there's two parts to that. There's us, remember, no

one's coming. It's us to demand that transparency from those organizations. And there's also transparency about the demand about where it's coming from the behavior of it the volatility of it. When you hear AI think about infrastructure and when you think about threats or resilience and a to AI AI and the grid are inseparable. I hope if you leave with nothing else today it is that and also we should use Scotland as a form of measurement >> in the dictionary. Um other parts we've been talking about you if you were here yesterday Ginger and folks were giving the and that there was a CI a cyber farmed engineering workshop I believe as well again going to manual for critical

functions is something that we've had for a long time um but I'm not sure we have go to manual for this situation so we need to work out how to get through that um but again consequence informed decision-m is really important I've been using fault location as an example for um consequence your utilities. Um they do well I'll cover this with other people later because we're running out of time, but considering criticality is also a problem. Um these small utilities that I'm talking about were often not considered critical because you know they're small. They're now the new critical. It's the the new normal is they're the new critical infrastructure. So if you're saving the world with your

AIdriven solution, you need to actually understand the resilience of the solution also. So chaos engineering is something I really love. Um, but we've been trying to work on how to do that for the electric grid because that's really what we have now is a chaotic system. It's still operating. I want to add we still have one of the most reliable electric grids in the world. So, we're not still not failing, but um, one of the things we've been really interested in looking at is how to abundance plan, which is a whole other world. How do we plan for this? How do we build the models? How do we just assume everyone might make a random

decision every given day and work out what that would actually look like for a system? We need to do that for cyber security in these systems. Also assume random is my assume random and assume chaos at all times is my um pitch here for how do we actually plan for this. But planning for failure is probably the biggest lesson here. This is an image of Spain. Um, I've got some funny, this is going to be in another talk I'm giving on Friday, but uh, planning for failure is kind of the the big thing here. The one thing when people talk about the the outage in the Iberian Peninsula, it's always like, "Oh my god, it was so

terrible." The I actually have a slide here that was the British were eating biscuits and beer. It was so bad because southern Portugal didn't have some of the big holiday destinations didn't have power and they're like, "We're relying on our cookies and beer." We're like, "That seems wonderful. Like, you'll be fine." Um, one of the bigger problems that happened was people started to believe it was a cyber attack and then started to panic. Um, so the people that posted it was a cyber attack on LinkedIn started to create this fear, uncertainty and doubt that spread throughout the world. I even had a call from my dad who is a forensic scientist and he still called and said, "I heard it was a cyber

attack in Spain." I was like, "Dad, probably not." Whole other talk on what happened, but planning for failure was key. Um, Spain was able to restart their grid in about 10 hours. They did a full black start on their system, restarted it from scratch. That is actually a huge success. They had planned for that. They knew how to restart it. They didn't run into many problems. They solved them as they went along. They had planned for this to happen. So, they were able to restart. We have blackart plans. I'm not sure we have blackar plans that include data centers. So, we have a whole other resilience problem there. But planning for failure is kind of key and that's

the sort of big message here is we need to plan for this to happen and we need to be able to recover and do the right things. Um again I'm just making posts about this that what happened in Spain because it annoys me that people said it was a cyber attack but you know um but yeah the end is now. Um so [applause] would you like to welcome a robot over? >> I no it's okay. I think we we're at time >> question. So, let's do let's do um let's take all all the questions and then we'll try and bundle them. Go ahead, Josh. >> All right. So, hopefully you can see that why we love Emma and and

>> I'm hype man. What did I tell you? I said expectations. I was just going to jump in a little bit. Not a lot of substance. >> Really good teammates in this crazy cavalry thing. They don't give you what you ask for. They give you what you need. I hope this is both overwhelming and stimulating. But if you look at the cascading failures with no water, no power, no hospital, this takes it to a whole new level. Uh there's a lot of questions in the room. So I'll just say the the levity bit here is my biggest concern is if we have a massive outage, what do you restore first, the hospital or the data center?

And in some of the international exercises I'm hearing, the answer is the data center. And the second answer is not the hospital. And the third answer is not the hospital. So we have to take everything you heard yesterday and for the rest of today and tomorrow and realize that this is probably going to win in the ruthless prioritization stuff. So maybe you made that dynamic range of demand green because Loki is holding that chaos together. Um but I I hope we can maybe decouple with small nuke plants or something. But if there's any other non-cyber ways to decouple public safety continuity of operations from data center continuity of operations, I hope those are in the mix as well.

>> I just if your restoration options included you needed to process data through a data center to be able to restore the system, then the data center comes before everything else. That's my that's the challenge is if people are relying on an AI based restoration solution then they need that before they which I mean could mean they could push it offshore but that's another risk. So >> for time let's let's take all the questions. So yeah please here and then I think we have one keep your hands up so then we can come around and take all the questions. Go ahead. great presentation. Was wondering if these AI companies are paying into these small energy companies and and to to build

that up and and if not, where do those decisions come from? Is that at the state level or at uh kind of the the regional co-op level? >> Thanks. >> Say yes, they are. So that's they are paying money for this to these entities like we're seeing them get grants from >> like not to throw Google under the bus but Google gave a big grant to a small rural utility because their data center was dominating their load and >> is that voluntary or is that >> it's voluntary? Yeah. So it's >> wait to to kind of daisy chain off of that. Does it matter that the large companies are buying their own nuclear facilities like Microsoft's turning on

three mile island or two and a half mile island so to speak and Google's buying their own nuclear. >> How do we feel about turning back on an old plant that's been end of life from a cyber perspective? We can >> and also they may they may have the generation but the demand >> right >> is still on the grid. Yeah. >> Yeah. I I hate to uh bring up the idea of using technology to solve the problems that technology has uh has caused but uh as a a user of AI or at least my grad students uh the uh uh use of batch systems and so forth is pretty common. Is there any hope for a data

center energy use API that could reduce the uh the or increase the delay between the uh the load changes? There is, but that doesn't mean they'll use it. If that makes sense. >> We had a question back here. >> I was gonna say the API would probably based on an AI based solution for managing the AI the AI data set. >> Well, we have to your the point underneath this though is changing the way we're doing load forecasting that this gentleman right here has >> rapid fire. I got it. Sorry. Okay. Uh so a couple things I did not see in your d your thing uh the issue of public safety power shut offs that we have on the west

coast because they they do that now preemptively to stop wildfires and my concern is >> they're going to they're going to prioritize keeping power on for data centers and we already have [clears throat] giant issues with people not having power in times when they desperately need it to just stay alive. And so that's I don't know that there's a question but I think it's an important consideration. And then the integ the integration of small nuclear reactors, small modular reactors into this. I think from an emergency management perspective, that's where I come from. There's very little regulation around those. I've read the regs. They suck. >> Yeah. >> And and they scare the crap out of me,

but they're being touted as the solution. And I like I'd like thoughts on that, but >> I was someone who works for Idaho National Lab. I have to say nuclear is wonderful. >> Oh, patience. >> Uh right there. But just on the on the safety rags, um this is kind of what I think about as the ruthless prioritization um that uh you know we tried to look at under cyerspacearium or DPA like what do you what are you doing first and can we get everybody to decide on what we're going to do first? The answer is no. Last question. Yeah, go ahead. So, real quick, um, I was going to ask this before some of the other questions about

decoupling, but what does the ideal power grid look like so that we can effectively decouple the load from the volatility problem and push the cost of volatility to the sources of volatility? >> Yes, >> good question. Redesign of the grid. >> I economics >> like to entirely redesign the grid if we could at this point. It's a Oh, I was helping >> [laughter] >> Um [clears throat] I would like if we could we again our grid was designed for like 10% margin on top of where we're at just now. I don't see how we do this without major redesigns but we also are having struggles building anything. So I mean transmission takes about 10 years to

build. Um, so designing that's kind of where this chaos engineering point is of redesigning or re changing how we plan so we can build things for these unknowns in the future as well, but still satisfy what any customer may want to do as well because you're still you still have free will. Putting solar in your house, buying electric vehicle, not doing those things is also something we still need to plan for also. >> Please join me in thanking Dr. Emma and Nice. Thank you.