BsidesLV 2025 - IATC - Tuesday

[Music] Heat. Heat. [Music] [Music] Down [Music] 34. [Music] There you [Music] go. [Music] Fire. [Music]

[Music] Heat. Heat.

[Music] Heat. Hey. Hey. Hey. Heat.

Heat.

Heat. Heat. [Music] [Applause] Heat. Heat. [Music] Heat. Heat. Heat. [Music]

Heat. Heat.

[Applause] [Music] Heat. [Music] Heat.

Heat. Heat. N. [Music]

Heat. Heat. N. [Music] Heat. Heat. N. [Music] Heat. Heat.

[Music]

[Music]

[Music] Heat. [Music] Heat. [Music]

Wow. [Music] Heat. Heat. Heat. [Music]

[Music] Heat. Heat.

[Music] Heat.

[Music] Heat. [Music]

Heat. Heat.

[Music] Heat.

[Music] Heat.

[Music] Hey hey hey. Heat. Heat.

Heat. Heat. [Music] Yeah, [Music]

[Music] hey. [Music] Hey, [Music] hey hey. [Music] Besides, I am the Calvary. My name is David Bots and I am here with uh Mr. Josh Corman and we are delighted to bring you this very carefully curated track. We are going to be talking about a number of topics today. Yesterday, we can't go back in time, but let's talk about today and tomorrow. We're going to be talking about a number of topics, some of which might lead to a level of discomfort. So, but that's okay. We encourage you to sit in your discomfort a little bit and say, "Ah, this conversation is making me think big thoughts." And that's good. We want you to think big thoughts,

societal impacts, personal impacts, family, neighborhood, community, all of these things. We'd like you to think about that because this is intended to help you think about not just the future, but also your future, your family, your neighborhood, your community, the the place where you live. That is the goal is for you to be thinking about all of these things. We have put together, we've assembled a variety of different people talking about different specific issues regarding the overall theme of I am the Calvary. And if you don't know what I am the Calvary is, I will say it very poorly compared to my co-chair, Mr. Josh. Um, no one is coming to save you. Nobody. If you're waiting for somebody

to come and save you, you will be sorely disappointed because they're not coming. Nobody's coming. Do you know who who can save you? You and your neighbor and your neighbor's neighbor and the people you bring along with you to help protect you and your household in your neighborhood. That's who can save you. Okay. Um so we put together uh a series of discussions to have you know some big thoughts. So today and tomorrow we'll wrap it up and uh at the end of the day tomorrow morning so we've got a forget if that's 90 minute two-hour block we'll we'll Mr. Josh is going to bring it all home and talk about Monday and Tuesday and

Wednesday and wrap it into a very elegant package. So we encourage you if you can to hang in for today and tomorrow just hang in. We're going to be talking about some very really interesting things and community things that are important. I am delighted delighted. Please step on stage a delight. Please join me in welcoming Dr. Emma Stewart and Manish Walther Puri and they're going to be talking about their own bios. So, I'm not going to I'm not going to bore you by talking about this twice. But I tell you, both of these people are amazing. And you're going to learn more about the electric grid and AI than possibly maybe even you wanted to know. But you should

know this. And one of the things that's going on this morning is within the the theme of I am the Calvary, we're going to be talking about certain externalities that may affect how each of us respond to the current emerging threat landscape that all of us live in. So, please put your hands together and join me in welcoming these two fabulous presenters. Over to you, Okay, making sure I've turned this on because that's my first problem with electricity apparently. >> Um, hi there. Uh, thank you everyone for having us here today to talk about this. Um, as David introduced, um, this is obviously a talk in I am the cavalry. So, we're talking about what's been

going on with the electric grid and how that applies to lessons learned, what the most emerging threats are. When we started to put this together, we actually had a slightly different theme. And then in talking with Josh and working out where the direction of this was going, we really came up with sort of new direction for this presentation and to talk with you all about how AI is actually impacting what our biggest threats are to the electric grid. Um I've been here a couple years and given where we're going in the electric grid talks um that's usually my this is where we're going and um you'll see in some of here like the direction has changed

drastically in the last year a way that I never even fully understood myself was going to happen. And so that's really what we're going to talk about and also what we think people could be doing um when they're out working with say small electric entities and others to help improve this and what we need to think about now when we're out there. So, tale of two grids is the idea here. And so, you have me. Um, I am Emma Stewart. I have worked in power grid my whole career. If you want to know how long that is, and I make this joke a lot, that picture on the bottom right is me in my first utility job many, many years

ago. It is an actual Polaroid from when Polaroids were really a Polaroid thing and not like a new fashion item. So, like it's so that's if you want to know how long I've been doing this, that is how long. Um but I've worked in power grid my entire career. Everything from uh working in a few national labs over areas like hydrogen defense, electric grid, cyber security. Again, always focused on how the electric grid works and things that we're adding to it, doing to it, using with it as well. Um I've also done all sorts of things to do with at one point I wanted to be a race car engineer. I just like to throw that

in. And then I changed my mind because I decided uh it'd probably be a more lucrative industry for me to work in power than to work in race car design. So off I went into doing that. But I also have Manish here who >> Hi everybody. Uh my name is Manish Walther Pori. Um and I'm here primarily this is my first time on the stage. So I'm here primarily as Emma's hype man. Um if any of you are fans of rap music, there's always like the primary performer and then there's a person off to the side who just sort of throws in stuff. That's my role. Um I'm a lot of style, a little bit of substance. Um,

I've worked in risk pretty much my entire career in some way or another. I started my career in nuclear policy doing now what's now known as OSENT. I've worked in geopolitical risk, terrorism, fraud, and of course cyber security, critical infrastructure, and supply chain. Uh, one of the best opportunities I had to really understand these kinds of challenges at scale was also one of my um, uh, most meaningful roles where I was the first director of cyber risk for the city of New York at New York City Cyber Command. So, I will give the disclaimer that I'm speaking here even though I no longer employed there. I'm still speaking from my experiences here and not on behalf of

that organization or the city of New York. Um, and I've gotten to know Emma because she's along with many people in this community and and I am cavalry. Josh and and Dave actually was one of the key people while I was at the city who have helped me understand the complexity around energy and we'll get into that but I hope to share some of that with you as Dr. Stew's hype man. So glad to be here. >> I was going to say you also put my name on the abstract and then told me I was presenting. >> That is true. That is how that happened. That is precisely how that happened. Yes. >> Well, so in the last few years, um,

we've talked a lot about the word cyber poverty comes up a lot talking about people that have and have nots in the cyber community. Interestingly, I called this here we go again. Um, because in the electric grid, we're also seeing this. This has always been a thing. We've always had energy poverty also. But now we're witnessing and this was a quote from the new emerald AI co recently that I really liked where he's basically saying we're witnessing the birth of two energy systems where essentially we're we have got one that's integrating massive amounts of load. We're able to serve things on big data. We're doing huge amounts of work for that. Then we also have a grid that's

basically failing the basic needs of humans at this point. Um we have lots of power outages caused by many things. And there's also a huge risk that these huge loads that we're adding to the system and prioritizing getting on there for supporting AI are becoming a threat to the average human experience on the electric grid as well. So I thought that's a really great quote to explain sort of why this talk fits into the we're witnessing two grids being birthed at this point and we're not sure which way it's going to go but it's not necessarily going the best direction. So this and I've now lost my microphone so >> hold on I will just hold this. It's much

easier. Um so when you walk around Black Hat for example just now or walk around this um any of the talks that you're seeing about AI a lot of what people are talking around is defending from AI or defending with AI. So that's the the two sorts of worlds we're hearing about in if you walk around the show floor of Black Hat for example that's all you'll see everywhere. Um but what we really wanted to talk about was just defending AI itself in particular looking at the infrastructure on which it relies because um if you look at all the people that are developing these real time threatbased how we're going to respond to all of the threats they're defend

they're depending on AI and therefore on the infrastructure on which it is based. So when you have models that are running in cloud-based infrastructure that's on a data center that funnily enough also needs water and power itself to be running, we need to actually defend that infrastructure for it to be useful as an emergency solution. Um we also are looking at the defending the infrastructure that everyone else is relying on from the AI infrastructure in that all of these things that are the giant large loads um they're essentially there and causing problems already on our electric system. So we need to also help defend the infrastructure that everyone else is relying on to serve their everyday need from the AI

infrastructure itself. So that's kind of the theme here is this is a cyclical problem. We're relying on AI for things that are relying on the electric grid. We're using AI to support the electric grid and then we're also potentially going to destroy the electric grid by integrating lots of things that might hurt it. So it's kind of a cyclical problem that we're in. So if you can the next time you hear AI add on and the infrastructure upon which it relies and see how that sentence ends. See how the story goes. See if they're talking about it. They probably aren't. So just quick overview. We're talking about the new frontier for the cavalry essentially is the AI and load growth

that's happening on the system. When we talk about electric, we can no longer separate these two things at this point. Um AI needs load load needs AI. That's it. Um but we have two grids. One is focused on the load growth. The other is ignoring the basic needs of every human at this point. Um, our threat models are cyclical. We're dependent like I said, but what can we do? We are going to cover a what do we think we can do about these things. You've seen some talks already on things like cyberinformed engineering. There are things that we can be doing to make this more secure. But the urgency and one of the reasons I'm having this talk here is this is

now. This isn't five years from now. This isn't even 2027. This is happening immediately at this point. So we have to look at ways to solve these problems as a as a community now at this point. So so 2023 just to explain where things have gone. I gave this talk hopefully it's showing up okay >> u just on where the electric grid was going and where it had been. Um this was from my slide. It's now on a dark background because I had to be more hackery. Um but the uh I gave a this talk where I talked about things going from I mean 20 years ago we had the dumb grid it got smart it

got clean like over the last 20 years we've moved through different problems every few years and come up with solutions interdependency we think we're doing great then we spoke too soon now we have cyber security as a common word then shields up sort of moving on to where we are now when I gave this in 2023 I was actually talking about we have huge infrastructure investments happening in the electric grid and there was I think $1.2 two trillion dollars was going into upgrading our infrastructure at that point and I'd said now what we have lots of money what are people going to do with it so move on um now we're here I have a now what

now this is my my slide for now two years later giant large loads everywhere AI everything and energy abundance and to understand this is a complete shift in understanding of how we plan the electric grid for 20 years if the one thing I should have had in this flowchart We actually planned for things like energy efficiency. One of our ways to help help grow and help the world continue moving forward was to make things more efficient. So we actually used less energy in our system. We're in the opposite space now where basically if you listen to anyone from Department of Energy um or from the US government, we're saying energy abundance, we must grow. We're going to use more. We're

going to have lots of energy for everyone. It's going to be affordable. It's going to be great. But there's just going to be tons of it everywhere. And that's new for me as well. And that's not how we planned the system. We planned it for >> um a series of uh um what's the word? Margins essentially that we're not this this is not what we planned for. So here we are. This is a fundamental shift in our understanding of the electric grid. So just for anyone that wants to understand why I'm saying these things, I had to do a electric grid primer for just very a couple basic concepts for people that don't deal with this every

day because I've been told I sometimes have a habit of just saying big electric grid words and then people stare at me. So um it most basic level in the long-term planning of the electric grid, it's balancing load and generation. We have the loads, we have the big spinny generators, we balance those out to maintain the frequency of the system. So essentially they must be equal for the system to be stable. At this point you must serve load and generation together. On the short term um we have when we talk about load and generation matching on the big numbers over a long time frame we're talking about the capacity of the system. And so when we say load

growth mostly we're talking about capacity growth of how much we serve. Um when we talk about problems with things tripping off say somebody's talking about their we hacked a system and managed to trip off this big generator. The problem they're talking about that causes a cascading failure is really around frequency, which is a short-term event caused by the loss of generation or the loss of load at the same time. So that's why we're talking about giant loads because these are now giant concentrated things sitting together that can cause that imbalance instantaneously by tripping off or doing something weird. So again, giant spinny interconnected system. That's the basics. >> Technical term. >> It's a technical term. I I have a PhD in

this, you know, so it's geez. Would you like to >> Nope. Nope. Nope. Nope. Can't. Uh the three things that we want to lay out for you and really is going to lay it out. Is the thinking about the demand here. So there's the type of demand, what that demand is like, its behavior, and who supplies that demand and the nature of that. Those are the three three elements. So while it is all about demand, there's some different aspects of there that are pretty important when you think about AI specifically. >> Okay. And just to explain a little bit more why this is interesting, especially for the I am the cavalry efforts. Um when we again this is the same picture

from a few years ago of the the spread of the different types of utilities in the country. We have 3,000 of them for electric, many more for water. um they are split in ownership between investorowned meaning the sort of big money efforts publicly owned like your munis and then we have cooperatives that are not for profit. Um the while the investorowned cover the most of the customers in the country the publicly owned and cooperatives cover most of the land in the country. when you're trying to build a a big data center for example, one thing you actually need is land because it's kind of annoying when they build them on your golf course, but um you need land and that's why a lot of

them have been moving out into these areas and why it's critically important for these smaller utilities to actually be thinking about it and knowing what to do from a cyber security perspective because they are going to have them all over their space. So to explain that um like to give some numbers and I Manish loves this because I apparently am now describing things in terms of the size of Scotland. So >> it's a unit of measure because >> it's a unit of measurement >> because we are in the US and in the US we like to measure things in weird numbers like hands on on horses or something I don't know and don't use the

metric system. >> So this is a postimperial um measurement >> colonial. Yeah. Yeah. So, Virginia, I I happen to live in Virginia. Um, but the entire state of Virginia serves about 20 gawatts of load. Um, they were predicting by 2030, which feels like it should be a really long time away, but it's not. Um, as it turns out, in 5 years they will be growing by about 35 gawatt um of load, meaning they'll be 1.7 the size they are currently being added to their system, which is huge. um which and I did the math is nearly six Scotlands. So, six times the size of Scotland being added to the state of Virginia in load, meaning that they also need six times

the size of Scotland in generation to be added to that system to be able to keep the system stable. They also need water and land. Um I was looking at what the largest data center that's been integrated. Again, data centers are used for training AI. They're used for other things, but this is like data center in Texas. The biggest one I've seen was 1.2 gawatt. That was about 5% of the load in Texas, and that was a single site. So, these things are enormous. And 1.2 gawatt is about a quarter of a Scotland. So, >> we're we're sort of joking, but not really, about measuring it in terms of >> a country or people. So think about this

capability displacing or a tradeoff of serving that group of people. You hear it a lot. 85% is controlled, owned and operated by the private sector. But what the the last slide and this slide show you is that it's not just about who who controls it, but who it's serving and what the trade-offs what the trade-off is if you're serving one group versus the few and the many. And you heard Josh talk about that last yesterday too. So explaining that a little as well, if you have a data center and you're a co-op for example, and do you know how utilities make money? They serve load. So basically a load is a giant income for anyone. And Carl, please stop

laughing in the back, please. But >> you're allowed to snicker because we're going to talk about dropping big loads. So now is the time in the talk where you can you can laugh about that. We're gonna keep saying it. So, if you didn't get the chance earlier, you'll get one later. Don't worry. >> We may have rehearsed this and had some problems laughing. So, it's okay. >> Um, >> we do serious work, but we do not take ourselves seriously. >> Yes. >> Thank you. Thank you. So, yeah, now I'm now crying, laughing, please keep talking. Um, yes. So, to follow up, so imagine we have a utility, a small utility sitting out there. They have a

giant load sitting attached to them. um they are getting the electrons through their site. So if something varies at that site drastically, say the AI training model steps up its load by 50% at that given point. It's that small utilities whose electrons are impacted. It's not the business model that's the problem. It's that if they have a dynamic event, they're the ones that lose all the power. And that's where their hospital loses the power. The water loses the power. Everyone is screwed at that point because of this one single item that's bigger than the whole utility itself. But they don't own it. They don't control it. They also don't know when somebody has decided they're going to train a model to create

pictures of people with three boobs. Like I they don't know when that's going to happen. So like that's a problem right now for forecasting these things as well. So >> yeah. >> Okay. So overall chip to grid risk everything from down at the chip level that's in these sites up to the electrons that are flowing onto the electric system that are caused by these things ramping up and down. I hope everyone feels guilty now every time they use something weird for AI. You should now all feel very guilty. That's it. I just want everyone to be guilty for all your useless applications that are just generating the pictures that are used in this presentation also. So

you know there's that. Um but yeah, it's interlin risk where small repeated failures can lead to a systemic crisis essentially. Now again, traditional planning is failing because we've gone from this um being an a sort of thing that was happening. I also gave a presentation at Blackhat last year that was around where are we going in the next year talking about data centers. I didn't predict how many there would be. So I predict a lot of things really well. Didn't predict how this was going to turn in the last year at all. Um, and then there's new models for fastmoving risks. We need better models for threat models for this happening. We need to work out both from a physical standpoint

and from a cyber standpoint, we need better threat models for these risks. And again, it's a cycl cyclical threat. It's going in circles. We are in defense of and from AI. So, just to lighten the mood, how many power outages have we had in the US caused by a cyber attack? >> Does anybody know the answer? >> And Megan's not allowed to answer because she knows this. Anyone? >> How many people think more than 10? I don't know if people think five to 10. >> How many people think less than five? >> Ah, good. >> How many people think zero? >> Okay, zero. We haven't had any. That's that we have had no power outages or

lights out event caused in the US by a cyber attack at this point. That's good. By cyber attack, I mean a cyber attack on the actual electric infrastructure. Um, the only one I would refer to in that way is maybe Colonial didn't really cause the lights to go off the Colonial Pipeline attack. It caused the gas to go away. And again, it wasn't an attack on the OT infrastructure. It was an attack on the IT, which somebody hit the big red button and they lost the the gas. So, you know, >> there's that. >> Squirrels, >> but squirrels are terrible. Again, how many news reports in the past four days have claimed cyber attacks are a leading

cause of power outages in the leadup to Black Hat? >> Anyone? 17 I went and Googled. So 17 have said this. So again, we also have challenges with people sort of talking about these in this particular array and not understanding in particular what we're actually dealing with at this point. So I read 17 reports and then I stopped. There's probably more. Walk around the show floor, you'll see lots of things. So you know, so who's afraid of the big lords? Me. This is I'm a power engineer. Um, usually, and Josh rolls his eyes at me for this quite a bit, but someone will say to me things, I'm it's an okay eye roll. It's fine. Um, David does it too.

But when people sometimes talk about like things like vulnerabilities that can cause a major cyber event, they'll go from, hey, we've got this vulnerability in this inverter, that was one I talked about last year, other things, it's going to take down the whole power grid. And it's a massive leap into saying that. And I'll usually be like, eh, there's things that will protect us. You know, the protection systems will action. Something will happen. I'm not usually as concerned. You'll see me kind of be like, eh, you know, it'll be bad, but not that bad. This will be bad. Like this is like this is my like where I'm actually starting to flag, I think this is a problem

because it's concentrated risk. >> Um, >> let me calibrate something here just from my perspective that I want to share with you. People like Dave and Emma are the people that I go to when I generate nightmare fuel. And I go, "Is this is this real? How real is this? Calibrate this for me. Where should I worry about this?" When I worked for the city of New York, had a lot of things to worry about and needed to prioritize very honestly like what should I worry about today? What can we do something about all that? What are the things we can worry about? What are the things we can control? And within within that environment and even

more broadly, of course, energy was super essential. One of the things that was hard to calibrate was the speed. So Emma just talked about it. She got the trajectory right of where things were going, but the speed at which that happened was way accelerated. And all the other things that come with that cognitive understanding are not accelerated. People's preparedness, the partnerships, the policy, the money to support these things is not also accelerated. Well, the money is >> on the other side, not on this side. And so again, when you think about those three things, there's a different kind of demand that's coming from AI. Number one, it behaves differently. You'll hear Emma talk about volatility behaves differently. And then

the rural utilities to a certain extent I mean keep me honest here need and and want or want to participate in supporting that because it's revenue for them and they need they need that to be a going concern and so calibrating these three against what we're actually concerned about is very very concerning. I'll also add that they're also being handied like grants money from these entities that want to install their data center there because they have the land or the water rights or something. Um they have options now for actually doing cyber controls in these locations because they have income that can actually help support it now and they can grow. But making those right choices

is a challenge because who are they going to they still need people. They still need choices in technology. they still need to choose the right things, but they're not catching up in that they don't still don't have the cyber teams to do the work that they're going to need to do to secure these things that are going to impact them. So, >> they could just use AI, right? >> Or they could use AI. Yes, that's a that's that's the solution that is also that's the problem. So, so yeah, or they can use an AI based solution that they don't know where it's actually being trained and if it ends up being the data center that's connected to them and they

have a problem, they're hosed. So, >> interesting. Yeah, that's that's kind of where we're >> that's where we're going >> cycle of crap that we're in right now. So, >> yeah. >> And again, I don't despise AI. I like that. Sorry. And patients over here is my AI friend and she is looking at me and I'm saying I don't actually hate it. I like it. I think it's interesting. I think it's a really cool technology that's moving the world forward in a really interesting way. But I think the work that patients and our team does is amazing. So I'm not just blaming you for it. It's, you know, um >> I could I do blame you often for that.

So um but I don't I think it's great. I just think responsible use of these things is really important for the system and planning for it as part of the threat model. Like if you lose that use of your algorithm or use of something that you were using to control your system because it was on a data center that was based on it, you're in trouble. Yes, you can move that load to somewhere else, but what if they have an outage? So, you know, there's a lot going on moving forward. Okay, so some rural and smaller entities are becoming the epicenter for this growth because why? We need land efficiencies, modernization. There's not that much

guidance for them adopting it safely. But for example, I've got a bigger slide on this later, but south of Virginia, um there's one utility, one co-op that's building 50 new substations currently, but to encourage that growth because they want them to come there. It's part of the data center alley going down Virginia, but they want people to come there because that's a business model for them. So, they're no longer these kind of side players of, oh, these small utilities don't don't have that much of a contribution. It'll be okay. our definition of critical is now changing because now they're becoming more critical to the system. So here's my Virginia example. Um as I think as people know I say Virginia a

lot just because I live there. Um but uh 70% of the internet traffic goes through Northern Virginia. Um increasingly the hyperscaler of loads that were that were there are starting to move to the rural areas because they're just losing the ability to build. We're running out of land in these areas just now. Um, Rapahhanic is the the small utility I'm talking about. Their public load prediction in 2023, going back two years, was that they would by 2040 need to add about 3.3 gawatts of generation. Again, let's go back to that being about half of Scotland. Um, and then themselves, they said in 2025, we're going to had have to add 20. We're like, wait, what? Um, so this is drastically

increasing for them as well. remember they're a small rural entity but they came up with a really interesting business model. They're one of the more progressive small co-ops that I know of. They have a really cool business model for cyber security as well that they built a a an offshoot basically a small subsidiary that's their cyber security business so they could make it for profit so they could actually pay people enough to work for it which is you know interesting point. Um but they also decided to do that for their data centers. They're having each one register now as a load serving entity, meaning it's a sort of style of utility in the country that separates the

financial risk from them. So they're their own entity even though they're part of the co-op that's serving it. So they isolated their financial risk of doing this. They're they're good. They that's not their problem if something goes wrong there, but they didn't isolate their cyber physical risk because it's still the electrons that are connected to them. So ownership doesn't really separate electron flow. and you're like, "Oh, why don't they just put a battery on everything and some big generators?" That's still electricity. Like, so we're on the same page. It's still they need electricity and the generators and the switches and all the same things that we're looking for for the rest of the grid as well.

So, it's not that separate. Um, I just like to add in these questions for fun, but how many priorities have we had in 2025 related to a woman climbing on a transformer? >> Same options as before. Who thinks more than 10? Who thinks five to 10, less than five, zero? >> Technically, we had one. Um, and it wasn't me, so we're on the same page. But in, uh, earlier in the year, actually, like talking about outages that have happened, a woman decided to climb on a very large power transformer in an area. Um, no idea what was going on with her. It's a really bad idea. I was joking about it though. I was like,

she didn't actually get electrocuted climbing on the transformer, which is kind of wild and impressive, actually. In the same way that people like the FBI wanted to hire the bad hackers, so they because they thought they were good at their job, I think they should be hiring her to work on power because I'm kind of impressed she didn't get electrocuted. >> Um, but there was an outage related to that. Not because she was on the transformer touching things, because they turned off the transformer to not kill her. So, they remotely shut down the transformer so she didn't die. also so they could throw things at her and get her to get off the transformer and

not break the transformer. So, there's a picture of this. I didn't put it in, but she's standing there. The police are sort of throwing bean bags at her to try and get her to come down, which I feel bad for this. It's not a great idea, but again, 800 houses lost power that day because they had to shut down one side of that substation to not hurt someone. So, again, we have physical events that also can affect this whole situation. >> Emma, do you expect this number to go up in 2026? just >> I mean just for women or >> that was the question but I think about you know people protesting right or um

trying to gather attention for it like this something to pay attention to. >> Y but to get back to that how many events have we had related to data center load tripping offline without warning? >> How many people think more than 10? >> Five to 10. less than five, zero to five, one to five, and zero. >> So, the last number I saw, um, there's reportedly been over 200 near miss events. By near miss, it doesn't mean the lights actually didn't go out. Near miss means there wasn't a reportable bulk system event caused by this. So, there have been around 200 events that have been caused by data centers tripping offline without notice to the

utility. like data centers are also sort of sensitive little snowflakes as well. The power electronic load is weird. Um it's particularly sensitive to fluctuations in voltage in a way that just like your normal house isn't or normal load hasn't been. So if you your voltage starts shifting around too fast the data center is like I'm out. I'm you're not going to break my stuff. I'm I'm going to trip offline. They're not really tripping themselves. They're switching to their backup generator where they can get cleaner power effectively. Um so this happened. Um, again I mentioned I live in Virginia. I live in Fairfax. Um, sorry, just to, you know, you can all come visit. It's fine.

But, um, live in Fairfax. I think it was July 10th last year. It's a really I remember distinctly what happened because I had also lost my AC for like three days to give a personal version of this story as well. Um, I hadn't had AC for three days because I'm not very good at making phone calls to come repair my AC apparently. But it was really really hot and uh I was slightly dying and finally got it back. It was over July 4th. Got it back. Day later my power goes out for four hours and I'm like, "Oh, come on." Like I'm going to die. The puppies were hating it. I was about to move them to a Marriott. It was all

bad. But guess but it was a really bad day. There had also been some big storms roll through and as it turns out um in the report a lightning artor had failed. meaning that the the thing that's meant to absorb if lightning strikes an electrical line, there's a lightning arresttor that's meant to stop it from blowing up other things like a transformer. Um, it failed. There's a whole story about why that failed, but um on a large transmission line in the eastern interconnect that resulted in a a permanent fault which basically took down a major part of the Fairfax area. Um, it also which was included my house. But the thing that also happened during

these lightning strikes, there was this kind of power wiggle. The voltage was varying because it was having an impact from the electric strikes on the grid um of 60 data centers decided to go onto their backup power. And this is just in Fairfax County. They decided to evacuate the building, get off the grid because they were going to be damaged. And that was a drop in power of about uh or of load of about a large load drop of uh 1480 megawatt. So 1.5 gawatt roughly just >> Scotland >> that's about a quarter of Scotland trucked offline at the same time in just that one county and so there was an event caused by that other things

started to go offline because they suddenly lost all of this. They had to get generators to go offline as well because it was so large it dropped offline and suddenly everyone's like what just happened and they were like did we not have a control on this to stop them from doing that during because that's really bad and they were like no. So there's now been reports and I've been going through the reports and about 200 other events related to this over the last few years. So you know it's happening now. >> So that volatility then so let me say this and see if I have this right. So the volatility uh data centers can't handle that kind

of volatility. No >> so they move to their backup power. >> That shift to backup power changes their demand on the grid. >> Yes. And that further exacerbates >> like this. And so in talking of my numbers of Scotland, I gave this as well a couple years ago and this was related to load during sporting events where uh Andy Murray was winning Wimbledon and everyone in Scotland during one of the set breaks went and turned on their tea kettles at the same time to go make a cup of tea because that's what we do. Okay? like went and made a cup of tea at the same time, but it caused a frequency event in the UK because so many people

did exactly the same thing at the same time that basically also everyone sat down at the same time and stopped doing things that involved absorbing load. So basically there was two things that happened. Everyone turned on their tea kettle and everyone sat down and stopped doing things that would mean they needed to generate load and that caused a frequency event in the UK because they dropped about 220 megawatts at the same time. Again, in terms of Scotland, this was Scotland. So they like this was Scotland. Um but they they dropped the load and there was a frequency event because it wasn't expected that that would happen. Um because we have to plan for these things as well. If you expect

that one day in the middle of the day when you forecast this, oh by the way, there's a sporting event, so there's probably going to be an increase in load at a certain time, the grid operators will be like, we're going to have to turn on those generators or do this at that time. And they will have a plan for it. is built into the dispatch schedule. This was unexpected because Scotland's really bad at sport and it wasn't like they were winning was unexpected for the everyone. So, you know, there was that. So, everyone in Scotland stopped and that caused an event. So, imagine that for these data centers unexpectedly doing things because they're training a

different model or doing something else. So again yep >> sorry. Basic question. Let me get you a mic. >> I can throw this one.

>> Yeah. I'm not a power guy, so this is really interesting for me. Help me understand. If everybody in all of these manufacturing plants turns off their power, there isn't a demand on the grid anymore. And so all of a sudden they would have 1.3 gigawatts to spend for everybody else. Um I would understand that there would be a big demand as they go back into the power grid and turn off their generation, but why would it be a problem when they start going into their own local generation? >> Well, load and generation needs to be balanced. So if they instantaneously take 1.5 gawatt off the system, um that's big enough to push against the

size of the system and the generation that's there. So there's a bias in every system of basically it's like 1.5 gawatt per hertz. Meaning if you drop that amount, you will shift the frequency of the system by that amount as well. If that makes sense. So everything is based on our system being balanced. And so if you if your frequency starts to drop, you start to lose other things because you can't operate that way. So it's similar to what happened in Spain essentially. So they started to lose the balance of the system. They had to start dropping different things offline to try and get it to balance back together and they couldn't. So like 1.5 gawatt is big

in that it's we did this math yesterday because we thought it was funny. I think it was it's around 14 million tea kettles. So unless you have some control in your system that would have you as a customer, we're going to turn on your washing machine all at the same time to help rebalance that. You'd have to drop a generator and have something that can drop that fast. So like coal can't drop that fast. It's a kind of base loaded generation, but diesel for example could drop offline instantly. >> What's being what's being used I had to learn this too. What's being used and what's being produced has to be close to each other if not exactly in balance. As

close as possible. there's only a little bit of tolerance, >> you know, if if it goes off. >> If you have a big frequency shift, what would happen? I think that's the question. >> Yeah. See the Spain outage like so essentially if you have a big shift in frequency, your protection systems and the in the grid start to action. So they're trying to save the rest of the system. So they'll start to drop off say this load, this generator, pieces of the system that were planned to be taken off to try and bring that back into balance. >> Bring the balance >> and if it keeps going eventually you get a Spain style event where you lose

everything on your system and you have to restart it because at some point that volatility can no longer hold and they they trip offline to protect themselves because damage is worse than the power outage if that makes sense. So >> should we take questions now or do you want to keep going through? >> Just kind of a quick comment. One second. Sorry. There there few folks with questions and I just want to be thoughtful. Do you want to try and >> Do you want to pause? >> One more question and then we'll go on to a couple more slides. Who wants to >> Okay. Are we going? >> This gentleman back here had his hand up

right there. Yeah. >> And then if you can just keep them. We're We're almost, but we do want to take questions right here. You you sir. >> Yeah. I just I think that one of the things that people are missing is that the tolerances are incredibly tight. I mean like we run on 50 hertz. >> If you drop to 49.9 >> then you're starting to trip stuff offline >> and uh and the damage that can be done is pretty incredible. I mean you can actually melt lines if you get too far out of >> if we drop like three hertz we start to lose parts of the system. So the balance is very tight on our system as well on

purpose like so. >> Okay, let's keep going and then we'll go through your questions. >> We'll just that is also relevant to the next slide as well. Just to bring that up on the tolerance, um, normal non-AII workload on a cloud data center looks like the left where it's it's not that volatile. It's not, you know, moving around too much. I I borrowed this from the internet, but um, the AI workloads are kind of insane. So when they start to train models or start to do different things, um, they can go through, this example was 15 megawatt. So 10 times as big a spike in the system and a rapid spike. That's why I'm saying volatile

AF. Um because >> yeah, like 10 times the size of a spike basically saying I've decided to train this model at this time or it hits a checkpoint and is pausing. That's that that sort of spike just locally on one data center processing is huge. And that's again a variable. You can't trip things off that line. There's only a couple types of generator that can absorb that fast a spike. Um, batteries and natural gas are two of them actually, but your coal plant can't ramp that fast. Your your nuclear can't actually do this either. So, you need a generator that's spinning to be able to do something to help resolve this. And that's really really hard to plan for

when the people training the models are not the grid operators >> in the in the city on a on a hot day or otherwise. Coned had hotlines to waste management to other places to say take this on or we're going to do this so that they could communicate when there were big things happening. How many how many of you think that the AI data centers or the AI companies are going we're going to retrain our model and give you a heads up that this is happening? >> Yeah. >> Go ahead. >> We keep moving. >> We need to keep moving. So um lastly, the interesting part is one of the the positives of all this is these small

rural entities are trying to modernize. So they're also adopting AI into their operations and into their systems as well at the same time. They're integrating models for things like managing their uh um what we call customer service. So half the time now when you call a customer service for your utility, you're probably talking to a chatbot initially. It's fine. They're probably pretty good at this point. They don't tell you weird stuff. It's fine. um or they're buying products with it integrated. So you can go buy an outage management system or wildfire management tool and it will have some really cool algorithms burn for wildfire management in particular has been one of the areas that's had huge benefits using AI for

faster detection of you know sparklike events that burn things down. See smoke in Vegas yesterday and then integrated to devices. Most of the meters that you get for your house now, the new ones that are coming out have integrated edge AI processing for different things as well. That's kind of insane to me actually. But most things you're buying now have something in it that claims to be a I'm not validating that it's real and it's not just marketing material, but they're claiming it. So, I'm going to go with it. >> Um, and again, they if they don't have cyber teams, a lot of them are saying, "Hey, you know where we can get efficiencies for um cyber security?"

again as you mentioned using AI. So here we are. So they're also adopting it in these entities. But again just to summarize our models don't account for this um for the AI supporting infrastructure. I asked actually a few of the AI related companies not looking at any people in here in particular just like do your models account for the infrastructure you're working on for your systems? No. Okay. Um, we're defending from AIdriven attacks with AI depends on this backup generators and off-site generation is still electric infrastructure. Um, and it's still electric and it's still water. It's still there. Um, and we're becoming dependent on this. Just one other slight worry. Um, when you go work in a utility

operations room, there's usually, and I'm not going to not going to lie, there's usually some old dude that's there who absolutely knows everything about he's the grid whisperer. He knows everything about that system. He's there. He's the the the the book of how you do this. We're now looking at those and people retiring and people are saying, you know what, we probably just need to use AI instead. I think there I jokingly called I think we're developing old white dude AI, which is unfortunate. Sorry, old white dudes. Um but you know, >> I resemble that remark. >> I know. Um so I think that's being developed and essentially we're going to have a brain drain. No one will actually

know what it was meant to do on the system. So there's this other issue coming in as well. And again, concentrated large load, we talked about this. It's enough to push against the system frequency and cause a resonate. So that's absolutely new to our system as well. So we need to win. You can say this. You wanted decathlon in here. So >> I wanted decathlon because I think people think about it as a race and it's not it's not one event. It's a bunch of events. And in in this case, China's planning it that way. think about supply chain, they think about cyber, they think about critical infrastructure, they think about people, they think about economic security, they think

about ownership down the supply chain. You want to dominate a technology, you dominate the infrastructure. You want to defend a technology, you defend the infrastructure. I'm sitting next to one of I think one of the countries maybe the world's foremost experts on battery energy storage and ownership of that battery energy storage. So when you hear this competition, geostrate strategic competition with China, it is not just about the frontier models or who's going to get to aentic AI or even who can buy chips. It's about the infrastructure underneath that. Remember what we said up front. If you want to talk about AI, talk about the infrastructure upon which it depends. >> So I think we've mentioned this one

already, but traditional planning is failing. We've these load forecasts aren't predictable. We've got transformer shortages. Guess where everyone's getting their transformers for like from in this country. >> Got a surprise for you. >> Yep. Yep. Yeah. Um we've got data centers without models. When I say not models for AI, we've got data centers without models for how they operate on the electric system. So predicting them is quite hard to actually do at the moment. Um and again, we've got these sort of static approaches to planning that were just never built for this. Well, so that all sounds terrible. Have a great day. >> Sorry. Um, seems pretty bad. There's things we can do. Um, we kind of need to

do it as well. We It's not We can't stop this. There isn't a We There's no going to a utility and saying no, don't do this. They are not going to follow those approaches. So there's no no here. There's a just a how we make this better. >> Um, so what can we really do? Understanding the priorities is important. understand that why they are integrating these large loads to their business models and what benefits that could give us to cyber security being that they have money to hire people and do it now. So helping people with those approaches is really important. Understanding it's there and why they're doing it. Um again encouraging good citizenship like of the customers. Again

we're talking about consumer and commercial load here are two different things. Um the world has shifted. We need to understand that in 12 months things have have shifted really fast. It's not what can we do. We do need to just know that this has happened. There's not just the small electric anymore. There's a small electric and giant load. Um our highest consequences are now dynamic and we actually need to understand the use cases they have for this. Like if somebody is buying a tool that says we're using AI, helping them understand how to responsibly use that in their own environment in a way that they can actually depend on it as well. So protecting AI's foundation is also

important because AI attacks are a thing. Um but don't tell people don't because tell them how to do it right because if you tell them don't you immediately stop them from wanting to even talk to you. So when you're a cyber defense type person saying well you shouldn't do that. Okay it's not the same as plugging in your phone to a different USB. It's something else. It's a bigger problem for their business model that they need to do this. Resilience planning needs to consider the supply chains for this. Um, and again defending the foundation of AI, helping small utilities understand their AI creep and demand transparency is really important. You love the demand transparency part. So that's

>> I I think this is kind of cool. The demand transparency. So there's two parts to that. There's us remember no one's coming. It's us to demand that transparency from those organizations. And there's also transparency about the demand about where it's coming from the behavior of it the volatility of it. When you hear AI think about infrastructure and when you think about threats or resilience and a to AI AI and the grid are inseparable. I hope if you leave with nothing else today it is that and also we should use Scotland as a form of measurement >> in the dictionary. Um other parts we've been talking about you've if you were here yesterday Ginger and folks were

giving the and that there was a CI a cyber farmed engineering workshop I believe as well again going to manual for critical functions is something that we've had for a long time um but I'm not sure we have go to manual for this situation so we need to work out how to get through that um but again consequence informed decision-m is really important I've been using fault location as an example for um consequence your utilities. Um they do well I'll cover this with other people later because we're running out of time, but considering criticality is also a problem. Um these small utilities that I'm talking about were often not considered critical because you know they're small. They're now the new

critical. It's the the new normal is they're the new critical infrastructure. So if you're saving the world with your AIdriven solution, you need to actually understand the resilience of the solution also. So chaos engineering is something I really love. Um, but we've been trying to work on how to do that for the electric grid because that's really what we have now is a chaotic system. It's still operating. I want to add we still have one of the most reliable electric grids in the world. So, we're not still not failing, but um, one of the things we've been really interested in looking at is how to abundance plan, which is a whole other world. How do we plan for this? How do

we build the models? How do we just assume everyone might make a random decision every given day and work out what that would actually look like for a system? We need to do that for cyber security in these systems. Also assume random is my assume random and assume chaos at all times is my um pitch here for how do we actually plan for this. But planning for failure is probably the biggest lesson here. This is an image of Spain. Um, I've got some funny, this is going to be in another talk I'm giving on Friday, but uh, planning for failure is kind of the the big thing here. The one thing when people talk about the the

outage in the Iberian Peninsula, it's always like, "Oh my god, it was so terrible." The I actually have a slide here that was the British were eating biscuits and beer. It was so bad because southern Portugal didn't have some of the big holiday destinations didn't have power and they're like, "We're relying on our cookies and beer." And we're like, "That seems wonderful. Like, you'll be fine." Um, one of the bigger problems that happened was people started to believe it was a cyber attack and then started to panic. Um, so the people that posted it was a cyber attack on LinkedIn started to create this fear, uncertainty and doubt that spread throughout the world. I even had a call

from my dad who is a forensic scientist and he still called and said, "I heard it was a cyber attack in Spain." I was like, "Dad, probably not." Whole other talk on what happened, but planning for failure was key. Um, Spain was able to restart their grid in about 10 hours. They did a full black start on their system, restarted it from scratch. That is actually a huge success. They had planned for that. They knew how to restart it. They didn't run into many problems. They solved them as they went along. They had planned for this to happen. So, they were able to restart. We have Blackstar plans. I'm not sure we have blackar plans that include data

centers. So, we have a whole other resilience problem there. But planning for failure is kind of key and that's the sort of big message here is we need to plan for this to happen and we need to be able to recover and do the right things. Um again I'm just making posts about this that what happened in Spain because it annoys me that people said it was a cyber attack but you know um but yeah the end is now. Um so would you like to welcome a robot over? >> Yeah. No it's okay. I think we we're at time >> couple questions. So, let's do let's do um let's take all all the questions and

then we'll try and bundle them. Go ahead Josh. >> All right. So, hopefully you can see that why we love Emma and and >> Emish Hype Man. What did I tell you? I said expectations. I was just going to jump in a little bit. Not a lot of substance. >> Really good teammates in this crazy cavalry thing. They don't give you what you ask for. They give you what you need. I hope this is both overwhelming and stimulating. But if you look at the cascading failures with no water, no power, no hospital, this takes it to a whole new level. Uh there's a lot of questions in the room. So I'll just say the the levity bit here is

>> my biggest concern is if we have a massive outage, what do you restore first, the hospital or the data center? And in some of the international exercises I'm hearing, the answer is the data center. And the second answer is not the hospital. And the third answer is not the hospital. So we have to take everything you heard yesterday and for the rest of today and tomorrow and realize that this is probably going to win in the ruthless prioritization stuff. So maybe you made that dynamic range of demand green because Loki is holding that chaos together. Um but I I hope we can maybe decouple with small nuke plants or something. But if there's any other non-cyber ways to decouple

public safety continuity of operations from data center continuity of operations, I hope those are in the mix as well. >> I just if your restoration options included you needed to process data through a data center to be able to restore the system, then the data center comes before everything else. That's my that's the challenge is if people are relying on an AI based restoration solution then they need that before they which I mean could mean they could push it offshore but that's another risk. So >> for time let's let's take all the questions. So yeah please here and then I think we have one keep your hands up so then we can come around and take all

the questions. Go ahead. great presentation. Was wondering if these AI companies are paying into these small energy companies and and to to build that up and and if not, where do those decisions come from? Is that at the state level or at uh kind of the the regional co-op level? >> Thanks. >> Say yes, they are. So that's they are paying money for this to these entities like we're seeing them get grants from >> like not to throw Google under the bus but Google gave a big grant to a small rural utility because their data center was dominating their load and >> but is that voluntary or is that like >> it's voluntary? Yes. So it's

>> but to to kind of daisy chain off of that does it matter that the large companies are buying their own nuclear facilities like Microsoft turning on three mile island or two and a half mile island so to speak and Google's buying their own. >> How do we feel about turning back on an old plant that's been end of life from a cyber perspective we can >> and also they may they may have the generation but the demand >> is still on the grid. Yeah. >> Yeah. I I hate to uh bring up the idea of using technology to solve the problems that technology has uh has caused but uh as a a user of AI or at

least my grad students uh the uh uh use of batch systems and so forth is pretty common. Is there any hope for a data center energy use API that could reduce the uh the or increase the delay between the uh the load changes? There is, but that doesn't mean they'll use it. If that makes sense. >> We had a question back here. >> I was gonna say the API would probably based on an AI based solution for managing the AI the AI data. >> We have to your the point underneath this though is changing the way we're doing load forecasting that this gentleman right here has a question. >> Rapid fire. >> I got it first. Sorry. Okay. Uh, so a

couple things I did not see in your d your thing. Uh, the issue of public safety power shut offs that we have on the West Coast because they they do that now preemptively to stop wildfires. And my concern is >> they're going to they're going to prioritize keeping power on for data centers and we already have giant issues with people not having power in times when they desperately need it to just stay alive. And so that's I don't know that there's a question, but I think it's an important consideration. And then the inte the integration of small nuclear reactors, small modular reactors into this. I think from an emergency management perspective, that's where I come from. There's very little

regulation around those. I've read the regs. They suck. >> Yeah. >> And and they scare the crap out of me, but they're being touted as the solution. And I like I'd like thoughts on that. But >> I was going to someone who works for Idaho National Lab. I have to say nuclear is wonderful. >> Oh, patience. >> Uh right there. But just on the on the safety rags um this is kind of what I think about as the ruthless prioritization um that uh you know we tried to look at under cyerspacearium or DPA like what do you what are you doing first and can we get everybody to decide on what we're going to do first? The answer is no.

Last question. Yeah, go ahead. So, real quick, um, I was going to ask this before some of the other questions about decoupling, but what does the ideal power grid look like so that we can effectively decouple the load from the volatility problem and push the cost of volatility to the sources of volatility? >> Yes, >> good question. Redesign of the grid. >> I mean, economics, >> I'd like to entirely redesign the grid if we could at this point. It's a Oh, I was helping Um I would like if we could we again our grid was designed for like 10% margin on top of where we're at just now. I don't see how we do this without major

redesigns but we also are having struggles building anything. So I mean transmission takes about 10 years to build. Um so designing that's kind of where this chaos engineering point is of redesigning or re changing how we plan so we can build things for these unknowns in the future as well but still satisfy what any customer may want to do as well because you're still you still have free will putting solar in your house buying electric vehicle not doing those things is also something we still need to plan for also >> please join me in thanking Dr. Emma and Nice. Thank you. [Applause] Now, so our first hour got us very heated up on AI and the grid and but the

uncomfortable discussions are not over. So, doesn't that sound great? So, uh for our next discussion, I'm pleased to welcome Mr. Joe Slowwick to lay down some truths about a very uncomfortable topic. Oh, people should come in and sit down. We've got seats up front. Don't be shy. Sit down. You can sit down. Everybody can be comfortably uncomfortable in your very fancy lounge seat. So, uh we'll get Josh uh Joe hooked up here and be on our way.

[Music] Heat. Heat.

[Music] Heat. Heat. [Music]

[Music] [Music] Where are you going? [Music] Don't

worry about me. I do not have a technical background, but I did uh do lots of humanity stuff and then I joined the Navy because that seemed like a smart idea. So, the way that I like to describe it is that I know how to swear and I know how to swear very elegantly. So, we've got that going for us right now as I work through this hurriedly. But the core proposition behind this is that Josh has already pointed this out that this is a set of discussions not about what you want to hear but about what you need to hear. And one of the things that I come up with often take your laptop,

>> okay, is that we talk a lot about cyber physical disruptive events and as Emma's noted earlier, how many outages in the US are not are related to cyber events? Zero. How many total across the world? Handful. Couple in Ukraine. You could align this to, you know, military and conflict zones and so forth. But if we're talking about real cyberphysical operations, what are we really limited to? The sword, which I don't think has been mentioned yet today, stuckset. It's funny that I bring that up because what happened just what a week or two ago, there was the congressional panel on stuckset. 15 years later, we are still talking about this that this notional boogeyman of the state sponsored threat

actor doing something to cause a cyber physical impact that's going to turn the lights off, shut the water down, etc. And it hasn't really materialized yet. You can make the argument that that's because thankfully we have not gotten into a shooting war between China, US, US, Russia, etc. But we are seeing a lot of disruptive events right now. So one of the things that for those who were attending yesterday and Josh already hit on this and last year this came up as well is that you know what is one of the main issues in terms of information security problem sets that we're facing e-crime, ransomware, BEC, etc. that these events are the day-to-day eb and

flow of what's going on as far as what's impacting individual organizations and resulting in disruptive disruptions to critical services. So to speedrun through my notional slides, which trust me, they do actually exist that we've already seen impacts. You can go back to the story from a couple of years ago about the ambulance that was rerouted from the hospital in >> Say again. >> Duddenorf. Yes. We good? >> We're going to find out.

Hit there's like three monkeys trying to hump a football or something like that up here. [Applause] [Laughter] >> All right. Damn USB. Going in the direction that I want you to, not the direction that you do. >> Or you can just hit the space bar. >> Yeah, you know what? That's fine. [ __ ] it. >> Okay. >> Okay. Thanks, Dave. Yay. Okay. So, I talked about this. Quick disclaimer. I'm representing myself today, not my organization. Like I said, Navy back in the day, bounced around to DOE, uh, bunch of product companies, worked at Draos, worked at Huntress, which actually will come up in from an MSSP perspective, uh, in a little bit in this

discussion. was at the MITER Corporation for a little bit. Now I'm a data miner, but I'm here as Joe, which means I get to say whatever the hell I want as long as you're not live streaming this to my employer and I say something that they don't like. But anyway, the state of the now, which I was getting into, ransomware more brutal than ever. Uh, you know, 44% of all breaches in the DBIR that came out this year had some ransomware component involved in them. Now, certainly that is a subset of already existing security incidents, but you get the idea. So talked about critical impacts of services and health care and education and this does have

important considerations to it in terms of financial impacts. Again it was mentioned yesterday in the intro to this track that hospitals don't have that many weeks worth of runway that if they're not generating revenue to the point where they are able to continue to operate. Then there's also the service impact component to things as well where if I don't have it up then how the hell do I operate? Whether it's electronic medical records, whether it's the, you know, everyone here who has a kid school right now, I'm sure, is used to having the Chromebook or whatever else and online education platforms that are supplementing or deleting a lot of items. To say nothing of getting into

incidents like just being able to do email, etc. All right, sorry we're just kind of blowing through this because of our technical difficulties here, but yes, getting to our Dusseldorf example, there was the Doppel P doppel payr ransomware event that resulted in an ambulance rerouting, which didn't, you know, the ransomware didn't kill someone, but the delay in service ended up resulting in impacting quality of care, which I think is the important way of putting this, which I know Josh has mentioned numerous times, including in congressional panels and so forth. And we've seen this even more recently in studies that have shown that as you start impacting the ability of hospitals to start providing service at a

consistent or in expected level whether you're talking about the blast radius item where hospital A in region has an impact resulting in patients being pushed out to other institutions which starts bringing down the quality of care more generally that we start seeing this degradation in care. But the thing is what we often talk about though when it comes to these items are state sponsored disruption. So, Vault Typhoon, which I do a lot of research on both historically and past jobs and uh currently kind of on my own because I'm a freaking nerd and this is just the kind of stuff that I dig into that we emphasize that this is this horrendous existential threat to the operations of

critical infrastructure that in a Taiwan straight scenario the lights are going to go off and other bad things are going to happen. or more recently or well concurrently cyber avengers which I like to say is computer network annoyance as opposed to computer network attack that uh you know defacing hm HMIs and similar sorts of items not in itself concerning but the fact that they're able to access these systems and targeting critical infrastructure certainly is concerning and what could mean from there the thing is though is that while there's lots of concern over state directed cyber impacts the results and actual impacts have been minimal to non-existent with certain exceptions You know, back in my day in DRAOS and

DOE and other areas, as well as working uh as an independent researcher, I've dug into the 2015 and 2016 Ukraine attacks, the 2017 incident in Saudi Arabia involving safety instrumented system malware. There's been some more recent things in Ukraine that we've seen, but you know, look at the panel that I referenced earlier on the 15 years since Stuckset and a lot of the expert commentary around that and we really haven't seen what we expected for something that took place over a decade and a half ago now. in terms of cyberphysical operations. So are we really focusing on the right thing because there's significant policy and political focus on state-directed adversaries holding critical infrastructure at risk for disruption

and destruction. But the thing is such impacts are already happening today. Now, it's not some esoteric exotic cyber physical payload that we're talking about, which maybe you'll get see something like that pres presented at, you know, up the road at Defcon or whatever in a couple of days, but rather we're seeing fundamental aspects of IT operations that underpin the capabilities of these critical networks to operate being held at risk, disrupted or destroyed as a result of criminal activity. And the thing is if you look at some of the like more impactful elements of cyber offensive operations uh targeting critical infrastructure or economies at any sort of scale you're talking things like nutpia which was pseudo ransomware uh operating as a

wiper effectively as opposed to an actual ransomware pay payload or other uses of wipers like we've seen in Ukraine that have actually had significantly more impacts than any sort of science project that speaks an OT protocol and can result after doing multiple sequences of steps to disrupt operations. So a couple years ago I gave a sky talk on like what we would actually need to do to disrupt the grid and it echoes a lot of what Munish and Emma were talking about uh earlier is that the possibilities exist but the degrees to which that you need to get a number of things and a number of things right in the proper order of operations to make it happen at scale is

non-trivial. However, being able to have localized impacts by simply targeting and blowing away the IT or higher level OT operations and eliminating the operations of all Windows systems in the environment, that isn't terribly hard. And we're already seeing that reflected in the way that ransomware operators, whether you're talking in Akira, Dragon Force, take your pick of the flavor of the moment, are currently operating, or rather the affiliates that are using these ransomware variants are operating today. So we have to ask ourselves, okay, if we're talking about allocation of scarce resources, are we really focusing on the right or the correct problem by looking at the notional cyber physical OT attack scenario versus what's going on already in the

day-to-day? And we've seen state sponsored actors actually piggyback off of in very impactful operations. So there's an important consideration about this though in terms of how these payloads are actually being delivered. So, the route to ransomware. How do I get to a point of deploying a payload? Who are my pentesters in the room? Yeah, I've got a few. Okay. Who are my offensive cyber actors either currently or in a past life? Don't raise your [ __ ] hand. Okay. Anyway, um we're talking about a lot of fundamental overlaps in tradecraft right now. So, what are we talking about? Things like credential harvesting and reuse, weak authentication schema, whether you're talking single factor or not very well

implement implemented multiffactor. We're talking vulnerable edge devices. So whether you're talking about the small office home office router device or whatever that's sitting in your environment or you know pick your network appliance vendor dour forinet uh sonic wall etc. Adversaries are identifying fundamental weaknesses either in how these devices operate or they have vulnerabilities in them that are either being exploited as true zero days or rapidly weaponized post discovery to enable access leading to living off the land binaries and scripts. l bin LBA bass activity uh but also seeing a division of labor in terms of ransomware deployment. The funny thing is is that a lot of these same sorts of items are reflected directly if

you look at how a vault typhoon operates or even how some of the higherend Russian APS are operating in Ukraine. It's not about developing the bespoke esoteric piece of unique beautiful malware or whatever that is unique to my organization, but following a script that looks pretty damn close to what you would see for pentesting 10. Well, maybe not 101, but like 301 or something along those lines in terms of operations because we have to ask ourselves, is this unique? And the answer after doing some research on this subject is that it very much is not that we're seeing adversaries from e- crime to AP increasingly rely on similar operational mechanisms. There has been a convergence

in tradecraftraft. I was hoping this was going to be available by time for this presentation, but I just gave a presentation at uh the first annual conference in Copenhagen that dug into the data looking at MITER attack mappings, looking at the DVIR, looking at the MTRS report and we see this reflected in an analysis of intrusions that whether it's due to efficiency, it's just easier to operate this way, general availability, but also defender failures. We're not talking about really rocket science sort of techniques here, but adversaries are able to leverage these common mechanisms in order to uh operate quite effectively with without looking very different from one another. Now, this presents challenges in terms of attribution if that's something you

care about, if you're a CTI weenie and you want to talk about, you know, whatever mythical AP or whatever you're tracking. But it also has potential benefits in terms of defense because as we start looking at converging tradecraftraft we also talk about converging mechanisms to try to disrupt or uh dissuade its use. And just to give a quick view because I know we don't have a whole lot of time here. You know, if you look at this was just taking the latest MITER attack release for CTI objects and like okay, show me what are the common techniques that are referenced after removing a couple of things that are just like basic you must haves and we see like adversaries love

using PowerShell, adversaries like using spear fishing attachments, etc. Um, and it looking especially at adversaries that have been updated most frequently, we see that that convergence applies even to a greater extent. And similarly in the m trends report uh looking at what the folks at Google cloud because I'm not going to call them mandantine anymore because it really bothers them. Um that's maybe not nice but anyway similar not quite as extreme but as you start looking at individual elements of intrusions and forensic investigations that have been performed uh by those folks we see again lots of commonality in terms of how these intrusions have taken place. Now this results in some conflicting defensive priorities though because from

a government military strategic sort of perspective u state sponsored cyber is the primary risk like we can't get around that or whatever if I am national government executive etc like that you know this is simply important to me thinking about the big picture disrupting multiple entities causing whole of economy impacts so I'm talking about voltton salt typhoon I'm talking sandworm I'm talking similar sorts of entities but the thing is is that for most asset owners and stakeholders the thing I care about is like, okay, the Taiwan Strait scenario is mythical in in as far as I'm concerned. It's notional. It doesn't necessarily impact me in the immediate sense in the sense that I'm just worried about day-to-day

operations, continuing to operate, continuing to generate uh continuing to make revenue. So, that means that concerns are generally about more immediate disruptive scenarios that impact operations in that direct fashion. Thinking again ransomware, BEC, etc. like your Akira, your scattered spider, your Dragon Force, etc. But the thing is is that if we could start looking at this as not being a eitheror proposition because whether we're talking about our mythical typhoons or talking about our [ __ ] little kids from the comm or whatever that are doing stuff in order to impact hospitals and schools if there's a lot of overlap in how these entities are operating it provides for some opportunities. So certainly whole of country and specific

asset owners have different perspectives and concerns. That goes without saying. The result is that there's a focus on different threat actors in defensive operations. But if we can adopt this perspective of convergence in activity, it opens up pathways for common actions that we can take uh that would help resolve or help address fundamentally different adversaries in terms of intention and in terms of operation. But looking at those single points of failure and adversary operations in order to kill a couple of birds with a single or a handful of stones. So in looking at this and we talk about things like do the basics, eat your cyber security vegetables and crap like that or shields up. I make fun

of those all the time. I blogged about it a couple of weeks ago or whatever and saying that it's a cowardly approach in approaching defense. So justifiably derided in a lot of ways um especially because these are often used in conjunction with abstract for the perspective of most asset owners uh threats that won't hold them at immediate risk. Again, Volt Typhoon, I will say right here, very concerning threat actor that has gotten into a number of environments in some very scary areas, but their actual record of both OT network access and OT impact is zero to very small, at least based upon information we can talk about right now. Um, so as far as the threat that it

poses to the everyday asset owner, whether we're talking that utility in New England that got hit that made a big splash in a register uh article from a couple of months ago to the island of Guam, it's like, okay, this is a concern. I need to be worried about this. But at the same time, if I can get ransomed and have my entire IT network blown away tomorrow, what do I focus on? But the thing is is that if immediate value and impact can be communicated across state sponsored and e-rime adversaries, looking at the ways in which these items overlap with one another quite significantly, the incentives for day-to-day operators to adopt the items they need to defend

their network, not just against the Akira affiliate, but also from a typhoon, become much stronger. um it becomes a much more easier cell, a much more easier way of justifying allocation of scarce resources where resources even exist in order to try to address these problems. Because the thing is we can start extending defensive guidance in ways that start identifying what are the key pain points for adversaries in operating and having and having having had an adversarial perspective on defensive items or whatever. Any of these items would represent maybe not completely eliminating but definitely crippling things in order to overcome. So starting first off with just authentication mechanisms. Uh you know I multiffactor everything and there's

almost no excuse especially for any sort of service that is external facing or accessible in any fashion that that should not exist. It is relatively easy in order to implement at this point in time. And yet we still see single factor off externally exposed or weak multiffactor authentication uh entities leveraged and abused whether we're talking scattered spider or some other entities that can do the social engineering route in order to get around the help desk etc. We could also talk about like okay if I can't do it myself you know do I migrate to a third party thinking about identity access management single sign on etc. little diceier because now we're talking about money that a lot of these organizations

may not necessarily have. But thinking about a lot of the at least modern suites of software and applications whether we're talking about M365, Google Cloud instances or whatever is that we need to start thinking about more options of leveraging what is thankfully being baked into the things that we're using and turning it on effectively getting us to things like access. So, who has an external facing SharePoint uh instance that was hosted internally or whatever uh that was compromised in the last month? Because if you have a SharePoint instance that is external facing that was also hosted locally, you probably did have it compromised in the last month. Um but thinking about items like well why is that asset even exposed

in the first place and doing things like again cyber hygiene items to try to determine whether it's my own network as I've designed it or what the integration contractor or similar installed at the point in time thinking through how are these items been put together and what is it that I'm exposing to adversaries to take advantage of and then applying things like that attack surface management and improved vulnerability management in order to try to get ahead of some of these problems as well. Finally, from a movement perspective, again, while we're talking about entities that don't necessarily have the most robust in terms of IT administrators and shops, if we're talking about a rural hospital or a

small school district, at the same time, thinking about what are the key items or whatever that I know that just simply are important and can't be compromised in some way. First off, thinking how can I migrate away from owning that myself if I know that I can't administer it effectively and going to a thirdparty provider. But then also thinking about ways in which I operate the things that are residual and left behind to try to improve the way in which these things are put out there and uh secured. So the funny thing is is that this is comes from the joint cyber security advisory put out by CISA and a bunch of other partners on Bolt Typhoon.

Uh bunch of network hardening strategies uh lots of really complex items but also some fairly basic things as well. The thing is if you start looking at this defensive guidance, it essentially mirrors the items that are necessary to defeat most ransomware entities at present as well. Again, we're seeing that commonality in trade craft that if we can start showing that the benefit doesn't just extend into this notional national security issue which may or may not manifest in two years, five years depending if you listen to Josh or listen to Bryson uh if you were in the morning talks yesterday but uh something that may manifest but I don't know that it will but also reflects on what's going to impact my

day-to-day. So in this sense, reframing a shields up language to empromise immediate tangible security outcomes should I think provide a greater incentive to start adopting some of these principles in order to try to get around the issue and reframe things in a way that would be more justifiable or defensible. So what's the call to action here since we don't have a whole lot of time here? So first off, I really am a big fan that communication is key for a lot of items in terms of being able to incentivize action or to prompt action by others. And so reframing emphasis when we start talking about the state or even commercial ways of communicating threats and in ceasing

uh our glorification of the potential AP uh state sponsored or whatever activity and reframe things into more direct tangible impact scenarios can start changing the conversation in a way that the incentives can start pushing for greater action by critical infrastructure owners even those that are relatively less welloff than others. The result should be improved defensive posture against multiple adversary types that fulfill multiple perspectives that in the course of defending us against a scattered spider event which is very prolific right now or pick a nasty ransomware actor that likes to pop off or pop uh school districts and so forth. Then in the process of doing so I'm also defending against much nastier things as well. So translating our messaging to

emphasize immediate security and operational benefits can enable improved posture for those strategic defensive items as well to sort of link the two together. However, there are limitations in this approach that I do want to acknowledge. So common defensive actions to address multiple adversaries is very good. But many of the organizations for those who've been paying attention throughout this track this year as well as in past years is that many of the organizations that operate critical national infrastructure remain below that security poverty line and lack the capacity to even implement these mechanisms. There is a significant disconnect between the resources, talent and motivation between adversaries, whether we're talking a vault typhoon or a ransomware affiliate and the local

school district, the rural hospital, etc. And that remains something that is difficult to try to figure out a way to overcome. We can certainly try to incentivize movement that hey by doing this these actions you're not just taking care of the five-year threat, you're taking care of the current threat. But if the resources still don't exist to even respond to that, it becomes difficult to try to figure out where we can benefit from this. As a result, residual risk will remain and must be addressed somehow. There are options for that. We'll talk to a couple items here in a second where doing the basics is certainly going to be necessary, but it may not be sufficient

in in and of itself. So, we do have to think about what is left behind as a result of uh these actions and what organizations are still not going to be able to implement this effectively. So, from a future consideration standpoint, heightened defensive posture is only going to get us so far. It's a necessary item, but it's not going to solve it all entirely on its own. Additional investments are going to be needed in terms of building resiliency. And I like emphasizing things like business continuity planning, disaster recovery planning, things that we do for physical security threats and other items that have applications for cyber as well if we're talking about the potential to disrupt or impact the delivery of

physical services. It's also interesting to note that higher level entities and authorities may need to pursue something a little different to try to dissuade or disincentivize adversaries elsewhere. So this is a point where, you know, the cavalry is not coming, but I don't think that we're ever going to be able to resolve this problem effect. we'll be able to perform harm reduction to a certain extent, but if the calvaryary never shows up, it's gonna be difficult to do harm elimination to some extent. So, we need to think about what that may mean. Uh because again, we'll have residual risk left over as a result of even some of the best actions individual organizations can take. The last thing

I'll say because I'm running out of time is we've talked about the fact that the cyber ship has maybe sailed already that you know we're already putting ourselves behind adversaries and so we should start thinking about things like defend cyber having moved away from being able to defend uh mitigate and build resilience because we're dealing with installed systems and we talked about things like cyberinformed or engineering in similar ways is approaching this. But we also have to ask ourselves like what is the possibility of being able to rapidly revamp or reconstitute a physical system in order to build more resiliency if it doesn't exist already. Again, going back to trying to, you know, you're fighting the battle with

the tools that you have right now as opposed to the tools that you wish you did. And how do we orient these conversations around things like building in resiliency and uh operational planning to take advantage of what capabilities are already latent to the environment instead of dreaming about what we may be able to implement in the future. So again, just something to think about. So finally concluding slide ransomware is effect is basically what we're dealing with in terms of social disruption right now. It may reflect upon what we can uh imagine a notional state adversary would do in the future. But the thing is is that e-crime behaviors are mirroring a lot of the concerning AP behaviors that we need to

be worried about and that as a result we can get take advantage of this overlap to emphasize immediate security needs to justify the investment and the action on the part of asset owners to basically uh operate against these threat actors and secure organizations against both the immediate threat as well as the longer term. But with the admission that passive security controls will only get us so far. And to to some degree, we're going to need to figure out something else if we really want to make these problems go away. So that's all that I have. Sorry for the technical issues at the start. Thank you, Dave, for being able to pivot very quickly and for me to use your machine.

>> Yeah. >> Well, I had a question, but you kind of blunted it. So, how about we all just say fantastic presentation, fantastic recovery.

So the comment offer request to you and anyone else that heard him is yes the kind of things you would do to stop ransomware might also help you with volt typhoon. The problem is most of these target rich cyber core utilities aren't doing it for either. So I guess the question challenge is if we get the consequence informed engineering cyber inform engineering resilience settled what's the very first set of crawl walk run cyber advice that should follow should fast follow the physical resilience stuff so rhetorical for now >> y >> but like we don't want to install super expensive brittle cyber that gets them hacked with kevs but we might want to prioritize the tangible win-win for

both. Thank you. >> Yeah, sounds good. Thanks Josh. And again, apologies for the issues at the start. >> I don't do we have time for questions or Okay, >> if you have a question, come on up. Come here. So, line up and we will answer a couple of questions. Come on down. >> So, some of the things that we talked about like um Volt Typhoon and difference between local and strategic interest. pull up my talk from critical effect or from BRUCON last year or whatever if you're interested in that and then the converging tradecraft presentation should be available in another month from first and the paper on that should be coming out in the next

six months if I can figure out who who will actually publish a damn thing. Anyway, thank you. >> Hi, thank you for the presentation. Um, so I completely agree that we like communication is key and I have to admit I hate that we're always talking about ransomware because it puts the problem somewhere where it doesn't feel like everybody can be a part of it and I mean to get to ransomware there's so many steps that happen that we can prevent that we can raise awareness about um and that is sort of lost on the way about like just talking about ransomware. So I hope that this is something that we can focus on a lot more. Yeah. And um I mean

you you said it so so one thing that I was also curious about is you were at MITER um and they're basically just talking about state sponsored but um how do you see like that transitioning and providing something like mitro does for everything else around it? Do you think there's a possibility to have that somewhere in the future? >> I think so. Uh I can tell you when I was managing the CTI portion of the attack framework up until May uh when I left MITER uh that was a big point of emphasis for me is getting greater reflection of e-crime and criminal activity within the framework because one it's important but two it's also difficult and we see a some of this

reflected in adversary operations as well where we see a division of labor between initial access infrastructure management and then actual onkeyboard operations. uh criminal ecosystems are much more diverse. Whether you're talking about your initial access brokers that sell access onto like someone who's going to then work with a ransomware affiliate to deploy a payload and that affiliate is going to handle negotiations that it becomes not just a unitary threat actor that we're dealing with. And so it's not just one throat to choke uh so to speak, but becomes rather which area of the problem set am I acknowledging and do I even have the knowledge to track these items effectively to know who it is that I'm

dealing with? And it's that diversity in the ecosystem which both on the government side and in the commercial sector has been difficult for folks to wrap their heads around because we like to think about the unitary threat actor as opposed to there being a very diverse division of labor behind operations. >> Yeah. And I mean also the problem that we've run into with um being able to communicate the the challenges and what we need to do and how we can effectively also reduce risks um to like the sea level and and so on. That's that's something that's stopping us on so many levels and um I hope that we will be able to like >> evolve in that

>> together. >> We shall see. >> Okay, so we've got two more questions. Uh each question and answer is 30 seconds and not more. Go >> speedun. >> What's up Joe? Uh so quick question. One of the biggest challenges in CTI is translating that strategic risk. Translating that strategic risk in like a briefing report to leadership um into like tangible actionable outcomes and getting them to action it while also like maintaining the balance of like not crying wolf. So do you have any like sage advice for making your strategic um risk identification maybe maybe even around ransomware land harder? >> Yeah. So, first off, I'm sorry to cut you short, but we got a little bit of

time. Strategic CTI is complete [ __ ] for 95% of the organizations that are out there. No one gives a [ __ ] who the hell is actually responsible for things in the vast majority of cases, unless you're law enforcement or the military. And CTI in general should be focused not on reports, but on detections, hunts, and informing security operations. That is my bias. I'm a very tact I teach this in my CTI training that CTI's focus. If you're not thinking about how the decisions I'm supporting can impact the day-to-day defense of my network, if you can't answer that question effectively, you're doing something wrong. And it's that change in perspective we need to adopt instead of going after the

geopolitical like this is what Russia's intentions may be or whatever in a cyber conflict like ain't no one care about that other than a few people. So, how do I actually ensure that this gets translated to a meaningful security decision? And we can talk more about that offline if you'd love. >> Yeah. >> Thanks. >> Yep. >> Let's see if I can make this really brief and to the point. Um, so I think maybe as a thought, you know, some some of this might just come from the fact that, okay, you know, from like the the news media reporting on APS and all that is maybe a little sexier than like talking about, you know,

>> the all the all the different ransomware attacks. And of course, the problem is too, the people who do are victims of the ransomware attack don't want this information out there. >> Um, just one thought I had when talking about this, >> um, because I haven't really seen anything of it. a good like anatomy like if somebody if we could get information about like the anatomy of like a full ransomware attack that the whole kill chain. Yep. Like in specific, you know, like a couple specific examples >> might emphasize to the people who might have to defend against this about you know what the actual actions instead of like just saying oh cyber hygiene shields up kind of stuff because that's

kind of like a what does that mean? >> The people need they need details. >> Yeah. And some folks do that. So uh not to chill for anyone but they're not really a vendor in the traditional sense but like DFIR report provides excellent examples of walking through from their case studies of how a lot of these intrusions have taken place that go to that level of detail. But I agree that there needs to be a greater emphasis on the part of reporting entities of providing the actionable details that relate to how these intrusions took place when those details are even possible to to identify because in some cases it's just like well we got ransomed and we have no logs we have no

forensic artifacts to look at or so but where possible providing as much detail as uh possible is what's going to drive that sort of actionable decision-making that I talked about. >> Okay, put your hands together for Mr. Joe Slowwood. Yeah. Woo. Yeah. Okay. Uh it is now uh time to go upstairs and see people upstairs uh get a beverage, maybe gamble. But at 200 p.m. 2 p.m. you want to be in this room to talk about the health system. And it is sick. So you want to be here at 2 o'clock. 2 o'clock for a two-hour health uh overview. And then we're going to come back at 5:00 p.m. and we're going to learn about food

and we're going to talk about the fact that end of life should not cause the end of your life. So, we look forward to seeing you at two later. Thanks. Bye.

[Music] Hey. [Music] Heat. Heat. [Music] [Music] down. [Music] Done. Here [Music] you go. [Music] Down. [Music] Tingle. [Music] Everybody

[Music] jump. [Music] Heat. Hey. Hey. Hey. Heat. [Music] Heat.

[Music] Heat. Heat.

Heat. Heat. N. [Music] Heat. Heat.

Heat. Heat. [Music] Yeah.

Heat.

Heat. Heat. N. [Music]

Heat. Heat. [Music] Heat. Heat. N.

[Music]

[Music] Heat. Heat. [Music] Heat. Hey, Heat. Heat. Heat. [Music] Heat. [Music] Heat. [Music]

Wow. [Music] Yeah. Heat. Heat. [Music]

Heat. Heat.

[Music] Heat. Heat. N.

Heat. Heat. [Music]

Heat. Heat. [Music] Heat. Heat. [Music]

Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat. [Music] Yeah, [Music]

down. [Music] Black. [Music] Yeah. [Music] Yeah, [Music] down. [Music] Black

[Music] Heat. Heat. [Music] [Music] Baby, [Music] baby. [Music] Hey,

hey hey. [Music] doing. [Music] Hey, down. [Music] Heat. Heat. [Music]

Heat.

[Music] Heat.

Heat. Heat. [Music] Heat. Hey Heat. Heat. Heat. N. [Music] Heat. Heat.

Heat. Heat. [Music] Heat. Heat. Heat. [Music] Heat. [Music] Heat. [Music] Heat. Heat.

[Music]

[Music] Heat. Heat. [Music] feel. [Music] Heat. Heat. [Music]

Wow. [Music] Heat. Heat. Heat. [Music]

[Music] Heat. Heat.

[Music] Heat. Heat.

Heat. Heat. [Music] Heat. Hey, heat. Hey, heat. Heat. Heat. Heat.

[Music] Heat. Heat. [Music] Heat. Heat.

Yeah, [Music]

[Music]

down. [Music] black hey black hey black hey black hey black hey black hey black hey black hey black hey black hey black hey black hey hey Yeah, [Music] you [Music] Down down down down down.

[Music] Heat. Heat. [Music] [Music] Down. [Music] Boo. [Music]

Black. [Music] Hey d hey hey. [Music]

[Music] Heat.

[Music]

Heat.

[Music] Heat. Hey, heat. Hey, heat. Heat. [Music] Hey. Hey. Hey. Heat. Heat. N.

Heat. Heat. [Music] Heat. Heat. N.

Heat. Heat. Heat. [Music] Heat. Heat. N. [Music] Heat. Heat. [Music] Heat. Heat. N. [Music]

[Music]

[Music] Heat. Heat. N. [Music]

Wow. Heat. [Music] Heat. [Music] Heat. Heat.

[Music] Heat. Heat. [Music] Heat. [Music] Heat.

Heat.

Heat.

Heat. Heat.

[Music]

Heat. Heat.

[Music]

Heat. Heat. [Music] Heat. Heat. N.

[Music] Heat. Heat.

Yeah, [Music]

[Music]

yeah yeah. [Music] black hey black hey black

hey [Music] hey hey black hey black hey black hey black hey black hey black hey black hey black hey black hey black hey Yeah, [Music] you [Music] Down

down down down.

[Music] [Music] Baby. [Music] Hey. [Music] Heat. Heat. N.

[Music] Heat. Heat.

[Music]

Heat.

[Music] Heat.

Heat. Heat. [Music]

Heat. Heat. [Applause] Heat. [Music] Heat. [Music] Heat. Heat.

Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat.

[Music]

[Music]

[Music] Oh. [Music]

[Music]

Wow. [Music] Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat.

Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat.

[Music]

Heat. Heat. [Music] Heat. Heat. N. [Music] Yeah. [Music] Yeah. [Music] Yeah. [Music] black hey black

hey [Music] black hey black hey black hey black hey black hey black hey black hey black hey black hey black hey Yeah, [Music] you [Music] Down down down down down down down.

[Music] I'm a [Music] [Music] Do you want hey? [Music]

Don't.

[Music] Heat. Heat.

Heat. [Music] Heat.

Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat. Heat.

[Music] Heat. Heat.

Heat. Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat. N.

Heat. Heat.

[Music]

[Music] Wow.

[Music]

[Music] Heat. [Music] Heat. Heat. [Music]

[Music] Heat. Heat. [Music]

Heat. Heat.

Heat. [Music]

Hey, heat. Hey, heat. Heat. Heat.

[Music]

Heat. Heat.

Heat. [Music] Heat.

Heat. Heat. [Music]

Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat. [Music] Yeah, [Music]

[Music] down. [Music] Hey, hey hey. [Music] Yeah,

[Music] down. [Music]

Black

[Music] Heat. Heat. [Music] [Music] Doobie. [Music] Da da da. [Music] Hey,

hey hey. [Music]

Heat. Heat. [Music] Heat. Heat.

[Music]

Heat. Heat. [Music] Heat. Heat.

Heat. Heat. Heat. [Music] [Applause] Yeah. [Music] Heat.

Heat. Heat. N. [Music] Heat. Heat.

[Music] Heat. Heat. N. [Music] Heat. Heat. Heat. [Music]

Heat. Heat. N. [Music] Heat. Heat.

[Music]

[Music]

[Music] Wow. [Music]

[Music]

Wow. [Music] Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat.

[Music]

Heat. Heat. [Music] Yeah. [Music] Heat.

[Music]

Heat. Heat. [Music] Heat. Heat.

Heat. Heat. [Music] Yeah,

[Music] down. [Music] Hey hey hey. [Music] Yeah, [Music] down. [Music]

down.

[Music] Heat. Heat. N. [Music] [Music] down. [Music] Boo. [Music] Doo doo doo doo doo doo doo doo doo doo doo. [Music]

[Music] Heat. Heat. [Music]

Heat.

[Music] Heat.

Heat. Heat. [Music] Heat. Hey. Hey. Hey. Heat. [Music] Heat. [Music] Heat. Heat.

Heat. Heat. [Music]

Heat.

Hey Heat.

Heat. Heat. N. [Music] Hey,

[Music]

[Music] hey, hey. Wow. [Music] Woo.

[Music] Heat. [Music] Heat. [Music]

Wow. [Music] Yeah. Heat. Heat. [Music]

Heat. Heat.

[Music] Heat. Heat.

[Music] Heat. [Music] Heat. Heat.

[Music] Heat.

Heat. [Music] Heat.

[Music] Heat. Heat. [Music] Yeah, [Music]

down. [Music] Black. [Music] Yeah. [Music] Yeah,

[Music] down. [Music]

Black

[Music] Heat. Heat. [Music] by far. Heat. Heat. N. [Music] Heat. Heat. [Music] Here [Music] you [Music] go.

[Music] Do you know? [Music]

Heat. Heat. [Music] Heat. Hey. Hey. Hey.

[Music] Heat. Heat. Heat. Heat. N. [Music] Heat. Heat. [Music] Heat. Heat. N.

Heat. Heat. N. [Music] Heat. Heat. [Music] Heat. Heat. N.

[Music] Heat. Heat. N.

[Music] Heat. Heat. N. [Music]

[Music]

[Music]

[Music] Heat. Heat.

Heat. [Music] Heat. [Music]

Wow. [Music] Yeah. Hey. Hey. Hey. [Music]

[Music] Heat. Heat.

[Music] Heat. Heat. N.

Heat. Heat. [Music]

Heat. Heat. [Music] Heat. [Music]

Heat.

Heat. Heat. [Music] Heat. Heat. [Music]

Yeah, [Music]

[Music]

yeah yeah. [Music] black. [Music] It brings you Yeah, [Music]

down. [Music] Down

down down down.

[Music] Heat. Heat. [Music] Hey, [Music] hey hey. [Music]

Heat. Heat. [Music] [Music]

Heat. Heat. N. [Music] Daddy. Daddy.

[Music]

[Music] Heat. Heat.

[Music] Heat. Heat.

Heat. [Music] Heat. [Music] Heat.

[Music] Heat. Heat. Heat. N. [Music] [Applause] Heat. Heat. [Music] Heat. Heat. Heat. [Music]

Heat. Heat.

[Music] Heat. Heat.

Heat. Heat. N. [Music] Heat. Heat. [Music]

[Music]

[Music]

power to thank. Good afternoon and welcome to Besides Las Vegas's I am the Cavalry Track. We'd like to thank our sponsors, especially our diamond sponsors, Adobe and Aikido, and our gold sponsors, Profit and Run Zero. It is their support along with our other sponsors, donors, and volunteers that make this event possible. These talks are being streamed live and as a courtesy to our speakers and audience, we ask that you check to make sure your cell phones are set to silent. If you have a question, there will be an audience microphone set up towards the back of the room. And so, if you have questions, please use that microphone so that the YouTube live stream and

everyone in the room can hear you. Very good. Um, this panel is so important. There's actually three introductions. So that helps you understand how important this is. So I wanted to touch briefly on photography. Each of the members of the panel have agreed to have you take their picture as makes sense for you. Um, so you can take their picture, but you can't take other people's picture if you do not have their permission. So, right now it is my honor to introduce the co-chair of this track, Mr. Josh Corman, who has some profound words of welcome for our panel. Mr. Corman, it seems so formal. Um, who was here for the opening yesterday? Okay, so some of you saw a video. We're

going to play a video because neither Dr. Christian Demf or nurse Dena have seen it. We don't have to play it yet. Um, but I'm if you've been tracking the cavalry since we started 12 years ago, we care about everywhere bits and bites meet flesh and blood, but we have a particular soft spot for healthcare. And it was really meeting someone in med school, uh, working with the FDA, Bose's origin story. The overwhelming lion share of the trust that we built with the federal government was with healthcare, medical devices, healthcare technology, delayed degraded care. So this is always my favorite topic block of them. And we have some people, you know, like Bo, some people you might

have seen in previous years like Christian. And we went from the idea of could cyber disrupt patient care, loss of human life, and now we've got legions of peer-reviewed studies talking about the impact of delayed durated care. So, we're really happy to have not only did Christian help found Cyber Med Summit as a 501c3 to pull together clinical technicians and nurses and doctors and medical stakeholders to meet them on their turf, learn their love language, find out common cause and common purpose. But that has kind of become the blueprint for what we're now doing with Undisruptible for water engineers and power. And you know, we had to build those muscles and trust. So, this is my

favorite topic each year. I'm really happy to do so. And then last year we have a brand new teammate from the medical world, Dena. And I can't wait to have each of you meet her. And she's a sponge learning this stuff, but like what does it look like to a non-cyber, nonIT person when we fail or when we have disruptions? So, please a warm welcome to these folks. We'll play a video and I'll get the hell out of the way. So, everyone. All right. Two-minute video. Picture a hospital. Picture your hospital. When was the last time you were there? Was it to welcome a baby into the world or to say goodbye to a loved one? No one wants

to need a hospital, but when we do, we depend on timely access to care. When and where we need it, irrespective of cause, delayed and degraded care for time-sensitive conditions can affect worsened outcomes and even loss of life. A 5minute longer ambulance ride has a significant impact on 30-day mortality rates. Time is brain where even an hour or few could determine if you walk again, if you talk again, if you even survive. Now, picture your hospital. What if that hospital was not available to you? If your hospital was disrupted, where would you go instead? Is it across town, more than an hour away? What if they are also down? The chance is not as remote as you'd hope. Hospitals have

become a top target of ransomware, cyber attacks that [ __ ] technologies in the vital path of care delivery. Worse, your hospital doesn't even need to be the one attacked to endanger you or your family. We've seen a 10-fold decrease in favorable outcomes for heart patients merely due to excess strains of a ransomware affected region. Now, back to your hospital, back to your family. You and your family deserve better. If we want timely access to patient care and more resilience in the face of accidents and adversaries, we're going to need to advocate for ourselves. Now, as we head into an era of hybrid conflict with threats to water and power, these disruptions stand to get a

lot worse. But we'll talk about that in another video.

As Josh mentioned, this has been a topic um a staple, if you will, of the conversation. And I bet many of you folks have been raise your hand if you've been to this event previously. All right, keep your hand up if you're bored of uh healthcare cyber. >> This person in the back. Uh I'm gonna take another little audience poll here. Raise your hand if you feel like we've made significant progress since the last time you were here uh in healthcare cyber. Uh raise your hand if you feel like we've really made big advancements in ransomware in hospitals or medical device cyber security. Raise your hand. And that kind of sucks. I don't think this uh what we're going

to talk about is intended to be a bummer. It often is because we're talking about some pretty serious things. And I loved that video and it talked about how no one wants to meet a hospital but we all most of us are going to and when we do uh we're going to hope that it is operating at its highest efficiency and then every doctor and nurse and technician in that place uh ate their waties for breakfast that day because it's going to be us or our loved ones. And what a dystopian nightmare we all live in now that we have to have these conversations about whether or not the basic technology that enables all this care uh might not be available for

you because of something like a ransomware attack. So we hope to do here in the next couple hours is give three varying perspectives slices of this um and bring some alternate perspectives into the conversation to try to convince folks that haven't been familiar with this previously that this is a deal. this is an issue that you should care about. Two, empower you to do something about it. And then three, start thinking about this from a little bit of a broader lens. Much of the amazing content that we've had in this uh track the last day and a half has been discussions about national security, international security, um things with increasingly sophisticated adversaries. and discussing it this as more of a

strategic or a larger goal, a larger kind of picture is I think something that's really prudent for us to talk about there at the end. So each of us are going to take uh a little bit of time to give our individual perspectives and then we're going to be able to open it up to some questions at the end and hopefully engage you all in a dialogue because I think one of the best things about bides is you guys aren't passive receptors of this information. you're really supposed to be active participants and collaborators. And if we had it figured out and we were just here to present it to you, um I mean we we have mansions and we would be like

on jets and stuff. No one's got it figured out. Uh and you might be the person that has that revolutionary idea and you being here in this talk and listening to this and then collaborating and working on this problem might be um the thing that cracks it or moves it or saves a patient's life eventually. So, we hope to empower you with that kind of voice at the end. You know, we talk about some pretty heavy stuff. I don't think we need like official trigger warnings or anything like that. Um, but I do think that this can be, you know, sensitive information. We're talking about, uh, scary scenarios and patients. And if that, if you are sensitive to

that, uh, just consider that. I don't personally have any materials. At the end, I'll have a little bit of a PowerPoint presentation. I don't have any videos or images that might be disturbing this year. This year, sorry about last year. Uh, but so I think that's it for our trigger warnings and kind of the introduction. Um, anything else you guys want to add before you kick it off, Bo? >> No. >> All right. >> All right. So, um, my name is Bo Woods. Uh, I started my career working in a hospital. Anybody here work in a hospital now? Worked in a hospital. Uh, worked in a hospital on the IT infosex side. A few people. Cool. So,

there are going to be some people in here who laugh when the rest of you don't or who cry when the rest of you don't. Um, but in I think I was at the hospital for almost three years to the week. Um, and I saw a lot of stuff a lot of stuff happening on that that network. Uh, over at some of the other big conferences they like to say it's the world's most hostile network. Well, they are right. It is extremely hostile over there. But not too far behind that is hospital biomedical networks because you have all flavor of types of devices, all types of generations and you have a lot of malicious software

that just bounces around the network there. You know, network worms that you haven't seen on uh in places for a decade or more. They still exist. They still live in these isolated environments like a a a niche um isolated environment where you find like um crypto animals like things that that uh time forgot, right? Um you can still find probably like SQL slammer worms which was back in 2003 I think. You can still find some of those old things running around some of these networks. uh just because of the nature of health care in hospitals like I said you've got all these systems of different generations that are there uh and a lot of times you know in a hospital I would

be working on troubleshooting things that weren't even uh malicious probably and so when in I'm the cover we talk about accidents and adversaries uh malicious intent is not a prerequisite for harm I remember one case where we had um all of the medical devices of one class I think it was a an infusion pump would flash every 15 minutes like clockwork. And so it was it was really really scary because these are in every hospital room pretty much. And we traced it down to there was this network packet that one of our IT systems would send out polling every device in the hospital to see if it was up to get a telemetry readout to

get just other basic IT information. It was it was like a proprietary version of SNMP if you're familiar with that protocol. And these medical devices didn't like that. So they would they would bounce. Uh so we disabled that packet being sent every 15 minutes. Um but you deal with a lot of things like that. Um who here thinks uh strong passwords, multiffactor authentication is generally a really good practice to have other things being equal. All right, pretty much everybody who thinks that if you can't do that, you should probably at least just have strong unique passwords. Everybody has their own credential to log into a system. Yeah, pretty much everybody. Who here thinks at a minimum you should have

shared passwords that everyone knows? One person. Who here thinks that uh that's probably a bad idea? Um well, I'll tell you how I found out that that was a bad idea and it might be different than what you expected. So, um, for certain types of systems, uh, we had a setup where the computers themselves had no passwords on them. So, anybody could walk up and they have instant access to desktop. Uh, I saw that as a pretty big problem. And so I went to my boss at the time and I was like, "Boss, we got this security thing over here. We should probably do a thing." Uh, and she said, "Okay, well, talk me through it." And so we talked

through it for a little bit, said, "All right, I tell you what. Uh, come back with me on Saturday night. We'll go into the ED and we'll just sit and and look at how these computers get used so we know that there's nothing in the way. You know, when you put passwords in front of things, you could put something in the way." So sure enough, I went looked and after about an hour or two of sitting there and watching, seeing, you know, every 15 minutes, you'd probably have six to seven people come and touch every single PC. And if they had to individually log out, log back in every time to that, how much time would be

lost? And this is the emergency department. You've got people coming in with critical cases where you know seconds can save lives and certainly the cumulative effect of minutes of extra login time between these things with the system having to log out come back up. Oh, somebody forgot their password. They got to call the help desk. 15 minutes for a password reset. >> It's true. >> That's a pretty big deal. So after that um I no longer asked whether we should put passwords on things anymore like that and have you know strong multiffactor multiffactor didn't really exist 20 years ago but like those are the types of things that you deal with in hospitals and in healthcare

environments that are unique and different from other environments that you might deal with. Um I also remember going in and uh we had a a push an initiative that all the systems across the hospital should have antivirus on it. Good idea, right? It's a at least a minimal protection against known bad software that will stop something when it pops up there. Uh but we had all these medical device vendors who would come in and they're all special and they know how to make medical devices really really well and sure they're on the top of their game in security but security isn't really important because it's safety that you care about. So we said all right well how does it react if you

put antivirus software on there? They're like well it's not really approved. We don't want to go through FDA clearance again for that. By the way that's a red herring. The FDA never says you have to go through clearance if you put uh antivirus on it or or send patches. Um but a lot of medical device makers uh don't know that. They're getting a lot better though. Uh so anyways, we went and we worked with the vendor. We did a site demo. They came out. They said, "All right, you know, here's the workstation that will control this device that sits bedside with a patient, and this is going to be the remote piece of that. you can put antivirus on this

remote piece of it and let's see what happens. So we installed, you know, semantic or whatever our our antivirus duour was. Um, and instantly their clinical app crashed. No warning screen, nothing. It just fell to the ground. Like, all right, what's happening? Let's try it again. Let's, you know, let's reboot the computer, bring it back up, start it up, see what happens. Sure enough, the same thing happened. So went back and looked through the logs. Um when we had enabled antivirus, we didn't realize it, but their software was using uh obfiscation techniques that malware uses to hide itself uh from the operating system um in order to I don't know protect their intellectual property from people being

able to see it or whatever. I don't know why they were doing this, but the antivirus would detect that and kill the software, just stop it from running and, you know, quarantine it, put it in this uh safe space. Uh and so we learned that, you know, you just can't put antivirus on those types of systems unless you can control it and make sure it's not going to accidentally uh kill the clinical software that's running to help patients. So again, an eye-opening moment. Um and then when I said, you know, hey, we've got uh the opportunity here to do some other things where we could put, you know, maybe um allow listing software that only known good apps can

run on the the system. Uh my boss said, "All right, cool. Create this, you know, business justification for this." Like, "All right, cool." I went went away, put together a one-page document. Um, after a few iterations, uh, it was non-technical enough that we could take it to the the business leadership team. Um, and, uh, the price tag on it was like $250,000. I'm like, you know, this is I went and I negotiated with the vendor. We got it down to the cheapest thing it could possibly be. This is the least expensive product on the market. Said, "All right, well, what I don't see in here is what's the trade-off for $250,000, we can get two to four more

nurses. We get another physician in here. which is going to deliver better health care. It's like a [ __ ] So I had to, you know, crumple up that that piece of paper and throw it away because I knew the answer to that. So those are the types of challenges that people in healthcare it face every single day. For me, the most profound one that I experienced, and this is where Josh and I bonded uh day one of I Am the Cavalry, right after he had launched um he and I were both in the speakers room upstairs here uh talking about different things. Um, and I told a story about literally my first day working security in a

hospital was uh we had a a network worm that was going around. It was Zotto if anybody remembers back I think it was 2007 2008 something like that. was this malicious software written by uh I think it was a Libyan and a Turkish guy and they were trying to steal banking credentials, right? So like this is supposed to go out and like hit grandma's PC, steal her banking password and then they're going to go drain the accounts. Um they were using IRC as command and control. So like it wasn't super sophisticated even for back in the time. Um, but it was going around our hospital and knocking systems offline left and right. Um, we managed to get it

under control on a lot of the systems that we had. Uh, we sent out patches. I think we patched a couple hundred servers within a day uh to update it um to to avoid this type of thing. Uh, and we're like, "All right, we're good. We're good. We're back up and running." Um the next day I came in and I had a couple of messages. So I called up the the department. It was the natal intensive care unit. I said, "Hey, you know, you left me a message uh calling you back. What's going on?" They said, "Well, you know, we we know you're not uh working on medical devices. We've got the whole biomed department that does

that, but they can't really help us with this problem with one of our devices. Um so we thought we'd call you." What happens is about once an hour, uh, this system randomly shuts down and then when it comes back up, there's a Windows screen on it. We didn't know it was running Windows. It's just one of our medical devices. It's the the fetal heart monitor, the thing that is making sure for those for those babies that we know what's going on with them. And uh, you know, if you if you don't know, um, those devices are a massive force multiplier in hospitals. they allow many more patients to be taken care of by many fewer nurses, physicians and

others. So when those are down, there is a um huge resource drain on the people in that department. There is also a consistency of care issue because if you've got to then attend to more patients than you're used to, you've got to move a lot faster. when you move faster, when you're higher stress, you'll make more more mistakes. And computing technology, they just do that reliably repeatably, right? It's why we use computers. It's why we use these devices is because we have clinical studies that say they're safe and effective for use, that they can outperform uh just nursing staff and physician staff alone when they work together with them. So they were like, "This is a this

is a big problem for us." And said, "All right, well, you know, let me see what I can do." Um, so I went up there, checked it out. Sure enough, you know, they were computers running the same operating system. I don't remember, it was like Windows 2000 or something. Um, that had this same vulnerability. It was a medical device. So, first thing I did is I called up a medical device manufacturer, said, "Hey, you guys have, you know, this problem. Can you uh, you know, do something about it?" And they said, "Well, it's a medical device, so we can't add patches to stop the malicious software from running on it." Like that. What are you crazy? You want

to like fix a computer? You want to fix a medical device that that's uh being attacked by adversaries? No, no, no. We'd have to go through a whole FDA cycle. And again, not true, but that was the line that they were sticking to. Said, "All right, well, you know, is it okay for me to add the patches?" Said, "No, you'd be out of your warranty. Are you crazy? Why would you want to, you know, put well- tested, reliable software on something in order to keep out the known malicious software? Like, that's crazy talk. Don't do that. So, I said, "Well, all right. That's that's not really how I'm wired. I'm I'm wired like a hacker. Uh I'm I see a problem.

I'm going to go fix the problem. And if there's something in the way, we'll look at the trade-offs. We'll worry about those. We'll make a business decision or, you know, some kind of a good risk decision based on that. and then we'll take action. So that's exactly what I did is again I wrote this business justification having learned a lot of lessons from the first time I tried this brought it to my boss. We took it up to the the CEO of the hospital asked a couple of questions like wait so you're telling me that we have a problem right now that may have an impact to patient um patient safety? >> Yep. That's right. and what you're

proposing might void our warranty, but our warranty is already void anyways because the devices don't work. He's like, "Yep, that's right." He's like, "Well, this is a no-brainer." Signed off on it. We had all our permission. And so I used uh Metas-ploit framework to hack these medical devices um to then install the patch, kill the malicious software, and reboot the box, get it back up and running. Called, checked in with the the NICU, the natal intensive care unit. They said, "Yep, those systems are all up. If we have any other issues, like we'll let you know about it." But, uh, the one of the reasons that Josh and I connected is I said, you know, I've been

in in infosc for 10 plus years at that point. And that was probably the day that I had the most profound impact on people's lives is being able to help get those systems back online so that the people who take care of other people, some of the most vulnerable patients in the entire hospital could keep doing their jobs. And so uh that for me was a very formative moment in my in my infosc career and also something that has made me extremely passionate about medical device security about hospital security about security of these other lifeline systems that we depend on like water like power like aviation uh like other things. So um that's that's my set of storytelling and

I'll uh hand the baton over to Dena who will pick up from there. Um, I just want to say I just want to say thank you.

Should I come this way >> because I have to get my my braille uh computer? Yeah, my little notes. So anyway, let me um I just want to thank you so much for caring that I feel like um our IT people try but like to to be a force and continue the battle all the way up to the CEO because they don't care. And I feel like um I think I'm the thorn among amongst the roses right now because I feel like uh I'm a science major, but I'm like knocked out by how smart everyone is and how much I'm learning. So um let me log on here real quick. And uh sorry about this.

>> She does have a password on her computer, by the way. Much more secure than some places. >> Uh no, I don't share my password. Um, at least I'm that far. Though Christian lectur le lectured me very kindly and said, "Dina, get a VPN." So, I will do that. So, I'm Dina Carlilele. I'm a critical care registered nurse. Um, I'm the president of a union. Uh, my story is we represent two different hospitals. And the first one is how I met Josh. Uh we were introduced by a uh reporter who was investigating uh 140 hospitals going down and one of them was who I represented. I represent two groups over at Ascension which is now Henry Ford Rochester. Uh I have a

large group of registered nurses and a group of radiology technologists uh at McLaren McComem which is about 13 hospitals in their system. They may have lost one uh that I have a group of the biggest group of nurses and a group of service group folks. The service group people are your phabotamists, the sitters who sit with you when you're not safe to be alone. um the people who register when you come in the hospital. So that's the kind of people we represent. It's all healthcare. Um my local stands for safe staffing, pure and simple. That's that's like our hill to die on every day. In 2004, uh McLaren Mcome went on strike uh indefinite strike. We were out for six

weeks. It was very painful, but we took a less percent of money and we got a safe staffing matrix at that house and I'm very very proud of that. However, from 2004 to now, there's a lot lost in translation. Three years ago, I put some pretty big teeth in the last contract. I put a big penalty if they violated the safe staffing matrix. They violate it daily. Never in my wildest dreams did I think they would pay later instead of staffing upfront. So I guess I'm trying to paint a picture of where healthc care really is. Our sister hospital at Ascension/Henry Ford has no safe staffing matrix and we are fighting very desperately for that.

Um, in June and July this year, both houses went out within 30 days on strike. Again, fighting for safe staffing. Uh, one needs a matrix, the other I'm trying to um tighten down. Christian and I were discussing like how EDs are so severely understaffed. So severely. Um, so um, and it's never about money. It's about safe staffing and protecting our patients. Um, we survived COVID. One hospital, McLaren, where I work, was hit very, very badly with CO patients. We had a refrigerator truck out back that was a morg. So, I want to tell you, I've never seen that much death in my life, and I pray I never do again. So imagine health care workers surprise

when a cyber attack took us to our knees again. You just don't see this coming. I mean I'm just now learning the things at such a basic level from all my new friends and like what a warm group of smart people you are. So thank you for helping me on this journey of learning what I need to do because it's taught me a lot. Uh the bigger hospital lost 200 RNs postcoid. Our ascension hospital lost about 100. Um and the service group lost about 100 folks also. Make no mistake there is not a nursing shortage. There is a shortage of nurses who want to work in these environments. It's not safe. So, I thank God for the

people who are fighting to make this safer for everyone. And you don't see it coming. We didn't see this coming that a cyber attack would [ __ ] us the way it did. So, you've all heard of an electronic medical record system, right? We have become very reliant upon it. I've been an RN since 1996. So back in those days we were chiseling away on paper are our charting. I'm aware of what to do. I know my dosages in a different way than these smart young people who are coming up. But let me say when this occurred the first one was I want to say let me see get my dates again. May 8th for

Ascension and August 5th for McLaren within months of each other. So we have new baby doctors and nurses who are coming in and have never written on paper before. We have hospital systems who have uh paper charting that's stuffed in like little areas where we don't even know where it is. Uh the Ascension Hospital, now Henry Ford, had a little bit better of a system that did not go down very frequently. McLaren, it stinks. It goes down like for a few hours once a month. Every time I think they reboot the system or upgrade the system, right, it like the next morning it's like chaos. We can't log in. So patient care is delayed. those ears not being available to us as

clinicians, these people came in. You would be shocked at the amount of people who don't know what medication they take. You would be stunned. And you men are the worst. So y'all better y'all better learn what you're taking cuz your women it's always like when I do a preop day, well my wife knows all my medicine. That's you know that could be a deadly mistake someday. So be on your best behavior. So we go forward. I've got young doctors who are used to clicking and I say this with great love, Christian. I that who are used to like we're all used to clicking boxes in the computer for our orders to to get things done to get

patients different places. There's no box clicking. These people had never written out prescriptions. you know, we do like the prescriptions go straight to the pharmacies. Now, this is a huge problem. So, now they're writing prescriptions. Um, we were I was discussing this with my executive board. What was the worst thing that happened to you when when we were down? She said, I had a doctor write Tylenol and like walk away like no dosage, no route, no nothing. You know, these things are important and can be life-threatening. So, you know, we were really helping our young people and our young nurses who had never charted on paper were were just as new. You know, they didn't know. So, then the hospital

has no preparedness, no communication. During COVID, we had a different chief nursing officer and a different vice president of HR. Every three hours, they pulled me out of my job. I was helping them make decisions on what we were going to do as staff in the hospital. This was a far different animal, different CNO, different VP of HR. They didn't care. They didn't want to include us. The nurses are I'm telling you, there's 50,000 nurses in Michigan who are not working, who have a license because they can't take the stress of doing this anymore. They can't do it. we can't not take the best care that's possible of our patients. So to go forward from that um

the EMAR it provides us with allergies, what medicine you're on, what surgeries you've had, your medical history, um lab work, diagnostic testing. um that system that was a great safety net for us since it's come on board like really decimated us when when we lost it. So we didn't know what your lab work was. And let's talk about our phabbotomist who are trying to care for you and draw your blood. It was so timeconuming. We had patients going into surgery with no lab results. people who are on different medications, we need to know their lab values. If you're going in for a patient uh a surgery and we think that you might be losing a lot of blood, that's

invaluable. We need to know that before you um go into surgery, but I'll tell you what, we work for corporation. They didn't care. And as all my friends here have said continually, I loved Joshua's mant man m man m man m man m man m man m man m man m man m man m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m mantra, no one's coming to save us. We all better rise up. And that's that's what my union is doing. So um let's see.

At one hospital, a pre-up nurse had to call every single floor to find the patient. We didn't even know where patients were. It was really, it was really dark. Um, and I believe that due to the lack of a computer system in place during this time, there were errors. I think there were people who had very legitimate bad outcomes. And I think it's going to be easy to just brush it under the rug because there's no clear record of it. And that is so disturbing. Um, medication administration. This is very frightening. There's a a machine that is in our bedroom and it's either called a Pixus or Omniell. Everybody may have a different type of

machine that they go into. My my fingerprint gets me in. We were overriding every medication and just taking it out. There was no second checks with pharmacy. There was no check second checks through from the pharmacy system that went into the Omnisell or Pixus, whatever anyone's using to get to stop a hard stop for any medication that could be wrong. Um, we had to be the ones to catch any errors. Um we found discrepancy from doctor's orders. Um we were used to pharmacy like being that second hard stop. Hospitals began asking the patients to bring in their actual prescription bottles to bring in their own lab work printed um printed scripts for testing. Um and as Bo referenced

the the connection between our computers and all our devices. Um, we've talked about before how certain IV pumps can can like deliver a lethal dose of medication if we're not careful. So, I mean, kudos to our our IT friends who are there for us. Um, the McLaren site is a level two trauma center. We continued through all that, taking in patients from all over. They're kind of like um a little hub and everyone would deliver all their patients to us. So, we're already overburdened there. The understaffing was horrific and we're still taking in critical patients from other facilities. I this just can't happen. And what is that? That's profits over patience without question. Um and and as we've

all said with those delays in care due to us searching for medication, verifying trying to find lab work, what what was the outcome? I think Christian was going to reference this later. What are the out or possibly do a study on it? I think you said, but like what are the outcomes for these folks who were sent elsewhere? And so I was I was jamming on Christian on some YouTube and he he did a study that really caught my eye that it it showed the um spike in surrounding hospitals. So if my hospital was down and we sent it to another hospital, that hospital can be overwhelmed. As as I love that little video, Josh, well done.

um that time is brain as in even with cardiac things, minutes make a difference in your outcome, whether you're going to speak again, be able to swallow, all of these things. And when you're diverting people and we're not ready and we're still not ready, um my key thing that is my biggest thing and closest to my heart is safe staffing. So every type of nursing has a a different uh person who or a association critical care associations for uh the ICU. It's like one to one one to two. Hospitals violate these things daily and you don't know it when you're a patient. And every one of you some at some point in time you're going to be a patient or

you're going to be a family me member there with a patient and to see these things is heartbreaking. So this is staffing ratios save lives. They do so many good things. Um and as I said hospitals violate these ratios daily and they don't even flinch. They're not even afraid to pay the penalties. They don't blink an eye over that. Um, some nurses in the med surge areas can have up to 8 to 12 patients. It's not possible to care for that many people. Not on your best day, not on your best game. So, I think people need to realize what's actually going on within hospitals. Um I want to add that staffing ratios reduce mortality rates, shorten hospital

stays, improve patient satisfaction, patients have fewer adverse uh situations, falls, things like that. Simple things that we need to take care of. uh faster detection of patient uh deterioration, reduce medication errors, um financial benefits, it's cost effective, um it decreases readmissions. Um how did my little local fight? We had a generous cyber journalist presence that brought our story forward. Um our demands were as follows. Uh because we had little interaction with our administration at both hospitals, we asked for unit shift huddles um to be able to have some communication over safety issues, what they were doing, what we could improve upon and none of that was occurring. Secondly, we asked the hospitals to conduct regular training sessions. I

have begged for this for the past year plus. It's not happening. So, this new wave of baby nurses and baby doctors still aren't learning those things that that are imperative for our patient safety. Um, third, weekly progress reports to update staff on the status of efforts to resolve the cyber hack incident and restore access to the EMAR address any safety concern staffing issues. Number four, patient ratios. We chose uh one to four for the bigger units and we kept the critical care units at what they were one to one for ICU, one to two, that kind of thing. Step down is 1:3. Um we tried to fight for reduction of elective surgeries and transfers in. It

didn't happen. They they kept doing elective surgeries like it was no big deal. And it was patient safety was completely depleted. And like I said, the biggest example, people coming in with no lab work. Um I'm just touching on what can happen with underststaffing. We all desperately need legislation. My local spent the last year there was um some legislation that was going forward for safe staffing. I know you all have been trying to get something through with cyber security. These things are imperative because no one is accountable. Um these are people lives whose lives are at stake. Um I want you to consider what this can do to communities that have multiple hospitals. Um our friend Dr. Dr. Damoth

states that these cyber attacks can cause our neighboring hospitals to be seriously overburdened. And thank you for that research. That was awesome. Um, so today what I'm doing I I brought we're trying to get contract language and it's appendix C, our cyber security contract language to fight for this to be in there. So if this occurs again, we h I have something to stand on top of. I mean, and we're one of the only um unions who had pandemic language. The women that I started out with in the in the union were thoughtful enough to throw in one line and it saved me. So you people think, "Oh, it's not going to happen again." at Ascension, there's a

large uh plaza right next to it and the vice president of that who's under me had gone in there either for food or for doctor appointment. There's a lot of little uh offices in there and she said it was week two of five weeks. She said people already thought it was over. Oh, it's over, right? It's I feel like we're kind of a fast food society. We take in what the initial input from the media is and then it gets forgotten. And I think people aren't aware of that. Um, downtime paperwork's not has not even been redone. It was so outdated. Um, I was I was talking with Christian about it's it's got like drugs that aren't

even used anymore on it and wrong dosages. Um, I love that the Josh Corman's and Christian Damoths and Bose are fighting for legislature and bringing this all to light because it's a dirty secret the hospital holds on to. that I have a big suspicion that if faced with a big fine for not being prepared or doing the right thing uh or training the healthare workers doing their due diligence with making our computer system safe for the healthare workers and most of our patients I believe they will roll the dice and wait and see if another disaster strikes there is no accountability um we never give up there's no one riding in on a white horse to save us. That's why as a union

we band together. We're fighting for legislation to keep our patients state safe. Um, and it's a good fight. I'm all in it. And with my over a thousand healthcare workers, I'm going to fight until we figure this out. I want to thank you so much for listening and thank you. I'm honored to be here. [Applause] Good job. That's a that's a lot of brave things to say and a story that um as difficult as it is to hear secondhand. Imagine living it for five weeks, >> showing up every day and working a 12-h hour shift and being faced with all of those struggles of taking care of those patients, knowing that even on your best

day, it's still risky to take care of patients. And then to have all of that completely taken away from you and having to navigate all that danger and all that peril and all that risk uh now leveled up by 20 just because of how bad these attacks were. Um, it is a that's a heavy thing. Um, I don't know if it's going to get much lighter, folks. So, if anyone needs to start drinking, just kidding. I'm a doctor. You're not supposed to do that. Um, we're going to kind of continue a little bit with this, but I think in some ways shift gears to what we've had is what it's like to operate on the

technology side of a hospital system. the constraints, the troubles with budgets, some of these uh medical device issues, uh that intersection, and then we've had a firsthand account of what it's like to take care of patients um during the heart of a ransomware attack and talk about that uh from a individual patient level to a unit level to a hospital level to a regional level. So, I'm going to spend the next little bit of is kind of doing a little bit of a recap um and then bringing two new things for you folks this year. And the focus for my remarks again are going to be a little bit more on the national uh sta stale scale or whatever. So I'm

Quatti. Uh that's what everyone calls me during the summer cons. Um that's my handle. I've been coming to Defcon and and the summer cons for like over 25 years now. And so it's weird when people call me by my uh by my real name, but you call me whatever you want. I'm an associate professor of emergency medicine, biomedical informatics, and computer science. I just made associate. Woo. Right. >> Yep. I'm tenured. I'm tenured now. They can't fire me. Just kidding. They can fire you. I found that out. Um, please don't fire me, boss. Uh, and I co-direct the Center for Healthcare Cyber Security, UCA San Diego. We're almost two years now on this mission of

kind of a academic healthcare cyber security research center. Want to talk about some of that work? I just wanted to say thanks for you folks for what you brought and just the amazing bravery it takes to talk about something like this. But I bet some of you in the audience are like, "Wow, what did I get myself into?" Um, I saw a lot of heads like nodding. I bet you folks are the ones that work in healthcare or have people like you you get this. But I think some of you folks are probably like, "Is it really that bad?" Like, what did I get myself into? What a mess. This map's going to come back later on,

but I've been doing this with Bo and Josh and all these folks in this community. It's great to see so many familiar faces with, you know, for for a while now, like over 10 years. And it's allowed me a chance to, I think, take a step back and and try to from hearing these stories of what's happened with ascension or change or um all these like gigantic attacks. It really poses you about it really causes you to think about how big the scale of this problem is and how if we're really going to be addressing some of these issues, we're going to need fundamental change in the way we think about this problem, study it, and the solutions

that we implement for it. And that even if we're so fortunate to solve it at one hospital, which is this is an unsolvable problem like cyber security concerns in a hospital, we're never going to solve this problem. But even if we get really good at one hospital, there are over 6,000 in the United States alone. Right? So it is a daunting challenge, a never- ending battle. But I think about this picture and it'll make sense a little bit later about really how much we need you folks to step up. And so I hope at the end to again instill that charge in you to kind of join us. I'm gonna do a re quick recap of last

year just because I think it has some in some stuff that's been mentioned and alluded to. We'll go real quick through this and just so I know how many people I'm gonna board that who was here last year for my talk. Okay. Well, not that I'm sorry, but hopefully this will be I'm just going to go through the takeaways real quick and build upon what the other folks have said. Takeaway one was that we are critically dependent on connected technology. Right? So how much did we hear uh about how much the nurses were needing the electronic health record and the pharmacy and these pixus omnicels like you cannot deliver even the basic standard of medical care in

the United States without a huge amount of technology whether that's be connected medical devices servers all the way up to like operational technology email um communication software uh VOIP phone lines in a hospital all of these technologies are critical for the timely care of patients So it's amazing it works just normally but let alone while we're under attack but lesson one we cannot deliver safe high quality care in any hospital without technology. Um and to exemplify that you know I talked last year about a study that just says like they followed some doctors around. I'm sure it would be even more so for nurses and just counted how many clicks they used to execute their job every

day. And it was tens of thousands, right? Like I'm sorry it was thousands of clicks. It's probably even more now. On just a regular shift in the hospital, a doctor and nurse is making thousands of clicks on a screen to deliver their job to actually accomplish their job. Takeaway two, healthcare attacks are rising. This is some great work uh from Dr. NRA um out of University of Minnesota. You know, an update to this study is coming up. I've heard it just talked about year after year the threats are not decreasing. The frequency is increasing and the severity of them. It's not ransomware lasting a day or two anymore. Like we all know it's these are

such devastating attacks and the recovery is so complicated. They're talking about weeks to months. You mentioned you were saying one five. I don't think some people recognize in the audience that she was talking about weeks of downtime. That's huge. Oops.

We've been having a lot of lately some contemporary largecale failures um with thirdparty stuff. So not even ransomware specifically getting hacked but you know whatever critical third-party vendors that health infrastructure has will get attacked and change healthcare is a really good example of that where a single ransomware attack on a third party vendor decimated thousands or it's not decimated impacted thousands of clinical operations like small clinics to hospital systems and by impacted what did I mean it means that financially devastated many of these there have been uh clinics private practices etc that closed because a vendor they used to process insurance reimbursements got ransomed. That's how fine a line so many of these organizations are really treading. Like

they are on the razor thin margin. Most of the time they're lucky if they break a 1% margin of profit year after year. Now I'm not saying hospitals should make a profit. What I'm saying is that if you're always on that financial edge of a razor, um these types of attacks that you're not even responsible for that impact a vendor could cause you to go out of business. And then what happens to the community around it if that hospital that goes out is the only hospital within 150 miles. Like that's huge. It has cascading rippling effects that last way longer than just the downtime of a single ransomware attack. Takeaway three, you know, cyber attacks

impact these technologies and cause patient harms. You know, I feel like we're getting I was talking about Bo and Josh, like we're getting old. Remember like a long time ago when we were like we don't have studies that show patient harm. We we get this constant feedback from like you're scaring people. There's no data that says that this happens. But we have I think that's changed. I think that it is hard to make the claim nowadays that cyber attacks like ransomware attacks do not impact patients. And I think one of the papers I was talked about a little bit ago, if you're interested in this, it's an open access paper. You can go review it, but we just measured what

happened around a hospital system that got ransomed at hospitals that didn't get ransomed. So think about it like an ecosystem. If you're in a city and there's five hospitals, if three of those get ransomed because of the same hospital system, those other two hospitals, uh, they don't just continue on their merry way. They take a lot of the rippling, um, ecosystem effects. They get overwhelmed. So, everyone suffers in a community um, when ransomware happens. And we looked at what happened to emergency patients. Their care was impacted. They waited longer. They took longer to get admitted to the hospital. They left sooner than they were supposed to against medical advice at higher rates. Like a lot of patients in the

emergency department at a hospital that wasn't even ransomed were impacted. Ambulances were significantly impacted. The whole uh system of prehosp care, this is a a graph that just shows the middle there is what happened during the ransomware attack. And it just shows the cumulative number of hours that hospitals are on diversion. And diversion, there's tons of papers to show that diversion is not good. If it takes longer for you to get to a hospital for care, if it takes longer for an ambulance to arrive at your fac at your home when you're having an emergency and then transport you to a hospital where you can get definitive care, that delay can kill people. And so ransomware attacks in communities

impact even the ambulance systems. And then that video I thought was great. It talked about times medical conditions like hearts, heart attacks, strokes and things. And when you look at these types of really vulnerable patient populations where like minutes matter, hours matter, um these patients can be disproportionately affected. And this is a study we followed up with that that just said like listen, if you have cardiac arrest, if your heart stops and they got to do CPR on you and they shock you, all that stuff you see on TV and you're getting cared for in a hospital in a town where ransomware is happening, you have a tenfold decrease in your survivability with favorable neurologic outcome. Just because there's

a ransomware attack in your town, that means that you, your loved one, your parents, whatever it's going to be, have a tenfold decrease in whether or not they're going to survive. and be able to feed themselves after a heart attack, after cardiac arrest, just because ransomware is in their town. This body of evidence that these types of things have impact well beyond the very obvious, rippling effects, diversion, complications in care, medication errors. We're building more of a a literature base to show that. Takeway four, hospitals are closing or consolidating. Okay. So, I think Ascension's a good example. They had 140 hospitals go out with their ransomware attack. >> This is only going to get worse because

healthc care is in such bad state financially in this country and it's so desperately uh funded. That means we have financially welloff hospitals and we have poor hospitals. When the poor hospitals no longer can pay their bills and they want to close, they get absorbed by a larger health care system. So we're just getting fewer and fewer independently owned and operated hospitals. Now we're getting mega hospital systems where like ascension 140 hospitals are on a single IT stack which means if they have a ransomware attack that effect cascades to all of the systems that are on a unified platform and that's just accelerating. Hospitals are closing. They're getting acquired. We're consolidating health care. And at the end of the day, we're

going to be we're increasing our risk for catastrophic failures across the country. And rural healthcare is uh particularly at risk. There's a list that got published a couple months ago that talks that there are like imminently 300 hospitals, rural and critical access hospitals in this country. They're at the brink of closure right now. Like that list is going to grow. All right, takeaway five. We're critically dependent on other critical infrastructure. I alluded to this. You know, it's been great and also terrifying to hear how much a an understanding I've had just learning about how critically dependent healthcare is on water, electricity, these types of interdependencies on critical re critical infrastructure that are shared among critical infrastructure

is terrifying. And then now we're going to go on to some of the newer stuff. So the last takeaway from last year was like these problems are hard to fix quickly and I made this analogy last year and I think it still kind of holds wherein a lot of what we talk about is about prevention and we want to prevent ransomware attacks, right? But like we don't just uh try to prevent heart attacks and cardiac arrest, right? These are the things that I'm supposed to tell you as your doctor. I'm not your doctor. As uh as a doctor, you're not supposed to do, right? like don't smoke, don't get old, I guess, don't drink, don't eat a lot of

unhealthy food that guy eats, right? Like stress, diabetes would be these are things that can put you at higher risk for heart disease, right? We're supposed to not do these if you are if you try to adhere to these uh recommendations, then your chance of something goes down less. We do that in cyber like MFA, uh no shared passwords, network segmentation. We could go over like all of the specific recommendations on how you're going to try to make your hospital system more resilient. You're going to do the good stuff to try to prevent the attacks. And we spend, in my opinion, a lot of effort on the prevention. And that's ideal. We never want to have a

ransomware attack hit a hospital. But how much are we really preparing for the inevitable when it does happen? If you if would you think it like if hospitals never actually had treatment for people who had heart attacks, we just spent all of our time in prevention. That's probably not a winning strategy. What's the right ratio? Do we spend 80% with prevention, 20% with response? Like what do we do? And there's like these risk factors. I talked about all the like the things that are against us, but what is the kind of CPR, if you will. What's the treatment for when a health care system gets ransomed other than just recover as quick as you can? Try to do

your best. Try to go on downtime paperwork. And the young doctors that don't know how, they know Tik Tok, but they don't know how to do prescription writing. And I have to confess like I think I've handwritten prescriptions like 10 times total in my career. I'm sorry. >> No, it's >> No, no, not all. But what is the acute response to these types of things? And that's what I left on with last year. You know, I alluded to this thing what we're doing called the healthcare ransomware resins response program. And I got a couple things to show you guys today. And so we're very thankful we're funded out of the advanced research project agency for

health. It's kind of like DARPA but for healthcare. and they took a chance on our pitch, which was exactly this, like, how are we going to respond to ransomware attacks in hospitals that can make patients safer? So, this is like a two-year research sprint, and we're almost done, and I got some stuff to show you. Um, thanks a lot to RPH. You know, the NIH would never fund this. The National Science Foundation would never fund this. I'm an academic, so I got to get my grant funding from the feds, and this type of stuff like would never have been funded. So, we're thankful for RPH. They don't pay me to say this. I just am very

thankful for these folks. All right. So, I'm going to just stop and say like listen, the goal of this is to say, how do we rapidly identify ransomware attacks in critical health infrastructure? How do we let the doctors and nurses have how do we build resources for them to deal with this when right now they have no playbooks? Raise your hand if you uh work uh cyber for an organization and you have a ransomware playbook already already figured out ahead of time. Raise your hand. Like you're supposed to, right? Like you're supposed to have a guide book for your technical response. We're going to do forensics. We're going to look at IoC's. We're going to do this, this, this, this. You

have a technical playbook, but the nurses on the cardiology wing on telemetry don't have a guide book for how they take care of patients during a cyber attack. They just don't. So, we aim to do that. I'm not going to show you that work today, but that's the second part of this work. And then the third is this thing we call uh like a crash cart system. Like how do we a hospital's been ransomed. What do I need to bring to a hospital to get them off of paper? And how can we rapidly deploy that within hours of a ransomware attack so that doctors and nurses can take care of patients uh to the standard of care

that the patients deserve and the clinicians are capable of doing. So, it's not just me. We have a gigantic team of folks. We have a bunch of really smart computer scientists, uh, graduate students, and I got to say like I these folks work every single day on this project. So, I'm going to show you some of that work, but please, like, if you ever see these folks, buy them a beverage or something like that. They've been working tirelessly on this for it would have been a really crazy sprint for the last two years. So, there are three technical areas. Um, they have little piffy code names. We're going to probably have to change some of

these names. Um, but right now, these are just kind of our research project code names. We're not going to talk about the tome today. The first thing I'm going to talk about is the thing we call ransomware. So, and it was this question that was really frustrating to me as a researcher. It got born out of the the problems we had Bo and Josh about like, hey, where's the data that shows what hospitals have been ransomed? Like, what's the data about how long they were down for? What's the data about what the patient adverse effects were? It didn't exist because when a hospital system gets ransomed, they don't want to talk about it. >> Their PR teams say, "Shut up. If you

talk to the media, we'll fire you. Um, and I'm not, you know, Liz, that's it's complicated. Why do they do that? It's because they are already under attack. They're they have these risks that they don't understand fully. They don't know how long it's going to be about. They have a lot of unanswered questions and their default position is to like not talk about it. They don't want to invite subsequent lawsuits. They don't want to have any more brand reputational damage, etc., etc., Those are reasons why they say don't talk about it. So I can't as a researcher go up to him and be like hey will you give me all of your data about this and be

like no thank you like sir this is a Wendy's get out of here. Um so I was really frustrated by this. So I want I had this question this idea like can we figure out if ran if hospitals have been ransomed without them ever having to tell us without having to find out about it on the news. So we did that. We built a system we call ransomware. It's like an academic project where we have a prototype system that scans over 6,000 hospitals um hourly. We've spent the last year and a half reverse engineering the uh public surface of every hospital system in hospital in this country and have a really good understanding of like what

services they're all running and all of this stuff. And we have been scanning it for over a year and we've amassed a gigantic data set over 6,000 hospitals we know and we've been kind of looks like this right little dashboard little red dots saying oh stuff's down yellow dots some stuff's down green dots everything's cool right and so we've been collecting this data and we've been successful in identifying three instances of ransomware attacks we detected on our system before it was ever publicly available. So, anyone here from Shytown Chicago? I like your pizza. I know people from New York hate it, but I love it. Uh, we detected Lur Children's Hospital when they got ransomed uh last year.

This this year, earlier this year, before they publicly announced it, our system saw a bunch of their stuff drop off. And we've had this benefit of scanning at a nearly hour. We initially were scanning every six hours, every three hours, but now our frequency is every hour. But we have like longitudinal data about all these services and when it's up and down and we've been measuring it. Now I'm going to show you guys a study we just published. I have to some important caveats I have to say ahead of time. Uh number one is uh science is messy. It's also sometimes controversial. Uh so I'm going to show you this paper and you can read it yourself. I have to

put out some uh disclaimers ahead of time which is I'm going to show you association data. Does anyone can quickly explain to me what an association is in research? Like something is associated with something else. Raise your hand if you want to quickly take a stab at it. Yes. >> Who pays for it? >> Ah, that's a different type of association, but uh that's a funny I like that. Uh all we know is that two things kind of temporally or spatially or something happened at about the same time. Okay, so I'm going to show you. So I it does not show causal data. I cannot tell you with certainty X caused Y. I can say X

and Y happened at the same time and I can show you the data and you can ask yourself what you think happened. All right. Number two, uh this is one of what happens. What I want to do is lots of studies about this type of stuff. So it doesn't answer every question you're going to have. So we had this system running for about a year. We were seeing evidence of ransomware attacks. It was pretty cool because we could also see how long they were down for. We could see what services they brought up first. Oh, their email servers came up first or hey, their their service now came up first. That makes sense. They bring up

service now first because they can do all the tickets for what needs to happen on the response. Like we saw these hospitals recovering as well and it gave us a lot of tremendous insight. We had the system running and then something happened uh a little bit over a year ago. Does anyone remember what happened before Defcon about a year ago in the news? I'm sorry. >> Yeah, crowd strike happened. So, I know I'm going to tell you if you want this QR code, you're going to have to trust me. This goes to this study. We published this last year and um I'm talking about association data. I'm going to show you some stuff. I really

encourage you to read this paper. Uh and I also en encourage you to read the limitation section of this paper about what this paper does and does not. And then if you have any methodology questions you can ask me but I ask you to read the paper first because it talks about a lot about what how we did this and what our data was. All right. So the question I had is what did we see go down around crowd strike? I can't tell you that crowd strike caused this. I can say I know what h what we had seen prior to crowd strike. What happened during crowd strike and what happened after crowd strike and I can tell you what I

saw go down. So the question is what patient care outcomes or technology outages were associated with CrowdStrike? Um there was also a Azure outage that day I've been told. Um so maybe it was that as part of it but we actually did 2200 hospitals in this data set and we looked at again before during and after a CrowdStrike outage. This was not an attack. This was not a ransomware attack. This is not a cyber incident. they had an outage. Um, and we identified 759 hospitals that had outages occurring associated at the same time as Crowd Strike. So, we saw 759 hospitals have some outages right during Crowd Strike. And then we went and looked at every single one of those and

what services were available before, during, and after and tried to characterize every single one of those services and say, were they ones that take care of patients? Were they the ones that took help the enterprise or were they we were not able to identify? I think this graph speaks for itself. Some people want to say that uh how did you know that they didn't do a firewall configuration or that they didn't do like an update or that something else? So this graph shows daily the number of there's a technology an API standard called fire fast healthcare interoperability resource. This is a healthcare specific kind of API standard that lets your hospital share your records with a bunch of apps and other

providers. So this fire endpoints is what we call it. We have been monitoring fire endpoints for over a year. On average, we see five, eight go down on a daily basis of the hundreds that we measure, thousands that we measure a day. At that orange line, uh that's when crowd strike outage happened and you can see an associated huge spike in these fire end points that went down. These are healthcare delivery organizations, uh not individual hospitals. And then we l looked at all of our IP data, not just our fire endpoint data. We went and looked at again all of our IP scanning data. And we saw all these are the hospitals that had downtimes

associated uh at the same time temporally as CrowdStrike. And we have the duration. We know when they were up and when they were down. And again, we're not scanning every minute. So I don't have minute granularity, but we have like hour granularity for most of this stuff. And then we went and looked at every single service of that outage and we c categorized them into four buckets. One was patient facing stuff. We saw prior to crowd strike um having we saw things like EMS ambulance um dispatch software. Um there's software that allows a ambulance to communicate patient information to a hospital. That was one of the services that we saw go down. We saw a whole bunch of patient portals go

down so you couldn't access your patient records. We um saw a whole bunch of other kind of clinically focused stuff and that was about 22% of all the stuff we went we saw go down associated with crowdstrike was patient facing. Then we had about 15.5% 15.4% of it be operationally relevant. These are things like email servers going down or uh service now portals going down. these types of things that matter to the healthcare delivery but aren't necessarily directly focused at taking care of patients. We saw a bunch of research stuff go down. We saw about 5% of what went down had to do with like recruiting patients for studies or educating patients on clinical studies

for new drugs and trials and things like that. And then about 57% of the time we couldn't tell what this particular thing was. Um so we just put it in this unknown uh or not relevant bucket. And I think this kind of shows the the a couple things. one that a single um issue could maybe cascade that's associated with this like really large healthcare disruption that we saw. But how many of those are there out there? If you would have asked me, I'm going to be honest. If you had asked me if I would have thought CrowdStrike would have had an outage like that a year and a half ago, I would never have thought it would have happened. I would

never have thought that would have occurred at the scale it did. Um but it did. My question and what happened was significant. It wasn't just healthcare. You guys saw airline industry all these other things happened. How many how many different systems that impacted? My question is how many analogous dependent third party vendors that are critical for the delivery of health care are there out there that we don't even know. We have no idea. Like if there's an 0365 outage, how many hospital and trauma centers are going to completely collapse, right? If there's an Azure outage or if there's an Amazon EC like how much stuff is hosted critical healthcare stuff that's hosted in the cloud

>> change >> change is a great example I talked about that before if you think someone knows those lynch pins those digital lynch pins and healthcare infrastructure in this country you are wrong and I I have not seen a single person that can coherently explain to you the scope of the risk no one knows how many of these are out there no one's mapped it so it's a kind of scary thing. The other thing I would say that this speaks to a little bit is um again just how critically dependent healthcare is on these types of thirdparty uh vendors. So that's the first thing I want to show you guys and if you're interested can

read the paper and the limitation section. I'm looking at you. All right. >> Next rapidly. Uh we're going to talk about the next project which is this like hey what's the CPR for a hospital that's been ransomed and the question is like what would you need to bring to a hospital that's been ransomed uh to like get them off of paper? Like what would you have to give the nursing staff and the doctors and the technicians and the registration folks and the phabotamists? What could you put in the back of an 18-wheeler, drive to their facility and deploy within two hours to say, "Hey, you're not going to be down for five weeks or you might be

down for five weeks with your own systems, but we brought this other thing." And you can work on at least this somewhat better system until you can recover. I have questions like that's a that's a scientific and a usability question, right? Like what do you actually need to bring and how would you engineer it? That's what we sought to do with what we call crash cart um disaster recovery crash cart and we spent the last year and a half building a prototype of it and it fits in the back of a 9 foot van and it's scoped to try to restore the technology of a 20 bed emergency department. We're not anywhere near being able to do it

for a whole hospital but we're like prototyping this for like hey the emergency department needs to function. we need to deliver patient care. If anyone has a heart attack or a stroke or got gets stabbed, what do we need to bring that basically to do it? And that's what we scoped it at. And so the other interesting engineering challenge for this is like you have to build this into an entire um you have you have significant engineering challenges because you are going to set up this replacement system right next to a hostile network that has active malware. So we can't touch any of their switches. We can't use any of their existing infrastructure. We can't use

any of their existing spectrum because they're going to be bringing up their own Wi-Fi, right? We So, we have to operate on a different spectrum. We can't use their back haul internet. They won't let us touch their fiber because they're worried about data xfill, right? They're worried about command and control. So, they'll cut their back haul as part of their ransomware response. So, you have to bring your own internet. You have to bring your own spectrum. You to bring your own endpoint devices, your own electronic health record, your own laboratory devices. You have to be able to quickly integrate with their own CT scanners and their ultrasounds and all that stuff. And you have to bring all

that infrastructure in a mobile set. And that's what we did and we call it crash cart. And so some of the stuff I'm going to highlight here is like we use Starlink as our internet back haul and we co-agregate a bunch of 5G. So we have a system where we can basically bond every uh commercial uh cellular provider in an area and then bond that also with the Starlink back call to get we've achieved some pretty significant um bandwidth throughputs to try to support the system. But is that going to be the same bandwidth that we're going to have in rural Idaho, rural Nebraska, downtown Manhattan? I mean these are constraints that we have to make the system

available anywhere in the world. We spin up our own private 4G, 5G um network. We don't use Wi-Fi. I told you we can't compete on their spectrum. So, we actually do we run our own private little cellular towers, our own little cellular access points. And we run all of our endpoints and all that stuff on cellular. It's not an air gap. I'm not stupid enough to say that. What I'm trying to say is like they're going to be infected on their endpoints. They're going to have we can't touch their APs and all their stuff, but we spin up our own things and run everything over cellular, private cellular. We bring our own laboratory devices. We bring our own

endpoints that are hardened. We have or bring our own monitoring system so that we can monitor patients if they deteriorate. We can't use their existing stuff. And so I'm happy to announce that we deployed this not during an active ransomware attack, but about two weeks ago, we went out to a small hospital in the Imperial Desert was 110 degrees with our U-Haul van. Um and we deployed this in our first we had done this in the lab we had done this in our sim center but this is the first time we deployed it in a real hospital system. Um so we brought it we set it up it took us uh 29 minutes no sorry 30 39

minutes from the time we opened the back of the truck to when we had a functioning electronic health record a our private private 5G network in all of our endpoints booted all that stuff. It took us 39 minutes. We print our own labels, our armbands, this stuff. And like what is this? Like this is proof of concept for a different way to approach the problem. We can spend a ton of money trying to prevent stuff from happening, but we don't have a plan clinically for when it does happen. The question I have is like if we had this and we scaled it and it was a national resource, like if we had that type of stuff available, would hospitals

pay ransoms? would ransomware operators go and attack them because they know hey there's a backup system we can rapidly deploy within hours of response that you know can we bend the arc of ransomware economics with something like this. Hey, this healthc care is a complicated example. You got to do labs and imaging and all that stuff, but can you apply this to other uh verticals that get ransomed as well? Like the idea that can we take the sting or burn out of ransomware enough with something like this that we can hopefully try to prevent them uh attacks in the longer term is kind of the goal. And so I'm like really happy to talk to you guys

about this. I mean, I know there's going to be a lot of questions maybe or like flaws in our logic and we welcome them all, but it's been like a whirlwind two years that we've had building ransomware, building crash cart, testing this stuff. And what we need our folks to like help us make it better because I think our patients deserve it and like you're going to be a patient one day. I think you deserve it. That's it. Thank you. All right. >> All righty. So, >> so I'll do uh Dave's job while I was getting the mic turned on. Uh we have a microphone over here. We have 38 minutes for questions. Um we've got two and a

half smart people uh who are ready to field them. Um got one question ready to go. Um Dina, if you have it handy, um one of the things I had socialized to people is you had a demand letter of five things that you felt would equip the nursing staff to maintain the quality of care and the communication. Uh could you maybe enumerate those five while people are lining up? >> I did throw it in my presentation, but yes, the microphone. >> Yes. >> Um >> and by the way, I loved them. we hadn't even spoken or met yet and I think arguably this room could maybe add to and refine these for intent but um

already on their own they're pretty impressive. >> Well, we threw them in a cyber attack uh appendix for the the contract. Um we we altered it. I kind of like our our our article even we improved upon it. If there is a declaration of an authentic, and I'm telling you, they don't want to say cyber attack. We had to cross it out, authentic electronic operational failure, bargaining unit RNs will not be required to take any more than 50% of the patient load assignment as designated by the CBA staffing matrix. So, I felt like that was an improvement on the 1 to4 at the time. Like we just did that for the bigger units. Uh the

immediate recruitment of service group ancillary staff through a letter of understanding is crucial to alleviating the burdens placed on our RNs, ensuring that patient safety is not compromised. Um it's essential that one calls for additional staffing need to be communicated in a timely manner within eight hours of the start of the shift for a successful response. We added daily meeting times as we did in the petition. We um put in uh daily unit shift huddles, weekly huddles with the union and uh HR addressing all the issues going on in the house. Um alternate measures of communication. One thing I didn't say that was mildly enjoyable, the VP of HR had to come to me because my text blast and email

blast, they could not communicate with the registered nurses in the service group. So they came to me and said, "Will you please send our emails?" And there was some mild enjoyment of that. Just saying. So anyway, that was what I I felt like I we improved on it in our contract language. >> I might be misremembering this, but a followup. >> Did you ask for simulations and trainings like fire? >> Yes. Yes, we did. And >> the question, >> uh, we asked also for training for these cyber attacks because no one's ready. I I sent Joshua a picture the the other day. In the corner in in my unit was this big stack of like just thrown

together papers that we're supposed to figure out if there's some downtime issues. So, it's just incredible. But they it's profits over patience. So, thank you. >> Uh so, hi Dr. Duff. Thank you so much for uh your uh display of Crash Cart. I've been excitedly waiting for Crash Cart because I got $40,000 waiting with your name on it basically for Crash Cart for my community. Um do you know when you'll be releasing any of the information or the the data from that because I really want like the equipment list or the training list that you've been developing? >> Yeah, we just uh finished writing a paper and submitted to a journal. So hopefully it gets accepted and then

it'll be available for the whole world. Um, our two-year sprint with RPH for the research funding is supposed to end in September. And so we have a couple extra like large scale deployments we have to do, some more proof of concept uh, kind of integrations with some of the clinical tech, but we're kicking the tires at the end of the prototype now. And now it's going to be about scaling, right? So, it's like I think there are a lot of unanswered questions. Like a hospital system is probably not going to be able to buy their own crash cart and keep it in their like build their own crash cart and keep it in their basement for when

they need it. So, like we have these questions about like well should it be deployed as a critical national resource. Um like we do this thing called the strategic national stockpile for drugs and vaccines and stuff that are like strategically positioned around the country. Do we need something similar for healthcare it? Right? like do we have this around so that we're within six hours of any large area of people in hospitals that we could drive a truck there? Like there's these questions about how do we roll this out? And then there's questions about scale. Our prototype fits in the like it's almost like 60% of a of a back of a 9- foot truck. But if we were to do that

for a whole hospital, it's going to be much bigger. It's each individual parts of the hospital are going to have unique considerations. like the ICU is going to need different stuff than like your family practice office across the street. So scaling this to be able to say could we roll into a hospital and replace an entire 200 beduh hospitals total stack. There's a lot of more work that has to do with the scaling of this. Um that doesn't answer your question. What I'm trying to get at is right now I think what we need to do is have people deploy this and do a test deployment in their area and then say have you considered this or what about this or

hey we need to work on what do we because right now during ransomware response we throw the baby out the bathwater. So you get you get hit. Your technical playbook says cut off your back haul, turn off all of your systems and then do forensics IOC. Like there is a a technical rationale for what why we do all this stuff. But what it does is it basically takes a system where we maybe could use 30% of it or 40% of it. You know, there are not many cases that are documented of CT scanners being primarily infected with ransomware. I'm a I'm aware of one. That means maybe it's the case that we can use your MRI

machine and your CT scanners. We don't have to not use them during a ransomware attack. Uh or other clinical I'm just giving an example. But this work of like what can we use safely in infrastructure that's been hit? What do we need? What assurances do we need before we can reliably use it? How do we do that quickly? These are all these questions and things that are still research questions that we need to do before this is ready for prime time. Um, but in the meantime, we're going to get it out there and we're gonna have people kick the tires on it and make it better because this was our first uh swing at it and we had to do it on a crazy

timeline. Like we had two years from start to finish and I've been pretty proud of what we've been doing, but we have a lot more to do. >> Good job. in in just in a minor response. Um so it's it's I'm thinking the hospital preparedness, our healthcare coalition, and then maybe our public health emergency preparedness grant um recipients might be able to fund these local implementations and to your point like the that NDMS, right? It's like can we deploy these mobile hospitals? Can we deploy these through the Arizona National Guard or other things like that? So yeah, thanks. >> Yeah, great ideas. one question, one uh analogy. I'll start with the analogy or story uh before. So,

there was a a very small little hospital in Yeah, I figured it was a little small for me. Uh yeah. Uh in uh Los Angeles, Hollywood. I'm sure many of us have heard for it. It supports, you know, very cool things and lots of important people. Um they had I was the incident commander for an event that occurred and it started with um basically somebody looking at X-rays uh going to sites that they should not. Um those devices were not really on the network. Those devices were vendor owned. Those devices were supposed to have a vendor managed uh antivirus. Uh they did not. Um and so that went reasonably well. There was no actual impact to any other surrounding systems,

very localized. We did find it because ransomware artifacts started appearing on other domain uh systems like the domain controller which started the panic attack on a Saturday morning. Um the sad thing is is we actually had that same problem uh the computer right next to it about six months later with the same you know medical technician kind of doing the same thing going to sites that they should not have. So obviously there's a whole aspect around vendor managed systems and everything else like that. And so you mentioned like I've never heard of CT machines and all some of these others of you know doing that like I I actually have it's it's exciting. Um and this was just for

clarity and everything else like that over 10 years ago. So if that's helpful in terms of explaining that my my question is completely different. We talk a lot about the medical stuff and ER and everything else like that, but how does behavioral health have any impact on some of those things? Because we know, especially coming out of COVID, uh there's been this big push for behavioral health having just such a an impact on our communities and everything else. You know, has there been anything around behavioral health being part of that emergency preparedness kind of conversation? I we have we have a really large ER in the hospital I work in and we have a huge

area for behavioral health. A very kind governor many years ago got rid of all our mental health services like virtually most of them. So it's very hard to to move these these folks to the place they need to get to. We hold them for a horribly long time. The one good thing about their issues is they're not on a monitor, but their meds are of the utmost importance. So, it's it's kind of the same as everyone else, you know, medication wise, because when you start messing up medication for someone who needs that for their mental health, it it's even more it well, it's just as important as everything else. So, I don't think anyone looks at

anything differently. I mean, I think they package them in with all the other important things, but mental health, it's a crisis. It's terrible. >> Thank you. >> Hello. Uh, new to the field. This is my first con talking about these sort of things. So, thank you for the amazing work you do. Um, the question I have is with the crash cart, is there augmentations or overlaps into more national disaster situations such as there's no power here or you know something of that sort? What does that timeline or effectiveness look like from your perspective? >> Yeah, great question. There's no reason you can't take Crash Cart and deploy it in like a gym at a at a high school

other than our engineering constraints. We did not assume we'd have to bring our own power. We gamed it out. we could do it with a pretty uh readily available commercial generator um that could run multif fuel stuff. So to answer your question about I I don't know if this is if I'm answering your question right but um crash car has multiple potential applications other than ransomware and we feel like that is one of the most challenging engineering cases for which if it would work under that circumstance and we've solved some of those problems then we I think we've solved a lot of problems for some of these other things. Now, there are undoubted I'm not a

disaster medicine expert. Um, I'm not an expert in earthquakes or floods or tornadoes or anything like that. I'm not going to pre pretend to be, but my my feeling is that we're going to get a lot more applications from this. And that think lends itself a little bit more to this model like strategically positioning the stuff around. We do have some analogies for this in like military systems. They rapidly deploy uh field hospitals and these types of things already. This is not an entirely new situation, but one of the things that we wanted to focus on is how do we leverage the existing people at an organization. So, if you're like a FEMA team and

you're responding to a hurricane, you bring your own doctors and nurses and you deploy your technology in a in a field hospital, but you are training your folks for that system and they know how to use it. It's an entirely different game when you have to use doctors and nurses and phabbotomists and registration folks on site. So one of the things that we've built the system around is to be modular in the types of things like electronic health record views. The idea would be we have uh there are several flavors of electronic health records like Epic and Cerner and all scripts and these other things. They constitute a majority of electronic health records. If you give a

Cerner nurse Epic and they've never used Epic before, you might as well kill the patients then. You might as well just kill them right then. >> It's true. >> Uh you give a cardiothoracic surgeon who thinks they're god uh Cerner and they're an epic doctor, they'll kill you and they will not be prosecuted. Um what I'm trying to get at is you these usability issues are really real. So how do you preposition cloud infrastructure to be able to be modular in your deployment? Your Cerner shop will give you vanilla Cerner. So, at least your doctors and nurses know how to use that. You might not have all your fancy bells and whistles and all the customization you

had before, but at least you'll have the basics of usability. You'll know how to use it at an elementary level. You have to design a system that can on the fly deploy these different things. Um, is another consideration to this. I don't know if this is all answer your question but >> Yeah. >> Yeah. All right. Sorry.

Uh question on the crash cart for the 4G uh cell back end you have are you using just consumer grade 4G or is it on the >> is the 4G on the first net first responder networks or just consumer >> great question. So on the I if I mis mis uh spoke forgive me our internet back hall is using Starlink bonded with we have we have like a cradle point router that will bond Verizon T-Mobile and AT&T FirstNet um all into one signal. So there's the cellular backhall side of that and then we deploy private 5G on the CBRS spectrum in the hospital. So for folks that know don't speak cellular, we basically spin up our own

private uh cellular network in a hospital. So you look at your phone, you look at the top, it says T-Mobile or AT&T or whatever. When you when we give a we have a phones that we issue to folks in Crash Cart. On the top it says Crash Cart. It's our own private network. That's a separate cellular network that we deploy in the hospital. Why did we choose that? We couldn't use the spectrum that they were on. But two, I only have to deploy one AP. So it has much better penetrance um in a hospital. I have to deploy one P instead one of AP instead of like five or six Wi-Fi APs. So there were other reasons. We more

quickly deploy that. Does that answer your question? >> Yeah. Thanks. >> Yeah. First net's been rad like not the fastest, but definitely deployed in a lot of areas in the hospital in uh nation. >> Hi. uh as a as an EMT and a cyber security uh engineer myself, I think the the concept of uh of the crash cart's really really cool. And one of the things I wanted to ask about um it sounds like crash cart is going to be deployed during an incident. So what does kind of like the incident management process look like when you're working with the hospital? >> Yeah, great question. Hey, I heard you've been ransomed. Can I drive an

18-wheeler up to it and deploy a bunch of crazy stuff in your hospital like within four hours? Is that cool? Uh to be to be determined. Uh my mental model for this is uh anyone's seen Field of Dreams? Come on. Like I'm getting old now, huh? Like I I have to say that and qualify it. Like none of my med students I teach or my residents I teach have seen that damn movie. So I'm now getting old. Like so many times I I I had this idea for Crash Cart and I'd tell people and they'd say it's too hard. So they wouldn't even like engage in the premise. Like so I almost had to build

it and they will come. Like I had to show that it was feasible before people would ever really wrestle with well how would we do that? And so I think if we can show technical feasibility, if we can show the benefit, if we can teach a hospital like hey you might lose a hund00 million on this ransomware attack. I mean there have been documented cases of hospital systems losing over $100 million because of a ransomware attack. There's been documented cases of hospitals being completely shut down from ransomware attacks, from losing all their funding essentially. >> So if we can get far enough with Crash Guard to be like, hey, this might be the difference between you shut down or

whether or not between you lose $100 million or $50 million, then it might make those types of conversations a little bit easier. But there's still a lot of work that has to happen. We we've built it and now we have to see if they come and play baseball. You guys are like haven't seen this movie. You're like, "What the hell is this guy talking about?" But anyways, build it and they will come is to the kind of faith if you will that we're having in this. But to answer your question, it'll probably depend a lot on the hospital system, the governor of the state, how it's actually deployed. Is this a commercial product? Is this a national resource? Is Congress

going to fund this? Will imple will impact a lot of what you're talking about. >> Hope that answers your question. >> It does. Thank you for all you guys do. >> Yeah. >> Hello. Um okay so first of all love crash cart I have two questions uh related to it one the first one I'm