Securing the Grid of Tomorrow

We got one more talk to go. Y'all ready for this? [cheering] >> We got a person here that makes fantastic corn beef. Has been in the industry for 9 years and is going to teach us about the secrets of the grill. >> Nobody's seen legacy. >> Let's hear it for Daniel Mu. Danielle Maguire. >> Yes. >> [applause] >> Thank you. And uh thank you everyone. Uh thank you Chicago for having me. Thank you uh besides 312 for being a fantastic event. Uh thank you all of you for coming out here because it's the audience that makes any presentation, any uh any conference, a good hook conference. Uh it's always the conversations that are happening out in

uh the seats and in the hallway that are so much better than the conversations that are happening up on stage. So, well, I hope that I can uh inspire some interesting conversations. So, why don't we go ahead and get started? Uh my name is Danielle Maguire or Danny Maguire uh she her and yes, I spent the last nine years working for an electric company in Pittsburgh doing OT cyber security for a transmission and distribution utility. So, not the people that make power, but the people that will deliver power to the end user. And uh as of a couple months ago, I am now uh at Guidepoint Security as a secops observability engineer. So where before I was very

focused on OT security uh now I'm doing a lot more uh cloud security um the manipulation of extremely large data flows. And with any luck uh we won't have OT and cloud for another three years, five years. Um so I got some time to learn. Um and then yeah uh at work I like to focus on how data flows uh but also the automation of previously manual processes and the integration of novel systems with the broader enterprise systems. Uh how many people here have bought a system or have been told that this system can handle all of the assets in your environment and then it turns out that there's still 5% that uh don't quite fit. I was yeah I was the person

then that would uh write the Python to make them fit. And so our conversation today we will be starting with a conversation about uh operational technology in general. Uh show of hands who would say that they're familiar with OT security. All right. I'm seeing a couple people who have uh seen it before but might not say that they're uh familiar with it. Um and then I'm guessing uh does anyone here feel comfortable with electric grid cyber security? Okay. one person with electric and cyber security >> specifically because of cyber force. >> Cyber force yes that is a great competition. Um I did that back in the day. It was taught me so much. Um and

then advanced grid technologies. Um just a show of hands not the security aspects but who here I mean we all know about distributed energy resources like uh solar and wind but uh who here has heard of advanced metering infrastructure? All right I'm seeing a few hands. Um who here has heard of battery battery energy storage systems? All right, some more people. That's great. That's great. [snorts] Who here has heard of uh I mean obviously electric vehicle chargers, but a concept called vehicle to grid. A couple people. Okay. Well, the good news is that by the end of this talk, uh you will all know all about it and you won't know how to secure it quite yet. That's

a bit much for an entire pres or one presentation to handle all of that. But I hope that we can start to think about the changes that this technology is causing to the grid and how the paradigms and assumptions that we might have had for 5 10 years um might start to break down as the grid modernizes to meet the ongoing climate crisis. So let's get started. What is OT cyber security? OT cyber security is the cyber security of operational technology. What is operational technology? It is related to ICS, industrial control systems and SCADA. What is the difference between industrial control systems and operational technology? This is where things start to get a little fuzzy. This

is where you start to see uh ICS/OT on the blog posts and the uh learning resources. So, SCADA cyber security supervisory control and data acquisition and industrial control system cyber security. The relationship between these two things is a little bit more straightforward because SCADA is when you have a lot of these systems spread out over a large ge geographic area. And an industrial control system can be as simple as a single PLC controlling a single physical process point. let's say a thermostat controlling a temperature, it becomes SCADA when now you have multiple industrial control systems at multiple locations that are now feeding that data back to a central point. Um that is what the electric grid has for

instance. The electric grid is the example of SCADA and also of SCADA cyber security. Um, and then I would say that just as SCADA is a subset of ICS, I would say that ICS is a subset of OT. And now this might be a little bit controversial because as far as I can tell, I'm just spitballing here, but operational technology, everyone understands, it is technology that monitors and controls a physical process. And so what I would say is that industrial control systems are systems that monitor and control a physical process in a way where they imagine that a human is going to be a part of this overall control. [snorts] Um as opposed to operational technology systems which

will monitor and control a physical process but the actual details of that physical process are never shown to the human. And the example I would give to this is a thermostat versus a coffee maker. With a thermostat, you set it to a certain value and then your house will try to or apartment will try to put the temperature to that value. If it is too high, it'll turn on the air conditioning. If it's too low, it'll turn on the heat. And you would then decide whether or not this is comfortable for you. And you would then adjust it accordingly. And as we'll see, this is the simple example is actually 90% of what you need to understand how

an industrial control system works. Um whereas with a coffee maker, there's a certain temperature and there's a certain rate of liquid flow and all of these things that you don't know anything about if you're just using a standard black percolator. Now, maybe you're one of those nerds like uh Gail in uh Breaking Bad who has all these uh meters and dials and meth labear to make the perfect cup of coffee. Well, then I would say you do not have an operational technology solution. Or maybe you do, but now that qualifies as an industrial control system because this is something that is trying to control a certain value at a certain set point. And this is the critical idea of operational

technology. It grew out of this discipline of cybernetics which was uh Norbert Weiner in the 1940s. And he was investigating how to control anti-aircraft guns. Because when you're trying to shoot down a plane out of the sky, the plane is moving so fast that if you were to aim where the plane is, you're going to miss you you're going to miss so bad that they're going to laugh you out and they're going to call in a mathematician from MIT to try and figure out how to do it better. And the critical thing is that there's a certain value that we want to keep. And the system is going to try and keep it at that value at that

set point by monitoring how far away the value is from where you want it to be. And then depending on whether it's too big or too small, adjusting accordingly. And this is the basic idea behind the basic uh control loop that you will see not only for things like thermostats and anti-aircraft guns but also the electric grid or nuclear reactors or cruise control on your car. All of these things anytime that there is a physical process that is being controlled by um a digital computer. It used to be back 50 years ago all of these things were using their own computers and that was actually good for security cuz none of the damn machines could talk to each other. Good

for security, bad for operations. Now the situation has flipped. We are at the flip tail end of several decades of IT OT convergence. And now everything can talk to each other. I can sit in my apartment and I can control a um gasoline pump all the way in Britain. Or I can sit in my apartment and I could control a water treatment facility in Mumbai. [snorts] or I could control an electric generation plant uh that is windmills off the coast of California because these things are all now connected to networks via routable protocols that can be routed over the network of networks, the internet, because that's all the internet is. It's a network of networks. And now that you

have routable protocols, protocols that can be sent over networks of networks, the world's your oyster. And so this is the first of the three main points that I want to get into today, which is that OT cyber security is the only cyber security that actually matters. And the reason for this is that it is the only cyber security that is going to have substantial consequences. Who here remembers the target breach back in like 2013? Uh who here remembers the substantial consequences that Target faced as a result? What are substantial consequences if you don't mind me asking? None. Oh, >> none. Yes. uh Equifax, the Government Accountability Office. I mean, we have breach after breach after

breach, and the only end result is that now there is a system to tell us, have I been pawned? And the answer is, if your account's been out there long enough, yes. Because we live in a capitalist society where our society is oriented around the maximization of quarterly revenue. That's it. That's the main concern. quarterly and to a lesser extent annual revenue. And so anything that goes beyond this quarterly or this annual thing such as a breach and the long-term ramifications of this given that a major breach of say the target or Equifax scale, the kind that we're talking about is maybe a once in every 10 years situation and maybe it takes several years of reputational damage to

re-reover, but you're not thinking about that until it happens because you're more concerned with your uh quarterly revenue streams. And this means that we can talk about the importance of security, but at the end of the day, everyone here has experienced, I'm sure, has experienced the feeling of being a cost center in a world where a business will only really value what the profit center has to say. The cost center is there to minimize, minimize, minimize costs as much as possible. So when we're talking about OT cyber security, security is the only cyber security that actually matters. I've heard people like Mike Hulkcom who has made some fantastic uh intro to OT uh materials that I'll be

linking at the end of these slides. I've heard him say that once there is an OC incident that has loss of life for in he was talking in the context of a Triton uh the Triton malware of 2017 which impacted a safety instrumentation system at an Iranian plant and he said if this had led to loss of life it would be all over the news and OT cyber security would be taken much more seriously than it is today and I say it would be all over the news and nothing would change. There would be some regulatory frameworks that would cause a lot of people to spend a lot more on systems and personnel than they might otherwise.

But it won't fundamentally mean that things will be secure in a way that they are not today. So what I mean to say about all of this is that OT cyber security is the only cyber security that actually matters because it is cyber security that is concerned with protecting preventing injury and death. There's also massive massive property damage potential. But the huge thing is injury and death. [snorts] All of these services that I have listed here, Tricus, which is an incident with a uh safety instrumentation system. They were trying to hack the systems that keep people safe and make these systems not work. Um I have two incidents from 2021 where Team Viewer was used uh to set a

value at a water treatment plant to an unsafe level that would have led to severe uh illness and death. Um, I've also got examples of three incidents that are believed to be attributed to the Russian government. Uh, two incidents with the Ukrainian power grid in 2015 and 2016. And then the next one, well, we'll get back to uh, those two slides in a second cuz I want to get into Frosty Goop here. Frosty Goop uh, shut off power to how many 600 apartment buildings in Liv uh, in the middle of winter, in the middle of a war. uh and it was the equivalent of a SQL injection attack expressed over the Modbus protocol. These nodes were exposed to

the internet. They were controllable over the Modbus protocol. Modbus is a very straightforward um OT specific protocol and it is considered I think probably the simplest of all of them. um for to expose Modbus endpoints on the internet shows that you're just a very immature organization in terms of your cyber security posture which might be you know reasonable for an apartment building. You don't typically think of apartment buildings having uh amazing cyber teams because they've got other things that they have to pay for. But what it does mean is that at the end of the day, there is a lot of work that needs to be done to secure these systems. And that is going to need to be

done by human professionals because I'm sure a lot of people in this room know, as Steve Shelton was talking about in his keynote this morning, [snorts] there are a lot of things where if a human doesn't argue this needs to be done, nothing will be done until something bad happens. and then the uh cleanup the aftermath may not be effective in the way that you would need in order to make sure that this doesn't happen again. So I said that the most important thing here is humans to secure the system. [snorts] The second most important thing for OT security is the air gap. Unfortunately the air gap is a lie. There is no such thing as an air gap. I

know there is no such thing as an air gap because I spent nine years working for the electric company and during co I got to work from home. If I can work from home and manipulate these systems then they're not air gaps. Um but the air gap when it is said to exist um is typically said to exist somewhere here at level three in this diagram. This is the Purdue model for industrial control systems. And anything that's it, a traditional uh enterprise uh environment is going to be layers four and five. Five is your traditional internet DMZ where you might put your website or a mail server. And then everything else is going to go in layer four behind some

kind of firewall. And now the air gap is a lie, but the firewall between the IT and the OT sections of the business uh has the potential to be an incredible [snorts] choke point that is the best opportunity for a lot of OT defenders to get a handle on all access coming in and out of the control center. And then from down there underneath uh level three, you get into level two to three. Uh and this is like the control cent's own IT systems. They've got their own Windows and Linux systems. But then down here in levels uh 2, 1, and zero, this is where you have your actual equipment out in the field. Um just one moment. 615.

Great.

[sighs] This is where you have your actual uh components out in the field. So these are going to be things like the PLC's that might actually uh monitor and manipulate the state of a physical process. Um that would be shown here as a field controller. And then it might be um the actual things that either take in data for the process these sensors or then um manipulate the process by putting something up or down. So for instance um you could have a relay which for the electric grid might uh energize or deenergize a circuit like a switch. Um you could have in a uh room that you're trying to keep at a certain temperature. You might have a sensor which is a

thermometer and then you might have an actuator that is like uh a rod that you can heat up to heat the room or you could also press a button for air conditioning which would then cool the room. So these are the physical components that are then maintained by these uh digital servers on top. And once again the air gap is a lie. But when people are talking about the air gap, they're typically talking about the boundary between level three and level four between the OT network and the IT network. And although this is not going to be airgapped because obviously if I can sit in my apartment and go through this firewall, through this firewall,

through this firewall to talk to field controllers, an air gap means just air. But it does mean that you can put some incredible controls in place and that this is the spot that you can watch to make sure that people are not accessing things that they shouldn't. or you can today because the point of this talk that I want to make is that there are certain assumptions that hold today when we're talking about electric grid cyber security that are not going to hold in the next 10 years. And part of this is the idea that there are going to be choke points like this where all traffic coming in and out of the system can be you know monitored, inspected,

controlled blocked passed whatever. Um, but we'll get into why I feel that this assumption that it's all going to be centralized is not going to hold for very much longer. >> Some of them in the solar panels, >> solar panels, that's going to be part of it. Yep. Oh, they're everywhere. Um, OT protocols. OT has its own protocols. Um, OT environments are going to use things like uh HTTP, HTTPS. They use HTTP way more than they should. Same with FTP. Um, they've also got SNMP, email, and a ton of things or SMTP, simple mail transfer. They've got email and a ton of things. They've also got SNMP. It's never configured, right? [snorts] Um, they've got SSH a lot of

the time that is often configured, right? Um, it should be. Um, but the those are all things to interact with the routable IT protocols that I was talking about. Um there are also these specific OT protocols like Modbus which is a general purpose simple protocol. You've got DMP3 which is specialized for SCA systems uh water and electric SCA systems especially. Um you got for instance SIP which is uh the common infrastructure protocol which is not the same as Microsoft SIP. It's not the same as the Nerk SIP standards. This is one that [snorts] Rockwell Automation uses to uh maintain their things. But one thing that I want to call attention to is that these protocols are almost exclusively plain

text. And there is a big push in recent years to encrypt these cuz obviously encryption is good, right? But, and I'm quoting Tim Conway here, who was one of the first responders in Ukraine in 2015 that the United States government sent after the I think it was the Kiev electric grid was uh shut down by who we now believe to be uh the Sandworm Group. Um yeah, read uh Andy Greenberg's write up on that. I cannot recommend it enough. Um but it's a double-edged sword. If we were to encrypt traffic, then we might be able to prevent an attacker who is already in our local area network from understanding what the traffic says. If the attacker is already on the same

local area network as our OT devices, we're going to have a really bad time because OT gear assumes that if you can talk to it, you're a friend. OT gear is used to living behind high walls and guards and checkpoints so that only things that it should talk to get to it. So if you walk up to OT and you are a malicious attacker, but you're there in the same local area network, you'll be able to talk to it. You'll be able to do pretty much whatever. Then if that traffic is encrypted, if the defender is trying to reconstruct it later, they will have that much more difficult of a time. So, it's going to be almost

exclusively plain text. And this might not be as bad of a thing as we've thought. But that also means that if we can't even count on the fundamental security control of encryption of TLS, then we have to start asking ourselves what other of our assumptions aren't going to hold in this very specialized space. And then the last thing that I want to call attention to before we start to talk about certain IT controls and why they may not be as helpful in this situation, process engineers, the people that actually program programmable logic controllers are not programmers. They hate and fear the computer. These are mechanical engineers, chemical engineers, electrical engineers. they went and they studied linear

differential equations and modeling and actual science. Um, and they don't want to worry about how to program an Arduino or a Raspberry Pi. [snorts] So, these five languages that I've uh listed here, ladder logic is by far the most common, and it's the one that you'll hear as kind of shorthand for OT programming. Um, but really what it just comes down to is that process engineers want like point-and-click visual interfaces. so that they don't have to worry about the details of the code. They can worry about uh implementing their process. Um but as uh Dr. Katherine Alman was just saying uh quoting uh Casey Smith who was quoting uh John Ericson uh hackers get their edge by knowing how the systems

actually work and OT engineers by and large don't know how the PLC's work. So that is a huge opportunity for then the OT professional and that is one thing that I then want to say as we're going to start to get into this and we're going to come up on this uh at the end of these slides. Uh but this is a solvable problem. Um Draos, who here has heard of Draos? Yeah, a lot of people. They're the big United States uh OT company. They employ like two out of three at least OT professionals, probably higher. Um, and we'll talk at the end about what the CEO says has to say about how small companies that do basic security

hygiene. So, not small companies who are Drago's customers, small companies who do best practices are beating APS. Um, this can be done, but there's only one way to do it, and that's to hire people, to hire enough people to pay them enough to, um, give them the resources that they need at their job. And yeah, their AI can't do this. Um, no computer system can do this. The humans will need computers to help them, but for the kind of things that we're doing, we just need dedicated people who have the resources they need to do their jobs. >> Preach. >> Thank you. um as uh we have here um an acquaintance once said at the NSA that this person

said if I you have a security product I'll buy it but none of them work and I would say they do work if they are properly deployed in the ecosystem uh that they are supposed to be a part of with a human being that can then take advantage of that tool to do the things that it was designed to do. Um and let's talk about why some of the traditional IT tools may not work as well in an OT environment. So, intrusion detection systems, you've got detection systems which just watch when bad things happen and raise an alert. You've got prevention systems that'll actually block the mischief. So, this is great because it'll actually stop the bad thing from happening. And

we can never ever do this in OT because who here has had an IPS prevent something? Who here has only ever had IPS prevent the right thing? IPS gets it wrong all the time. All the computers [clears throat] do. It's why we talk about things like precision and recall and false positives and false negatives, and that's all great. But in an OT environment, stopping the service is usually just completely unacceptable. And it can be unacceptable because you might be a um colonial pipeline who has their payment system hacked and then decides to shut down their entire uh gasoline delivery because they can't collect money for it anymore. Colonial Pipeline, if anyone remembers the shutdown in the uh southern east coast a

few years back, it wasn't the OT systems that got got, it was the payment collection system. But then if you had something like the electric grid going down, um, one thing that they always drilled into us, the electric company is there are people throughout the city who are on at home life support that if they don't have power for more than 24, 48 hours, they will die. >> There are people, hospitals, I come from Pittsburgh. Pittsburgh has a lot of hospitals. Our hospitals all have dedicated substations because we've had surgeons come in and tell us if a power if there's a power outage while I'm doing surgery and it's more than 5 minutes that person's probably going to

die. This one's going to be more controversial, but um air conditioning is a human right now because there are people that live in buildings that if they are in them during the summer and they don't have air conditioning, they will overheat and they will die. And uh first and foremost I have to say um people in jail and in prison people in jail and in prison often do not have adequate air conditioning and they can die from this. So and that's just electricity, water, um medicine, all these things. It's often not acceptable to have any kind of outage either because it would cost you too much money or because people are relying on the service. So we cannot use intrusion

prevention in an OT environment. we can only ever use detection. And then this gets into um the the flip or not the flip side related to this is that with endpoint security. So once again, endpoint security, we're not going to have it in prevent mode. It's only going to be in detect mode. So whereas the intrusion detection or prevention system is typically listening over the network, something like a uh Cisco ASA, um the endpoint security is going to be listening on the devices themselves. And this is I mean back in the day it was Norton or McAfee but it's mostly like uh Crowd Strike as we saw in the previous slide or [snorts] you've got um Microsoft Kasperski. Yes.

So whereas in the previous slide I was talking about how we cannot turn on prevention mode. The other thing here is that actors in uh in OT environments love to live off the land. They love to use LOL bins, especially this one group that I'm going to call out, Volta Typhoon, which is a suspected Chinese aligned threat group that has been observed uh probing into um electric systems all over the country, critical systems all over the country. I think it was recently announced that they were found inside the OT network of a Massachusetts uh electric and water utility, a tiny one that does both because that's the kind of places that these are going to target. [snorts]

Um and yeah, so if you're trying to do um endpoint detection that is based around looking for suspicious samples of malware, um you're going to have a bad time, especially in an OT environment where one thing that OT has going for it is that it's such a regulated environment that OT operators often have much more visibility and asurances about what their assets are like that um IT people don't have. Uh, I just left an OT job and I'm at a job where I advise Fortune 500 clients now. And I got into a contract and I said, "Okay, can I see your uh, configuration database, your CMDB?" And they laughed at me >> because that's only something that you

get when you're an OT network that will then have a federal regulation to have a good database of all of your systems. And then you pay people like myself to maintain that database over several years. It's expensive. It's tedious. It's absolutely essential. But what it means is that I had a list of every single port that was authorized to be on a device, every single piece of software that was authorized to be on a device, every single user that was authorized to be on a device. I knew which devices were supposed to be talking to which other devices. All of this combines to mean that if I come in and I see um a weird process executing with admin

rights on a machine that is only ever supposed to like send network time to the other servers, I have a sense that something bad is going on that it might take someone in IT who has less of that level of visibility a little bit longer to find out. And that brings us to the third thing on the list, which is the SIM. Um or the seam, as I've heard people say. Uh show of hands, who calls it a SIM? Who calls it a seam? I'm right. Yay. Um, you need a seam. >> If you preempt it with NextG, >> there you go. Thank you. That explains a lot. Um, you need a SIM. You can't get by doing

cyber security in 2025 without a SIM. And we've all got SIMs. Now, the problem is we don't have enough SIM for the amount of data that is flowing into the system. And this is a problem for IT departments, especially IT departments that are embracing concepts like, you know microservices containerization Kubernetes. Um, but especially for OT environments, you might not have that kind of advanced technology. You have the opposite problem. You have devices that don't support CIS logging. You have devices that can't tell you or they'll tell you when there was a successful login but not a failed login. Why would you want to know? Why [snorts] would you want to know? Um, IT SIMs might not support the OT SIMs

out of the box and then you have to have uh custom configurations. And the big thing, the really big thing is that you are going to spend so much more money configuring and maintaining this thing than you ever did just, you know, buying it. So once again this gets back to the point which I was trying to make at the beginning which is that with electric grid cyber security we understand well like a basic orthodox classical electric grid cyber security um but now the problem is not all of these assumptions are going to hold but when it comes back so as we've been talking about so far you need to pay humans you can't have AI

automate the entry level people out and then say oh we'll just pay mid-level and senior level people. You get mid-level and senior level people by paying the entry level people. Um, and that doesn't mean that there isn't an incredible opportunity with LLMs or any of the other technology that's been coming out in the last 5 to 10 years because generally OT networks are going to lag behind it networks in terms of technology for a minimum of 5 years. Um, but none of it can replace the human. It all has to augment the human. And as I said earlier, um, small orgs can beat these advanced persistent threats with basic hygiene. A, for instance, a Massachusetts water and electricity,

water and light, which uh just had uh just found volt typhoon in its OT network, if you believe the Draos report. Um, organizations like that do beat back uh, attackers with things like acid inventory, acid inventory, network segmentation. Um, don't put your critical, don't put your OT stuff on the internet. put it in a network with a really really strong firewall as strong as you can get in front of it and then your IT network and then the internet and then also you know password account management security patching lease privilege all of this is can and should be common sense stuff at this point. Um so yeah that's where we're at today. That's how a tiny um a tiny utility can

[snorts] beat back an AP. But there's two problems with this. Um, number one, the reason that it's possible for small utilities to beat an AP is that if someone, if an AP wants to compromise the electric grid, then they are going to find out of the hundreds, thousands of independent organizations that make up the uh, North American electric grid. Um, they are going to find the one with the shittiest security and they're going to get in there. They also are just people with jobs. they want to do as little work for the maximum reward. So this means that when we're talking about an a small worker can beat an AP. What we're saying is you can outrun you

cannot outrun the bear but you can outrun your friend. If anyone's heard that joke, well I think we have that two campers in a tent. Uh there's a bear attack outside and one starts lacing up their boots and the other friend says, "What are you doing? You can't outrun the bear." And the friend says, "I'm trying to outrun you." That's the thing is that a small electric utility or maybe not water so much but a small electric utility which is part of this very interconnected ecosystem is able to protect itself because a resourced attacker will just go somewhere else if it gets too difficult. So this means that in my experience OT professionals are some of

the friendliest most uh helpful people that you have ever met [snorts] because they want to secure this stuff and they know that that the only way to do it is cooperation. It can be done. It takes diligence and cooperation and so OT professionals the nicest people in the world. They will help you out however. I'm no longer an OT professional so I'm not talking about myself. [laughter] Um, but let's talk about electricity because now we're going to start to get into the fun stuff. Um, real fast because I hated freshman year physics. I'm sure you all did too. Electricity, flow of electrons. How much push are behind those electrons is voltage. And how fast those electrons are moving is

current. And when you uh multiply voltage and current, you get power. And that is what an electric company sells you. It's selling you electric power. the rate of energy transfer over time. Um, and then there are two refinements to this. You can have a direct current, which actually means that it has a constant voltage, or you can have alternating current, which means that it has a varying voltage. It's a very helpful naming system. Um, and alternating current is what's going to come out of your wall at 120 volts, 60 Hz if you live in the United States. Um, and the single most important thing when you are running an electric grid is that the amount of electricity that you make

always has to be exactly the same as the amount of electricity that you use. There's not a good way to do grid scale storage yet because we don't have batteries that can handle electricity at that scale. Um so transmission is what is going to be taking uh higher voltages from substation to substation and then distribution is where you go from the distribution substations to the residential customers. And the critical thing that I want to highlight on this diagram is that in the real world the grid is a grid. It's a bunch of interconnected nodes. Um in this diagram they have chosen to model it as a line as a series from the power station to the residential consumers and

this is an assumption that for 100 years was pretty good for the grid. Um you had consumers who use electricity you have whether they're residential consumers or industrial consumers and then you had the generation people. But now with uh wind and solar, you are starting to well, we'll get into that in a second, but these ideas about how the grid ought to work are going to start to be challenged by some of the technologies that we're going to look at. Oh, and just real fast, um, the regulation of the grid's a [ __ ] mess. Um, long story short, in Chicago, it's in RF's territory, and RF designate RF handles it with PJM. And at the top

it's FK and NERK. And this whole system was set up after the 2003 blackout. I will be nice to you and leave it there. [snorts] So, as I've been saying, the grid is changing and the way that we think about it has to change. So, some of these assumptions that I've already discussed, power is going to flow from transmission to distribution to the customer. It's going to be generated deterministic. You're going to know exactly how much power you're going to get before you get it. Grid operation is going to be human controlled and human legible. And when I say human legible, I mean uh everyone here understands that like when we have a modern LLM or a

modern machine learning system, under the hood, it's just math. It's just a linear algebra. But then if you open it up under the hood to be like, all right, what you got numbers? Oh, that is so many numbers. I have no idea what any of those numbers mean. That's what I mean. Um, if you are pro, if you're manipulating a system, telling a system to do something with traditional programming, you can read the source code. Um, with machine learning, it's just numbers that make no sense to the human, only to other machines. And that is the destiny of the grid, I believe. Um, and we already talked about uh the air gap and control systems theory.

So that's all fine and good, but that doesn't allow us to do certain things that we want to do. And we're going to go into all of these questions in detail. But the big thing that I want to bring up is that these are requirements for the grid and for society to meet the challenges of the 20th 21st century. Uh in terms of the climate crisis, but also the fact that a lot of technologies that we just want to deploy in general are having much much more electric consumption that it requires a grid that can do much more than the grid was ever designed to do. So the grid has to get better. And the

thing is, the grid hasn't been getting better. There's this really great book, um, The Grid by Gretchen Baky, and what she talks about is that for a hundred years, the grid's pretty much been the same. Like Edison and Westinghouse figured out a way to do it. They had their war of the currents. Like we put wire all over the country, and now no one wants to pay for that anymore. Um, no one wants to invest in significant new infrastructure. Um, but we have all these things that we need to do and we're trying to do them on a grid that's basically been static since at least the 1960s. So, the big one distributed energy resources, solar panels and wind farms

because first of all, um, these are going to be spread out everywhere. I said that the traditional grid model assumes that everything that uh we care about is going to be behind a air gap or can be put behind an air gap. But what about when you've got a solar farm that spreads hundreds of square miles with a minimum of like a mile between every asset or what about offshore wind farms? But that's just geographic distance. The much much harder challenge here is that these are non-deterministic devices. If you have a coal plant, you're going to get as much power as you burn coal. It's the same with natural gas and nuclear. Um, but with a solar panel or a wind

turbine, that depends on how sunny it is or how windy it is. And if you know how to predict that um, perfectly, like at least 24 hours in advance, um, please come to me. I will pay you $500 for that information. You're not getting a better deal anywhere else. Um so in order to deal with this um where we don't know how much it's going to generate and also the customer when their deer isn't working if the solar panel isn't powering their house or the wind turbine isn't powering their house they want to use the grid. Okay. Um and then if it is working and they have too much power they want to send that back

into the grid. Well, okay, because a second ago I told you that we've got an assumption that power is always going to be flowing from the generation to transmission to distribution to the end user. But now these people want to send power back and they want to do so in a way that we can't predict in advance. Um, so one of the ways that we talk about how to do this is battery energy storage systems. If we could store a lot more power at a magnitude and scale that we don't have the capability to today, well then when there's not enough power, we take from the battery. When there's too much, we put it in. Great. Uh we

don't have those yet or at least not at the scale that we need. And they bring their own challenges with them. Um doc, I told uh not this talk. I told uh Dr. Katherine Alman that I was doing a talk about electric vehicle chargers and she said, "Oh, oh, electric vehicles. Did you know that I'm an amateur firefighter? [clears throat] Have you ever heard of something called thermal runaway?" Yeah. When these batteries start burning, they cannot be put out. And that is the challenge of a battery energy storage system. But um with deers before the battery energy storage stuff, what we're really getting to is that They're spread out, so you can no longer put them behind a single wall. Um,

they're going to be organizationally chaotic. They're going to be owned by like minimum of like three organizations. It's going to be a patchwork of individuals with solar panels, individual wind farms. You've got the small scale uh like startup wind turbine people. You've got like the major oil companies that are trying to uh diversify their portfolio by putting renewables in there. like so it's not also all going to be under the control of a single utility with a single org chart. [snorts and clears throat] Um yeah and then these also are going to bring new devices with them like for solar uh that's a direct current thing and the grid is alternating current so you have to put a device called a smart

inverter in there and the smart inverter can bring its own vulnerabilities. Um we're going to get into some of these other assets here but maybe I'll just get into some of these other assets. advanced metering infrastructure, smart meters. Who thinks these things should have TLS encrypted web servers on them? All right, so people aren't even really taking the TLS encrypted web server, plain text web servers, bad. Yeah, FTP servers, email servers. These are in your home. This is right outside your house. This has an IP address. someone if someone could get into any of these things, not only can they control your electricity, they can also see things like uh when do you have spikes that might correspond to, you

know, lights are on someone's home, um dishwasher, washing machine is running, someone's home. Um the privacy concern is ridiculous. Also, I was told this nine years ago that, oh yeah, I just got a job at the power company. We're working on this great new thing called AMI. What it means is that when we need to shut off a customer's power, they won't run outside and wave a shotgun in our face [snorts] because that is before we had smart meters, we still had the like direct analog wires that someone had to drive past and look at what the value was. Nowadays, it'll be on the internet. They'll be able to shut off your power over the internet.

That's what these things were designed to do. They were designed to take a command from someone and shut off your power. And of course, only [snorts] the authorized people will ever be able to do this. Um, and also there's only a couple vendors here in the space. And so what this means is that if you find a vulnerability in one vendor [snorts] before, if you wanted to attack the electric grid, uh, unless you could find like maybe you found a vulnerability in a PLC or an HMI, these are very specialized systems that are deployed in very specialized situations. Um I've heard it said that the the big victory of stuckset even more so than the

technical aspect of the four zero days uh that went into it was the human intelligence, the signals intelligence to know exactly what zero days to choose um to know exactly what the settings needed to be for the centrifuges that stuckset was going to destroy. Not the same with smart meters. Uh from the perspective of the internet, someone with a smart meter in Chicago is no different than someone with a smart meter in Pittsburgh is no different than someone with a smart meter in Lev, Ukraine. Um so one person, one vulnerability would then have very broad geographic impacts. Uh EVSSE and V2G electric vehicles obviously becoming a much bigger thing in recent years. Um they've been growing an enormous amount.

They're probably going to continue to grow. Um, fun fact about these, um, their chargers are absolutely chalk full of bugs. And we're talking password injection or no, I'm sorry, [sighs] command injection via the Wi-Fi field. Via the Wi-Fi password field. >> Christ. >> Yeah. They had a competition in Tokyo and people were finding uh, route RC's in 30 minutes and everyone that was there agreed, oh yeah, this is just something that no one's looked at before. Um, another one here, root in 30 minutes over a set UID tn net shell listening o on the local network and and the control protocol for these things completely unencrypted. So that means that if you're sitting on the Wi-Fi network uh like so let's say

you're at a parking garage or a car dealership or something and you're sitting on the network uh and you're watching the messages go through. There's one like integer ID that you need to capture while sniffing because that is what corresponds to the human who is trying to charge their car. [snorts] Um 3 minutes. Great. Um capture that. You can do whatever you want. Come talk to me afterwards. Vehicle to grid. We don't have enough batteries for the grid. We've got a lot of batteries for all these new electric cars that we have going around. And so people said, "What if we use those as one giant distributed battery?" Very cool idea. Um, I've actually been talking with professors at

the University of Pittsburgh that are researching this thing right now. Um, it's a problem that's very much being solved. Um, security people don't seem to be a part of the conversation as much as they really should be. So, if this interests any of you, there are so many ways to get involved. Please come find me. [gasps] This is coming in 3 to 5 years. We will see SCADA in cloud because we live in a capitalist society where the most [snorts] important thing are the quarterly balance sheets and someone at a bunch of organizations is going to save a lot of money by moving off of on-prem SCADA into cloud SCADA and then things very bad things will happen in 5

to 10 years. Um anyways uh the long story short two minutes thank you. Um just as we are understanding the traditional electric grid, it's going away. Um the grid today is considered an industrial control system, there are certain set point values that operators are trying to keep at stable areas. I argue that tomorrow it's going to be an industrial internet of things. It's going to be a network of networks, a system of systems. various organizations like your electric utility, your uh building management systems, your for electric vehicles, charge system management servers, all that. Um I can talk about this in the hall if anyone wants to hear it. You can learn 90% of OT cyber security

in a month. Um, my buddy in Pittsburgh, Mike Schroeder, told me this and he said, "The la the next 10% will take the literal entire rest of your career because the last 10% is things like, oh, I didn't know that this one thing only works under blood moons with uh like middle Indian data or some [ __ ] I have seen middle Indian. Um, it's cursed." But yeah, if you want to learn the basics, uh, watch this video series. Um, build a virtualized SCADA lab with open PLC, do replay attacks, and then watch from the other side. Um, go online and find Triton malware, that stuff I was talking about that damage security systems. It's just on GitHub. Um, yeah,

and listen to experts who combat fear, uncertainty, and doubt. Whenever there's a problem, people love to say, "Oh, I bet it was evil Chinese super hackers that, I don't know, made it so that this light is red when I want it to be green." Um, or more likely, something blew up. Was it hackers? A br a boat crashed into a bridge. Was it hackers? I heard that one a lot. >> Um, squirrels. >> Yes, squirrels are a bigger risk to the electric grid than hackers. I'm completely serious and that's what I want to close on. Squirrels are a bigger risk to the electric grid than hackers. Thank you all. [applause] [cheering] >> It is. Yes.

Okay. Uh any questions? >> Yes. >> So obviously there are a myriad of like uh IDS IPS tools for you know traditional IP uh being protocols. Do does anything like an IDS exist for uh like these OT specific protocols like Modbus and whatnot? >> Uh yes and Drago sells the most popular one in North America. There's also uh Clarity which is an Israeli company. There's also uh Nomi and I think that industrial defender is also trying to uh get into this space. But yeah, the big one is Draos and everyone else is comparing themsel against Draos just as every other SIM vendor is comparing themselves against Splunk. >> Yeah. [clears throat] >> Tables. >> So are the majority of the SCA systems

still limited to an 8 character password all caps and no symbols? Are the majority of SCADA systems still limited to the 8 character password? All caps, no symbols. I want to say that more and more systems are no longer limited to eight characters. But the flip side is every SCADA operator password I have ever seen. Eight characters, all caps, no symbols. Again, these people have their own problems to worry about. They go to school for years so they can learn how to design an electric grid without it breaking the normal way. Grids love to break. um they're not thinking about attackers and that's where OT cyber security professionals come in. >> Did they still explicitly follow the

requirement that the um password be not default? >> That is one of our big ones and I wish I did not have to tell people change your default passwords but and really this comes down to asset inventory and knowing where your things are because everyone changes the password on the system that they know about. >> Yeah. >> Thank you Danielle. Danielle be happy to go ahead and answer questions out in the hallway. >> Yes. Thank you. >> Absolutely. Thank you very much for Thank you all for attending all these talks.