OT Cybersecurity and Defending the Grid

otherwise let's uh introduce our next speaker my friend and cooworker Mike tedo hopefully we stay off the green screen all right and uh he's going to be talking on uh operational technology cyber security and defending the grid so Mike take it away all right OT cyber security defending but it sounds amazing um I tried to think of how to spice these slides up um I if you guys there's a South Park episode where they try to make School news a lot more exciting and I was like that's inappropriate I'm um all right we'll you know we've got a half hour a little bit less so we'll we'll try to rip through this we're we're just going to be doing a little skim off the top here we're we're a quick intro if you will all right what is OT operational technology Puro not going to be quizzed on it we'll go through that real quick um you know what we do uh you know some basics of what we do and how we approach protecting uh our electric and gas systems uh and some of our challenges uh details of my life are quite inconsequential I'm Mike I'm OG cyber security manager at ever Source uh my power goes out I don't get to counts uh sorry and I'm sorry um last year I was able to participate uh Department of energy had started a a program a couple years ago called the OT Defender Fellowship um so that was pretty neat you know got to uh have a number of sessions with other OT security you know peers throughout various uh you know energy sector uh entities um got to beet with uh every three-letter agency uh no no surprise they all figure they're in charge if something goes wrong and everyone's kind of pointing at each other like this um yeah but that's uh government so we got to figure stuff out on our own uh been doing this for a while uh you know from networking coding audit compliance security um rised in Connecticut so you know hit Insurance hit defense now in the utility uh I don't know if there's another one to to add to the list for the Connecticut like bingo card yeah right so all right so we're just gonna Jump On In what is operational technology so um until I got to to eversource right this wasn't really my my primary so I'm coming at it from more traditional it and it security background right um but been in here for you know four years now uh know enough to be dangerous know enough that we've done some good things and we have some things that we still need to work on right but at the end of the day right operational technology it's uh the technology that controls physical processes and very often that comes along with some life and safety impact so um you know on a small scale your thermostat your smart thermostat right that's an OT device right you know that's you're you're interacting with that thing and you're fing the heat in your house um I think there was a good um Dr M Mr Robot episode on hacking a hacking the temperature controls and erasing the backup tapes but it's been a while and I bailed on that show but anyway um so so we really need to evaluate things through through that lens like we're controlling physical processes here and there is life and safety impact so what's the worst that can happen well pretty bad um death um we don't want that um you we we know there's you know there's the kind of traditional CIA Triad and so we all love our confidentiality integrity and availability uh kind of you know up until getting here was like oh they're kind of like three equal legs on a stool uh well change that slightly now in the operational environment and really it's safety so this is all wrapped up in safety and number one one is safety onea availability so these systems have to work they have to run uh not that I don't care about confidential confidentiality and integrity uh they're important just a little less so and to the effect that you know how do they impact availability and safety all right so at the top like it's starts it it really kind of looks like it just you know you've got your servers you've got your workstations we've got our switches and far so like within our control centers I mean we're running pretty you know off-the-shelf stuff right some custom software but you know to run to run the Grid or what have you but otherwise it's stuff that we're familiar with and stuff that we know how to interact with so you know we're going to install our endpoint protection tools on there we're going to collect all the logs we're going to uh you know do all the things that we normally do uh with it infrastructure uh So within our within our data centers within our control centers High degree of visibility uh because that's stuff that we know and love and how know how to interact with know how to protect and then as we leave our data center walls right and now we're starting to talk out to equipment that's on pole Toops we're talking to equipment out at substations wherever else uh we're converting to seral we're you know stripping protocols we're not no longer doing IP communication ation we might be doing radio we've got devices at at substations right so rtus remote terminal units hmis human machine interfaces plc's programable logic controllers radios relays these are no longer it devices we're we're not in Kansas anymore so uh we are not installing Crow strike on those things we are not you know uh really interacting with them very much at all um they have a job to do and they have to you know keep the safe reliable flow of electricity throughout New England so um it gets to be a little bit more Challen we have a lot less visibility down at that level not no visibility but less um you know there's a particular device that you know we've you know I know that we've tested we've run end maath stands and boy we get to Port 20,000 I know that device is going to topple over and there's no coming back from it unless you do a hard reset of it right so we're not doing that a lot of these devices right they're long lifetime devices so uh they were probably designed 10 20 years ago they have you know they're doing their job they're a protective relay they're doing what they're supposed to do right so when these things were designed and the pro protocols were written that this is all running on security wasn't even a consideration right uh or if it was it was a very minimal consideration uh that's changing that you know that has changed over time um you know equipment manufacturers product manufactur like I they've gotten the memo right that that it is a you know critical to critical infrastructure that this stuff have security um but that's not it's not like that server that's getting refreshed every few years or your workstation that's getting right this might take 10 20 years to before that device is touched um firmware updates boy there better be a darn good reason to touch that thing and update the firmware on it because again this stuff if it's working it's like don't touch it right uh so in in the operational technology like OT cybercity so there's references to What's called the per model um again the the details of this um not terribly important but at a high level like you know when I saw this it kind of was like okay I kind of get where it's going in terms of like we've got Network segmentation here right we've got layers of networking here and and you know trying to segment and kind of filter as you go you know work your way down through the level so you know the exact device names not terribly important but towards the top I I mean add I added to two boxes about like cloud and DMZ area right but Enterprise corporate Network your email your HR your business functions right that's kind of happening at the level five level four right now we're going to start to cross over into the OT space uh and we've got this OT DMZ layer in the middle sometimes they refer to it you know level 3.5 but now we're starting a crossover right and we're SE we're trying to separate what we do from an operational perspective from anything that's going on in the corporate Network right and then further down from the DMZ right now we're getting into what would be the scada or the control center aspects right and even there there's lots more of isolation um I don't think I mention it later but boy do we love firewalls we got a lot of them uh we do high degrees of network segmentation uh for very good reason to keep all that stuff away from malicious activity and away from the internet um let's say you have an issue you know our log for Jil you know you get popped in your DMZ what are we going to do we're going to cut that thing off right we're we're going to protect our corporate Network um let's say we got ransomware running around our our Enterprise or our corporate Network well I'm prepared and we have plans ready to go in place that we're going to cut off all the skada and operational stuff and we're going to run in an island mode and we are not going to talk at all with our corporate Network right um now maybe that otot DMZ layer gets popped right I hope not that's going to be a bad day but we still we can still pull the rip cord and really really isolate and at the control center level and so um we're gonna we're gonna come back to the idea in a minute of you know defensible architecture that's if there's one takea away from this it's not that it's called the BR new model but that's kind of our approach to defensible architecture here right um I will never promise that there's an air gap there is no air gap the air gap's a myth there's very valid reasons for things to either Traverse down or up um from a trust perspective our most critical are like our most trusted stuff is at the bottom and so I'll I'll typically say like the corporate network is our internet from an OT perspective right so we don't trust the corporate Network so I want as few connections originating from the corporate Network into OT as possible not going to say it's zero uh but there's a need business need and you know just to operate things to for I'll say for data to move up right and so data that you know starts at a sensor that's monitoring you know a piece of equipment at a substation the information about that sensor and hopefully it's accurate information and don't have time on how to get you know Ways and Means to mess with all that stuff but um it can be messed with but it's hard uh right we we we need to get data going up right so um I don't know if anyone's ever you heard of data diodes uh you know they they enforce onewe data Communications that's something that we're looking at and I think is going to give us some again not going to be an airap but give us some some ways to move data up the chain in a one-way fashion without introducing network connectivity from the north side uh I know you had a couple of card slides in there so happy to see that right uh this is what we are trying to do and a lot of our architecture and philosophy right if there's something happening at our corporate Network Colonial pipeline right they got hit with ransomware a few years right we need to be prepared to fully separate and and the lights need to stay on and hopefully with no interruption whatsoever right um we're pretty darn close to to being able to do that with no interruption mod I'll say the lights won't go out if we ever have to do that but um you know we might have to drive into the office for a couple of things but we're getting pretty close to the point where it's like we're going to be able to run indefinitely in this mode if we have to right I hope we don't have to but uh that's why this need do good work so all right there we go all right [Music] so some of the things that you know we we talked about defensible architecture so you know we I I think so first of all you know don't pack right um and my predecessor would you know said he's like a lot of the stuff in this space it's like a lot of a lot of waxed on waxed off right it's a boring environment and I want it to be a boarding environment and I hope it stays a boarding environment right um so there but there's a lot of attention to detail so we talked about the defensible architecture and that's very core to our our our approach there um secure what you can so right I'm not going to be able to bolt on security onto a device that was never meant to have it but there's still things that you can you can do and you can really just make it as hard as possible so um you know starting at that more it infrastructure of the OT and the control centers and whatnot right it wasn't that long ago that uh you know patches were maybe a couple times a year right it took a while it took effort it's really what what this comes down to is there's no magic bullet it's hard work right so uh we worked very closely with um you know the skada teams right to um prove that we can do vulnerability scans and the machine's not going to topple over uh there's still this kind of Legacy perception that if you give a machine a sideways glance it's GNA fall over right so we spent close to a year testing and validating and non-production environments that you know we can scan this stuff we can do it safely we can get accurate vulnerability data and oh by the you know we can deploy at a minimum the Microsoft you know monthly patches on a monthly Cadence right so you know we're very up to date on that stuff right you at the control center level right um we're not running Legacy os's at the control center level we keep things up to date right secure what you can that gets harder the further you go down the layers into the more specialized equipment but even then right rtfm like you you know get to know the equipment a little bit and and what it's capable of and if you can get some security out of it that's better than no security right um I think everyone's heard know your assets right we take that very very seriously so um and again like you know we know our assets to a high degree of fidelity at the at the control center level level uh one of the challenges is that it's harder to know what the assets are you know out in the field but um you know for example at the control center and part of it because of how heavily regulated our environment is but I just consider it good security um we know for any of our servers and workstations in these environments um we have established Baseline configurations of the installed applications the ports the services the users right at a minimum and so for any of those servers and workstations for our switches for our firewalls right uh we have a focus like we get reports every day that'll say what's changed on those assets so if there is a program that you know whatever Adobe went to the latest version or you know um whatever the case may be the latest Microsoft Patch um my team gets reports daily and we investigate that and we tie it back to an authorized Change Control something that was tested and approved in a nonpr environment 99 times out of 100 that all checks out right oh this change is because of this Co and this was authorized this was tested we're good right one time out of 100 somebody goofed made a mistake uh so far we haven't seen that um something malicious has been going on right um but that's part of why we do that as well right you know if all of a sudden Port 80 starts listening on an asset that has no business listening on P 80 we're gonna we're going to be all over that right um that gets harder the more specialized the device gets so honestly one of our challenges is you know I keep I'm laughing because Mike's heard me say this a ton of times like I need to know the make model and firmware of the thing on the other end like great I know the thing in the control center we know it's patched we know like we're protected from the you know the north side of that diagram but I need to know the make model Fromm where the thing on the other end uh at least once a month our cesa will be like hey Mike saw it was reading on uh you know bleeping computer whatever we got CBE 20231 1234 and it affects uh schwizer electric relays what's our exposure like well I don't really know uh and that's not a great answer by the way um we have some I mean we have databases we have uh you know substation engine Engineers tell you and folks that uh their job you part of their job responsibilities is to go out and manually touch and interrogate these devices once a year and update that access database somewhere once a year that's not good enough I'll check it you know that's the best I have at the moment but um one of the big challenges that you know as long as I've been here and I think it's getting closer to you know we're looking at a few options for how we get that make modeling firmware of that relay of that HMI that PLC um some tools are being are are in the marketplace now where that's becoming an option um and that's kind of like that's what's I'm G to be able to help answer that question I'm gonna be able to assess risk I'm gonna be able to say you know yes we have that device no we don't and then kind of feeds into the next one right what's the exploit path so this device at a substition that's you know fenced off multiple you know badge readers PIN codes like you know if the exploit is one that you have to do standing in front of the device okay I mean it might be it might be a 10 out of 10 it might be really bad but like you know we've got Gates and well no guns but um you know cameras and and right there's we take physical security very seriously like uh if the only way to expl that is by standing in front of that device or physically being present on the same network segment right we've got some compensating controls there so you know being able to know what those asss are are they affected by a particular cve what are the details of that so that we know hey okay yeah we do have it but I'm not as concerned or we have it and oh by the way like yes it can be remotely exploited uh you know from a presence uh you know in our control center then we're going to take that much more seriously right IR plans and drills so my perspective has been that um you know me my tetto I am not going to personally prevent you know Russia China North Korea you name it you know from from breaking into the grid if you will but we're gonna make it a lot harder so kind of like you know you and your buddy running you know running away from a bear just you know don't be the slowest one right we're not the slowest one so that's good um you know um I lost my train of thought on that oh it was a good analogy um oh yeah right so uh make it harder of course right but um I'm not going to necessarily be able to prevent that but I I I'm pretty confident that we're going to be able to detect it right because of the of the attention to detail that we we very high attention to detail to any changes the level that with that we pay attention to our systems um I feel there's a good chance that we'd be able to you know detect it right if we can't pre it prevent it I think we can detect it and then if we detect it right how are we going to recover from it um been able to you know there there have been a variety of exercises um one sponsored by the Connecticut National Guard called cyber Yankee uh really nice event that they put on where it's a red team blue team event they set up a a you know virtual ska system and you know the National Guard members are trying to penetrate a ska system the I think this year was a water plant but um you know so so my team was able to participate in that and a lot of other utilities and other entities in the region um like I said it it day-to-day it's quiet but events like that are invaluable because now they can actually get to see and like okay this is happening for real on these systems how am I going to respond how am I gonna you know what am I going to do in that scenario um there's another one coming up uh not next week the following week uh sponsored by Doe out on Plum Island and in New York and on Lo