ICS/SCADA Defense

all right folks we're going to go ahead and get ready for our next speaker uh so by way of introduction uh yesterday at the security conference i had someone come up to me afterward and say hey uh is there anybody that's like deploying security on you to ics and scada networks and i was like yeah you need to be here tomorrow as a matter of fact uh because we have one of the world's leading experts on ics and skater networks to talk about that so just not curious would anybody be willing to admit that you are responsible for defending an ics and scada network that you can talk about a couple of folks so we know how critically important this

stuff is even if you are not necessarily responsible for defending it we know that it does affect all of us and can affect all of us if it falls into the wrong hands so please join me in welcoming mr chris sistrunk thank you for coming to besides augusta today it looks like it's been a great event so far and let's kick things off so let's talk about protecting scada networks a little bit about me i'm a registered professional engineer i was with entergy for instance power company in the south in louisiana mississippi arkansas and texas and i was there for almost 12 years and i was the about the last nine or ten years of that

job i was running their scada systems from different pieces of it and then in february i started with mania on their ics scada consulting practice i'm a skid expert i've been doing a long time i love security i'm also a member of the dmp users group which is a open protocol we'll be talking about today and some other protocols as well i'll be talking about this too and i'm a button pusher i also like the blue team and how i authenticate the systems i love that picture uh if you've ever seen that movie up well you have to to do bluetoon you have to know a little bit about red right so another good picture i love the show

what happens when you use nmap or fuzzer on an industrial control system you get training ghosts that's because a lot of the systems are 10 or 15 years old they have network stacks that haven't been totally tested and when you do even pings a packet of thing of death you'll get your devices just fall over and what happens when you your device falls over that didn't say fudge oh and that was me whenever i started accidentally uh breaking my stuff because i was like well let's just hook up stuff to it start testing in the lab when i used to work at energy and me and my colleague of mine on the dp users group we

started testing dmp3 protocol he wrote a fuzzer and i'm going to talk to about that in a second and we found so many vulnerabilities they got in the press several of the articles that came out they're like oh man this is pretty serious he wrote a buzzer put it up on the uh github and said chris uh or dp user group you send an email out to the dp users group he said i've got this closer that i wrote for dmp3 did anybody want tested itself on github so i downloaded it first thing i tested it just you know broke i was like oh boy we should take this off of github and so that was in april 2013

uh adam crane and i started project roblox which is latin word for bulwark or strong we wanted to give it that name because we wanted to take a protocol that was widely used in north america in the electric industry water industry also in the uk australia and south of africa it's the most widely used protocol for those skate systems as dp3 we wanted to make it stronger so in the course of since that time we've come out with 26 advisories from ics server we've got some more that still haven't been released so we've got 32 tickets in now total 24 dmp3 advisors one with modbus and one with a serial only protocol called telecare 8970

and we wrote the ics fuzzy framework called aegis we used to call it medusa because it turned everything into stone but then we're like well we need a new defense so if you've never seen the shield the aegis shield has medusa's face on the shield so you're using a little red team to protect and be blue and if you want to go to the website automatic.com forward slash robust it gives more detail about our project and also the code for the fuzzer it's open source so we've released it it's got dd3 in there we're about to be releasing it later with modbus and we're getting other uh ics researchers to add more protocols to the

framework so automatic.com four slash aegis so a little bit about dnp3 anybody ever heard of dmp3 before okay well for those who don't know it's a scada protocol that works on serial or ip stacks and it uses it's the same protocol no matter what the only thing different is the tcp headers are slapped around it for the tcp side or udp you have data link layer transport function and application layer and then the newer version of it that just came out in 2012 we've got secure authentication so you can do secure authentication built into the protocol even over serial networks and then you have maybe a master database like the command center or you know

the operation center for a power company and you have its database like binary inputs like breaker status or are the lights on things like that analog inputs like megawatts or amps and then it's databased in the remote terminal unit the thing that's out in the field every substation it has the same database so they talk to each other over uh tcp or cereal to dig into it a little bit deeper this is a high chart on purpose this shows that all the fields that are available in this rich protocol you've got all kinds of things for the link layer you've got the destination and source and you've got crcs for everything because there's a lot of

error checking involved you have application headers you have all these different function codes for the master to send like read right select operate freeze restart open file close file and then the rte response was three function codes in here got all these different just smarts in the protocol and you can go up to 16 data blocks so it just shows you there's a lot of features in this protocol but there's also a lot of area to make a mistake if you're doing coding so the fuzzing we did had tested things where they had never been tested before so we found some vulnerabilities due to some actual screenshots of the errors we got we actually caused one rtu to lose its

configuration they sent an unexpected message it went 100 cpu and all of a sudden it rebooted and whoops configuration's gone oh and then they pop the calculator now they're in your system okay don't know if that's actually going to happen it could if you know what we know so the state of ics and scada security it's for those who are not familiar with it it lags it by about 10 to 15 years so the things that you guys are working on now on your enterprise systems well guess what we still have serial networks you know i had 1200 ball lengths at my previous game now 735 stated related vulnerabilities on open source vulnerability database since

2011 and jericho if you know him who runs that site he says it's like kicking a puppy i mean it's so easy to knock these things over the reason one of the reasons why is for so long the scada companies make these devices have done positive testing so when you buy this new system you go to the factory and you do a factory acceptance test and we say okay send control one okay send control homework okay great hit control two readout analog one ready to analog two and they go do the positive testing but they never have done the negative testing so like the front yard is mowed but the backyard has never been mowed

and when you mow it and you find like cars and toilets and who knows what you find in there so please fit the grass that's kind of that picture there so now what we found out that you know your scada system might have some vulnerabilities and more than likely it does so what what can we do we can ask ourselves some questions what is the risk if this device is compromised okay you can go to the standard risk analysis i did this in energy for every rtu we had and actually did a presentation on this at s4 digital bonds s4 in 2013. so you take the probability of the device failing times the impact equals the risk so the

probability might be the age of the rtu does it have any vulnerabilities uh is the circuit that it's talking to is it analog based land or is it happy or you know open 900 megahertz radio and the impact is maybe a little patient-based like how many customers are at this substation or does it have a nuclear power plant connected to it or how many transmission lines are coming out of it or you can do for water too i mean it's like how big is the fight coming out of your water system there so the impact may be the big factor there what is this uh scattered device talking to is it talking just to its

local devices or is it talking to a wide area network like a skate assistant for a power company any states and is it segmented at all is it is all of your critical stuff talking to each other or you've got critical and non-critical stuff talking does it use serial or ip protocols or both that's a big thing a lot of people used to focus on security of ip protocols but we found that you can fuzz over cereal how do we how do we defend unsecured protocols most all ics protocols are unauthenticated and unencrypted so how do you defend against that because you can't just go and build a new protocol right now and go install it

and it's going to take years and years for that to happen so how do we deal with that and let's think about physical security um that's an important thing can i just walk up to your substation and just walk in and plug in you know and nobody will know lastly and this was me and my previous company will you be the one that gets to call two in the morning when something goes down so that was that was totally me so let's anticipate the mitigations what type of medications exist which ones will you use how will they be deployed so you've got to answer those questions on the previous slide with what we have so

keep calling the blue team is here we are here we can help we can defend scada systems even though they are very normal so how can we do that we can do software firmware patches device upgrades you know pretty simple as familiar in it systems we can configure the rtus and other scada devices to have robust configurations like turn on things that are not needed robust network configurations is everything wide open any any on a flat network we could probably do a little bit better job of making that better do we have any control system protocol tools that can help us out they're starting to be more of those proper physical security and employee awareness for those of you we have any

vendors in here that make ics stuff okay well the last one is secure coding and secure secure development life cycle for the vendors and if you buy gear from vendors make sure that you start making them do sdl because that's new to ics stuff but it's been around from microsoft and other companies for a long time so that's kind of what where we are hey we tested the software let's ship it um yeah that's kind of what happens and we find the vulnerabilities if you look so will the regulations and compliance help us will emerge sip help us with this mitigations will nerks that protect us against serial attacks no there's nothing in there but it's all

routable protocols they completely forgot about uh serial and the last revision so they're you know i'm thinking about that now see facts for the other folks and like all i guess they don't really have any regulations and compliance versus cyber security so can we rely on that well we don't know the first thing we can do is get it get the bug fix and that's pretty important you know there's software updates coming regularly now from ge abb siemens they all have cert teams and a lot of the others a lot of the other vendors are starting to get to get the picture we should start doing negative testing and doing white box testing at the factory so now they're

trying to come up with patches you know vendors have told me they've come out with patches for their software and scada software but no one's ever downloaded that patch before so it's up to the user too you need to go ahead and get the patch and then obviously if anybody's done any uh patching on a control system you can't just patch it like i t you have to completely test it before you roll it out because you may have dependencies in other parts of your network that break if something upgraded and i've seen that happen and it's not not fun when you have to go roll everything back but you have to have a plan for it

and if you've never done this work with your vendors because they probably have more experience in it they probably made the same mistakes before and can help you out so the next thing we can do is robust device like plcs and rtus we can configure those more robustly and also the scada master stations for instance you can use dp3 secure authentication it's that's one way to make it more robust is you you can correct against uh replay attacks because now you have to authenticate hey i'm a so-and-so master and you're the rt i'm supposed to talk to an artist yes you're the master i'm supposed to talk to and great now we can talk but it won't protect against things that

are in the lower layers like link layer and transport layer if there's bugs in those so at least will protect against the application layer now i've seen this so many times if you've looked at a configuration for rtu well if it's not used it's but it's still enabled in the configuration whoops that means i can go and use it um so disable all those unused serial and network force if you have the next one is a workaround so you have a windows hmi you know human machine interface and it's got some windows services and it's vulnerable well can you make it or restart if it crashes instead of having to send a man out there two hours

to drive that might be one small uh workaround and then check the default settings another big thing is well let's say we bought this device and it's maybe a recording voltmeter and i've seen these on on the internet a recording voltmeter that they hang on the side of your house to if you've got voltage problems but some of those have dmd 3 enabled by default and they're just willing to talk so if you're not going to use the mp3 and or any other service just turn it off we've seen several things on showdown and you can go look that have the same configuration like they're talking sending the exact same messages and you go from one ip to the other they're all

the same output that's not supposed to happen at least it wouldn't on my system anybody familiar with showdown okay well it's been a pleasure knowing john matthew he's a good friend of mine and we've been helping him to add more ics and scale stuff to showdance so this is one of those new reports that you can do showdown.io and i did a search on port 20 000 that's the default port for dp3 and it's over 600 devices that show up you got this neat little map that shows up us has about 415 at this point and most of them are on verizon wireless like it's a cell phone radio so it might be like who knows it could

be wind farm out in the middle of nowhere talking and it's wide open and what john is actually doing he's actually and this is some of the he's one of the protocols that he's using he's sending an actual dmp3 command a short message that we helped him put in his system and it just it sends out a real name of your message and whatever message he gets back he puts in the headers and showing so i'm going to take this time stop here to do some and trivia got two things to give away but i've got some more questions so uh the harder questions i give the the prizes too so what does scada stand for

does anybody want to raise their hand okay uh uh

yes all right nice job okay let's ramp it up a little bit what is the standard tcp port for mazdas does anybody know okay well i've asked too hard a question okay okay you got one close you're very close go down about 10. 502. all right so i'm going to give you this book book hunter's diary

all right you can actually search for my bus on showdown and i'll show you that in a second does anybody know if they were paying attention to the beginning of my talk i kind of showed it briefly uh whoa the two starbucks for dmp3 it's always the same and wireshark couldn't understand this anybody having this i saw the first one just 54. okay it's close you're really close okay it's oh 564. it's always so 564 and others ics protocols have different um start bytes usually okay what year was stuxnet discovered

no 2010 2010 who said that yeah so you you're gonna get a pick set from here

and anybody ever heard of headaches it recently came out from mount new malware it's the second ever malware against the control system does anybody know what protocol havoc's used tcp3 no no it's actually used opc opc is um ola for process control it's kind of like a microsoft thing that allows microsoft enabled plc's and hmis to talk but so you can google it pretty good wikipedia and then uh anybody recognize what this is are you sure yeah that's an hmi human machine interface this is one that my friend uh dan tintler he's this on on twitter he did a vnc scan of the whole entire internet at defcon and then a couple of days later he was going back through the

screenshots and this is this is a control system for a caviar company and like it's called callaway i think it's in sweden so he says the happiest little statement because this little person here is really smiling so if you look at this timeline uh going back about the right after defcon uh you'll see like he put a lot of these different ones that he found and we've tried to report him get them all you know hey you shouldn't put your control system wide open on the internet but there's an example of one and he didn't actually try to go and click on any of these things they even found somebody's curves you could open and close

if they would have had a webcam showing you that'd be pretty cool look i'm hoping here's a shot of ics radar that showdown has and we've got several different protocols bacnet which is building automation control so this building probably has magnet in it well there's just over 10 000 backhand devices on public internet about 600 dmp3 ethernet over ip that's mainly used in oil and gas as almost 4 000 of those modbus almost 14 000 of those uh niagara fox which is another building automation uh there's 23 000 of those and there's higher fox with ssl 159 and then siemens f7 that's the same controller that they targeted in stuxnet almost three thousand of those

and actually he's actually pulling each one of those real with the real protocol and getting a response and this is a little uh just a graphic that you can show and it spins around and you can see where all the industrial control systems are and some of these are actually honey pots so he's he's figured out which ones those are and he's marked them so let's keep going down device robust configuration so when possible in the protocol let's disable things that are required for instance in dnp3 there's a couple of function codes that it's probably not a good idea like cold and warm restarts that's functions at 13 and 14. uh cold warm resorts just reboot starting

so that'd be fun and just reboot the rta until it doesn't work anymore i mean there's a real reason to have it back in the old days but now i don't know how many companies are still using this that's probably not a good idea start stop application kind of does the same thing it gets stopped rt's save configuration activate configuration those function codes and then the file transfer stuff probably not a good idea to allow those unless you've actually designed for it if you've designed for it in your system and you need these function codes well we need to be able to monitor them and when they're used and make sure the right people are using those function codes

and if you can't disable them there are several ways you can monitor them or block them with ids or ips or deepak inspection firewalls okay so the next thing we can do is harden our net ip networks on scada systems you can add routers firewalls dmz vlans you know same things that you use on ift systems and they apply to control systems if you don't understand your network the bad guys sure will because they're gonna take their time they've got all the time in the world to figure out your control system network and you may not have time you may not even know what you have which is sad there's a lot of companies that i've dealt with that don't know

what they have it's just they've inherited companies from murders and acquisitions and they just don't know they just didn't have good record-keeping of their their databases so another thing we can use this dmp3 since it has authentication and there's several companies that actually can bundle dmp3 with encryption in the device so you can use dp3 secure authentication and tls encryption on you know in the device and in the master so let's do that if you had that availability also enable encryption and authentication on your remote access vpns and radios if you have you know most of the industrial radios have some kind of encryption on them so might as well turn it on there's a standard called ic 62

351 which dovetails with secure authentication that's a good standard to look at if you have a control system it's got a lot of good details about encryption and authentication and another thing don't allow control systems protocols on your corporate network why would that ever happen or maybe conversely why would you allow some enterprise traffic on your scada system you know do you need http on your scale system i don't know if you don't then don't don't allow it okay we can talk about some ics aware network tools some of these that you already familiar with like wireshark it has i don't know at least 10 scale protocols in there and they're adding more all the time

it already understands i mean right out of the box i mean a lot of people that work for companies vendors that make control systems have actually put the parsers in wireshark well there's also the asc test set and another rte test set from tracker microworks that you can actually download free versions of and you can try to if you don't want to pay for them they're going to last like 30 days but you can download these tools and there's some actual free tools for like the ep3 and modbus and some other open protocols you can download free protocol analyzers then ever heard of snort and bro all right well they understand ev3 my buff hello

so there's also some other ones that are new cyberx is a new company and solid defense from um security matters they both do d-pack and inspection on team p3 modbus and a few other protocols those are new mcafee adm baser networks and checkpoint also understand dp3 modbus so if you've got those systems let's start let's start looking at our scare traffic routers such as the cisco connected grid router it understands dmp3 function codes um you don't want to block all your function because some of them are always going to be needed at least you can monitor it it's something you can do and then to feel firewalls like industrially hardened firewalls that go out in the

field from secure crossing into athena they actually have modbus and secure crossing has dmp3 and ic 6870-104 and maybe a few others that they've added but the customers are starting to ask for new features and new protocols for control systems monitoring because they're 15 year old rtu's you know accept all this traffic and if it's bad it's not doesn't validate inputs like we've shown when we did our fuzzing and guess what you can put a security engine in a control system and it's put it in a monitor mode and boy who knows what you might find if you have a sock now let's start doing that at your uh that your you know if you have an

enterprise level sock well let's add in the security pieces from your control system then you have enterprise wide network security monitoring because hey your control system's a network too then you gotta protect that so put security on you in anybody remember who this is yes click okay so i made this little graphic i don't always see the hacker log in but i'll sleep at my desk until i catch him that is when i read that book it really a light came on me he was so determined to get this guy and some of you in here shaking your heads you probably have the same drive to look for this stuff first time i ever uh

you know well it was just right after i started mandy i got a new laptop this laptop here and i downloaded some tools i needed for my control system stuff and then i got an email from doug burks like two minutes after i downloaded it he's like here's an md5 of what you just downloaded or dude was that malware i'm like what the heck two minutes after i downloaded that's pretty cool that really impressed me that you could do that if you watch your system well like in the cuckoo's egg queen's data security are in 1986 no one and or maybe i shouldn't use absolutes very very few people are monitoring their control systems i think you have only known about

um since me talking about it you've got uh joel engel scared hacker he's got a link to it on his site and there may be a few others that are wrong that you don't that don't talk about it that might have installed it or any other nsm but um securing's free so just grab a box put it in there and away we go well uh if you've got the old version of the book he's got this on the inside cover shows how the akron hanover got into the launch berkeley lab system and then pivoted to these other sites well is this happening in your ics system right now yeah are they coming in to your company onto your scada net i

said arpanet or milnet are they going to your dmz they're going to your corporate network are they coming in to your rtu or the pumps or whatever you might have your power plants or you know your refineries or are they going to a third-party customer connection you might have do you know are you sure well i wrote a blog post i write for a liquid matrix anybody her liquid matrix okay uh the go to liquidmatrix.org and i've got a blog post it's their cuckoo laying eggs in your control system and the two one big breed is you need at least one person who really cares like coastal did and like some of you do houston have a security onion or any

other network security monitoring tools and i suggest using a honeypot like compound or even build your own because that's what cliff did he he actually wanted to catch this guy and had to put files in there that took two hours or three hours to download over 9 600 blog connection so that the germans could you know hand trace these calls because it is all electromechanical equipment so and you can even do full packet capture over serial channels you don't have to use printers anymore you can use terminal servers there's a couple of terminal servers out there that allow you to do port mirroring and send it to a tcp socket so chris why haven't we seen any ics

incidents because you can't see where you aren't looking so there's i mean that's really the truth the reason why we haven't heard of many iss incidents is because nobody really has monitoring on their control systems so put nsm in your ics data now and you can do it you know free almost free and it won't take a whole lot of your your time just to set one up because of doug's and all his colleagues doing the next next finish thing so let's talk about proper physical security you know if you have a critical scada master like your transmission grid or your i don't know your your water or scared for in the city of los angeles or something

well if it's talking seriously to a little bitty distribution or to you about some uh in all know nowhere that has just a little name a cabinet box on no no lock on it or anything does that really matter is that okay because i could just open the box and uh denial service your master station and take down your critical system do you use locks that are hardened not just a little brass padlocks that's been around for 50 years that everybody has a key to and all the contractors have q2 and maybe you know people that are not even employees that have a key to hard external barriers better defenses you have the more time

it buys you for your response team to get there so you know real world if we could afford this at every substation we'd have a castle we'd have dove and have machine guns and green white men put a digital bottom and we had close stall up there we had some alligators in the mode you know so you can use things like 3 8 inch mesh because you can't cut through it with a pair of bolt cutters you have to use little wire marks here to cut 350 of them to get through the fence or use you know like when that subsection was shot metcalf substation well let's put some uh uh firearm uh it listens for the shots and

it helps locate where the shot came from use uh hardened keys you know more not bump proof or pick proof or these grade six locks that are you can't really break into i've tried and uh it it took me longer than 15 minutes just to try to do some one thing i was like okay it's easier just to do something else okay so if these hit our store last night this is how they circumvented the door alarm

uh yeah just because you have a good lot doesn't mean you can't get through there this is my friend santa pong on twitter uh he took a picture of the pipeline and says please don't climb on the pipeline because you know what happens next he wouldn't let me have the picture or he was actually on top of the pipeline but people don't follow directions and the last few things i'm going to talk about is employee awareness training your your employees on security being aware uh if you have nerf sip compliance well the compliance kind of helps drill that into your brain you can send the security conferences uh ics cert has lots of free training uh

opportunities and they even have stuff on their website you can go take little exams for free to test your knowledge on you know opsec or or security basics and get the new gi s uh gicsp certification you know some people really like to have certification so there's a few of the big oil and gas companies that are now requiring it to have you be a consultant on experiences and you have to have that served security awareness is all about questioning attitude now just because you say it can happen can it really can you do something else if you find something suspicious is there a hotline you can call it work uh it's not you call your supervisor what's the

process um do you call isis cert and there's a number so to finish up i have let a few questions at the end dmp3 and other state of protocols are going to be here a long time ask your vendors to put secure authentication in and put encryption in their devices do negative testing do white box testing required in the procurement language you know put it in writing and say you will be more secure by doing this this this this send off these devices to a third party and let them be hammered on like you know somebody like how active they've got a world-class team that breaks stuff or just others like digital blind others it doesn't matter send them off

have them tested if you can't do it in-house you know do positive testing you know fat set and then also do negative testing and you can download some of these photos for free get ours for free the hs and add to it it's not new it's been around but it's new kind of in control systems but to be honest since we haven't really had any events and people like electricity i'm still kind of more worried about uh about these guys because that's really um to bring it back down a little bit i'm not my level of worry is not really high right now but we can go ahead and do these things now these defensive things

now before something really bad happens now protecting in squirrels you know there's many ways to take out squirrels if you'd like so to conclude nap3 is not a special case all these other protocols are going to have the same problems and we're going to be adding these to the buzzer but and trying to get people to do these testings and patch you can defend your data you do have all these tools available to you put security onion in it's uh we've got a heck of a team that's supporting that do remember that compliance does not equal security but the culture is kind of important and don't count on the government to protect your criminal systems it's your

job to do that not anybody else's does uh anybody have any other bad ideas on how to protect the control systems any ideas or questions all right uh front i have a question about patching okay one of the problems in the past has been that vendors take a long time together certain pieces of the system that where they have to test the patch and try the patch and it takes several months or a year before they provide those patches how do we convince them or move them along a little faster because that could be a serious problem it depends so if like in my case i was uh before i joined mania i was a power company we owned

millions of dollars of their equipment so if your vendors uh if i say jump they usually say hi but if you're not in that position where you can say that you can also leverage ics search and others in the community even just call me and i will just start banging on them and say hey we need to get this fixed we actually with one of the dmp3 vulnerabilities is it almost took them a year to get a patch out the call the fastest one was like 17 days so some of the companies are starting to get the idea but some of them still have their heads in the sand as far as understanding what uh patch tuesday is

really supposed to be like some other questions in the back okay uh well you in the blue shirt um since the vendors aren't hatching as quickly as they would want to be you showed the way that they you know got into the system but to reduce the number of points into the scada network to make it a closed network and reducing the endpoints is that out of focus of the energy companies

yeah it all depends on the company um yes sometimes they some of the companies are reducing the amount of the perimeter and sometimes say oh we've got 900 users so some of them are pairing that down to who actually really needs access to those so it's really dependent upon the companies they may actually need that many endpoints for their business so it's it's going to be a case-by-case basis so did that answer your question yeah it's really gonna happen i mean every control system is gonna be completely different and you know yes it's good to have a smaller footprint but if you've got you know a hundred thousand sensors that you've gotta deal with uh for

whatever process uh it might be important to look at it in the big picture of youtube okay uh blues running back how can you say there's been very few essences so underserved

yeah there's not very many that do it all and i know this uh well um i can't say my how many incidences have happened i'll talk to you later if you want to talk about it but there's been you know not very many actual real incidences from like css well again just because it's open um may not mean it's a target right now but again that's again where we don't have security engine or other nsm deployed everywhere or and we may not just know that i know for a fact that there's um config are still floating around several oil and gas companies right now and they're having to deal with it because they sort of xp and windows 2000

systems yeah some some of them are and some of them are completely flat so great shirt yes how do you

right okay so in the power company i can speak to that it's uh you know during the summer they try not to have many outages at all uh as a matter of fact uh they've reduced a lot of the electric utilities to reduce the number of database updates to like you know less than 10 a year so it's all completely coordinated and they try to eliminate the amount of database roles that they do another thing that the electric sector has done is there's this exercise called gridx where they have a cyber you know like a almost like a hurricane drill but for cyber events and they get all these different um connected utilities and pretend

and have real role play scenarios it takes like two or three or four days and actually role play oh hey southern company just got hacked oh and they've pivoted over entergy and you know they've taken out this system what do you do so they role play that so the other companies like oil and gas and water i'm not sure if they have those yet but the electric sector definitely does do uh planning for that and each company in the electric sector is usually thinking about it in in terms of disaster recovery and incident response so uh reducing the amount of databases and then um you know working through your configuration management and things like that and then

your disaster recovery so i think one of the biggest issues that we're seeing is the what i call the ostrich effect both from our vendor and especially from our people and managers who really they don't know but it's it's i you know i guess oh that won't happen here we won't have that problem here we don't have that problem here what what do you think or are what would you point us some good strategies to point out to them or show to them that you well you actually do have these these vulnerabilities in your in your systems whether you know it or not yeah that's so it's a combination of um strategy does your c levels care about

security um if they don't then it might be time to start getting them and there's different ways to do that um you know if they don't have compliance driven security mandates like with narcissism and power grid um and then there's also grassroots you know if you see something say something about it hey we you know i don't know if you have a whistleblower but whenever i found these vulnerabilities we didn't i was the first person in my previous company to find a vulnerability in our control system and they're like we don't know you we just trust you so i did the responsible disclosure route and said okay i'm going to get ics certain involved because that's the

right thing to do and just maybe education is the first start um because it's more in the mainstream now and show them some of these news articles even though they're kind of hand wavy and but at least it brings our attention it's like we're having things with target and home depot and all these things it could happen to control systems yeah exactly and there's um threats targeting it but we don't know like i said we don't have any eyes and ears everywhere so we don't know what's active right now and a lot of times too they point to the physical security which for granted is pretty good

yeah there's a it's a if you think you trust somebody and then you know okay oh wait well we didn't think about that path oops yeah so maybe uh maybe maybe you can think of some ways in your own organization to raise a level of awareness for security and if there's other like-minded people like you get together and start because you've got no one else is gonna do it for you are you seeing in yes to what you see is a lot of vendors that like this digital and wireless instrument are the vendors are they preparing and securing these big wireless histories yeah um yes and and no like smart meters they're all wireless usually uh the first

few that came out were not very secure and they've had several revisions and they've all been tested supposedly by third parties now and they're more secure and some of my colleagues in other security companies have been doing pin testing on those devices so and as far as the electric sector yes uh and even the big companies like ge that makes radios and maybe mx radios and all these others are starting to get the hint if it's a smaller company they may not know it again it depends on the the wireless but yes they're they're starting to do more security on those for sure i just wanted to comment further on what she said in response to

compliance a lot of c-level types really want to keep their compliance and one of the things that we don't do in compliance is auditing and so the cna process gets lost because we're we're quantitative compliance whether it's pci therapy or whatever the case may be so most of the times if you can get auditing done on a regular basis uh that will enhance your proposition for maintaining your compliance and things like that but it's also uh up to the security engineer or the project managers to also define the effective plans for auditing uh networks uh anything so in our case we usually start with this 800 as a as a mechanism for cna and that gives you uh

leverage to get auditing done so a lot a lot of companies just don't do auditing they just try to to meet their compliance right and again it depends on like with electric grid they do a lot of auditing and narcissism and the perkins are in in the south of cirque they do have circles and then internal audits but if you have a company that doesn't do odds is again start somewhere and you gotta you know take it into your own hands if it might not be a bad thing to do to start because like with me in my previous show we had the compliance but we didn't have a security button so we would have to

sneak security in with the compliance stuff to leverage those funds things like that if you don't even have a compliance budget well then maybe it's time to start talking about you know auditing yourself or having some third party come in and audit you just because you care i mean it's just because we have all these systems and they're not vulnerable but they are vulnerable and we just need to uh somebody kill the flags of security and they saw a moving on what caused last week yeah that's right any final questions all right well thank you so much for coming