Chris Sistrunk - The Power of the OT Security Playbook

all right thanks so much uh test test I think you can hear me okay awesome [Applause] so uh we've had great weather uh we used to be expecting a hurricane so we got to think about that but how many people here have OT or work with OT or have it in somewhere and where you work okay a pretty good many of you okay hopefully this talk will resonate with you and if you're an I.T um background you can kind of hopefully get this message across to the folks that do OT where you work or your colleagues and that's kind of the uh and since it's football season right A lot of people love football I kind of want to tie

those two subjects together so my name is Chris sistrunk I'm a technical manager at mandiant for the last almost nine years I'm on the ICS OT Consulting team and done assessments and critical infrastructure around the world but before that I was a power engineer for Entergy over in uh and Arkansas Louisiana Mississippi and Texas that's where that company's based out of they've probably sent all their linemen over here and towards Florida and the East Coast there but I was in scada and substation automation also was the founder and I ran besides Jackson in Mississippi for about six years and it's still continuing on hope not unfortunately not this year hopefully next year they'll start besides Jackson

back up and beer eye sack napcon dad joke says a service so and then now Mandy and we're kind of becoming Google so I'll probably get one of these hats here in the next month um so I want to start off with some wisdom I like sharing um good nuggets that kind of help tie the idea together so Mike Tyson anybody a fan of Sports boxing he has a really great quote about having a plan everyone has a plan until they get punched in the mouth and then you know throw your plans out well the the the guy that interviewed him Mike Bernardino or berardino um was got his quote and he asked them for

some context around that and his response to that was it's how you react to that adversity that defines you not the adversity itself so that's something really to think about as we're doing for those of us who are in instant response cyber security infosec any of these things even Engineers we're trying to defend what we have and make them better and that's kind of that adversity is what we're going to be responding to the last quote is from my senior safety guy when I started the power company he had a quote that he told everyone even the linemen safety rules are Written in Blood and so think about that something bad happens we've got to write that as a

safety rule so it's the same thing with cyber security if you do something bad that's going to be a check or a mitigation that we're going to implement so response plans and playbooks aren't aren't new right sports teams have been using playbooks to win games with great success you know government entities have been using structured instant response like FEMA started in the 1970s from wildfires you know deal responding to that there's a part under FEMA called Nims National Incident Management System that's a little screenshot there of that so anytime there's a disaster of any kind now just about the government spins up FEMA spins up Nims and there's a whole bunch of different parts under Nims now National Incident

Management System the CDC used it in responding to Coba 19. we also have playbooks for Disaster Recovery incident response playbooks are already invaluable tool to win or not prevent them or minimize an impact but in the case of a cyber security incident or Foe here industrial processes like this one from um but a BP you know they had the big rig explosion in the Gulf of Mexico I believe was it 2012 that was the uh yeah the deep water Horizon right that was really something that they had to write playbooks around and that's a an example from one of their instant response playbooks and then of course we have the the the nist standards for

instant response right so basically we want to say is they're not new how can we apply them to OT environments with cyber security in mind so there's an incident there should be a plan right not just someone going help help so we have these instant response plans and playbooks like nist SBA 161 standard right and this is a standards that they've been publishing around that vulnerability response playbooks there's also this incident command system structure from FEMA that I talked about which is in use right now uh after a hurricane in and this is kind of I want to dedicate dedicate this talk to all of the incident responders this is a screenshot from just a few

days ago in Tallahassee of just one Cadre of line trucks uh lineman and think about it there I think I read on Twitter some say there's over 30 000 line Crews incident responders going to help restore the power after the the hurricane hit but also you have you see all the the buses there you have Logistics you have porta-potties they're all lined up on on the bottom left there you have the feed and house all these people you have your your First Responders like your medical and ambulances you have tree trimmers you have cable TV and internet service providers going to restore so all of that stuff follows this incident command structure um that is really something to behold I

got to experience this when I worked at Entergy doing hurricane instant response we had a tabletop Playbook that was about this thick you know four inch three ring binder that would take two to three hundred people to practice during a whole week we would have fake storms coming from The Weather Channel or whatever they would email us the updates and then we would practice it all the way down to which power lines were going out what was our triage how do we get the quickest power on the fastest and get the most people's lives back on so I really want to dedicate this talk and this idea uh to what's going on now here's an uh a screenshot from last

night at the beginning Florida when they were hit they had over two and a half million out last night they were down a million they had got a million lights back on but Puerto Rico that whole island was out they've got half of about half the island back on and now we see North Carolina and South Carolina and Virginia affected as well so this is from the website poweroutage.us and it collects all the outage data from all the power companies and you can actually go watch it on Twitter as well if you keep up with Twitter uh earlier last two years ago I'm sorry two years ago we had this talk at a cyber security conference called S4 it's

in Miami every um spring or a late winter it's in February generally and that this idea of taking this incident command system ICS and then applying it for instant response for control system environments which is a great idea and so we we created this group I don't say me it was led by Megan Samford and she grabbed all of her incident command background got all of her folks involved got the ISA which is international Society of automation involved we got Global people involved there's over a thousand people involved in this now and you can actually become credentialed and be an Incident Commander you can go take the trainings online and become as part of the FEMA process you can be

credentialed in this now which is really great uh organization change so well the idea of that was is using an existing system and then adding cyber uh instant response to that so back to uh Playbook idea right this is some playbooks from usafootball.com and it talks about playbooks that if you want to have for your team like if you're a high school coach or something like that basically you have to know your role and what you are playing as a team if you're on offense you've got to know your role in what plays that you can do to take advantage of the opponent in defense you have the same thing you've got to know which plays that

you're going to play when you see the offense have a vulnerability and we can take advantage of that right so we think about it in that way as the sports teams do we do that in cyber security instant response so part of that knowing your role for Control Systems let's say this is The Incident Commander this is from Nims you have your like I said before you have your different planning Logistics Finance who's going to pay for all this incident response who's going to pay for all these hotels same with OT who do we call and when when an incident happens do you call mandiant well great if you don't that's fine call somebody um but you're also going to have to deal

with folks instead of just normal I.T Security in your sock team there since it's an incident perhaps involving control system and physical environments you're going to have to call the subject matter experts that are like Engineers operators the safety department because if there's like for instance if you had malware infecting a aluminum smelting line or aluminum pot line where they're producing ingots of molten aluminum you have to act fast you don't want that aluminum to to freeze otherwise you're going to have to throw out your entire operation so you have to act quickly think about safety and also one thing I do want to highlight here in addition to your other things like legal hrpr you also want to

talk to your vendors because a lot of times in control systems the control system equipment is very specialized it's not windows it's not Linux it's going to be some real-time operating system that's maybe custom and so you have to use their expertise and call them and have them help you like from Siemens GE ABB whoever it may be even health care we can even consider Health Care kind of like uh related to OT in some cases so we've got hospitals all around us right so you're gonna have to call the GE folks when you have something dealing with maybe malware that's impacting these Health Care Systems so I want to review a talk that a I did

several years ago at Defcon and a few other places and that talk was called what's the difference between I uh for for what's the difference for OT and that's a dad joke uh so if you like Dad jokes follow me on Twitter uh so we have the seven steps of instant response so the first step uh at least from Mandy's point of view is assess the situation triage where how is this affected it's similar to when you have an I.T incident you're going to have to do the same triage it's going to be the same as a hurricane response what's damaged what's effective what's not who do we have available to help us respond to that that's going to be similar

the next defined objectives is going to be what's a little bit different we've got to return the control system or the manufacturing plan or the hospital back to normal as quickly and safely as possible sometimes safety is the key driver there not confidentiality you're going to have to get things back safe where people are okay and the equipment is okay so just remember there may be physical processes involved the next step is collect evidence so you got to collect your evidence to do the proper scoping and get the indicators of compromise as you're going through that remember that control system devices and embedded devices have real-time operating systems and you know protocols that are not you that are not used in

I.T so you have things like modbus and dmp3 and other protocols profinet that the it tools that you're used to when you do instant response those tools don't understand those protocols and so you may have to collect these things manually unless you have one of the recent companies there's several companies out there that do network security monitoring for ICS so you can leverage that if you have that now the next step is performing analysis on that collected data and to verify anomalies or finding evil like we say at mandian and there's no ICS specific dfir tools out there you're going to have to use the tools that maybe the vendor has so one like when when we did the instant

response for the Triton malware in 2017 it targeted the safety instrumented systems that were from Schneider Electric that's one of the vendors we had to partner with them to use their lab and their tools and their memory analysis with their software developers and their Hardware developers to to understand what was going on that was really important to have as a partner communicating what's happening that that constant uh call that conference call every day or the Roll-Ups that's going to happen every day twice a day it's going to be similar to where you do regular instant response and then lastly developing the remediation plan when to kick the bad guys out and regain control remember control system devices may have

constraints like hard-coded backdoor passwords you can't change and you can't just re-image like a PLC like you can a laptop the lastly the uh after action report is going to be the same thing you just got to document that the same way so this IR plan objective they're made be no right way to do it it may be dependent upon your control system how it's designed how it's engineered and the safety may be number one the incident response plan and play books are going to be different probably for each type of network you have so since we're talking about coaches and playbooks this is a screenshot of coach Sean Payne who used to be the coach of

the Saints my favorite team and you can see it he's got quite a bit there on his Playbook there you can see you know third and long third and short on offense when to pass when to you know what the defense looks like they're going to each of these plays are designed for particular part of the game and using the strengths of his players and taking advantage of the opponent's weaknesses so that's the whole idea about having these playbooks for OT so we break down the incident response plan into use cases or playbooks or run books and hopefully you're all familiar with those and now we want to do the same thing for OT that whole step-by-step instructions

that maybe the IR teams used to but also the engineers they may not be familiar with some of these things so or they may be they may know more than the I.T security folks so they have to work together to write these playbooks down and and have that custom tailored instant response playbook for what's happening some example use cases that I highly recommend that the control system half is these four commodity malware what do you do if commodity malware's in your OT environment often it's just finding where it's coming from and cleaning it with some kind of malware cleaner because a lot of times these Control Systems may not even have antivirus so you're gonna have to deal

with that the second one is this credential compromise what if your OT credentials are compromised like the operator credentials that's what happened in the first Ukraine attack in 2015 the attacker stole all their operator credentials and you have to be able to to know what to do if that happens and sometimes you can't change the password sometimes there's hard-coded backdoor passwords so you have to have other mitigations in place another one is if there's some kind of destructive attack like kill disc you know not payche uh want to cry any type of ransomware we've been seeing a lot more of that impacting control systems in the last several years and so you have to have a a plan to you know

emergency segment that Network or have restore from backups is a critical response the last one is the ICS protocol attack if someone's using ICS protocols to do bad things to your control system what do you do in that case so we have like stuxnet drink and Destroyer one Triton and Destroyer 2 which was discovered this year in advance actually Ukraine cert caught industry or two before it was dropped and then this also one from this year's ink controller that we wrote a blog post about and I believe several others wrote blog posts about I think dragos called it something else they called it pipe dream there's a lot of good things that we caught it wasn't ever found in a victims

network but another use case of a control system protocol is being used for evil so having these redeem remediations for each play restoring backups resetting passwords sever or isolating the OT network with the emergency red button script to to segment the network utilizing critical spares that's really important review the tape so like we said tabletops like with the power companies we practice that emergency plan for Hurricane response every year so God will take those exercises and do them in your OT environments because often they may not have one very often and may not have an incident they may have an honest mistake more often than a malware incident so take those Lessons Learned take the

Past aars after action reports doing threat modeling reviewing what real world events have been happening to drive your preparation for the next thing right here's an example of a free Playbook that's out there from Power industry so APPA I'll post these slides so you can get the link later if you can't take a picture of it now but Public Power has this really great Playbook about how to do instant response for like the mom and pop electric company that's out there that's not like part of the big one like here would be Georgia power and others they already have playbooks but the one the smaller power companies they don't and so this is why they wrote this one

good free resource so all right stop gotta collaborate with your ICS vendors like I mentioned before they have those steps documented that you could probably use in your response playbooks they may have it in their manuals they may not you may have to work on them together with your vendors but we really highly recommend you work with your customized equipment vendors to design your response to if something bad happens and if there's an incident more likely they're going to have to be involved because your engineers and operators won't have time to learn what that system does under the hood we don't want to over complicate things I'm getting close to the end of my talk

here so more wisdom from football the best defense against any offense is your defensive playbook no one else's do what you do best this is from joedanielfootball.com it's really great quote do what you do best have that good defensive playbook you don't need every coverage every coverage has a weakness so choose complementary coverage that takes away that weakness so have a set of five playbooks that cover most of your things I had four there listen if you have two of those that's even great a complicated Playbook won't win football games for you if the players can't execute you should strive to keep your playbook as simple as you can now run the Playbook you've got those

designed for each phase you got your player strengths and exploiting your attacker's weaknesses remember that what you practiced and react to that adversity so anybody recognize these players here there's quite a difference in size you know David and Goliath right yeah that's it's really something is sometimes the smaller Defenders can win against what would be a scary vote right knowledge in preparation are powerful so I'm not going to read everything here just the three main points create an OT incident response plan if you don't already Define use cases for each of your different scenarios like I said like those four have commodity malware playbook for OT what if your passwords are stolen or compromised and destructive malware in

your OT environment those three is really important to have one about the ICS security protocol I mean ICS protocol one is a good one to have and then practice we want you to redefine the wind so since I'm kind of close to Atlanta I've got a dog on the Falcons a little bit with this play here uh minimize the impact by preparing early detection and response so this is the lineman from the Saints he was able to catch that football and return it to almost a touchdown in that that particular instance and he just wipes out uh Matt their their quarterback right that was due to his preparation halting the attacker somewhere in the cycle stops them from achieving their

objective and transparency in information sharing within your own company will help targeted orgs and even other Prudential victims within your own company and with others that you are partners with with that do I have any questions

and oh yes

yeah it's it's important with hurricane response it's a regular practice in the South right and that's why our Playbook was so thick because we had so many injects and variables that we could throw at what we've seen same with instant response of any type on your cyber security insulin response plans it's good to have a practice across teams every year you had a question yes

so the the best tool is having people to even start using what you have but if you give them a tool that has that knowledge I mean even Wireshark understands 30 different control systems and protocols uh and bro now Zeke has a whole bunch of different parsers for control system stuff so even if you don't have those tools that cost a lot of money right and there's a lot of Market competition out there um you can start somewhere with free tools but if you do get one they're all have strengths I would suggest that you look at differ the different tools and do a Bake Off and for your environment this particular tool like a nozomi Clarity dragos

whatever cyberx which is now Microsoft Defender for iot they all may have different strengths for what your network is like so I would I would test them and Hammer them um before I answer any more questions I'll do some trivia so anybody uh this is a practical iot hacking so can someone name a one of the uh recent control system malwares in Destroyer okay who was that yeah all right I'll just leave this here all right uh second one does anybody know which protocol industry are you used there and I'll say industry or two the one that was revised this year does anybody know what a protocol control system protocol modbus no no not close uh so this is I'll give you

a hand it's European Standard protocols they're not what's used in U.S even though modbus is used everywhere any other guesses that's the winner all right you get that uh this little uh Alpha network wireless adapter there so any other questions before time okay one question here

okay so how I transitioned is uh stuxnet um I I was doing some cyber security uh back in 2008 because of nerkship regulations but then when the news about stuxnet came in 2010 it was simultaneously scary and sexy at the same time and so I was like oh this is terrible wow this is really cool you know kind of deal and so that really sucked my brain and then I got sucked backwards into doing some research on my own I was on the dmp3 technical committee someone one of my friends wrote a fuzzer for it and we tested it out and I told him immediately to take all of his fuzzer tools off of GitHub immediately because we broke like

30 different things and then I got an offer to come work at mandian so I actually wrote I gave a talk back at besides 2015 of how to get into ICS security so if you're interested in how to do that even though some of the links are probably outdated from 2015 that's still a really great talk if you want to go check that out what was that uh it's on YouTube but it's also my slides are on my SlideShare for that yeah now besides 20 besides Augusta 2015. it's on their Channel so thank you for attending I appreciate it hope you have a great lunch right so [Applause]