Robert M Lee and Jon Lavender - ICS SCADA Threat Hunting

I'm very privileged to introduce our next speakers uh these are some amazingly smart guys and they're doing some really great research and doing some really great work into I and skate environments uh sobody please join me in welcoming Mr rer we John ler all right thank you thank you so today we want to talk to you about industrial control system or IC threat hunting uh industrial Control Systems just as we get started so I want to do a whole a lot of in material uh industrial control systems are the type of systems that end up operating a lot of the infrastructure that we deal with on a day-to-day basis uh water utilities power grids oil and gas refineries uh

petrochemical facilities just think of hardened devices these these physical rugged systems some of them are Windows systems we have Windows systems running you know the human machine interfaces or hmis these environments so the thing that really sets them apart though is that ability to impact and control physical environments right that and actuating in these environments that's really what sets them apart so uh little intro uh John and I are at a company called dros I also authored the threat intelligence class at s and the IC instant response and active defense class uh but most importantly I write the the comic uh for anybody that came at the bsides presentation or excuse me security ending presentation yesterday

you'll have noted that you would have seen little Bobby I always like to put him in the slides uh throughout the material I found at my time in the government that it was really easy to use Comics to explain complex topics to congressmen and other senior officials uh so I continued that on a weekly basis now uh John go ahead hey good afternoon everybody uh so I'm John lavender uh as Robin mentioned I'm at dros as well uh I am also an avid Starbucks mug collector if anybody else in the audience is is one of those as well I have kind of a um U yeah something I do way way too much um but I've also got a pretty thorough

background in the US Community uh just recently left back in May um with my backgrounds in everything from Red teaming blue teaming uh but I actually met Rob and we are working in the IC space um doing threat intelligence Perfect all right so here's what we're going to talk about today first up we're just going to cover the three things what is threat hunting uh how do you start hunting and specifically why and how do you do it in ic environments so for our first little Bobby comic uh little Bobby asks why do people try to redefine active defense to involve in and offensive operations and Matt says well many would like to call those types

of operations or to do those type of operations but if they call it Act of Defense it just seems more reasonable to folks and uh little Bobby says well fine well doing homework is now defined is playing Minecraft and I started with this one specifically because with this topic of threat hunting and threat Intel and active defense and any other word that we start throwing out there is a real concern that we start talking and getting into buzzword land right so you probably heard that as a theme through a number of different presentations today but there is real meaning to a lot of the words that we use at times I find that they get spun out of control either

by vendors by folks trying to appear to be Security Experts uh or just those folks getting involved in the community I generally find that nobody's being malicious buzzwords generally form because people want to get involved and do something whether they're on the vendor side or not uh but we still have to be careful for them so we return today and talk any of those terms what they actually represent and not that buzzed out concept that we might have heard over the years no actually Rob before you get started I want to do the first trivia for today so oh we have yeah we have the four gifts uh so the first one go ahead sure the first one's

a lockpick uh by Southward they were nice enough to donate uh so the first trivia question uh I'm actually going to refer to little Bobby comics for those of you who know Rob and have seen Rob do um have seen him do presentations in the past uh if you had to guess which one would you think Rob represent the most little Bobby or the older mature Matt I saw your hand first perfect there you go yeah no so it was actually modeled after myself as well so it's easy when you give conference presentations and get involved in places sort of let your ego get out of check I like to always return my ego uh and try to be reasonable

headed so I like referring to myself as the little Bobby the kid that doesn't know anything since we're all people that are learning I hate the term expert there are no experts um so cool so what is threat hunting um as we go through this here's our next little Bobby comic if you will and little Bobby is starting to get a little concerned because he hears all this stuff about cyber war and hype in the community he says will people go to war over network security and mass says no not likely uh unless there's some sort of physical grave damage maybe you know loss of life or something and little Bobby says well I saw experts saying differently and Matt

said well were these experts selling Security Solutions and little Bobby said yeah their products look really nice uh and what I wanted to note here getting into this next couple topics and especially with threat hunting is threat hunting has been around for a long time it's that human dedicated approach to actually searching for adversaries in your environment over the last two years we've seen it sort of go fully to the right in the sense of marketing and presentations and vendor products and things like that uh and I just want to note that a lot of the folks selling Solutions have typed on Hunting when it used to be thread Intel and before before that it was just a storage

platform but now it's a hunting platform and back and forth we see all this Jazz take place I just wanted to note again that there's some real value here that John will start us going off with yeah and like Rob said that as far as as far as getting value out of tools and different types of platforms there is definitely an advantage to having them and going with them but at the same time a lot of this is a people process um so if we refer to one of the first of I think six or seven models that we're going to use in our slides today um um the sliding scale of cyber security so

just kind of walk through this quickly um if you start with architecture and you start with the simple hygiene facts on a network um you're going to get a lot more value out of doing that as far as being able to do threat hunting later down the road um but it's definitely something to focus on First and up front um and then kind of As you move across the scale you get into passive defenses so being able to deploy ids's ipss uh different types of platforms or systems on your network then to be able to take advantage of alerting data collection uh then moving into more of the active defense piece so as your organization matures um and as you kind

of build processes get people in place then you're able to do things like active defense which includes threat hunting um we'll get a little bit more into the active defense piece here in a second but um for those organizations that are able to kind of put all this together and do a very good job of it then you can see potentially moving down the Spectrum a little bit further into actually generating threat intelligence um but very rarely will you ever see an organization going on the offensive um it's it's it's definitely a myth I w't say a myth but it's definitely something that's um that's kind of preached out in the market that isn't exactly something

that organizations should do or or even um have the resources to do yep and so I just want to add on there as well one of the reasons I created the model early on in the paper behind it for anybody interested is specifically to push back against the offense discussion as I've seen sort of the hype in the community take control of what we should be doing next and next gen and next nextg or whatever gen we're on all of these things we're supposed to be doing I do want to note that we do forget a lot of the basics and I usually joke around that the key to being a security expert is just renaming the basics into

something sexier uh but really at the end of the day we need to stay away from doing the stuff that doesn't return a lot of investment to our organizations uh there was a large company that was compromised uh some of you will know its name but I'll leave them out of it large company that was compromised had millions of dollars in loss and there was serious boardroom discussions going on about can we go back after the adversaries and they decided to do so so there are private companies that are starting to do the whole hack back thing and they spent millions of dollars to go back after the adversaries and most of it actually or a significant portion of

it was in legal fees just to see what they could and couldn't do inside the confines of law um but they spent a lot of time and money doing that and then you looked at their security organization and they had 11 people 11 people on the security staff for a global organization and six of the 11 were management positions so this is Rob Le's rule of thumb that if you have more than a one to1 ratio of managers to analyst your breach will be embarrassing okay so when it comes to what they could have been doing in terms of the security Investments that they had if they would applied all those resources to hiring and training the staff to start moving

along appropriately it would have been much better for them one of the reasons we bring this up today is yes threat hunting is in that active defense construct and even though we're both big Advocates of it I would never recommend to an organization that that's the first thing you got to start with no get the basics done correctly make sure your passenger systems logging uh taking care of that cyber hygiene stuff if you will uh but don't just jump to training 15 people to go searching for APS in your network when you're getting hit with uh unauthenticated requests into your environment you know I've seen systems that have had unauthenticated vpns and to a Windows 10 system running

Powershell across the environment while they're trying to deploy nextg firewalls and some other segment of the network it's like maybe you should just enable to form first and then we'll start going along so with that some other models if you will oh a couple more models actually three I think to be specific but um since the talk is about threat hunting uh we'd like to actually apply some models to doing threat hunting itself so how do you get started with uh how do you get started down the hunting path uh I had I had a conversation this afternoon with somebody who's who's doing a little bit of it in his organization and just listening to um

trying to figure out where to get started it's it's a very difficult process if you don't have a lot of background and experience in it and part of this talk will get into hypothesis Generation Um to be able to start doing that but if you look at applying some models to the actual process itself we've kind of divided it out both into an active defense portion which is more the people process U which is more the proactive means of going after and trying to do discovery trying to do the threat detection and then figuring out how to clean it up and remediate it but pairing that with the threat intelligence piece uh where there's tons

of models available um at your fingertips these are just two that are very common very widely used in the industry um so being able to kind of pair those two together don't rely too heavily on one or the other um is a good way to kind of build your process around throat absolutely um I will also note that when I I did not make the uh the diamond model it's actually a paper really good paper written by Sergio and crew uh called the um Diamond model for intrusion analysis it was a model that we used on the intelligence Community side when we were looking at large representations of data across large amounts of intrusions and trying to find

out who or where the adversary was and what they were going after worked very effectively it's still something that we'd use and teach out to today um and of course you've heard the kill chain before uh I will admit though when I made the other model the active cyber defense cycle that the only reason I threw the word cyber in there was cuz it made the acronym ACDC and I thought that was clever uh so that's the only real reason for that but uh this aspect of seeing what's on the network finding it remediating it looking into trying to uncover the threat not malware we usually Focus way too much on malware malware is not the threat the human on

the other side of the keyboard is the threat malware is just one of the capabilities they might or might not deploy so that's a good thing to keep in mind so when we talk about uh hunting it's kind of hard not to refer to David biano at some point really really smart guy uh he came up with this hunting maturity model one that I like to look back to when we look at hunting that human approach to finding threats to your environment there is a scale of maturity to it on the far left hand Scot side and when you're not really hunting that's why it's level zero but when you're not really doing any hunting

you're just replying on alerting right the IDS says something the firewall says something the ab says something and you respond to it it's not hunting it's a very reactive approach and one of the pitfalls in it is that you're probably not taking a lot of data off your environment to begin with anyways as we move across it you'll notice that theme that it's a lot of data collection that's going on and that's where a lot of organizations get in trouble that's where a lot of organizations have problems is how do I get all of the data how do I get it back into my environment and search through it without paying an arm and a leg and stay on top of the

constantly changing environment and Industrial Control Systems as we'll talk about later that is probably one of the biggest challenges in the community is how do I get data out of control system environments while respecting the safety and reliability concerns of the facility uh as we move along as well you'll not only notice that there's a higher retention of data and a higher uh need for data but you will see that you can start relying on indicators of compromise instead when you're just doing sort of the basics you move along to actually taking processes and models that other people are using and ingesting them on your team and then at some point actually creating the new

analysis procedures the new analytics to find unknown threats the the last one is one that gives a little bit of a confusion in the industry you'll note that it says automates the majority of successful data analysis procedures there are a lot of companies and marketing out right now trying to say automate all of the things that you do for hunting just automate everything that's not hunting automation is very important to hunting you should be automating all of the things that you can but if you are actually going through an automative process it in itself is not the hunting because you've taken the person out of the loop and that's the piece that is obviously uh

core to it so in effect when you're at hm4 what you really should be doing is figuring out everything that it takes you to find the threat that you did and then automate that away so you don't have to deal with it on your next hunt so that's actually sort of the process there yeah it's more it's more of an enabling force and something that allows you to create a repetitive process than than actually taking over and doing it for you and another thing that we sort of see is uh problematic is a lot of organiz ations have the shiny object syndrome right they go to the latest newest thing and you'll walk in and

you'll see an organization that's got a threat hunting team but they don't have a network Operation Center or they don't have a security Operation Center but they've already jumped to the far right hand side of active defense if you will to that threat hunting so be sure not only in this model to uh sort of stage your maturity but also against that sliding scale aspect of actually taken care of what's on your network so this next slide I I wrote it as a joke at first and then I thought it was actually pretty useful um David biano if anybody uh knows him he has this pyramid of pain you've probably seen it at some point in your

life talks about hashes and domains and the ttps and things you can sort of move up for the Pyramid of pain for adversaries so I give you the the threat hunting jawbreaker so the entire idea behind the jawbreaker here is that there are at least five things that I see wrong in the industry today that we need to sort of adapt to when it comes to hunting if we really want to embody what it means to be a threat Hunter so these four things in addition to the fifth at the top uh that we see problematic number one it has to rely on the people and this is really hard for a lot of organizations to understand they'll

usually try to take somebody that's new throw them into a job that's threat hunting and then be disappointed when they don't get a lot of value out of them it really is a specialty to do threat hunting uh you should have some sort of specialty that you bring to the table an instant responder security operations folks somebody who knows Mac forensics if you're a Big Mac environment somebody who knows how to do IR across the Enterprise someone who knows how to do Sim alerting different Specialties are ultimately coming together both red and blue team onto the same team to do these threat hunting uh approaches the second one is your job focuses on those human adversaries you

should be finding them but you might not and that's actually one of the most depressing aspects here so I had a customer that set up a beautiful Network they were doing a security Operation Center for a gas pipeline and it was a really really good environment they had uh wh listing across the environment they had every bit of monitoring they could have full packet capture from everywhere inside of the IC it was a beautiful setup and this guy had trained and trained and trained him and his he and his staff and it was by all accounts a very impressive organization overall and then they saw nothing and continued to see nothing they actually locked down

their facility very well inside of this confines of the IC they never got to see the big bad AP that they kept hearing about in the news and everywhere else that was almost disheartening to that staff and it's also very hard to do retention of people when someone trains up to be this threat Hunter and they sort of embody this and things it's going to be awesome and then find out that their job relies on the adversary being there and again that can be sort of problematic we've seen that even in the news where uh firey fire ey has done a lot of good stuff in the community and then you see discussions about oh you

well China is not coming after portions of the government anymore which is debatable but uh a significant portion of the Chinese activity fall off well that's going to have a significant impact to all the people at firey who were focusing on Chinese threats so our adversary Defender relationship is much more of a bir directional relationship than most people realize the next one there is I wanted to note that you need to be open-minded but you can't be a new person so going back to that kind of original discussion I don't think threat hunting is the place to drop your new analysts I really think they should get familiar with your organization they should build up their security skills

they should develop a specialty find what's interesting to them and then be able to come over to that style of team but they have to be open-minded so we had a an analyst that worked with us back in the uh intelligence Community he was a lance corporal in the Marine Corps definitely did not belong there by any stretch of the imagination did not belong on the team that sounds harsh but he was a new person and we were doing more advanced stuff so he went off to got some training try to get some Specialties came back still disheartened still a new guy and I'll and I'll sort of note to you that the questions he

asked even though he wasn't able to investigate everything that he was supposed to even though he wasn't able to uncover everything that we needed him to do with the tools we had the questions he asked as that new person was more impactful than anybody else in the team that open-mindedness that he had of not really understanding why the world worked in specific ways he started asking those questions that revealed adversary data we never thought to look for so I don't want to be discouraging to the new folks we need more of them we need a lot of them and I think they have a huge value an organization but for the benefit of your new folks I would like

to see less plopping down at a threat hunting team to start off with you could always you could always pair them up like if you're trying to get them more more expertise or you want to get them to work with your senior folks that have a lot of experience it's helpful to let them do shadowing for a week or sit down on certain types of um certain types of processes in in the hunting um process yeah absolutely I'll also Note S your next one there is that product vendors will pitch hunting but it's not about the product it is that human Focus so we see a lot of products in the market right now about hunting platforms as an

example there's really no such thing and a lot of us can sit down and be like well there there's hunting platforms because this that no it's you're really just talking about different tools to enable data access analytics and automation that's great and that's useful but really the hunting itself is focused on that human person so we can relabel things in the industry if it gets people to use them that's fantastic but the at its core just remember it's always Central that person and then of course as we hit on already you need to rely on automation but you can't fully automate hunting all right so how do we start so the ciso here says to little

Bobby I need to ensure that no attacks occur little Bobby says cool what's the security budget the CSO says no no no we had to cut it all out so little Bobby says well I assure you we will never see any attacks uh so when it comes to starting to hunt a lot of it is just initiating what you're doing but it's nice to actually have a guided approach so these next couple slides we're going to go through how to do hypothesis generation a hypothesis is nothing more than a testable idea you come up with and it really is Central to hunting it's how you should start a good hunt coming up with an idea of what the adversary

might be doing in your environment or where they might be located because that's going to drive how you do collection in the environment am I pulling registry keys or am I pulling memory samples well if I think that they're doing a root kit in the registry to remain persistence well then it's going to be really important for me to grab that registry key so we'll see how that sort of takes hold actually before we jump let's do another uh let's do another trivia I'm going to be the trivia guy today I guess um so for those of you I guess has anybody in anybody in the audience ever been involved in threat hunting or done hunting in their

environments show hands perfect cool we got some all right so let's go with um we we have listed three specific kind of genres of of hypothesis Generation Um before we jump to that slide I'm me see if anybody has an idea of what one of them might be all yeah all right so intelligence driven hunting is absolutely one of them uh one of the ways that you can generate intelligence I'll go and flip as John's walking one of the ways you can generate hypothesis will usually usually come off of either threat intelligence or friendly intelligence we often leave out that friendly construct if your forces your blue team are doing something in your environment what they're doing and

how they're doing it might encourage you to hunt differently so uh we're going to walk through some examples of these coming up we'll just walk through the the generation aspect itself um we also have domain expertise at the end of the day there's always that something that Hunters have or experienced security folks have and it's kind of hard to put your uh finger on it it I would usually rely that or sort of say that that would be domain expertise it's something either from your past job or from your current job and over time you've learned tradecraft you've learned adversary capabilities even without intelligence even without any prompt to do so you've learned something that you're going to

bring with you into generating hypotheses so that is what I would refer to as domain expertise as an as an example of that um there was a guy that Rob and I worked with when we were back in the government who he would come sit down beside of us as we were going through either either looking at Network traffic or um just doing some sort of investigation and he'd be able to sit there and just kind of like tilt his head a little bit he's like that smells like it's X andx actor or somebody who's done this in the past and nine times out of 10 the guy was actually right which was just odd uh he had been doing this

for so long for him it was like staring at the Matrix with a little um with the numbers and letters falling down he would just kind of point and then we would start investigating down that road but just having those people like if you guys know those people or if you guys are those people um it's very helpful to get up and I can share that with the rest of your team share that with the community um but that's definitely one of the I'd say that's probably one of the biggest drivers for being able to generate a hypothesis and actually make it come to real or be able to test it and find out that it's actually true or

um is actually happening in your environment and then the next one I would say is situation awareness and this is really un understanding your own organizations but I would drive you not to just think about tech and this is going to come in in a big way with the IC it's not just about the networks it's about the overall systems and organization and people around it do we have things going on at a macro level that are going to influence what we're seeing on the network uh I would pontificate to you that Saudia ramco the oil and gas gas company in Saudi Arabia has a different threat landscape than JP Morgan does different threat actors

different capabilities different things coming after them situational awareness around those things not from a threat intelligence perspective but just from understanding your own organization is going to drive how you generate those hypotheses um so let's step through a couple of these examples uh the first one here is this threat intelligence inspired hypothesis so maybe we've got this sparkling Panda group no sparkling panda is not a real one uh I just like all little names that always come up like Jedi squirrel and stuff but uh sparkling Panda utilizing SE shells based C2 servers so most of you probably I don't know if you've ever heard of the country of SE shells but it's probably not common in your environment so if you

reached in and you found in this intelligence report that you were consuming that there are SE shells based C2 servers for this adversary it would be reasonable to generate a hypothesis that if SE shells or if the sparkling Panda actors in your environment you would expect to see IPS or domains and proxy logs of your environment related to se shells so that's how you're going to hunt for that a little bit differently but you've got to have that testing to make it a hypothesis right so it's not just the good idea theory on the team it's actually going and testing it at that testing stage you might actually pull the proxy logs understand the data flows and try to evaluate what

you see in the environment and again this is really where we bridge the gap that it's not just IR it's not just security operations it's a senior individual who really understands how to explore and test a theory in your own environment related to what the Avary activity might be and that also has the also has the um the technology and the data to help support those as well um another example with the situational awareness uh inspired hypothesis so let's say you're let's say you're working for a large company or small company for that matter and you know there's a merger about to happen and your company is acquiring uh a globally distributed company that you know

networks are going to have to be integrated over the next 6 months um if you kind of took that in to your thought process made made assumption or a hypothesis that says I'm pretty sure that whatever networks we're taking on there's either already an infection in it or it's going to it's going to introduce some sort of vulnerability to our environment um and being able to test that by identifying how that network is going to be attached to your current Network whether or not you're going to rearchitecturing

prove that hypothesis that if if if something malicious is going on into the network that you're that you're bringing into yours you're going to be able to find that out ahead of time then for an example of the domain expertise maybe an analyst understands industrial Control Systems very well so they understand how they interact they understand what Normal and abnormal looks like from that specialty that theyve brought to the table from that domain expertise they generate hypothesis that there's likely new function codes going on in something like a modb TCP or dmp3 protocol that was going on in their environment and it's adversary related so they generate this hypothesis that there will be abnormal and potentially harmful

function codes in the protocols themselves that ability to then test it is going to require them to know how to get data off the environment effectively P packet capture understand the protocols well enough to break them M past a wire shark dissector trying to understand it for them and then determining the legitimacy of those function codes itself so that's sort of a process that they might go through and if you think about it sounds very simplistic at that high level the thing about all of the components that go into actually testing it that's where a lot of folks get sort of in trouble and another flaw that I see with a lot of uh more Junior Hunters is they'll start off

thinking about the test phase more than anything else they get so technical that they think I want to pull peap I want to pull log i p proxies and they never came up with a really good idea about why they were doing that in the first place that in a lot of circles can uh influence what government circles will call the self-licking ice cream cone to where they do stuff over and over again it's really not benefiting anybody but they think they're doing the right thing and well darn it looking that ice cream cone sure does feel good um so let's jump into the ICS portion so little Bobby here says is it unfair that China

is always getting blamed for computer breaches and Matt says yeah other countries do the same or similar activities and you know what we have to keep an open mind and keep critical of the data little Bobby says okay so which country is most responsible then it goes China so I do want to note that you know as we start talking talking about the industrial control system aspect one of the things that I really want to leave you with is we do not fully understand the IC threat landscape there's a lot of reporting there's a lot of uh folks doing some good work in the community we fully don't understand what's going on in these environments so to sort of

articulate this this usually is a pretty good slide to do so from the IC Sear's uh annual reporting can anybody pick out what's wrong with this first graph or what would stick out in your mind so exactly yeah so for the rest of the audience noted was the high percentage of unknown every single year when these metrics come out and I wouldn't say they're the best metrics but they are the authoritative ones in the community in terms of a central repository every single year the number one attack Vector into IC is unknown so they have no idea how it got there let alone if they're actually cleaning it up correctly if you didn't know what the infection Vector

was it would be very hard to say with a matter of fact that you actually clean it up the second highest attack Vector into an IC every year spear fishing what's wrong with that that's absolutely on the business side of the network we don't have email servers inside of a ska environment so when we see stuff get into an IC we're detecting it in the business networks coming through the DMZ not in the ICS itself and when we actually do understand what the attack Vector is it's something like a spear fishing email which we don't have on the IC side itself as well so I would push off to you that even though we've got a lot of

people doing some really good work in the community we're not historic L getting into those environments and collecting data and understanding what's going on uh as another way to sort of break this down when we think of those big vendors who have thread intelligence it's not just indicator to compromise right it's that understanding of patterns of malicious data related to adversary intrusions uh what we're seeing though is the big companies that are AV companies or instant response companies the folks who have had access to intrusion data into lots of companies and they're able to extract out those patterns and knowledge in ic there is no such thing there are no teams out there no companies ours included nobody who

has a large repository of instant response data in ICS environments Community doesn't like to share and at the same time we have not historically had antivirus and ids's and things recoring out in industrial control system environments so none of the vendors actually have everything that you would hope they would have that they historically have done for it networks so it's one of the gaps that we have and one of the reasons that threat hunting specifically is so important and that open mindedness is so important in ic environments because we're coming up against things that we do not know what to expect it kind of plays back to the to that sliding scale of being able to

kind of start from the basics of the of the architecture and the infrastructure piece and build that out so we're noticing that it's happened and it's been happening for a long time on the it side but on the OT side it's it's really becoming more of a um it's the awareness is really being raised I guess in the marketplace for it um in certain facilities and and companies are definitely taking it on absolutely so let's take two fake case studies or two case studies that I would call hype and then we'll look at some real ones so I've got some real case studies from instant response we take a look at so this one was one of the first ones that

gained a lot of media attention uh so 2010 I could make it through an ICS presentation without saying the word stuck net 2001 you know 2010 stuck net happened in Iran everyone was like Wow SK is a thing uh and people started getting interested so in 2011 when we heard about a potential other breach it made national Level news and there was this water facility in Illinois um that potentially got compromised and the state Fusion Center put out a report that said a Russian Cyber attack took down a water pump in Illinois which if that was the Big Bang thing that we were waiting for like a single water pump failing in a water utility like we won

congratulations um but uh everyone was concerned about it what it turns out that it was was the contractor learned from the naal media reporting about the incident the contractor was the one that remoted in to the environment while on vacation in Russia and remoted into the environment and made some changes and then like three months later the pump failed and so the analysts on scene were like wow Russian IP address in the logs from three months ago and a pump just failed obviously those things are related um obviously that's a misapplication of a causal analysis they're not related but you can see that a lot of people wanting to do good in these environments and maybe not fully

understanding the environments themselves uh report a little bit of hype that gets pushed out there in the community probably the one of the bigger cases that's very hard to kill is the BTC pipeline case so in 2008 the BTC pipeline in Turkey was attacked there was a attack on it there was an explosion and at the time the Turkish government reached out and said it was these extremists and the extremist replied yep it was us and then seven years later bloom ber came out with a big article in the media said nope it was a Cyber attack and it was the Russians uh and the problem with that I don't I don't blame the reporters for

this I don't think they were doing necessarily uh hurtful things on purpose but part of the problem with this was that instant responders misled them into believing that this was a thing and they didn't understand the environments well enough they're not industrial control system experts to push back on all the things that were being said and didn't make sense in the environment as an example The Story Goes that the necess found Internet connected surveillance equipment that was monitoring the control system environment they compromised the Internet connected surveillance equipment they pivoted over into the control center they blocked the satellite Communications that the control center had as a backup relay they went down to the compressor

stations and they changed the alerting and they changed the the pressure on the compressor stations to allow the pipeline to blow up and three later three days later it did uh the problem with that story besides the fact that there's just no way they got logs from that environment uh one of the incident responders said that they pulled CIS log from the type of controllers that were in that facility except for those type of controllers didn't have CIS log even as a feature you could enable until 2013 so it definitely wasn't taking place in 2008 uh the other thing that really sort of set everything apart was the Turkish government had installed the cameras after the attack in response to the

attack so it couldn't have been the attack vector and of course we had TNT and like wrench marks and physical cameras understanding that extremists had come up to the pipe but you can again see how it's very very sort of prolific that hype will generate in this community very quickly when people are interested obviously in things like National infrastructure so to explain what a Cyber attack in an IC actually looks like a little bit of repeat from yesterday but we'll bring it to a new Point um I developed a a model called the IC cyber kill chain so John's going to walk us through it here so out of curiosity how many people know of the

Cyber kill chain the regular one yeah traditional the one being represented here so all of these phases all of these actions still take place it's not anything magically different in an IC environment with the exception of when you're actually talking about being able to have influence uh being able to either manipulate or take down or deny in an actual control system you're going to have to do a bit more work than just getting a footprint and getting a foothold uh and keeping that foothold into an environment so the actual piece that we wanted to kind of highlight it's the second phase of that when you're actually being able to um being able to either xfill data out of an environment

to learn about what's happening in that so when you get into actual Control Systems a lot of I should say uh a lot of the red teams and a lot of a lot of nation states might not necessarily have the expertise to go in and turn off a water pump uh turn off a generator just because they simply don't know how it works they don't know what protocols are driving it they don't know what function codes are going to actually make it turn off and what they don't want to do is basically just shut everything down and lose their access so being able to define the steps and the extra work and just the amount of resources that it

takes to be able to have a an actual defined effect in in an in an ICS environment it's very very challenging um you step through the phases of once you get in get a foothold being able to actually develop and test your attack or be able to to test um to test what kind of effect you're trying to have um in the environment and then being able to actually deliver that and carry it out in a successful manner um a lot of these environments are I wouldn't say unstable but um a simple a simple flip of a bit could change the entire process what happens could change the entire control system itself um so being able to

actually U being able to actually do that correctly is extremely challenging and I would to add on here as well you know when you think of a control system most people think well it's very insecure well if you gave me a control system I could turn it off you just connect to the network and I'll turn it off that that's entirely true I guarantee you I can put a PLC on a network in this this environment and every one of you could figure out a way to screw with it no problem but to understand the physical process that's going on and understand the larger system not that individual system is actually very difficult especially remotely over something like packet

capture so if you wanted to let's take a water distribution facility as an example if you wanted to take a water distribution facility and overflow the tanks you might find a human machine interface that says like drive more water to this location water done and if you would click on that as the attacker nothing would happen you're thinking like why why wouldn't it well because there's altitude valves at that location and they're physical devices that aren't networked and they aren't connected to the network and you wouldn't understand that without understanding the engineering behind the system and those altitude valves say hey there's already too much water in the tank so if any more water tries to get pumped we're

just going to flip the valve and it's going to be bypassed to some of the location understand that physical aspect is more engineering than it is cyber prowess so for an adversary they've got to step through more so we think about this for a second I'm going to flip back and forth what that basically says in an IC is we have an extended kill chain there's a longer amount of time adversaries have to persist in our environments and more steps they have to achieve to have a reliable effect inside of that environment so let's take a case study of this Ukraine ends up being a perfect case study I was fortunate enough to be one of the investigators on

the Ukraine attack uh the report that I wrote out at s that went out to the communities up in the top there so what essentially happened is December 23rd 2015 uh a Cyber attack took down portions of the Ukrainian power groups specifically three different regions 6 months before that adversaries sent fishing emails they did the sort of the normal thing sent fishing emails uh had a Word document the word document had macros on it the email was like Hey guys do you want some more features just enable macros and people were like I totally want more features click and black energy 3 dropped the system uh black energy 3 started siphoning off information especially related to the

user credentials associated with the vpns from there the adversaries use the VPN to get into the IC environment while there they spent the majority of that six months learning that environment understanding how it worked each one of those oos or control centers had a different distribution management system so they learned how to intera how to interact on those three different distribution Management Systems when it came time for the attack they had also identified that there were serial to ethernet gateways the serial to ethernet gateways communicate the ethernet based protocols the control center to the physically separated serial running control system at the substations so it acts as a bridge between the control center and the substation so they found

that they developed uh malicious firmware for each one of those devices so that when it came time for the attack they went down and used the distribution management system against the IC it wasn't malware that took down the power plant or uh Power in that environment they used the system itself against itself then they blew the bridges by doing the firmware updates to the serial ethernet devices which left the responders unable to remotely access those control environments they had to send people out to each one of them to manually operate portions of the grid to return electricity back on and in addition uh just because why not they went through the environment found Network connected UPS devices and then

shut those to go offline and dro the interface so that power would go out in the control center as well and then they launched the deny telephone service against all the call centers so a lot of stuff going on for these Defenders and they actually responded and did very well let's talk about the sort of the hunting aspects of it you know for these CS environment there's a lot of smaller more static environments that we deal with data collection and good hypothesis generation are two of the most important things to get through because once you have the data there's a lot of things that will stand out as abnormal as an example uh in the environment if you had

black energy 3 deployed uh that actually made it into the IC you'd find that one of the first things it does is reach out to microsoft.com msn.com bing.com things like that that all malware does or most malware does for internet checking the problem is there's no way that would happen in an IC normally shouldn't have internet checking going on inside the IC we shouldn't ever see bing.com in an IC if you ever see bing.com in an IC you are compromised okay there's your number one hunting tactic um same thing here we see it's going over ntp but ntp Network time protocol right so we're trying to get time checks except in most IC locations we utilize GPS because we need

very high reliability of time synchronization so we definitely wouldn't be going to Microsoft .c to get the time server we do an internal time server that was likely linked up to GPS very simple things that would stand out if you had the situation awareness or the domain expertise of that environment uh in terms of the chart over here that is a simple IO chart of doing a remote firmware update across the control system network does anybody or anybody without me explaining what control systems are on the network Can you spot the firmware update right it should be kind of obvious I'm not even going to give out anything for this because I would hope everyone understand

that there's a big jump in the io and the bandwidth on this network these networks aren't doing things that normal Enterprise networks do I wouldn't expect to see a controller going out to Twitter and using bsize Augusta hashtags right so it has very set patterns and data flows and things that it's doing all right so let's take another one as an example cool another case study that we have so there was a utility we went out to um that had they had identified they had configured running on the network just like just about every other organization in the world um they found out well they weren't able to determine the actual infection Vector why it was there in the

first place they tried remediating it it kept coming back and they just couldn't figure out what the heck was going on so as part of the process of actually trying to help them do that trying to hunt um we started with the the hypothesis generation so we've listed three up here on the board uh basically the customer was infecting themselves either the vendor was remoting in and for some reason the vendor was infected and they were passing that on as well or they had or maybe they had some sort of uh some sort of transient or mobile devices uh that were coming that were coming and going from the network that were infected and I guess based on the

picture of matter and cars up here if anybody could take a guess as to which one of these hypothesis was correct we have a price to give you away this is a all right so raise your hands all right yes oh the first one the customers infecting themselves that's not it next uh there's like everybody had their hands up I'm going to go with you in the blue cuz you're like right in front of me yeah uh no was not the vendor so we're gonna hold off on this one great job no no um so what it ended up being in this environment is it all those are very reasonable and those happen all the time the person's

infecting themselves or the vendors doing it we see a lot of those infections but actually it was the fact that at this facility they had line trucks that used Windows XP embedded systems and they were connect to the Wi-Fi and so the instant responders would come down remote in clean up the conficker on the network the trucks would drive away they clean everything up they leave and the trucks would drive back like a Ben Hill theme song and infect the network right so that was unfortunate for them but it was important to understand going through that process of hypothesis generation of not only what's most likely which these two folks got those would be the most

likely ones but also the type of data and the type of collection and the type of logs and system analysis you would need to determine what was the actual infection Vector just think about what all goes into doing that all right so let's take another one I used this one yesterday but this has got to be my favorite one uh so we had this Nordic wind farm so this wind farm somewhere in the north uh was operating the turbines you see here or some similar ones and each one of these turbines they have Windows embedded or Windows systems operating and they just have like L little terminals for them and they had this problem that they were seeing

abnormal behavior so what was the detection Vector the abnormal behavior they were seeing well the systems were patching themselves definitely weird right so nobody was initiating patching in the environment yet the systems themselves were being patched to maintain so these hypothesis were generated maybe it wasn't communicating to the operations technology folks and they were just going out and doing patching which is entirely likely uh or Rogue operators uh were patching the systems like bad guys were breaking in and they just wanted to be super friendly uh or it was actually the adversaries themselves not your Rogue folks inside your company uh doing the patching of the system oh okay we we'll do we'll do another one

there right which one was it yeah all right avar patch the themselves all right guys nailed it all right so what this ended up being was the adversaries that gotten the environment found out that this was a perfect environment to do Bitcoin mining and they were using the spare process space to do Bitcoin mining on these systems and as a way to defend it against other adversaries they were keeping the system's PS to maintain now there is a sad part yeah go ahead Change Control right Change Control caught via Change Control absolutely uh the downside is despite all of the recommendations possible they decided to leave the adversary um yeah I'll let that sink in for a second they decided

that the adversary is patching their systems more effectively with less failures than their it Department um so they left them don't do that all right all right so we got another one here as well to just go through and start closing out here so this was a naval ship this one there's a small reference to it in the public Community but this one's not very much discussed but there was a non us-based nabal ship um that had sensitive control systems that went offline they lost access to portion of the control Network um and so when trying to figure out what malware was involved and what was causing it they generated some hypotheses one of the

hypothesis was direct interaction from the adversary over remote connection via the satellite Communications somebody actually came in interact with the systems took him down that's a scary thing especially if you're on a ship right the second thing was that an operator plugged in a USB and infected the network and the malware spreading around took down sensitive control systems and the third was that targeted malware embedded in the supply chain right embedded in the control system themselves finally activated so we went through this process and what they ended up recovering this was not one that we actually got to be involved with but what they ended up uncovering was that it was just simply the operator but to

understand that they had to go and pull registry keys they had to understand the USB plugins where they were being plugged in around the environment and they had to track down the USB and figure out what malare was on it so they could analyze the initial infection Vector again from simple hypothesis generation each one of these are going to take you down a different path of collection and different path of adversary activity now once you figure out any of the ones that we went through you can figure out remediation of that you can automate that process out of either detecting or blocking that activity in the future and that's what a good Hunter should and will do they'll

automate their previous hunts if you're at that mature level so you have to do this every single time we don't want to play whack-a-mole we want to continually put P our security to get better and better in our environments so we'll leave you with some of the sample hypotheses um and then we'll do a recap and exit out here um so go ahead John sure so in an ICS environment and I think we mentioned a lot of these already but um in a typical control system environment you might not necessarily want to see or should see internet traffic so it's a it's a pretty simple thing to to detect as long as you're as long as you're looking for it

you're set up to do collection and actually monitor to see if it's happening or not but it could be a very very big indicator that something bad is happening in your environment and a pretty quick way to go back and actually trace it down to figure out what's going on b.com okay yeah so uh another one is adversary well one hypothesis I usually operate under is that adversaries will gravitate towards the HMI the human machine interface so it'd be very difficult to learn the process the physical things that are ongoing without things like engineering documents but one easy thing is a human machine interface it'll show you the layout of facility everything that's ongoing is it

a water facility is an electric facility what type of systems are they using so you get a lot of information and also it's not some weird embedded system it's usually a Windows system that adversaries are typically used to and that system that can actually interact with and operate the environment so a good hypothesis is that if you have an infection you might want to start at the uh HMI first patching and updating like one of the case studies that we said um and it controls his environments usually very very well organized and planned out um having downtime in a production environment is usually kept too minimal um and is is done in very controlled manner so if you think about patching

happening or updates happening that aren't scheduled or that either haven't been fully tested so a lot of times you'll run you'll run older firmware on Old devices specifically because if you update them they might not work in your environment so all that stuff's very controlled so if you're generating a hypothesis that if I see updates I see patches um that's probably that's probably malicious or that's something that's out of the normal that we should look into yeah another one would be an adversary on the network filtrating out process historian data so my hypothesis is if I've got an advisary in the network they're exfilling out process historian data so I'd have to figure out where on the network I could collect

where I could actually put in some sort of monitoring capabilities to pull off packet capture or flow data and then go through and check in my process historian data if I'm seeing more of that system beaconing out or communicating out of the network that process of stor is going to tell me what everything that is physical in the environment is doing tell me things like frequency or water levels or things like that that advisary would need to know if they wanted to achieve a stage two style attack and then if we refer back to the IC killchain model um the the first phase that's built into the second stage is is doing your reconnaissance doing

your background check your background checks U doing your reconnaissance and and basically learning the environment that you're in or trying to get into so a lot of times we'll um we'll go out and label or we'll generate a hypothesis that says wherever we keep our engineering documents is probably a very likely Target for an adversary um and be able to set up monitoring around that and the interesting thing there is in a lot of environments your engineering documents aren't on the IC so you actually find that your engineering documents could even be in contractor networks in third party networks so is your ICS security staff getting notified if the contractor gets compromised and they lose engineering documents related

to your infrastructure is your integrator compromised how might that be affecting you if an adversary is trying to come after you that two-stage kill chain doesn't have to happen just in your environment so in recap threat hunting is a process of humans actively searching for adversaries in their systems it is a very human based process threat hunting itself Falls in that active defense category good threat hunting begins with a hypothesis a testable idea it's going to drive how you do everything else in that environment and then lastly hunting for threats in an IC is absolutely important not for security aspects necessarily but for improving the safety and reliability of those operations so we have little

Bobby here and I Ed this one specifically because you're always concerned as a speaker what people take away from you saying so I wanted to note a little Bobby one that he doesn't quite uh take away the right thing uh little Bobby says Matt can you virtualize a ska system Matt's like absolutely you know engineering workstations operating stations the actual supervisory stuff that's windows-based totally you absolutely do that and help out with instant response help out with a lot of things little Bobby says cool so virtual skate in the cloud Matt says no that's not what I meant at all so with that and I have to do one more little Bobby as uh uh we're giving away everything here

Matt an expert says I have to change my security approach Matt says well who called an expert well he did uh experts are people that are recognized by their peers not necessarily themselves little Bobby says yeah but it also said so in his Twitter bio Matt says even worse so no matter who tells you anything ourselves included always be skeptical to at least enough to go research it got a lot of fantastic people in the community doing some great things a lot of good research always do your own research on top of it though uh I've got the car hackers handbook here donated by no starch press let's give this one away uh a name

the three tyes of hypotheses that we discussed

sir yep so domain expertise friendly or threat intelligence and then the uh situational awareness so give him a round of applause and thank you so much for sticking around