Matching Your SOCs: A Discussion of Joint IT/OT Operating Models for Monitoring and Response

and dive dive straight into this I know no you guys have a lot of other options for talks and workshops or abilities to the answer thanks for coming coming to this talk my my name is Trevor Hawk I've had the opportunity to work in a various number of cybersecurity roles in my career I started out as a systems engineer at lock at Lockheed Martin doing a lot of secure system design on some cloud and cloud environments for various government customers and I started working more as a cyber security analyst for cyber force our commands I had just so many opportunities to to learn and opportunities to expand and then I that that for a for a couple

years I gravitated more towards a towards a penetration testing role I was able to do that in both the IT and corporate environments and then more of the industrial ot environment which which will dive into here a little bit more and I've seen a lot of examples of Sox those security operations center working in both the IT and OT environments and I've seen a lot of a lot of companies doing it really well a lot of companies doing it not so great so I really had the opportunity to kind of pick and choose that's the best practices in each model and I just wanted to kind of above it aggregate all that all that experience knowledge

together present this before we do that I want to give a little overview of myself in my spare time when I'm not looking through firewall walls and flying to incidents I enjoy the woodworking so it's nice to just take a step back from all the technology and work with your hands either some of the projects that bill mostly at my wife's direction for for furniture around the house and so it's a nice nice way to detach so if you're anything like me when I started my career I started a company called law Lockheed Martin and then we were sold off to another company called Wyeth and then we had we the opportunity kind of create our own company of revolutionary

security and when I started with them we were doing both itno t work and I started my interview and they asked me I was like what's over T I have no idea what it is I know what ideas and so I kind of wanted to kind of set the set the page here to kind of go over the IT all of us we're all here at IP security conference there it's gonna be the computers servers file servers VoIP phones networking infrastructure that's really used in a in a corporate environment to process and store data or if you're home with your xbox to play Halo where as ot is gonna be more focused on control system Hardware is

kind of perform certain control functions example we all have our houses or apartments as a thermometer we we use that thermometer to set up a certain temperature of like what we want the house to be and that that setpoint is going to determine how hard our HVAC system is going to pump out air whether it's colder or higher and how long it's going to do it for until that sensor that thermometer reaches that at that point another example is a simple garage door opener or any any smart home devices you have Smart Switches smart plugs there are kind of really small small examples of a control system order and this over here is an example of a

PLC a programmable logic controller those are gonna be used to really control real time process functions and chemical actions in a in an industrial context you're you're gonna see ot and like a power plant or hydroelectric dam or new nuclear site site operations and you're gonna have operators or engineers working on on HMI that are going to be monitoring like the flow of water flow of electricity I mean able to respond to those different life life situations in that arena I kind of wanted to go over what what the how ot met network topology has been in the past and kind of where we live the current state I'd say probably a deck yeah a decade ago or past they've been

trash and air gap or should have been air gap where you saw these systems that are isolated from your corporate networks or the internet they're gonna be standalone and environments with there's been more of a shift towards kind of stepping away from the completely air gaps and the more connectivity in the name of efficiency so you have the introduction of third-party Internet connections set up by hard hardware vendors to go in there and monitor or maintain or you you see more connections moving into the corporate net Network environment so ok-kk monitor metrics or performance about how this and industrial is working and obviously with that with the growth of those new connections there's going to be an increase in the amount of cyber

security concerns connections so I really wanted to kind of highlight the progression of ot or ICS both from our ICS is industrial control system for this talk you can think about them so obviously in 2010 we our first example that really put it put on the mat with Stuxnet this was the first example of computer code that caused damage to a physical device it wasn't just a matter that was stealing your data order or an anchor in Europe it was causing physical destruction the hands Iranian it contained a number of zero days that exploited a Windows system and once the Windows system was exploited it was able to exploit a Siemens PLC for their their

program logic controller which which controlled the mechanical actions of a centrifuge and and what the Malheur was doing was that it was it was spent spinning the centrifuges really unsafe had unsafe intervals it was telling the system operators on the HMI that are that were monitoring it okay when reality it actually halted the nuclear operation so it was it was unprecedented at the time and it was and the delivery vector was a simple USB connection the next was 1/2 X and 2014 this leverage tenon and the industrial protocol called OBC was the open platform controller which is basically used to scan a network and in this case it was specifically looking for hardware related to Rockwell Automation or

Siemens Hardware those are those are two major players in the in the OT industry as it was it was really fine to and enumerate that and it it targeted multiple different industries that targeted energy aviation pharmaceutical and defense primarily and and the Europe and the delivery doctor for this the hackers didn't directly attack those companies but instead they attack the vendor website the website and compromised those websites and injected their own malware masquerading as operators and engineers on the power plants were thinking they were just updating normal but they were actually downloading malware thong and giving hackers access to those association and operating the coordinate we had crash override in a store in 2015 five fact

this is called two different names the entire industry calls it in destroyer but all dragons that's that's really heavily in the OT space they call it crash override as hackers there's such a big player and the in the industry that we just call them both and in 2017 we had threatenin crisis this was the first time that we saw now are specifically targeting safety systems in these industrial environments now when when we have humans working South's a very very dangerous environment temperatures hovering temperatures of batch processes going out this count countless numbers of opportunities for harm to human life and they and there's hardware specifically designed called the safety instrument system or SAS to prevent harm

or death of humans but this malware was specifically targeted to override the safety protocols in that hardware and so that was obviously a really big big deal really just wanted to highlight that of the ever growing OT threat landscape to really paint a picture of me oh the obvious need for these modern response capabilities traditional socks you know you think thank you our traditional IT sock they're gonna have almost full visibility into everything I write they're kind of like in charge of everything from the server they want alright but but if you're a company that also is not always a case that your IP sock is gonna have the visibility those types of environments and it's hard for

you as the as a sock to say I have the mission to protect to protect my entire company's enterprise from cyber security threats when you have zero visibility into into your operational environments there's this there's just too much of a possibility of actors hiding in those environments and oftentimes if if you're a company like if you're if you're a pharmaceutical company you're the main like business driver for you is the manufacturing of and so that's going to be an obvious target for hackers that they wanted to drop your business operation they're gonna be where your crown jewels are another way to combat this threat landscape is a collaboration between IT teams and I don't say that

like that right like we I know that it's much easier said and done especially when you start to consider you might think that both of these teams might have competing priorities but it's I'm gonna get into this later on they're talking kind of approaches I found work to kind of bridge that gap but I think it's important to realize that this is no longer should we do this should we collaborate the communities but it's more of a let's find the best way to do this and third I found that a lot of companies try to apply traditional security monitoring approaches to this and and it's at some some successes and some some shortcomings but I think in general that the traditional

approaches include isolated efforts that aren't really holistic I want to kind of dive in to really illustrate what I've seen different companies summarize all what I've seen in integrated I dedicated or a hybrid approach and each approach is gonna have their their pros and cons this integrative approach this is my favorite approach this is where you have your your physically combined itno t sock space gonna have your your ID stock analyst sitting with your Okazaki analyst site side-by-side physical space share physical space you have your your dashboards to the screens on the wall that that that used to impress all the executive company but you don't actually use in a security incident you're always working on their own desktops your

shared words a better gun going to be displaying information and data from both your IT and enno team wrote technologies and with that you're also gonna be able to cross trainer analyst you're gonna be able to create instant response processes better tailored to both and ideal well it's challenging what are some of the challenges we've had to work through our finding finding cyber security analysts that are really good usually a lot of people who have those skills are very happy in their job so it takes a lot of convincing to get them to to move move over the other approach that is the beginning to cross cross trying analysts and in the OT domain knowledge other challenges is

creating processes that that work for both both domains it can be a lot a lot to overcome especially when you're talking about interest regulation and political norms and battles say that I have to fight through and also avoiding information overload with with aggregating log log sources from both your IT and ot and environments the other approach I've seen people take as a dedicated approach where you have your your IT sock you have your ot sock your ot analysts are only monitoring traffic from the perimeter down down to endpoint devices and you have your IT sock where you're monitoring only traffic from your network perimeter down the endpoint devices and not looking at any kind of

lateral movement in between the two environments they're gonna be physically separate you're gonna have separate dashboards separate analysts separate skillsets and really the only time you begin to integrate is when there's a really serious threat and and obviously that's not ideal I was it's ideal for addressing the incident higher up in the kill chain right not not down down low when they're when they're working on data or or command control and then I've seen the to be a hybrid approach where it's kind of like a la carte option of like I'm gonna choose this part of the dedicated model or this part of the integrated model I'm not gonna do this and it was trying to combine just like

certain functions and then maybe you might have analyst shadow shadowing in the other sock but there was never really like all an approach for an integration so I never really seen it worked well work well and I think I I alluded as to why I think the the integrated approach well works well so I just want to kind of expand on that a little bit we've found a lot of improved situational awareness in these environments when you have a well structured sock operating model things just there's gonna work well when you have your detection sources aggregated together in a central sim so you're able to do some real advanced or advanced correlation you're gonna have your your

co-located IT and ot cyber security analysts you're gonna have that domain knowledge at a real real easy access know and when when you're sitting next to someone it's gonna be a lot easier for you to be like hey do you see what I'm working right now like what's what's your opinion on this based off your domain knowledge instead of instead of picking up the phone and calling someone and you might think I don't want to bother them with this when in reality like that might be the simple phone call that could have prevented the next infection into your teen varmints so just having that the the the access to that knowledge co-located with you just

makes for so much more for it allows for faster event correlation and remediation which which brings us into the quicker response times when you're when you have your holistic approach to be able to see the entire enterprise and different tools were reporting in it's gonna make it so much faster be able respond because when you're able to see oh I don't have to call someone to ask for proxy logs over here or access logs to a field component I can see this all real time and I don't have to waste time with the my new show process it's going to be able to prevent a delays and incident remediation also its cost-efficient this is one of the biggest biggest sellers if

you're trying to win top top level support you're gonna be able to combine management structure is gonna be able to combine physical spaces and I think if done correctly you're able to reduce the cost of yours two security stack if you're able to use technique two technologies and these different environments that we talked I talked a lot about why this is awesome and you might think wow this is this was really cool but there's no way I can ever do this in my in my my environment but I want to kind of go through some some examples or what we found works best to kind of get us there towards this path of integration I know it doesn't sound

sound sexy but one of the first things to do is to understand your your baseline so really really taking time to to understand how your organization is set up understand reporting structures so really they think of it from a RACI matrix point of view think of functions think of positions who's in charge of what understand what what your current snapshot is same thing with your tech with your technology stack question

you

i their their absolute absolutely challenges to to go through especially when you you're talking about traditional IT analysts aren't gonna be as familiar with a packet capture let's say ZigBee traffic or where my Modbus traffic or or knowing how to read a syslog file that's and think that's kind of where we have the cross cross-training come in and come into place and being able to understand the the norms or base lines of each environment but well will I dive into that deeper to later on and so really when when when you begin to understand your snapshot for your or your or your current snapshot of your organization your them your limitations in your tech technology and your security stack

you're gonna be put in a much better position to understand the gaps that you have and be able to you know it's kind of like a you don't know what to fix until you know it's until you know what's what's broken sort of approach to help you better understand your your capabilities and your process and technology gaps second is your vision now I say I say the integration approach is is the industry best standard approach there you might have some real tough tough battles to fight at the corporate level at the top top level you know mine works best I would I would Lobby to say have a vision of of integration and work towards that and see not not a I like I

like to follow the approach of wow it before you how it say wow that's a really good idea let's see how we can make that work instead of instead of hearing idea be like I don't know how to do that so that's gonna be impossible I'm not gonna I'm not gonna even try here's where you're really gonna map out your your like you had your baseline RACI of roles responsibilities now that you're combining functions plan out a new race II like who's going to be in charge for what talk about processes and responsibilities in integration with the business next is the is the plan this is where it's gonna be a little more in the and

Andy gritty text actively but see okay if we're gonna have a shared responsibilities different sources into one central environment and there's gonna be things you have to overcome like like industry regulation there's a lot of electric utilities I work for but there there was had to there was a lot of work had to go around just sending logs from from an endpoint device sorry it was just a simple come it was a Windows 10 server but it was operating on it was operating in front of Quitman I was controlling the electrical grid so there was a lot of compliance that we needed to work through to send those logs over to a non protected space in

the corporate environment it was able to be done but not without a lot of challenges and so once we had that we just had so much more visibility into those operating spaces I identified log log sources you're gonna have your application and an operating system logs from so from a lot of the IT infrastructure that's going to be in the OT environments that network infrastructure your HMIS your operator workstations time servers process data and telemetry from satellites or a field field devices that are time to are important using all that to aggregate events in your corporate and your corporate soccer just kind of it's going to be very awesome when you start to when you're playing when you're able to

write rules in your in any kind of central sin that you have to have to aggregate and correlate all of these close data and here to in this plan approach I've found that it's it's easy to look at the let's let's go find a tool to give us the latest greatest detection Zoar or let's let's pay for this really cool threat right a threat Intel feed but one of one of the things I really like to kind of hit home when when at this stage is like let's instead of like let's instead of trying to run let's just learn how to walk in this new kind of environment so I like to take a back

back to the basics approach and I know the kids on reddit these days are saying okay boomer when talking talking about stuff that's been been done in the fast old methods but well that's okay you can call me you can say okay boomer to me but I found that following these these steps I've never led led me astray and as has really said a lot of people out for success that's it management I know it doesn't sound sexy it sounds boring but when when you're able to do this correctly you're gonna be set up set yourself up for so much more success when you're ingesting through things like that sexy thread all right Intel feed when when

when you're able to group systems and into logical divisions so you know that like okay and this in this in this organism I wind owes 10 servers in this subnet I know they're this this patching level okay you might look over here and your roti environments is saying this is I have all this siemens step7 hard hardware running on this this four more version okay so that when when I get a tip from my threat Intel feed and they say oh you this this firm from where is out-of-date this firmware is vulnerable I don't have to go through and and worry about scanning my entire system to try to find this Rimmer I already know and I

can the next next basically like Apollo is a vulnerability scanning if you have really good asset management I wouldn't even I wouldn't do any vulnerability scanning and your ot button before you have asset management

yes yep yeah it's all it's there's there's obviously some concern about more around industry compliance and like there's like there's there's one utility I would work for that it's good to have a government clearance to be able to have access to some of the equipment so there's a lot of sensitivity around there and I think that that company's approach was to get everyone in the saw it cleared so that it basically turned in turned in to ask if they had to have access to get in the sock but everyone who who was able to step foot in the sock was was cleared by the DoD or sorry ideally so with vulnerability scanning there's a lot of times where I've seen

corporate our corporate scans from the IT environment traverse into the OT environment because it wasn't proud although we segregated and a lot of this equipment is old legacy stuff that isn't designed to handle traffic it of them know and so it knocks it over and you halt operation and that's a real relationship breaker but between IT and ot right so if you yeah it's your your stuff them off yeah so don't don't blindly throw your your own your vulnerability scanners over to the ot environments you will break something and you will break for relationships like I said before it have it into logical groups so that if you want to conduct scan you know exactly what you

need skin we need to scan and it's going to allow you to have quicker scans and more efficient scans more targeted you can make it the it's gonna be secure architecture design there's this model called the Purdue model I don't know if we don't I I wasn't exposed this at all when I was working in only IT it wasn't until I started working in in ot environments that I really became familiar with it and this is basically just saying you you have your top level and you're like corporate lands up here in level five and then what this is trying to say is is to have a DMZ in between your Corp IT networks and euro T networks we're

going to be level three and down and to have enforcement zones or firewalls in between each and only allowing the minimal amount of traffic that's that's required to go in and out of these zones restrict viral roles only necessary traffic ensure that logs are being sentenced to securely if it's if if if the device doesn't support an encrypted transmission set up a tunnel do something that yeah and that's just going to put you in so much a better position to be able to ingest those logs yeah

you we've I wouldn't say there's a simple answer to that it's always based on the hard hardware yeah there yeah I don't I don't think there is one I've never been able to give a simple answer that and I've never had a one-to-one application of this working in every Sitz always gonna be slightly modified and edit medication plans you can't avoid having I could see a quick quipment and you're in your own environments vendors just aren't fast enough with with upgrading boat both the software and hardware to combat the the threat landscape and I guarantee you if you start doing this you're going to have to work to collaborate with with your ot engineers to mitigate a 20-year

old box running running Windows NT that has some look and that has some control function right it is going to happen you can't avoid it so talking about collaboration this is more of a soft skill approach but I was able to really break it down into three three different approaches here one is understanding and appreciation well as much as we want to integrate and talk about let's come together let's be happy let's let's do all this stuff together we have to realize that the environment are different for a reason so we need to treat them differently and think about identified vulnerabilities as in more or less in the context of the environment so if we go back to this example if I'm

if I'm running an IT sock and I see incoming Boehner incoming SS traffic into US internet-facing server in my DMZ I'm not gonna be concerned about that because it happens all the time kids throw scanners or that stuff all the time I have pretty protections against that I'm not worried about that but if I see a scan coming in through the internet all the way down to an engineering workstation in level 2 that's a really bad thing so it's the same it's the same voter but it's the same incident happening but in a different context and it has much it has very different consequences because it's the difference in the environments to is relationship building traditionally has

just been if we want to kind of bridge that gap between these two we can just have our IT security awareness go over to ot and say this is why this is why you should be so concerned with with phishing and this is why you shouldn't use USBs I just be like listen to me this is the way it's gonna be because I own cyber security responsibility for the entire enterprise where that's a real leadest approach and that's not really trying to understand the cultural differences between but instead try to take of take an approach of what can we learn from each other what can we what can we do in terms of knowledge transfer

that we found works well in our environments and learn so one example I've found is OD teams are really good at backups they're really good at backups because the business will come to a halt if their operations are stopped and so they have really good right readiness operations and procedures built-in I think IT could learn a lot from that and that could help IT respond or recover to a ransom learn and infection and reduce their time to recovery another example is ot could learn from from my IT about Y USB use gives them so much heartburn now to talk about talk about Stuxnet talk about delivery vectors talk up show them a rubber ducky you know how dangerous like

me and how and how easily that can be done as soon as it's inserted another thing I found a blog on this later too about site tours bringing your IT people in and talked into that too that one system engineer and say what would ruin your day what piece of hardware going down would just ruin your day and it shows you this piece of hardware maybe it's a maybe it's an RT you maybe it's some kind of field field component say if that goes down I'm gonna be working a 24-hour shift to get operations back up so like okay that's a really important piece of equipment what can we do a cyber security to protect that but not

halt business operations because if business operations are halted you're not making money I be security is something that's only around because companies have money right so don't think of cyber security where we're kings where royalty no it's it's a support function um then obviously there are there are these like tensions or challenges to overcome now we talked about the cultural differences but between two groups ot might be in a sense of like we never upgrade we never change this is the way it's always been done so that's the way I'm gonna do it when in when in reality that doesn't work anymore because there's this this rapidly growing threat threat landscape and they're constantly evolving TTP's and

just growing growing threats ask the question of who owns cyber security responsibilities me in the OT environment typically when you're running an OC sock you have both the responsibility and liability for any cyber security incident across samurai's someone at the top is not gonna care if you're like oh well that infection was in the OT environment um I just over here hi T they're like no we're giving all this funding where you're gonna protect the entire enterprise and but you have to balance that with some some grumpy plant plant manager of a site that's bad his his box has got knocked over because someone ran a necess can so every chance he has he says no to IT

coming in and and doing anything so it's really trying to find find the find the balance there something that we've worked really well I think I talked about it about it earlier about organizing tours you know organize a tour for your fort for your IT analysts to come over to an operational site and get briefed by the engineers and the and the administrator there's like what's your daily operations what what what's your mission what struggles do you deal with on a daily basis and and the opposite you know I'm I'm working as a consultant right right now in a and in an IT sock and they're constantly giving tours to upper-level management executives and any other employees of

the company who wants you to just get an overview of the operations and I understand how they're doing their mission what they're doing why they're doing it and just well when you start to get when you just get briefed like that and understand you become a little more understanding of why things are done in a certain way and it just really starts to open up a communication when you just start to just talk about it talk about the concerns and notices I know is that I see folks we don't really like to talk talk about a lot of these things but it really starts to a dividend than that in that operation so I just wanted to kind

of go over again like the benefits of this you have your improved situation where as you're gonna have much more capabilities to respond to the adapting threats going to lower response times and and if done done correctly can can really start to drive down costs so I just want to say thank you guys there's my eye I always used a gif of keyboard cat and all my college powerpoints so thank you and I promise these QR codes are not links to malware it's my twitter twitter handle and email this has been my first talk at a conference so I wanna thank you guys for suffering through it with me

I'd love love to take any questions I know I ran a little short but any questions