Five years of OT security assessments by Derek Dunkel-Jahan and Justin Turner

we're back all right now our second Talk of the day is as you see on screen it's called five years of OT security assessments horror show myths and practical advice to improve security and safety we want to welcome Derek duncle Jan and Justin Turner for this talk let me remind our attendees uh you can ask questions using the Q a feature if there's time at the end or if they want to do it during uh our speakers will take your questions that way and then they've agree to go head over to Discord to their talks Channel and answer more questions and interact with you there take it

away hi this is Justin um it looks like I am not able to join on video so I don't know if there's if there's a way that we can looks like dereck's may be having the same problem when we try to start video um can you hear me audio wise okay yes I can jeffon we sure can all right um so let me uh I will uh I will share my screen again here and we can we can work on video and uh get started on the on the intros um while while we wait uh on that I'll keep kind of clicking um and see if I can get this video going um as we get

rolling um but excited to to join you all today appreciate the opportunity to chat with you about operational technology security um it's something that um Derrik and I are both um really passionate about and have been spending a lot of time the last several years um focusing on um assisting companies in evaluating security risks around their operational technology and helping set a road map um for better security and safety and so uh this isn't a heavy U you know Death by PowerPoint or academic uh type of a presentation today really what we wanted to do is just kind of do a bit of a brain dump and talk about the things that we've been seeing um

and basically just kind of our our Tales from the field and the things that we've learned um things that we've seen that have worked um things that we've seen not work or some some uh large risks or vulner ilities we've seen in our clients and so um that's kind of what we're going to walk through today um so if we we'll we'll do some quick intros we'll level set on what is it versus OT and how there's some some differences there um the inherent vulnerability um and and the vulnerable nature of those devices themselves and often the networks they are connected to uh we'll show you some of the uh scary things we've seen um

just in the last like 12 to 18 months some of the sort of the pictures and anecdotes that we've pulled together to share with you today um and then Derrick's going to kind of go through what are what are some of the things that we have heard um and that have sort of myths that we have um debunked right or some things that you you commonly hear as push back or adversity um to moving the needle on maturing operational technology security and uh and how um we can sort of move past those those potential roadblocks and then just a very practical road map of what are what are some good things that you could be doing now in your

organization um to help secure those we didn't pull in the kitchen sink of everything that you could be doing but a lot of times our clients just ask us like hey we we want to we want to make progress in the next six months or the next 12 months like what what can we do right now what are some achievable and practical goals U and so we we'll go through um some of our experiences in that as well um so so so with that um let me kick to Derek uh quickly and Derek if you'd like to do an intro and then um I'll I'll pick it up from there and go through the next

section yeah absolutely so um Derek dun Jan associate director for pertivity out of our Houston office um I guess it's a good thing you don't have the video of me because um I'm in enemy territory I have both ANM and an OU diploma behind me so um luckily I won't have any virtual Tomatoes or anything thrown at me without the video um been working in OT security uh for the last 10 years since I started um with an upstream o gas company Devon energy really excited to talk with you guys today I have attended bides before and very excited for to be a speaker on one of them for the first time so looking forward to the

talk and looking forward to hearing any questions and comments that you guys have because this is a very emerging area both from a regulatory and just from a interest perspective given the potential impacts uh over to you Justin yeah thank you Derek um so I'm Justin Turner I'm a director in the cyber security practice tivity um I have been with the firm for a little over 11 years now um I uh have been focused on cyber security for the majority of that time but early in my career um spent some time uh cutting my teeth on on it audit and some it socks sorts of things um General it you know controls I spent a good amount of time

as a pentester as well so I was doing more offensive activities and then being based in um Houston um we we do a ton of work in the energy and utilities industry and that was really uh my in into starting to um evaluate and advise on um operational technology and then over time we really started to do things in in a number of other Industries including our manufacturing uh consumer products and chemicals um even some Transportation right and and and a lot of use cases uh for this technology so um I uh I attended UT Austin I try to get back to Austin as often as I can I know we're not all uh in person today um but

excited about um future years to to be able to uh to get back together um uh physically so let me uh let me move on to just a quick level set on when we say OT what are we really talking about so so so it in the traditional sense right this is this is what's supporting our business data our our business uh applications and so you think about your um uh you think about your key uh Enterprise systems you think about your email you think about databases um data warehouses uh things of that nature um the traditional back office support operational technology um these are uh devices that are connected to a network um but often times they

they're non they're non-traditional in the sense of we're probably not talking about a Windows Server we're talking about things like um programmable logic controllers or flow computers or rtus um the things that automate the business right or and automate operations so in the in the case of of manufacturing you could have a production line and there are devices that are automating um that that line right things that are like sensors um measuring temperature or volume control there's a lot of individual working components that go into that and over time those have become increasingly uh more connected we've had more Network connected devices um we've moved from manual ual or pneumatic type of controls and these are things that we

can monitor in real time we can get data from in real time we have the ability to remotely control from an you know an iPad from from the couch right I've kind of seen that in in real environments where folks can sort of monitor what's happening on the manufacturing or the production side uh in a in a remote capacity so that's kind of when we start to talk about the it and OT convergence um which is a very sort of common buzzword or ter you will hear and really what this is getting at is with the increased connectivity and convenience we do have additional risks that we need to consider um with these devices and

these networks because the the impact U is pretty high if someone's able to compromise that device you're talking about um something that can directly disrupt your operations and pretty easy to kind of see how that ties um to your bottom line um why are they vulnerable um when we go through some of our stories and anecdotes here a lot of this is going to come to light so I don't know that I want to spend I don't want to spend time going through every bullet and reading to you um and I maybe will just go through the headers here but we have inherently insecure um because in a lot of cases we see that the the the uh oems

right the the manufacturers or the vendors that create these devices um don't always have security at the Forefront right even some of the things that are coming out uh today newer models um don't necessarily have the highest levels of of security and there you see a lot of Legacy devices in these environments because it can be difficult to replace because you have to coordinate an outage or there's a domino effect of other devices and other components that you have to replace if you want to get rid of like your Windows XP machines so we see a lot of Windows XP out there a lot of Server 2003 that are supporting those shop floor operations um plc's from the late 80s or

early 90s I have I still see those in production environments today and so right there some of the stuff was made before security was even we didn't even have the internet when some of these things were made and starting to um you know sort of establish network connections and then we I hit on on the previous slide on the increased connectivity um you know another vulnerability here when we talk about lacking collaboration um there in a lot of organizations what we see is that there is a bit of a um Power struggle between the folks that are on the it side and those on the OT side so your plant managers or even your more technical

folks that support uh the networks and devices um in the manufacturing environment um there's you know either some some opposition and opposing ideas there right with security being seen by operations as something that inhibits their progress and makes them less efficient um and then you also kind of see a bit of an a you know not apathy but um you know it it's a bit of a black box right what's happening in the field and so I hear a lot of a lot of cases where um the ceso or the it organization says yeah I don't know what's happening out there right you know the field guys they they buy their own stuff they set

up their own stuff I don't know what they have we're not looking at it and so when you don't have tools you don't have the same security appliances and network visibility and and logs and um you don't have all those things on the OT side that you have on it then you you that kind of leads into the last um thing on the far right here which is we don't have good visibility into those networks we don't know what we have we don't even know what our indicators of compromise might be um because we don't know we don't have a good inventory of what's out there and we don't have eyes uh on that um because we're it's just a bit of

a a bit of an unknown right or politically um we haven't been able to work together between these two teams right our our our Enterprise it or security and the folks in the field uh to come to a consensus of how can we make this more secure uh and not impede what needs to happen on the on the operation side um and then th this slide is uh you know a bit of a uh just kind of a level set on why does this matter right and I don't I don't think anybody is going to be surprised on this but uh one of the things that I find the the e easier right to communicate or tie

um when you're talking about these risks versus some things on the Enterprise it side is that these uh outcome it's it's a little bit easier to see okay well if this if we have an outage of this system right or this facility is made unavailable what is the risk or the financial impact of that it sometimes can be difficult to say well if we lose our AP system for the day right what does that really mean right uh you know for a revenue or ar if there's certain things we can't recognize you could probably back into what the financial impact is but it's a little easier to look at um you know a a an oil and gas

production facility and say hey this thing makes us $5 million a day right and this that facility is unavailable for a day we pretty pretty sure it's going to cost us $5 million and so um you not only have the the financial impacts of production line or or um the the devices in the field being made unavailable but you could have Financial loss from loss of confidence right if you're if you're if you're publicly traded and those those breaches need to be disclosed right we've got new SEC U proposed um regulation for for public companies around what they need to report on Cyber breaches and and um the timing and how to do so and so there's

costs that are SOL around reputational impact or perception in the market Ransom payments that may need to be made if you you could still have some you know an attack on your infrastructure lead to the compromise of something like sensitive data and so you definitely don't want uh to see an instance where you you're paying for you know breach notifications and credit monitoring services and things like that for for your customers and then um can I can I jump in really quick because there's a really good question question in the chat that I think would kind of um tie in well to to where you're going um Ryan asks uh what's going to be the

best way for organizations to really get an understanding of what their true attack surface looks like for the for the last place I worked at I know that part of what we did for companies was dig deep on finding an entire attack surface for them is there a better way than hiring pen testing consulting firms to get a true grasp of a company's structure

yeah so um I'm reading the question as well here Derek why don't you start I know we're gonna we've got some things in here when we start to talk about our um our sort of our road map that I think are going to go for like what are the things that we're going to uh you know how do we enumerate the risks and then what are our practical um uh what are our practical Solutions I know you're going to cover that so anything you kind of want to share as a as a bit of a preview for what you're going to cover later yeah absolutely so I think um when I kind of think about the attack surface

from an organization with critical infrastructure or OT it this slide's a really good landing spot because it's what is the business risk that you're trying to reduce and mitigate is it Financial loss uh production capability impacts to health safety so I'm going to pick a you know let's say I'm a oil and gas company and my main concern is negative impacts to health and safety then I'm going to be focusing my initial attack surface on those safety instrument systems the things that if compromise could um could make things go boom in in our oil fields I also would look at and we go through this later on in the presentation but I kind of go

through a Continuum of like like what are the risks that are most likely to cause an incident so first it's what's remotely accessible if anything is remotely accessible from your site that is your stop what you're doing get that secured and taken um taken off the internet if possible after you have your internet accessible attack surface then go to like Remote Access VPN and segmentation between it and just kind of moving down the attack surface stack and there are tools out there like um Clarity Nomi dragos that can help you get a understanding of your actual attack of your actual asset footprint within an environment and those tools are really good at doing that but I'd

say from an attack surface perspective i' think outside in uh Justin is there anything you'd add on to that no I think that's a I think that's a great overview you um and I think one of the things we want to when we when we get to sort of our our last section when we when we talk about you know again emphasizing the the Practical remediation um we see a lot of our clients that get sort of stuck on here's everything I have to do like they'll go find the nist you know 80082 framework for securing you know industrial control systems in OT and say I need to go do these 200 things and I have a 100

facilities and there's a bit of paralysis of like how are we going to do all this we don't have enough people and not that any of those things are are bad things um but but Derek's point is is a great one which is there are there are certain things that that bubble above others and if you look at um instances of in case studies from OT attacks and breaches over the last several years we have a couple of them highlighted here um there are definitely some themes tees that emerge right it is an email compromise somebody clicks on a fishing email right and they're and then they don't have maybe all of the network segmentation that they should have and

someone's able to Pivot into the operation technology environment right so you you don't often think about security awareness and needing to train your users um and how that can impact operations because it feels like it's maybe a little further remov from that but logically there are sometimes paths to get there right USBS um and and having sort of uh not a lot of control or security around removable media in these environments there's a couple of case studies you could highlight where somebody plugged something in and it resulted in in a compromise or an outage and uh so we we'll we'll walk through a couple more examples but um you know I I I think what I typically recommend is um let's

let's look at like Derrick said remote access uh let's look at the the most uh you know five six most um likely Avenues where we could get impacted and compromised and let's ensure that we're locking those things down and let's do that first um before we start worrying about other things uh because the other things are just going to build on that and enhance our maturity um and just create more um points of mitigation if one of those other methods is compromise we have to lock down um the we got we got we got to take the log log me in and the things like that off of the um the shop floor PCS right we've got to we've

got to start with those things and we've got to get the the vendor appliances that someone drops out there and they have a VPN connection right directly into our our OT Network and you know we're not governing that connection we don't know who from the vendor we don't know how secure it is but we know they have a remote connection point we need to inventory and make sure that those are secure and so um those are just a couple of things that's a great question I appreciate it and and certainly uh we we welcome any additional um questions as we as we go through this uh before I move on from from this Slide the last thing I I I don't want to

Discount here is the you know the the health and safety of employees right and we you know again I don't want to I don't want to um downplay the impacts of an attack um on a business Network or an Enterprise Network um there's certainly hassles and there's costs and there's inconveniences associated with that um but we're probably not worried about anything uh detrimental happening to somebody physically um if we lose our you know our our financial reporting system or we lose email um in in the case of compromise of an operational Tech technology environment particularly if it's a motivated attacker um that knows a little bit about the environment is looking to cause um disruption or

even harm um you have you have potential loss of life here and so we're talking about something very very real and it's really it's it's impossible to put a price tag on that and so you're you're right at the ground floor of some systems that you know again Derrick mentioned Safety Systems there are uh sa mechanisms built into many of these devices some of the older historical stuff um it it doesn't have that or those safety mechanisms sometimes fail there might be ways to bypass those and so it's definitely not something that you you want to gloss over and that's a that's certainly a risk to highlight here which is you know if we're if we

are um creating or refining hazardous materials or chemicals we need to make sure that our cyber security um is strong and that doesn't lead um to something like an environmental event like a a spill or an explosion or something along those lines okay um so I'm going to transition to the next uh section here uh which is just a couple of photos we've taken from um some of the engagements and we've done over the past most of these have been taken in the past year to to be honest um and so I won't spend a lot of time on these if you if you have been um working in the OT world for any amount of

time you have seen the passwords posted somewhere right um I probably have a hundred different pictures I could could have used here I grab sort of the the ones that were the most accessible or the most recent uh but it is very common to see um passwords written down places um when you start to walk around on a on a shop floor uh or you're out you know in the in in the field with a pipeline organization or oiling gas um I think the most memorable example of this that I have personally is is I was in West Texas um with an upstream drilling oil and gas company um and and uh was actually one of their their plants or

their Midstream facilities and so uh I I walk in the gates open right you you could just drive in and there's several Portable Buildings and trailers around um and I walk into one of them and it's like a bit of a makeshift control room and there's a PC in the room right in the middle of the room and there's a whiteboard on the wall um and it has a um what looks kind of like a password on it and uh so I I walk in and I say the the person that's giving me the tour I said please tell me that's not a valid password and they were like well you know it might be so I get on the system

and I think they were using Delta V Emerson's Delta V which is a uh pretty you know Common um industrial control system sort of application that we see in the field so I was able to log in with that password and I was like okay well I have full control of the facility now right and so you know it's just it's this wasn't a social engineering type of an exercise that I was doing but it sort of got me thinking it got the rest of their organization thinking you know somebody drives out there in a you know in an F250 with a hard hat on and some you know flame resistant gear and sort of looks the part right there's a lot of

people coming in and out of this facility it's pretty big walks into this building I mean you can you can shut the whole facility down from there with a password that's on the Whiteboard you could do that in literally in five minutes and uh so that that picture ended up being something that was pretty impactful went to very high levels in the organization and they sort of said look we we have to get better about this we we know that most people that work here and work at the facility might have access to that shared password of that shared account but we do have to think about the risk of somebody coming in um that doesn't belong there or a

disgruntled user or an employee so um definitely risks associated with this and given this audience I don't know that anybody's going to you know going to disagree and they're not all created equal some of these accounts may be more benign but even then um you know thinking back to my days as a pentester we would we would absolutely use any little foothold we had even if it was for intelligence gathering right or Intel and surveillance that we could potentially use we see a lot of non-standard devices in the field um so um devices that you know maybe are not they're a Cisco shop certain devices that one facility somebody decided to purchase it or went

over to Best Buy and bought a Netgear Nighthawk and plugged it in and it's this one I think in particular was also running um was broadcasting with with web um which that protocol has been deprecated I think for over 10 years now and so you have uh you know one of the risks is you have folks with a PE card or with the ability to go procure their own devices and they're well-intentioned and there's probably a reason that they need the device there's probably a use case there but again we I I talked about lack of collaboration and coordination earlier this is definitely one of the uh one of the outcomes of that which is

they they're not security folks they don't know how to secure these devices it's not that they are trying to make things insecure um it's that somebody who's an INE Tech right is just going to kind of plug this thing in and if they can connect to it and get whatever other devices they need to connect to it then that works um these these devices specifically if I remember correctly were also these were dropped in by a third party who helps provide at this particular manufacturing site helps provide some some support and they were using these as remote access points and they kind of had their own network out there and no one sort of on the

Enterprise you know shared service it or security team you know knew that that this was happening so right we have this kind of this Rogue Network there um with these devices that are unmanaged and really there's not a there's not an inventory noting that that these devices even exist um we also see a lot of examples of when we do have Windows systems out in the field um for good reason we we we should you know we often don't see and shouldn't don't have a use case for having internet um from being able to get out to the internet from from the field or from our operation side and so you're not going to be able to have you know

automated updates um you're going to be able to set auto update in in Windows and so uh patching is absolutely a challenge I don't want to I don't want to uh uh act like it isn't it's often a manual and painful process um but this is an example of and again this was these were both taken just a couple months maybe like in May or June a little bit earlier this year um so you're talking about some some windows devices like a window7 workstation that has been patched in 10 years um passwords that haven't been reset in you know 11 years um that sort of control the entire facility and you've had a lot of people you know if you inis have come

in and out of that facility um in the last 11 years that that likely know what that password is um and so uh definitely we see this a lot um in environment and then you know you gota you got to love the uh internet access uh for a machine this is a this was also a manufacturing client um that I uh we were we were they were very adamant that they did not have internet access on the shop flooor and we were able to disprove that pretty quickly and uh even were able to see some of the recent searches and things that folks had seemingly been watching uh from some of those machines uh on the

floor and so uh a lot of risk with these having internet access where somebody maybe is more likely to click on a malicious link or visit a malicious site or download something and if we don't have our security tools on those machines and visibility um then that could be sort of the thing that uh results in a compromise of a of a site and and possibly you know our entire Enterprise U and then the the last thing I wanted to include here not directly cyber security we don't often think about it but this is sort of a growing you know an emerging area around cyber which is the the concept of resiliency and we have a lot of other things that

can cause an outage and we it's going to be hard for us to isolate whether something's a cyber sec incident or maybe it's an environmental type of an incident if we've got a bunch of trash next to right our our Network rack that could possibly catch fire right and not good temperature controls we've got Rat's Nest of cables and we don't we have to troubleshoot and and we're not able to um to to sort that out easily we've got sprinklers um and evident water damage in ceiling tiles right above where we sort of have all of our networking and and local compute for for our facilities and so um this this concept is is certainly sort of over the over the last

couple years has has been woven in a lot tighter um with cyber security and this concept of of operational resiliency um so I definitely wanted to kind of show a couple of examples of some things that we've seen um from that side as well uh let me kick to Derek um I'm there's a couple of other things here uh uh that we we we don't necessarily have pictures of or good ex illustrative examples of but some some things that we that we often see um I mentioned asset inventory um being something that the majority of companies that that we walk into have no idea what they have in their in their OT environment um and I I

highlighted a couple of these other things as well earlier around user security awareness um you ask folks hey if there was a breach like if you on your on your um operator workstation in the control room if you saw like a mouse moving and people changing stuff and like what would you do right and we hear a lot of well I guess I'd call it okay well who in it right do you have a contact like do you and it's it's in a lot of environments it's very clear that the escalation path really hasn't been established for if you see something say something that might be preached but what are we looking for and who do we

need to say something to um and then uh right again just to kind of to to hit on lack of security monitoring and visibility um definitely a big a big gap and indicator of risk in these environments all right um Derek let me uh kick over to you um to start going through uh some of those security Mists and then uh start talking about the road map as well absolutely thanks Justin and thanks for sharing all those those photos I think um especially your story about the the Upstream uh provider in West Texas the interesting part of that too which gets into the visibility and layered controls which I'll talk about later is that probably looked like legitimate

traffic it looked like a legitimate login from your admin on the HMI and how would you be able to see if it's just a your actual admin or any anyone coming up in um like he said an N F250 and um some flare gear um it's it's an interesting thing for these environments on how you need to layer controls because not everything these environments are not the same as our it environments um and OT secur is getting more and more notoriety just due to some of the high-profile hacks like um JBS foods and Colonial Pipeline and emerging regulation from the TSA so I want to combat some common myths that that we hear kind of talking

to to either clients family members or um just other cyber Security Professionals um and if you can go to the first myth Justin if it's if it's air gapped it's secure I understand that there is absolutely a risk reduction um whenever you um Justus if you're able to switch the next slide I'm not seeing it on my own um I understand that there's absolutely a risk reduction in air gapping your environments and I have a definition at the top which is restricting all Communications between it and OT networks there would be a A reduced risk if that were to take place but that does not mean it is completely secure for your West Texas example

Justin that could have been an air gaped Network and without physical security controls to and additional monitoring controls the air gap would have would not have stopped a random attacker from just driving up to the site and logging on with the password that's written on the Whiteboard I also I cringe a little bit inside any time I hear somebody say oh we have an air gap Network I'm like do you really because I think it is um in my experience having done this the last 10 years I don't think I could count I think I would I don't even think I can count um the number of times that I've actually seen an air gap fully

implemented because there's just a lot of operational needs with the you know itot convergence with needing to get production data into your data historians and to run analytics and for accounting purposes on the corporate side it's just it is very rare to see an actual air gap fully implemented um other security controls besides network security are still needed you still need host level visibility you still need to restrict the usage of USB to only appropriate workstations and individuals you still need physical security because it's one of the it's one of the you know golden rules is that if you have physical access to a device you now own that device um and back to the visibility component which is going

to be a very common theme for for Jess and I in this talk is how are you getting visibility if it's air gap do you have an on-site sock that is actually monitoring all the host and network level alerts that are coming in do you have a cyber security team with forensic capabilities on staff to identify your you know air gapped and siloed Network and Insider threats can still have a massive impact even if you have legitimate access with malicious intent is still a big impact something that still could happen even if your network between it and OT is air gaped if you go to next slide please Justin um another common thing that I

hear a lot is well why don't we just use the same controls for OT as we do for it I mean we've we have meaningfully move shifted right the defenses that we have in our cyber security industry with like endpoint detection and response and network monitoring why can't we just do some of these same things for OT there's a lot of kind of issues with that one is Legacy operating systems may not be compatible with modern cybercity controls and tools you may not be able to use your current EDR software on a Windows XP or Windows 2000 machine and attacker goals are different for OT systems compared to it systems with your OT systems you're trying to

typically as an attacker cause harm or drive some sort of response from the company whether it's like a ransomware and you're trying to get financial gain or you're trying to cause reputational harm those goals are different than in the it environment and you need to have controls that are appropriate physical security is more important in my opinion in the OT environments compared to it environments a lot of it environments you have a lot of you have a lot of network security controls like for for my organization I'm working from home right now with my you know local with with my internet connection at home there's a lot of you know secured VPN Etc EDR type of

controls that are all internet based and orchestrated to be able to protect my device for an OT system that that HMI is not going anywhere it is like it is physically connected with serial ethernet Etc cables to these sensors and controllers within its environment so that's why physical security becomes more more important also these systems are typically less stable and able to handle agents or able to handle scans there can be operational impacts if you try to run a subnet scan with nessus you you could easily take down a plant or you could and again that could you know draws in identifies its own vulnerabilities there but um these are typically less stable um operating systems and environments so

some of your security controls are going to have to be more detective and less protective in nature finally in the OT environment safety is the Paramount goal it is making sure that no matter what whatever happens on your HMI whatever happens in your it environment that no matter what we have a safe reliable operation for this environment because otherwise there could be impacts to health and safety um and next slide if it's a legacy system there's nothing you can do to secure it we might as well just throw in the towel at just do whatever we can using the last talk uh for guidance on like getting getting our EX Executives to sign the 78 figure

check to make sure that we can just replace it um although modern Cy security controls may not be compatible with Legacy operating systems there's still things you can do layer defense and depth controls such as network security controls um physical security controls identity and access can significantly reduce operational risks if there are particularly fragile services and systems segment them and segment it to the point where there's only the authorized connection one of the things that's good about baselining and OT environment from a network perspective is it's pretty there's not a lot of variety it's a lot of the the inputs and outputs and connections are going to be the same dayto day if there's a new sensor connecting to your

control system I think you can find out pretty quickly from your operators on site whether that's legitimate or not when there's an inability to use digital security controls physical security controls can really be a help let's say for example you have a HMI software that you just are unable to have multiple accounts on it or the password is ver is can't be changed and we see software like this and it's very expensive and costly and to update it and even if you there else has to be aoft two update to we're running these environments have a very specific operational purpose we're building products we're we're mixing chemicals we're making food and beverage like there might not be a software

that's been created outside of the one that's Legacy and horribly insecure so what what can you do to secure it you can use more physical security controls making sure that you have CCTV cameras cameras on the workstations to see who's actually making those changes you can restrict remote access to where it's only physical access provides you the ability to manage these control systems and sensors have badge access to and from the room so that you have better assurance and understanding of who's actually making changes on these systems if you go to the next slide please um the next myth is our organization isn't important enough to be a Target um I could not disagree more with

that myth every organization has a mission and a purpose and there can either be a disruption for monetary or reputational gain and there is absolutely a reason why an attacker would want to disrupt your OT environment attackers also typically Target companies with weak controls regardless of industry or Financial importance if your Seaman system is available over the internet I guarantee you you're going to get a lot of attack on it because attackers go for what's easiest first and then go for higher financial and reputational gains after that operational incidents are also highly disruptive you can think your organization may not be important enough but when an operational incident happens it's really disruptive to operations it's costly it impacts public reputation

negatively and there could be an impact to health and safety of the Personnel on site so in when that happens it's going to be a bad day for everyone involved and Le last one your Executives will not feel the same way about how unimportant your organization is after there's been an attack and we'll go to um the next slide please Justin and the final myth is we need a local cyber security team in the latest Tools in order to have a safe secure environment not necessarily teams don't your OT teams don't need a eight figure budget in order to secure your OT environment and your and your important control systems and sensors layer defenses for physical

security physical access logical access can go long way towards reducing the attack surface onsite Engineers are also very well RIS very well versed in operational risks um as part of their job as a process control engineer or operational process engineer they need to identify what the particular risk levels are for operations if we use the SAR Florida example your engineers understand for a water treatment plant what levels of lie in the water are unsafe what level of particles are unsafe and they have a risk-based methodology and how they build their processes out they just need some coaching on what some of the cyber security risks are and how we can work towards shared goals together

to having a more safe secure environment um safety controls can typically be um implemented through the through the applications without the need of expensive cyber security tools and monitoring overhead and logging in monitoring I like using the fancy tools like Clarity Nomi dragos um table OT they provide great visibility and great value ad but you can still use out of thebox monitoring capabilities to see what's going on your environment to Baseline expected connections you're many of the OT applications your control system servers have some level of like application monitoring in place as far as like where where connection where authorized connections are in place you can use your network switches and routers to understand what traffic is in

what traffic is currently happening within your OT environments and your firewall traffic to understand what's attempting to go to and from your OT networks and most of your host operating systems even if it's that old Windows 7 machine there's still host operating system logs that you can leverage to better get logging and monitoring capabilities if we go to the next slide Justin and I'm going to take a quick pause because I see a couple of uh questions here um in the Discord Channel um the first one I think is more of just a comment was is is your environment resilient enough to withstand human error probably not which is why you need to have all these additional monitoring

capabilities in place so that you understand that whenever there is like for example in the ultimar water plant whenever there's an attempt to um add too much lie into the water that people can catch that because even if a administ an authorized user on an authorized software made that attempt it it's still not good and there needs to be checks and balances to understand and identify human error um how or can backup configurations in your OT environment how can you check if it's changed um from a configuration perspective it I would first start with your OT applications some of those applications have tools already in them in order to have change logs to understand who made

changes when and to back up your configurations you can also save off your project files and just do discs between the to files your configuration files to see what changes were made and map that back to your management of change process um and then um last question is um visibility how can you capture net flow to and from your devices I'd say kind of best practice typically start with your span ports and um of your switches and just try to mirror and get um Network traffic off of there and then use net flow analysis tools like Sil to better understand kind of your noisiest transactions um if we can go next slide please so let's talk about um practical

um recommendations um one and these are risk ranked um one is prioritizing your locations and Facilities by business risk a couple different ways to do that one is what are your Revenue generating facilities what are your supply chain impact facilities if you have a if you have if you're an organization of you know larger size let's say you have 50 50 manufacturing sites and one of your sites builds components for the other 40 sites to use then that's going to have a higher risk from a business perspective since it's a critical path and a critical um point of failure for your other sites identify assets of the N environment you can't protect what you don't know about so that's definitely an

area that we we view as very highly important securing VPN or remote access if you cannot have secured encrypted channels into your environment it's going to be abused and that's going to be one of the first ways that attackers try to try to gain access into your OT environments we talked about um Network segmentation this one is highly important just because there's a lot of there's a lot of risks and threats that can come come from the it environment your it environment has much more connected devices a l larger attack surface and working with your process control Engineers to understand what that appropriate segmentation in DMZ should be I'm certainly of the opinion that we

should not air gap all the things because it and OT shall never meet there are a lot of really important efficiency gains from having your it and OT networks being able to communicate in a very restricted appropriate business approved manner so that you can get that production data out of your OT environments into um into your it environments for greater analytics and greater understanding of Trends restrict access controls permissions on a leas privilege basis if you're able to enforce role based segmentation on your applications fantastic please do that as often as you for whatever applications systems domain domains that you can gaining network monitoring visibility we just I harped on this on a couple of

slides here is just it's it's really important to be able to understand what you have in your OT environments what's happening so you can respond appropriately and then finally once you've kind of gotten all these other things now let's get to the fun stuff let's identify what are vulner abilities are start with your publicly available exploits start with your vulnerabilities that are most accessible from an outside from an outside in perspective uh Justin I'm GNA ask if you have any additional comments on the slides that I've gone through or any other um recommendations towards building a safe reliable OT environment yeah the the I appreciate you you walking through this Derek and and I and I I love this list um I

think there you know this is there's other things we can be doing but but this is um this is going to offset we we fix these things or we put we put in controls around these areas and we we've addressed a lot of what we need to defend against um the thing that I will say I I throughout the presentation we mentioned a couple times about there being kind of a a lack of collaboration or maybe competing priorities between it and OT um you know I one of the things that I would maybe you know that ties all of this together is my experience has been that the folks in the operations or the automation teams are

not opposed to cyber security in fact they're open to it especially if you can articulate how a more secure environment helps with their metrics that they're evaluated and compensated on right which is going to be efficiency and uptime and safety and cyber security absolutely can be a contributor into ensuring that those things happen um and so there's a there's just a big education piece and a lot of opportunity for for a middle ground right I have heard too many times that a cyber team walks a plant site and they see a server and they go that's not my server I don't know what that is you can't have it right and they and they rip it out or they they try to come down

in a heavy-handed manner what that really needs to be is hey what is that server doing right we need to understand there's there is likely a good reason that something was put into place to achieve an objective um to help get visibility to pull data out of something to communicate with a device and so there's likely a way that we can still achieve what operations needs um in order order to support the business um but then do it in a secure manner we' likely have got in our organization tools at our disposal and and um things we can use um to help but there there's going to have to be um a common understanding there there's going to

have to be communication there's going to have to be open dialogue there's going to have to be um sort of uh established roles and responsibilities and so I know that that kind of feels Elementary and not very technical but that is a it's an often gloss over aspect of how to make sure that you have a successful program and more often than not I don't see that happening and you throw all the tools in the world at the solution and Technical um you know you you you implement a framework and you assess your facilities and you start trying to push remediation but unless you're getting um input from the folks on the operation side it's just never going to

be successful and so um that's kind of be the last last thing I'll add to to tie all this together any any final questions that you see um Derek in the QA looks like the QA nothing in the QA here anything else in Discord or anything else you'd mention as a as a wrap up we like right at time now uh no nothing in the Discord um we have some uh you know clapping emojis um so I think um good um good feedback from the group and the Discord throughout um thank you everybody for the time um and um as I mentioned in the Discord I'll mention here um if you guys have any questions um that pop up after this call

feel free to reach out on the Discord feel free to reach out to us on LinkedIn we this is all we do every day is talk about OT security so we we love talking about this stuff and there's a really important Mission around that which is ensuring safe reliable operations for the most important things in the world world so if you if you all want to connect afterwards please reach out and thank you again for the besides um besides committee for helping us uh get set up with this and for putting on this this this conference thanks yes thank you everybody really appreciate

it all right thank you very much for that excellent talk uh we're just going to make a transition here so give us a couple minutes