De-mystifying Zero Trust in Industrial Control System Environments

all right welcome everyone I've got a lot of content so I'm going to go through it relatively quickly I will share the slides afterwards uh so don't worry if I'm skipping ahead because the point is for me to talk to you not uh for you to read from slides so you know I I was originally going to call this debunking zero trust in uh IC but the vendors don't necessarily like it when I say that so I don't know why I just did anyways um as you probably know IC and zero trust are second only to AI in terms of buzzword so I thought uh I'd come and talk to you a little bit about

uh what that means to me because I'm seeing a lot of um talk about zero trust in ICS environments um but no one's really quite sure what that means I think we can all agree that we do have some challenges um these are some 2022 statistics in the OT World um you know like to think of ourselves as as Backwater little places right no visibility shared credentials lots of outside connections um we're certainly seeing an uptic in ransomware and uh segmentation some some some cases good some cases not so good um but you can see from from this these uh statistics that we're not even at the 50% mark on any of these uh security capabilities in ic

the top five security exposures in ic today are of course the business to IC connectivity as well as remote access the internet both that you know about and don't know about segmentation in general and visibility monitoring whatever you want to call it um we're starting to see some visibility and depends on the organization but uh the deeper you go the the darker it gets shall we say so I thought I'd start by figuring out what zero trust really means well if you talk to the firewall vendors it's zero trust network access it's firewalls and segmentation if you talk to Identity vendors well of course it's all about people and identities that's that's what zero trust means so I thought I'd ask

chat GPT because you know why not get some AI into in with my zero trust in my my OT and actually you know not not a bad answer from chat GPT it's the the never trust but always verify identity verification is important and the identity and integrity of users and devices which is something we have a lot of in ic environments uh is key so you know what chat GPT kind of came through all right there what does it mean to implement zero trust well there's a few approaches to that right uh and unfortunately what I'm seeing is people implementing zero trust based on capabilities of products or capabilities that they have today and not so much

based on their requirements or their desired outcomes right I can go buy zero trust anyone I'm sure sure we can solve that problem by buying something there's lots of guidance right everyone's got their opinions uh I want to talk a little bit about maybe how we can wrap our heads around this though and and get started with zero trust so let's go right to the source nist 80027 right it's it's it's a long document but the core is of zero trust according to nist is lease privilege per request access decisions in the face of a network viewed as compromised and excuse me one of the words that's missing there is continuous as well um I think it's on this next one

these are the seven tenets of zero trust I'm not going to go through them all but it's all about Dynamic policy continuous evaluation of of health or trust or whatever you want to call it founded in actually knowing what you have on your network because we all do that too right that's where we were supposed to start with security so when when we try to do this in ic we have some technical challenges our cyber security maturity is pretty low overall right uh so are we even thinking about zero trust we got you know let let's start with the basics um is is kind of where we're at with cyber security and ICS we don't know what we

have right at least we might have some ideas but when you get down in the weeds we don't know what we have on our networks and identity I think is the big gaping hole in most IC networks we you know operator as a username operator as a password goes a long long way and then you get to the organizational challenges right there's limited appetite and also limited opportunity for change right to be fair like you're you're pumping whatever keeping the spice flowing you don't want to take the time to to uh stop trusting everybody right I've I've been there you know you got the guy I've been here for 40 years and we've been doing it the same way

we've had no problems go away leave me alone right and then um templates or or best practices right every site is unique it's one of the things I was talking to a customer and they said yeah you know what we've got 100 facili ities they're all unique so we lose one not a big deal we got 99 other ones like it'll take them a while to figure out how to take down the next site right so anyways um yeah we have a few challenges when it comes to implementing zero trust and this is just some of them there are quite a few others one of the um things or tools that we have is the sliding scale of

cyber security so this is stolen from Sans and we've got architecture on the one end and we've got offense I hesitate before saying that cuz that is not really offense but it is taking active measures to uh protect yourself and in the middle we've got a bunch of other things right architecture self-explanatory we've got passive defense which is basically going and buying a product plugging it in light turns on that's all you got to do right uh and then active defense I'll talk about a bit more later but that's people you know that's where the the magic happens knowing what you have leveraging your knowledge of your environment to get rid of the attackers and as we get up to the the uh more

mature organizations we have threat intelligence and like I said actively um uh deploying counter measures of course as you would expect bang for the buck you get a lot of it you get a lot more bang for the buck with architecture passive defense and it decreases as we move our way up so that kind of says we should be starting with architecture right and in fact there are five critical controls for IC I guess we decided 18 would be too many uh so let's let's start with the basics here crawl before we run and start with the five critical controls for ICS cyber security and these are instant response defensible architecture visibility and monitoring secure remote access the Achilles heel

of many many in environment and risk-based vulnerability management I'm going to talk about these a little bit fairly quickly um although it takes a while to go through them all uh but again this is uh a white paper that you can go and download uh that dives into all of these controls uh in detail and what you need to be thinking about when you're uh applying them in an ICS environment and I I will talk about that at a high level as well first one as I said bang for the buck this is this is my favorite topic which is a defensible architecture and this is what um the critical controls say you want to build an architecture

that supports all of the other things I'm about to talk to you about right supports the visibility and the logging and the asset identification segmentation DMZ control of communication within the environment right if you don't have a a defensible architecture that you're stuck you can't Implement any of the other controls you can't do anything to help yourself uh unless you have a foundation to build on So speaking of architecture high value low cost talked about that now I also talked about the rate of change in industrial environments let's just say it's not fast so when we talk about architecture we're usually talking about where do I want to end up where do I want to be

what does good look like and you know what I'm ecstatic if if I can get a picture of what good looks like and then have an organization that doesn't move further away from that every decision you make should not move you further away ideally should move you closer to that but it may take a very long time it's a long game you're looking for new new sites site upgrade site refresh um you're not you know you're not going out and retrofitting architecture in any environment not to mention IC environments and of course we want to start at the itot internet boundary right that is you can I can make up statistics like any as well as anyone

else right 80 90% of um IC attacks come in through that boundary sure people can walk on site there's Wireless there's lots of other stuff but uh uh that itot internet boundary is where we want to start so when we talk about secure architecture we want to understand and document our requirements right and there are as many uh as much as in any other case we have cyber security requirements absolutely but we have business requirements as well right we are protecting the the um the money making engine of the organization and if we screw that up well then we we don't have anything left to do right we want to inventory we want to know what we have have you ever heard a

framework talk about inventory before okay well we'll move on uh we need to know what we have and that means in especially in an IC environment and especially when we're talking about architecture that means grouping and labeling everything and I will talk about that a bit more on the next slide we definitely want that strong boundary we want secure remote access see this is foundational for all the rest of the controls we want to segment our networks and not just it from OT but different different processes different types of systems different functions within the OT Network and especially because it's really hard to get strong endpoint Protections in an ICS environment we need that Network visibility monitoring

uh which leads to response endpoint visibility is great I'll take it all day I'm just saying it's not that easy to uh to get this is a version of the Purdue model right I hear I wanted to throw this in because I hear lots of people talking about oh yeah our architecture is based on the Purdue model no it's not Purdue model is for for lack of a better term it's a labeling system it's a way of putting sticky notes on everything so you know how to categorize them how to group them how to refer to them it has nothing sorry I shouldn't say that it is not an architecture it is a tool that

allows you to talk about architecture right with the Purdue model we can say all of my my plc's my controllers let's call that level one let's call all the local supervisory systems that are are right next to those process local supervisory let's call that level two Regional or sitewide supervisory let's call that level three and so on it's just putting a label on everything so that then I can say things like well you need segmentation between level two and level three within level three between three and four etc etc right so no one implements the Purdue model it's just a way to refer to things so that we can have an intelligent conversation about architecture and everything else

so key tenets from an architecture point of view let's keep the internet out of OT now I'm not saying it's completely I mean if we could operate an environment completely isolated sure great but that is becoming less and less possible uh these days right heck almost all of the vendors have apps for phones and tablets and so on um they monitor from the cloud they want to collect Telemetry in the cloud but we do want to very very strongly control internet noty like it's that is if you ever wanted to implement a positive enforcement model deny by default and permit only what you need this is the place to do it right it's not a free-for-all we certainly

want to control it you should be using your business systems for regular email internet and so on we want those enforcement boundaries right because again if we create and establish enforcement boundaries that supports all the rest of the security controls that we want to put in place so that means unique credentials not your business credentials because how secure are those right how many times do people get their usernames and passwords stolen on the business side MFA of course to get in we want secure file transfer we again want to know what is supposed to be talking so that we can block everything else or sorry block everything else and permit what's supposed to be talking and of

course monitoring and logging that because again we don't do a great job of collecting logs from endpoints or getting visibility on end points right we are uh kind of stuck on the network so we need to architect a network that gives us those choke points that enable that kind of visibility and of course I'm I always have to hesitate when I say it but I'm a fan of active directory in an IC environment it does bring you a lot of centralized control authentication policy and so on but it's got to be managed by someone other than an engineer please and it's got to be completely independent of it and of course we have antivirus we have

patching we have all of those other infrastructure activities as well that should be managed independently of the business so those are I mean I could go on about architecture forever so I won't but those are some of the key things that I do talk about which leads into so now we've got a defensible architecture that enables us to do uh monitoring and again the control says we want continuous network security monitoring of the I CS environment with protocol aware tool sets and system of systems interaction analysis capabilities that's a mouthful so understanding what's going on on your network to inform operations of potential risk to control there's lots of ways we can do this well not really we have passive

monitoring that's that's our go-to again we don't touch anything we don't interfere with anything we don't take outages we just Monitor passively and we can leverage those choke points that we created with our defensible architecture and it has an added bonus that hey maybe we can get close we can approximate an asset inventory that's a win we can group our systems based on their configuration labeling anything anyone right we can identify vulnerabilities to a point and we can certainly uh improve our our logging maturity right I am ecstatic if I go into an environment and there are logs somewhere even if they're scattered across all the endpoints hey that's I got something to look like like

that's that's a benchmark obviously we'd like to collect them centrally and then build some use cases and alerting and correlation and all of that but hey let's have some so for visibility what are our requirements we want a full inventory at least of Ip connected things like I get when you get right down to it to to field bus and serial networks and so on you that's that's a challenge but we want to know not only what do we have but who is talking to who because one big advantage that we have in ic networks is that they are relatively static right they've been doing the same thing for 40 odd years right so we can

actually to a point catalog all of the systems all of the communications paths we want to know we want to have visibility into the actual IC protocols right what commands are being sent who's connecting where especially remotely we have vendors out the wazo connecting from who knows where doing who knows what with who knows what credentials so again come back to those choke points that's where we can get the visibility that's where we can force the authentication we can put our our passive monitoring and like I said I'm happy to have any logs but there are a lot of Windows systems I say OT or I say IC and people think process control we have

controllers we have machines we have Automation and yes we do but guess what it is supported by a ton of Windows and that windows is I can't say it I can't make a blanket statement but it's not generally processed critical it's valuable it's important we don't want to do without it but it's not if you you know if you have a window system going down you're not going to be taking out the entire process unless it's that one system of course uh but we we can bring a lot of our traditional Windows management capabilities into the OT environment and benefit because those are the systems that are also getting compromised and attacked right so where can we put visibility in

well firewall and network infrastructure logs that's easy that's free um you could add net flow to that list right every packet or every session that's going through a firewall we should know what it is absolutely there's lots of vendors lots of products that do passive network monitoring which will help us as I said with the vulnerabilities the inventory the communication Partners got Windows event logs if you have centralized Authentication it's something we generally lack and I would like to see a lot more of that's a great source actual logs from OT systems maybe uh as well as the systems that are managing them they there we might get a little bit more traction which takes us to our third

control secure remote access let's put an end to what I like to call RDP and free how many of you have those environments where you can RDP or you come in via remote access you RDP into something and that's it you can do whatever the heck you please you the whole Network's your playground at that point right let's please please please put an end to that we we there's a whole bunch of controls whole bunch of ways we can do that better again according to the the critical controls we want to identify and inventory all remote access points including that hot spot plugged into the back of that server you didn't know you had um we want on demand access we want

MFA of course and this all comes down to some sort of jump host Bastion host whatever you want to call it as long as you don't call it Citrix um environment sorry that was out loud wasn't it uh so key components distinct credentials for OT again your corporate credentials will get compromised period it's happens all the time so you should need a separate set of credentials MFA um actually I was watching uh the the latest season of the the grand tour with my kids this weekend uh and long story short um what's his name Jeremy Clarkson started a separate show where he started a farm and I actually figured out what a sheep dip was um because

that's where they actually take the Sheep they take a little hand crane they crank the Sheep up turn it over there drop them in a vat of some sort of antiseptic haul them back out and they're nice and clean so I I never knew where the sheep dip was until I saw that the visual was was very informative there that's for USB devices though or file transfer not cheap um yes we this is one of the key points where we can do identity based zero trust right you get on to a jump host you get access to a jump host based on your identity once you're on there you get access based on your identity

and that host gets access based on its identity right we can get lots of auditing and there are solutions today uh and they are getting better where we can get credential vaulting and recording and sharing and approval workflow and all that kind of stuff as well I'm going to go through this fairly quickly because I still got a few things to cover um again I will share the slides this is my version of a a relatively ideal remote access architecture first step terminates in the DMZ I don't care what credentials you use but these are not your OT credentials they get you to a DMZ that's it realistically all you can talk to is a file transfer solution and a jump host

that's it dedicated IC authentication happens after that so I shouldn't have combined those but MFA to get in then dedicated IC authentication to actually log into that jump host so one credential to get into the DMZ one to get onto the jump host and as I said I got this totally out of order didn't I only get access to the jump and file servers the file servers there I don't care how you structure it but some concept of I upload a file it gets scanned it gets copied over to this other file server once I'm in the environment I can copy it off that file server and back and forth and not wide open to anybody and

everybody because we don't have any usernames or or access control please it's great when I can you know we're pen testing and it's like oh I go to the file share there's every single project file every single architecture diagram every single anything to do with the OT environment jump servers authenticate against an IC active directory then they you know you might have multiple vendors you have a vendor one jump server vendor vend two vendor 3 vendor 4 and those users only get apps that they're authorized for and those servers can only talk to the processes that they're there for right multiple levels of control uh and then the jump server network uh so the jump servers are

controlled at the network level as well in terms of what they can get to when we talk about segmentation and the Purdue model I want to see segmentation between you know levels two and three three in the DMZ DMZ and level four but also within especially level three I've got IC servers I've got file servers I've got active directory I've got jump hosts I've got Sim and Antivirus and patching and all that all of those should be grouped into separate segments as well vulnerability management I think we all know what vulnerability management is but risk based uh is the key word here uh we want to mitigate the impact and monitor for possible exploitation the thing is risk-based

so if I patch a vulnerability in a system that does everything I tell it to do that will accept any command over the network that doesn't do any authentication doesn't do any logging have I solved a problem no I think is the answer to that patching things that are insecure by default is a waste of time we don't have enough time to do all the things we need to do so that's why we talk about risk-based vulnerability management the things that could be exploited to gain additional access we're seeing lots more vulnerabilities but we're seeing that the advisories don't have accurate information uh the vulnerabilities are relatively serious but only 13% are extremely critical so again risk-based

let's start with the things that really matter 83% are deep within the network yes I would like to fix those but it's easier for me at this point to stop people from getting to that point in the network so that's where I'm going to focus my effort so start where attackers are most active right level three close to the business Network close to the internet on common operating systems that are familiar to attackers easier to update for you right again that Windows stuff closer to the business network not as critical for operations so let's patch it and do it based on impact to your environment what are the things you can't afford to lose those are the

things where you start there's a couple links they crown jeal analysis and consequence driven cyber informed engineering our mou ful but also really good ways to focus your effort I'm going to go through this I'm not going to belabor the vulnerability um bottom line is prioritize have a process to oops that's not my pointer and I can't see it anyways on this screen so I won't worry about it but the bottom line here is figure out if it really really really really needs to be patched right now and is that more important than whatever operations you have going on otherwise mitigate it and Patch it when you have a chance there are some that need to be patched now

absolutely but otherwise you know I don't want it sitting forever but let it wait patch it when needed again though stuff like your active directory your jump host your antivirus all those kinds of utility servers those aren't process critical patch them keep up with them because that's where attackers land and lastly instant response um you need an operations informed instant response plan you can probably leverage more from your real world in uh Emergency Management than you can from your it incident response plan although they're both relevant right and practice practice practice practice practice practice because you will find so much that you um didn't think about or didn't consider when a real incident comes along IR is important regulatorily now

you must have you must report incidents you must have a plan there's reputational risk there's operational risk and Financial Risk right we have a huge dependence on it systems now whatever you want to call it systems computers uh so everyone is a target right you drive your car to the mall someone's pulling on your car door handle right you're you're a Target uh it's not a matter of if but when which means that prevention is ideal detection absolutely necessary but if I run around here screaming fire that doesn't help much we need somebody who's going to come and respond and do something uh for that to put that fire out active defense I mentioned it

earlier this is where uh this is a very summarized version of it but you're leveraging your home field advantage you should know your networks your environment better than any attacker ever would and you need to use that knowledge to um get an advantage to get a leg up on the attacker use your knowledge of your environment to evict them from your environment so from an instant response perspective be able to Island your network I think of instant response in terms of thresholds if something happens I've already made a decision to take an action and I know the impact of that action I know how I'm going to take that action and I know both the impact of

doing it and of not doing it for example isolating it from OT networks I know how to do it I know how to operate in that situation uh and I have pre-made the decision about when I'm going to do it based on some thresholds uh we want to you know they call it minimum viable operations or Turtle mode right uh what can I do to isolate myself and still keep operating communication is Paramount as is collaboration is your your it vendor or OT vendor going to show up to help you you know Cisco has famous L shown up in 18 wheelers when people have data center fires uh has your uh will your OT vendor do that for you

right so bringing this back to zero trust because this is this is kind of the the the focus here the critical controls all support zero trust in some way incident response we've got that minimal minimum viable operations to restrict access to Enterprise resources that is a one of the um tenants of zero trust right restricting access to Enterprise or in this case OT resources as a part of instant response we should understand how we're going to get to that position our architecture of course as I said supports Network segmentation supports zero trust network access supports authentication and identity management and much much more our visibility and monitoring again we need to see everything that is the

that is just straight up matches the requirement from uh the zero trust framework secure remote access same thing identity based only get getting access to what you want uh and then the risk-based vulner vulnerability management feeds a lot into knowing what you have into your asset inventory into your exposure your integrity and so on so how do we do this well let's start with the easy stuff right defensible architecture easy things we can do to Implement zero trust or move towards zero trust in an IC environment control Communications at the boundary and then once you've got that problem sorted segment out your network uh at the lower levels of the environment as well and establish those

choke points for network monitoring right those are all relatively easy things to do Network architecture is not complicated it's just the time you you have to do it at the right moment in time right where when you can um imple you know Implement those things then if we talk something a little bit more difficult authentication and authorization of remote users really should be table Stakes right it shouldn't be that hard certainly if we want to get into an identity based defensive posture remote access is where we start with IC push as much of that out uh from an architectural perspective and then you can use those jump hosts use that remote access architecture to authenticate and authorize all your

users and control what each of them can do get your sheep dip going right secure file transfer not open for everybody but um um you know not so provideing means to upload and download files that are known to be safe that applies to USB as well as uploading and downloading and improve your visibility and asset inventory getting passive monitoring in place again is not that complicated there will always be work to do to improve it and alert and so on but getting started is not that hard so secure remote access and network visibility monitoring they they directly support zero trust and really they are solvable manageable even better problems today then we got some harder things

vulnerability management and patching again for certain systems really easy for other systems maybe not so much but something we need to be working towards continuous authentication and authorization especially of devices and that's the big elephant in the room when we talk about zero trust in OT how many devices do you have what is the identity of all of those devices who's well there is no who there is no person there's no one logging into them necessarily but those devices are your critical devices on your network and um doing continuous validation and authentication and authorization of device to device uh communication there are starting to become um Solutions both product-wise and architecturally for that but it's

it's tough and so as active defense I've worked with a I don't know4 billion doll company that had an engineer spending 50% of his time running all of it for OT for that organization uh so getting people to do things is they may want to but that's uh no mean feed so here's what I get told reasons you can't improve ICS cyber security can't take an outage yeah we hear that time and people constraints different priorities the potential of an impact either during a change or as a result of a change ancient hardware and software interdependencies between all of that lack of vendor meaning the OT vendors Buy in so we got lots of excuses built

in but let's let's look at how we can do this a gradual approach is okay this is especially in OT it's a long game right we we want to know where our destination and just try and crawl towards it architecture is free right you don't have to pay anything for that virtualization gives us a lot of advantages in an IC context vendors are you know what vendors to be honest if you're buying new OT products from vendors they've got security capabilities you just got to work a little bit to get the security guy in the room during the uh planning phases uh because you don't get it if you don't ask for it start with your new

initiatives your new sites your site refreshes where you can get as close to Green Field as you can SE Suite certainly has visibility that is helping our cause uh and we can start with the boundary not like let's not start with the process leave the process alone there's a ton of work to be done at that itot boundary at those with those Windows systems and of course monor ing isn't disruptive so why not it it does take some people time but monitoring is something we can do that isn't disruptive so there are lots of reasons that we can improve ICS security cyber security most important thing start do something i' I've heard for too long

that this is too difficult of a problem right there's there's it's all those no reasons that we can't do anything well there's lots we can do zero trust isn't Magic it's not a product it's not a thing um I don't even know that it's a I don't even know what it is it's not a thought process it's not a journey it's just a bunch of Concepts that can help us uh improve the overall security of our environments and there's lots of things we can do to inch closer to that ideal so that is the the short version like I said there's lots of content there I will share these slides there's some extra slides at the back

end too um so they will be I'll make them available however that happens worst case hit me up on LinkedIn or something like that and otherwise I think I got a couple of minutes for questions if anyone has any yeah yeah I just a um just around itot convergence big buzzword obviously in Industry a little uh for a little while now uh just wanted to get your thoughts maybe on itot convergence and um you know maybe dos and don'ts or if you had have any recommendations on where to start if those align to these domains that you were talking about or yeah uh any priorities there uh well I'll start by saying I hate that term itot convergence but uh I

mean we are seeing a lot of cloud obviously even in OT and we like to look at Cloud the best solution we've got today is think of cloud as yet another process so isolate it from all your other processes put it over here give it controlled access to the internet and so on I would argue that sure there's some iot type stuff in our it environments you know your video cameras and meeting room things and all of that but we don't it and OT convergence is not something that we really like what it sounds like is not something we want we want to keep our OT environment separate we have a lot of it stuff in OT and that's where

we are seeing a lot of that convergence and that's where we see a lot of the opportunity because again all of that it stuff is stuff that we can secure shall we say without direct impact on the operational side so that's that's my my two second version but I I really hate that because we're not converging it and OT we're getting more it in OT but that's my my two cents think we got one more over here okay I'll tell you what I'll come over and talk to you because we've got to move on to the next next presentation here thanks very much everyone