Rick Kaun: Think Global, Act Local in OT Security

uh what we're going to talk about today is think global and act local and ot security um what i'd like to do is um walk through a little bit of let me get the right session here i'd like to walk through a little bit of an introduction to both myself and the topic um what i'm hoping to do today is bring some practical lessons learned that came from um many many years in this space doing ot security and i know that we've had two full days of great security content so when i talk about introductions i'm gonna assume a certain level of understanding um i wanted to make sure that uh you know we sort of had a base level but i

want to show some some push some ideas some new newer concepts that we're seeing many organizations adopt and so i'm going to walk through some case studies but also some framing primarily of how to uh be more effective in ot end point management um you know in light of what we have for for risks and we assume that we have the risk but in light of what we have for real world constraints in ot i work for a company called verve industrial protection um verve is uh an industrial systems integrator uh so our founder is assistant electrical engineer former ge westinghouse um and we grew from that into reusable tool sets one of which is a security

platform and so one of the things that most people don't know about us is that we've spent about 26 years now 25 minutes we have today about 26 years in the ot space and for 13 of them we've been doing security solutions including the deployment of an agent-based approach to security management very safely at all oems and so that's part of the stuff i want to stretch people's minds with is you know we have a lot of i.t assets in this space and so the more we can adapt it types of tool sets uh but with a very ot unique um spin the better off will be hence the the concept of thinking global but acting local and i'll leave it a

little bit better virb itself is a vendor neutral organization we have a number of ot system integrator type of projects like plc upgrades and dcs programming et cetera we also have a lot of historian capabilities um and we also have of course the security so we actually um partner with our clients over long term as most systems integrators do we're not really uh you know commodity or widget sort of sales thing and so some of what i'm sharing today is actual not some actually all of it is real world organizations and their and their experiences in the space myself by the way i've been around for a while you may have seen me locally uh cc chapter uh meeting uh

president with uh you know paul petrovsky and naja when she was still we were fortunate to have her near us here in calgary um i've been doing ics security for about 20 plus years having worked in matricon and of course honeywell as well so getting started um this was some of these are you know we're using from another presentation or different levels of maturity in the in the audience um but most people it's very rare that we find somebody with nothing and so some are very very rudimentary that they have some sort of inventory maybe some patching or what have you but at least everybody has some level of program investments and policies right some sort of inventory

and we can't just throw out the script right but what we have seen happen regardless of how far along you're on that maturity is you've usually seen a patchwork of tools and therefore a patchwork of coverage and we find that sometimes that's because of what we call the mouse in a cookie scenario anyone who's familiar with this story but the most if you give the most a cookie the idea is that you want to grab the mouse you offer most of cookie and he says oh this is interesting uh but could have a glass of milk to go with this and now that i've got a cookie and milk it's like well that was good but maybe i've got some

crumbs on my face can i look at a mirror and the idea is it's it's like pulling on a thread right and so many organizations have started down a path in some cases oh my biggest issue is input let's go get white listing or my biggest issue is inventory so let's go grab a passive tool that gives me a low touch you know first pass of inventory those aren't bad um but in reality taking those uh piecemeal bits and pieces and trying to put them into a sustainable scalable program which really if we pause and look forward to three to five years um we're going to be completely if we're not already uh overrun in terms of asset uh account to head

count uh and in terms of risk and and the requirements to get better uptime and drive more visibility so um what i'm advocating today is if we stopped and looked at what our end objective was and then use that towards uh analyze what we're looking at with an eye towards things that probably aren't going to change like the increase of ip addresses in an operational environment and the decrease of skilled ot security practitioners and the need to automate and manage some of these in a way differently than we typically have been um and so what we find many organizations emerging in what we call first generation security um i as i mentioned i've been doing

they're except for a while i was on nurkship guidance committees when i worked for matricon back in 2010 2000 uh sorry way before we got bought by honeywell 2005. um and so it's been around for a while at the time you know ot security really was piecemeal and so it's no surprise that people have a collection of tools that we call first generation security um what we're advocating though is you know there's some challenges to that um there's a real challenge to maintaining the current security program like that uh tasks such as patch review configuration management logging etc they just continue to grow as patch tuesday becomes more frequent and more difficult um as i mentioned the

number of ip assets that we have continue to grow this is a rather old anecdote but one of my friends at a pipeline company not far east from here when we first started working together he had three a staff of three plus himself so four of them and there was exactly three windows-based ip enabled assets in his environment um he fast forward a few years and he's down to one uh additional staff member so just him and one other person and they're up to 150 um ip enabled assets that need to somehow take care of so we are getting we are getting outnumbered um regulation corporate standards continue to improve whether we're trying to do things for

best practice or because uh we have a corporate or regulatory need and what this means again is that many organizations have disparate solutions we found one client actually i spoke to recently they have 55 different control security packages of software 55 different security tools and i think eight of them were patching tools i can probably name about three or four myself but um i don't know about about about eight and so um the problem is that you know different vendors also drive this sometimes i mean um you know if i have a multi-vendor environment i'm probably gonna have more than one version of antivirus and maybe one more one or more versions of backup segmentation

um is something that everybody strives for but maybe isn't necessarily uh you know consistent um and vulnerability you know anyone doing vulnerability scan based um hats off to you because usually scan based vulnerability in ot is is sporadic it's spread out um amongst outages or only on redundant systems it's often scaled down because the intensity can be difficult and so uh you know getting real value or real-time value vulnerability data is always a challenge and so then what often happens is security and maintenance is often best effort or manual if if the threat is big enough we re redirect our people to go work on day-to-day stuff or sorry from day to day stuff onto you

know real-time um risk reduction and patch deployment but otherwise that stuff often gets uh left to the wayside because it's so much further down and the other thing we're seeing is changes in the organization um you know when you talk about uh what a multi-disciplined security program looks like you're going to have people that want to know data to be able to pull into different functions now whether it's a traditional i.t sort of siloed environment where you have endpoint people and vulnerability people and patch people and server build people and network people and security people and perimeter people or whether you're looking at within an organization you have regulatory data you have risk data you have life cycle data um

you can always see um you know that there's different views and different needs but it always goes back to the endpoint and so the result is you know current generation security is patchwork there's complexity to it to the environment to the to the management and the capability lack of centralized visibility if you have multiple antivirus tools and backup tools um you can probably you know dive into different you know visibilities and sort of piece together what your overall status is um a lot more insecurity than you might think and you have a lot more manual effort and redundant tasks etc um and as an example i'm sure most people have heard it was a little bit

older now but you know duke energy they actually negotiated a settlement and they know narc sip isn't security but compliance is at least a framework that's starts to give you the building blocks uh for a security program like do you have inventory and software you know what your users are etc duke had an exactly this sort of patchwork scenario of different tools in different locations some people had gone and bought you know particular oems tool another but different you know piecemeal tools etc um and at the end of the day they were sent such systemic lack of consistency and adherence to the standards that they simply negotiated a single fee for the entire organization and it was a 10 million dollar uh

negotiation so i can just imagine if they actually you know enumerated the list of issues what that would be now that's a very punitive example um and i'm not trying to you know i hate talking this topic with you know fudd but those the reason those fines were led me was because there were real risks and real lack in their security controls and their awareness and understanding what they had so that really is maybe a worst case scenario but but it's very much um a real possibility that you know your best-selling plans have some friends that you know are in this local market and they're very well spoken and very well understood but when i see an actual

network diagram of one of their facilities through one of our other initiatives you know it's like very wide open and so i asked my friend a needle in the little them a little bit they say well what we tell them to do and what ends up happening um can often be different so it's a challenge i get it not pointing any fingers so what do we do and how do we get started um well the first thing is we need to revisit some key learnings these are things like i said that we have seen time and time and time again whether it's a first pass client that had really rudimentary manual efforts or wants to go into

something automated or whether it's somebody who has this peaceful bit of information has realized they're really not driving it forward um or whether it's somebody that that's you know wanting to look at what they have and really reassess what they're doing um the first thing we need to do is what we call start with the end in mind and i know that seems very very simple um everybody with kids you know school age kids probably cares about the seven habits they teach the kids you know start with in mind what are we trying to can't i can't overstate it um quite frankly uh if you look for example in on the diagram here um these are recommended you know groups

or categories you typically find in most organizations and we've just mapped them in the orange bars across the categories in this csf so you have a good crash records against whatever standard you want to talk about but you also see that over time you typically sort of add these more advanced things near to the end but then eventually you're going to need all of them and so if we're making a decision today on how we're going to collect inventory and hope that it is as suitable today as it will be when we're building playbooks or doing security analysis we're probably going to be disappointed have to come back and revisit that unless of course we look at how we're

going to do inventory relative to not only today but sustainable over time to keep track of ads moves and changes and to help to continue to feed additional a more mature security practices as we go so one of the things i always want people to think about is you know if you if you're one of these people that has a program and you're kind of part way in it you know what's your wish list if you were to get a do-over would you want to automate more of this data and pull it together would you want to aggregate data from different data silos um do you think you could drive a single pane of glass i know a lot of people

that invested a lot in splunk are trying to get that way splunk is of course you know interesting it does have its own benefits maybe there's some challenges to actually pivoting that data into other functions but better along the lines of these sort of aspirational uh bigger picture second generation sort of approach um and then there's a you know was a debate about scope um what should you do or what do you have to do right uh and that's really where a lot of the patching you know latency or the or the debt that we have is is in place because some do bare minimum and some you know push forward quick example uh we had a client that that scanned a

system and they found 1200 known or 1200 missing patches 1200 and some they hired the vendor to come in and patch it and the vendor patched everything that the vendor supported and handed the system back to the vendor or to the owner operator and they rescan and they found that there's still over 800 missing risks now that's not a problem for the vendor that's because the vendor took care of everything they knew and supported on that system but that end end owner still owns 800 plus risks on those assets and i know that the vendor didn't quote approve anything beyond what they patched but is it acceptable to keep that risk in your environment simply because

um you know vendor approval didn't work to be honest we helped that client patch the other 800 and they got it down to zero and nothing happened um so you know i just want to push people on what are we trying to we're trying to do the bare minimum vendor says this and that's enough and my hands are tied uh or are we actually trying to do what's more important that is reduced risk for the companies that we work for so um i mentioned a couple different case studies i'm going to dig into a little bit here and i'm going to start with one and then i'm going to drop back into uh this is how a client decided to

revamp and reassess how they did it and then i want to get to the second part which is more around the maintenance so this is the planning and the and the the refresh or the level setting and starting again case study the second case study in my presentation today will be how that looks in a day-to-day basis in a couple of use cases so from the planning perspective we're talking about a client that multi-function environment um they had uh loved the cartoon yeah i liked that one too um transmission distribution they had generation and renewables as well uh they had at least six different um dcs vendors um so they have different oems different levels of support and capability

different levels of complexity uh they had multiple segments within each facility not every segment was always connected they had you know islanded subnets of protective relays for example they had multiple backup and antivirus platforms uh and at different systems at different locations as well um and they had tried you know again that scan based vulnerability assessment but they were doing it once a year during annual outages and so their data aged almost immediately and wasn't necessarily that valuable um and the inventory that they had was built at the beginning of a subsequent project they built a project and they added some extra service dollars so people walked it down um and you know these generation or

1950s and 60s coal fired so they're ridiculously complex with lots of assets and then what they tried to do was use the walk down and use change management process to keep the inventory up to date well you'll see where i'm going with this by the time we came in to work with them on this project that inventory they thought they had was wrong by 592 to be exact um and so some real challenges um and what they did was they decided to come in and build a multi-disciplinary roadmap so they decided at the top that they wanted to um first establish a standard they said we are going to pick the csc 20 and we're gonna have a maturity level of

two um and so you can see that their objective was to get to an automated maturity level which was not just do i have an inventory but i have an inventory and it's automated um do i have a regular review of administrative accounts to understand what software's installed etc um and so they first built a multi-disciplinary map and as you can see here i'm not going to go too terribly much into the specifics but they had different teams whether it was design teams installation teams evidence collection to grab the data and push back into the reporting and the tracking then they had a very real uh timeline with multiple disciplines to allow them to sort of um

achieve it now what they did was by establishing csc we should shout the csc 20 and now shalt have a maturity level of two they then had a starting point and admittedly they had a lot of discussions with the cso's office to say look we can't practically do this in ot or it's not technically feasible and some were pretty easy you would just throw it out i mean for example antivirus on a networking gear for example um but the other ones where they wanted to look at encryption for example they had legacy stuff that maybe didn't support it or maybe they couldn't um delete or remove or adjust uh certain administrative accounts because of the way they were built the oem

but they negotiated those and they realized that what they did in the first pass maybe wasn't the end game um right out of the gate but the end game if they couldn't achieve it at least get as close as they could and apply some compensating controls to get get even closer um and so what they did was they had a very good discussion through multiple divisions and departments with the security and the risk officer to then say okay here what we collectively agree is going to be our path forward and where we're going to take exceptions and where we're going to you know cross things off the list and they effectively move from being reactive in that oh

you know here's the latest concern or something and reactive into now being proactive and what they realized was that uh the best way for them to maintain it because it's one thing to build it's another to actually keep it alive and living and breathing and up to date they found that they needed an inventory but again that inventory had a very different context when you looked at the whole program that inventory for example needed to add asset context when we go to plan a backup solution or a patching solution or reducing risk from the latest vulnerability update um we don't start at the top of the alphabet or the lowest number on the ip address

we want to start with the highest risk assets right so asset context not only is its books its impact operations but who's the owner where is it is it redundant what vintage is it what other levers do we have to pull on it um they found that another key component was the ability to take actions too much of that first road map i was showing you people look again at the inventory today and say let's go grab an inventory tool um and so now they have an inventory but things are still happening while they try to figure out how to fix it when i see things are happening um you know a new new security update is

released and suddenly they go even further backwards in their patching level and the risk level goes up um and so um when we talk about anything management like vulnerability management or asset management it's not enough simply to have read-only data you have to read only data that directs your action and then that actually has to have tools to allow you to execute them um alerting for changes and and deviations and and it's great to know what is happening to the what the asset is but it's also important to know what's happening to the asset or with the assets so alerting and then finally reporting um this is i think one of the biggest sort of

things that many organizations feel is more of a throw in i throw in that okay great i got a dashboard i can look at a pie chart well in reality when we're talking security in ot and we're really truly trying to make sure that we are identifying where the risk is and we're reducing and targeting uh the important stuff as opposed to just trying to boil the ocean reporting is a fundamental aspect of being able to say here's where we are today and now here's where we are you know in three months or six months um but also being able to use that reporting to drill in and say i think we need to do this or not and to

be able to then pull data to verify or to debunk that particular perspective reporting goes a long way towards justifying the effort we're making but also showing your progress it can also show the opposite for example um if you had a decent risk profile and then something like blue key came out being able to bring that in and see what that did to your to your environment and again with the context portion not just the raw risk of i've got this many boxes and they all need you know blue key i have this many boxes that are of high impact to operations and maybe they failed their last backup right and so you're able to provide

context and say these are the ones that are the highest risk but you can also take that and pivot into your management which is part of my second case study so i'm not going to tell you all that story just yet but they realized that you had a stack and it always came back to the inventory um and so what that looks like you know when you build that and automate it is they're able to have this visibility across their inventory and they pull inventory in from multiple um asset classes os based networking as well as your embedded right and that inventory in the context is what you then are able to mine from that

single user interface into vulnerability data versus configuration or patch or user or network and again this sort of putting all of your data into a single perspective allows you then to provide context at the viewability of the reporting capability so your information comes up um and then as you take actions and go move on the devices they then refresh and update the visibility so you effectively are building a feedback loop right like we're talking about an ot right it's uh here's here's what we're checking here's what we think here's what we're getting back and it allows you to take that same sort of hmi and trends that everybody's built with their different historians and their in the process

and put it now on to security uh system life cycle management and day-to-day maintenance um and it it's it's it's remarkably effective at taking what is uh supposed or what is a hunch in terms of risk and putting real context to it so for example um i walked into a facility and the plant manager said you're here to do an assessment isn't that great you're going to tell me that i don't change passwords and i don't patch anything how does that help me it's a great question whereas if we have an automated data profile in inventory and we bring in the context of saying here's my asset here's the missing patches here's the known vulnerabilities here's the

compensating controls we've done on it and here's how we produce the risk now i've got a completely different perspective um and it's all dependent upon building that ecosystem and it's a change from that first generation of security because we're going from siloed tools that collectively give you a bunch of functions to at least pulling those tools in together and having them paint a picture of what we call a 360 degree view of the asset so what do we recommend for the second generation approach obviously to approach is a multi-dimensional or multi-disciplined program if you're looking again at what you want to do for inventory tools or for endpoint management tools or for user accounts or even just a backup

selection what exactly do you have from your inventory that will help you decide how you should probably architect a backup solution maybe some of your inventory um takes some options off the table or puts other ones in um that inventory plus that analysis of the backup will help you to design your recovery uh process not all assets you're gonna get a full backup daily with a weekly off-site storage so again you know as you build that multi-dimension and have the data from the different senses uh you'll have a much better approach and a much more streamlined uh result uh always practically work with vendors to fit into current uh requirements or remove barriers um many people say oh yes well oems they

don't allow agents well the oems have a lot of their own agents agent isn't a dirty word in ot working with the organization though is in the example i gave earlier to say look you know there's 1200 known risks you can't just reduce 400 and let me live with 800 we need to find a way to get better aggregating data sources you've invested lots of money you also have a lot of people with tribal knowledge all of that stuff can come together to give you a better view that better view allows you to have better priority prioritization and triage of your risk and it will allow you to the fourth point or fifth point here incor

incorporate the management portion of the planning decisions um i can't say enough about the fact that you know there's lots of ways to understand how many patches are missing or how many users haven't logged in or dormant admin accounts there's a whole other way there's a whole other um level of expertise and work required to actually management and the other thing that really that i just wanted to sort of underscore is the spanish armada note there i watched a discovery channel thing the other day so suddenly i'm an expert in in the you know the spanish reminder now but what really struck me about the example was the spanish armada was much smaller in numbers and in size but if you didn't

know the story what they did was that they built all their cannons and their uh ammunition as sent one central place and then outfitted all their ships with the exact same stuff um many of their opponents didn't do that they could build whatever size and shape they wanted and so when they came to battle uh the spanish armada was able to crank out two and three times uh the volleys of of cannon fire in the same time that their opponents were getting maybe once it was like three to one or four to one and so the smaller um smaller numbered and smaller sized ships on the armada actually took out significantly more opponents than you might think they would

simply because they had an automated uh and and streamline process with consistent tools made a huge difference um so and one of the next thing we need to do organizational principles now there's lots of things in here and i'm not going to go into all of them as i mentioned i borrowed some slides from some others this is part of more of a larger workshop that we offer where we talk different um categories of how you build different things and almost every one of these you could probably spend about a 30 or 40 minutes session on i'm going to mostly drive into think global and act local but these are all key components if anybody wants the slides after happy to

share them um but really a program is multi-disciplined multi-department and requires you know understanding multiple of these and again the data the automated data drive from the from the ot environment is a key component and contributor to almost all of these in different ways and spaces again i'm going to for the sake of time make sure i focus on the think global act local so um what does the team look like when we get to think old black local what sort of things are we looking for um this is from cyberseek job database and i pulled this up just for interest i mean a lot of the cool stuff and don't get me wrong i love

the passive tools and threat hundreds i think that's really super cool stuff and they're you know they're worth every penny of what you can get but it that's really like the the the scouts that run on ahead and find that the giant army is waiting over the next the next thing i mean it's valuable and it's useful but we really do need to sort of uh minimize our risk right and so if you look at the cyberseek database um these top three categories represent let's say 30 to 45 so about 80 percent almost 80 percent of the current jobs are otherwise considered management so endpoint management configuration management av updates the care and feeding the stuff that you can design solutions

if you know your inventory but even better if you have automated tools you can actually have a running running start and a fighting chance at keeping up with them and so as we get into this i'm really advocating you know that risk isn't just about being aware of it or monitoring when something happens it's actually being proactive and reducing it as well and so you know when you get into this environment on the ot team you're probably going to need to pull multiple people in and i'll show what that looks like here what we call think global and act local we realize that a lot of those people on the previous slide don't have the ability to to devote this

as a full-time job and so what we need to do is we need to be able to leverage what we do have and so the way that we've talked about our automated approach um to inventory and context coming up to that single pane of glass you can you can layer that in within a single environment or you can do it across a multi-site environment as well the concept is true in both cases and what the premise is is that at the plant level we make sure that we have that automated inventory to cover all three assets of of classes of asset os based where your traditional risk comes from your networking gear and even your embedded

stuff and you pull all that data up in an automated fashion and then you add the context you add the ot context is it critical is it critical asset to your safe operations you layer the national vulnerability database over top of that inventory so you're doing a full analysis of risk against your inventory and offline status and it allows you to dive into the data without touching the systems um you add the other context to the things you've invented you invested in like your antivirus waitlist in your backup you take all of those concepts together and you bring it into a single pane of glass at that smaller centralized team this team can then pivot into any sort

of function or analysis that you want from life cycle planning how many windows 2000 server 2008 or windows 7 boxes that we have to go get my extended subscription support from microsoft um what does the vulnerability uh database say relative to what we have let's drill into my plc's how many rockwell plcs of exploitable firmware we're going to do an upgrade in this facility or take a turnaround can we flash some some some firmware while we're doing it and so this aggregated centralized data allows a small specialized team to be able to be very very exacting it turns to for me it turns from a shotgun approach to a sniper approach we're able to now say look this is my

risk today or my concern or let's just look at a dashboard and drill into it and so what happens is that this higher level of capability um is is streamlined and centralized in the smaller team this team can then do the research and the planning once with a small team with empirical data and they can take those actions again using the automated you know environmental infrastructure we've been talking about and they can deploy those actions in an automated fashion now the really cool part is that you can stop short of actually doing anything and so one quick example um i'll give you is um we had a client the same client that the first case that it was about

they were asked on a particular saturday afternoon by the ciso if they had been done the task of removing a certain foreign built antivirus software um and they haven't and so the cso and said well i'm reporting the board on monday so i better have a report that it's completed by monday morning it's saturday afternoon six coal fire generation facilities four different states um and what they were able to do was look at the single view across the six sites and say filter on this particular piece of software and they came up with 146 endpoints that had it and then what they did next was they took 146 endpoints and they sent the commands to those 146

via the agent that was on those devices and said uninstall this piece of software but they set a flag and the flag said make this an offer so what that meant was they then went to each of the six sites and they sent them a printed list of their list of assets and remember we put the other context in there the assets criticality operations the assets physical location which unit which rack which rack room where the owner was and a tech at each of the sites was able to walk down to each and every one of those assets and see in the system trading offer to remove the software and so the ot tech at each of the sites

then accepted the offer watched it remove rebooted if it needed it saw that it came back up checked in like it was supposed to moved on to the next one so we had this think global this one small team identify and automate as much of the process as possible across 100 and almost 150 assets but you had ot oversight that last mile and so this particular organization was able to identify and remove and send that report to the cesa like you'd ask within 90 minutes not days weeks or months like who typically happened in ot environment 90 minutes later they had it and they verified it that's what this automated you know infrastructure and that insight allows them to do and

that's what think global act local is but i want to show you a bigger um example because it compounds it's not every day that the cesar phones and says uninstall your software let's take a more real day-to-day scenario client company profile large operating company thousands of miles assets you know large transmission distribution company distributed team of scada all throughout you know north america relative to the various uh sites along the lines uh larger ones obviously were man smaller ones had people at least within so many minutes or kilometers of a drive um over time you know due to mergers acquisitions investigators instead of thousands of assets comprised of varying os's and vintages and all the assets were spread across

hundreds of physical sites um so what happened was um and the automation didn't work we're supposed to share the cool part for the right but go ahead and read ahead the without the global act local program and this this is actually the same client you know a couple different times once before they deployed this automated months after so it's a very good before and after sort of uh analysis when the when they first had a result uh a security result uh come out i.e blue key as an update um a security update that was required um without this sort of insight and this automated uh infrastructure they pulled dozens of people in spreadsheets into a single meeting

um they ended up having multiple meetings in the next few days just to sort of understand which order they were going to go into the meetings stayed up you know but they weren't as frenzied or as well attended over the next few days as they got to the execution phase but they didn't really start with a concrete understanding of scope they were kind of discovering as they went um they also absolutely had to make many guesses and assumptions like system criticality owner location i distinctly remember at one point in time they were looking at a list of ips they weren't sure who owned it so they actually remote desktops to those ips and went into the um start menu to see

what programs were running on it if they saw maximo for example they go talk to the maxwell people and ask if they knew of this particular name like it was it was really um you know unfortunately typical but very difficult for them to figure out what they had and where they were going they started manually patching um and they also started manually tracking in those same spreadsheets and combination thereof um the total effort and this was rough because you know at the beginning a lot of operational staff were pulled off their day jobs and then you know as they started to move down the road someone sort of peeled off but in general uh they estimated that they used six

full-time equivalents for about ten weeks um and that overall they spent collectively with the management overseeing reporting and tracking and providing air cover about 2500 hours to reach about 90 coverage and it was either patched or not and that was it um once they've gone through this they went and deployed the technology a few months later something similar came along where they have a need to understand and identify and patch and roll up multiple places they then had a very different starting point they went into the dashboard they found their 360-degree view of assets they were able to see instantly which assets needed the patch and which didn't they could also sort that view of assets

by location by asset type by criticality by owner by vintage by os by whatever so their kickoff meeting was a little bit different they had a small set of key personnel that same sort of global team um and their first task was they used the agent to disable remote desktop on all the endpoints this is first thing monday morning when they decided to do something send an email out to the entire field staff saying if you need something enter a ticket we'll open it for three hours we'll turn it back off when you're done and said the ability to put real granular immediate control in place while they bought the time to um deploy their patch uh

program um they then while they were sitting there in the meetings during the planning they'd also cued all the files locally on the endpoint so in this scada distributed environment with low bandwidth issues they had the architecture set up to slowly trickle files across low bandwidth so give the files a head start so when you come out to actually do something they're already there for you again automating as much of this as possible they then did a combination of stuff that was in a data center or a manned facility or newer vintage of assets or less critical systems or redundant systems so really sort of test and see what the the environments were and they were able to triage which ones

they wanted to patch first versus last either by criticality to operations and needing to protect it or fragility of the system and being able to feel more comfortable what they were doing on what type of asset but it went from a patchwork of i think i can do this i think i can do that to know here's empirical evidence as to the system criticality whether it's it's a manned location um whether we have had previous issues with it or what have you so um non-patch systems though they were also able to leave the remote desktop turned off if need be uh they could disable the guest account from initiating remote desktop so they in all cases were able to normally

deploy the patch um but it increased the security profile and the ones they couldn't get to they at least were able to minimize the risk uh all the progress as they went was reported in the live dashboard so as you patched a system it came out of the big long laundry list of assets that were in scope and so as that list reduced and your overall red green yellow gauges got more green you could see the actual progress day in and day out overall the total effort they only needed three core staff and then field texts here and there they were put on notice hey you may be asked to drive out to the site tomorrow and

reboot the system and make sure the patch took and off you go but those techs were able to roll that in because they had a heads up they knew what was happening it was all scheduled overall they took probably just uh just a little less than a 75 percent savings in the hours spent so you can see there they spent around 600 hours to reach the same 90 but the other 10 were also locked down so you can see how you know this central team building the solution or a path forward once being able to take that path and have it match criticality system fragility ownership um you know it's a huge difference if i show you today all your risk in an

environment i'll show you 30 000 risks understanding that you probably want to take that critical risk on high impact assets that maybe failed their last backup is your 50 of the 50 000 that you absolutely need to focus on that's what this starts to build and so we're really advocating that as you continue to evolve you know know what done looks like and look towards automation it's the only way we're gonna fight it so what are the benefits well the cost is greatly reduced i mean you just look at the 2500 hours versus 600 the lost opportunity of the other projects of those operational staff were not doing for 10 weeks central view leverages scarce resources

you can only have so many people that understand that get it in ot security and there needs to be on that central team both it and ot because it can tell what the risk is and what sort of things can be done to maybe reduce it and the ot can tell you whether or not you're allowed to do that on that end point um and so that the other thing about a specialized team with global reach is it reduces duplication one of our clients today is looking at 6 000 assets in 57 countries around the world and so they're able to take that and really supersize this scaled approach towards you know smaller team having a much wider impact in an

automated way from that central visibility the other thing that's key is that ot safety is insured as i mentioned you can have really granular control over what you actually do i can just i could stop the central team at the analysis the research and even the preparation but the final mile is always handled by ot should you should you require and having that same group at the top means you can have an itot lockstep decision making process in support of exactly what you are or not doing um providing context this is key not all assets are created equal in ot um you just ask any operator which system that when you see the alarm for

it you know that your phone at home and saying you're not gonna be home for dinner versus which one can you live without for a few days or weeks until monday morning and they all know that it's it's there but it also aids in the decision-making process and then tracking um i always used to joke that in security is a consultant if you do everything they tell you nothing will happen right it's not really the the reality but but roi and and understanding your true risk whether that's purely from a risk perspective or whether that's from a budget and a staffing justification or prioritization perspective is key one of the things that i haven't talked about but this

type of approach um effectively negates the need to do an annual paper-based exercise of having someone external come in and do an assessment and hand you the list and hope you you know improve between now and the next time they come you still could have somebody come in and look your stuff to make sure that you're you're you're paying attention to doing the right things but it's near real time so the client i gave the example of with the the any bar software they're updating their data every 15 minutes their inventory and the risk associated with it is never more than 15 minutes old it's a completely different game than what we're typically playing in lt today

there's a little bit of the dashboard we're nearing the end here i want to leave some time for some questions um but you can measure this is across the csc how they're doing relative to the different controls in csc 20. um and options this was just talking about we can't boil the ocean we've seen clients do everything from let's first look at our os based devices or let's first pick uh so the one client that is in 57 countries they're just doing windows everywhere that's what they want to do first they're going to come back later do networking on the contrast we have a global pharmaceutical client that is doing um by as far as their

concern from a financial perspective the top five facilities um that are the most important for their financial bottom line and then the next five and the next five until they roll through you can manage this in any way shape or form we're not trying to boil the ocean and say throw everything out and start from scratch and build an entire program it's understand what that input looks like and grow into these in whatever fits your organization benefits of this twofold just quickly on the project side you end up with multi-division multi-department involvement it is a way to draw people together there's so much discussion about itot convergence the reality is it's a team uh nit it's a

team you cannot have deeply siloed security practice in ot because uh there's such a wide potential impact across so many operational aspects um so you get better support and upkeep during the project into maintenance mode um quite often a project team comes in deploys the technology high five themselves and leaving the people left around are going well what's this who's going to manage it right so you get much better buy-in uh and you can anticipate and manage cost if i can tell you in real time exactly how many assets you have and how many need to back up or how many need this patch you can very clearly either with your outsourced organization or with the project based um

support or summer interns you know you have exactly 327 patches to apply or you have exactly 83 rockwell plcs with explodable firmware you have definitive scope to manage now the maintenance side you get much better increase in accuracy visibility security compliance across the board um and so in summary risk is on the rise it's not going to go away we continue to increase our technical debt um and there's growth in the technology and the exploits an absolutely programmatic plan is required siloed tools and and individuals or disciplines just aren't going to aren't going to cut it and all roads as far as working certainly this you have to have some sort of comprehensive real-time inventory i mean

the the reality in ot in any environment is that security is a fast moving thing and so the better we can tap into what the current status is as changes either improve as we lock them down or or devolve as new risks come out is key most of the cookies how we got here um that puts a few legacy challenges like disparate tools and styles of responsibility so really the second generation of ot security uh would realistically and significantly be improved if we just adjusted sort of the way we looked at this slightly uh that think global act local hopefully is is of value to some people be happy to talk more with anybody who

wants i think we got about five minutes left i know i am between you and uh the weekend but that is what i brought today i thank you all for spending some time uh there is a chat here i've got a c uh on the chat side i've had some comments as we went which was interesting uh thank you for the for the feedback uh but uh in the session here if you had any questions please throw them in here otherwise i thank you everyone for coming and there's my contact information if anybody wants to pick this up um after the fact during the next week i'll be more than happy to share some more case studies and more stories if

anyone's interested thank you so much for the time and thank you besides for the platform and everybody spending a little bit of your afternoon