ICS SOC Pyramid - Gabriel Sanchez

[Applause] all right all right so let's jump into it so IC soft pyramid that's what the presentation is going to be about so IC industrial Control Systems you're going to hear me talk about this as uh critical infrastructure so think about power plants chemical plants think about any ufacturing oil rigs power utilities and then soft Security operation Center so a lot of us are familiar with that term right and then pyramids this is not a pyramid scheme I'm not going to be selling you anything but kind of our traditional pyramid our tiered system so big part of this presentation is really kind of defining the differences between an it soft which a lot of us are

familiar with and then what is an actual ICA then how do they coexist together right so we're going to be discussing that at a high more so real quick the agenda we're going to hit on approximately about 10 different items I'm not going to read this to you some of them will take a little bit longer than others and so we're going to just jump right into this safety moment so safety moment is really big in the culture of critical infrastructure so we try to do safety moments at the beginning of uh either a conference uh pretty much every day at the beginning of a meeting right the beginning of a presentation so safety moments is really really important in

the it world we think about the CIA Triad right confidentiality Integrity availability all those pieces are important in ic which again I'll kind of say critical infrastructure sometimes I'll say OT operational technology all those are important probably even availability is extremely important especially if you want to make sure you're getting power to your home but safety really does Trump all of that right I mean safety no matter what is going to Trump all that and what we're seeing in the IC world is there's actually this is where actually cyber meets the physical world and so you can actually cause harm to individuals you can actually you know have people without lights without water um we were

discussing yesterday at dinner and uh you know someone was bringing up you know do you remember the last time that your thing got hacked do you guys remember the last time that happened most of us can't necessarily pinpoint that time or really actually even remember remember it because it's just kind of a second thought you know the money is going to be there eventually but do you actually do you remember when Colonial pipeline happened right I mean if you were on the East Coast you definitely remember it I mean mass chaos right people you know running up with different bags trying to get gas we all remember that clo pipeline now even though technically you know from what

it's reported the OT side of the network right the part that controlled the pipeline wasn't necessarily breached there was a ransomware on the it side and so it still had that same effect that same impact of shutting it down right and so um I think it was actually the billing system and some other stuff that that got breached and that still had a impact so that's that's what we're talking about you know how do we form an operation Center that's going to protect things that have a real life consequence and try to do it in a way that hopefully you know we're not going to make mistakes right so the quick safety moment um decided to kind of add a little bit of a

cyber twist to this right you know in safety moments we do cover safe driv all that kind of stuff obviously that probably occurred because you guys are sitting here today so um I'll presume that there was safe dry we that that that happened unless you were late maybe you were going a little bit faster but this kind of cyber twist to the kind of the safety moment that I want to kind of talk about a little bit here comes from some massive campaigns we've been seen in 2024 and so my team and I we've really been seeing this kind of uptick where they're breaching it networks in the critical infrastructure and obviously the next next logical step once you reach it is

breaching the OT side of the house right so just a quick safety moment on 0365 account takeover so if you or if you know customers or individuals that have Office 365 one of the things we've been seeing large campaigns on is attachments getting weaponized with the link and so this is not like a like an executable that gets dropped right on the computer system and automatically detonates this is more of an attachment that has a link and that link takes you outside of the network and it's going to take you to a site that mimics Office 365 okay so this is basically like credential harvesting right at that point the thread actor is going to take

your username it's going to take your credentials and then it's going to that to the real Microsoft Office 365 site now of course there's a a large sense of security with a lot of the organizations because we' got two-or authentication right and all of us know that two-or authentication can be have right right all right good I'm glad you guys allowed um so what the threat actor does says Hey user let me just ask you for that two-act authenication and now I'm going to provide Microsoft that two Factor authentication right at that point they now are in inside of your 0365 account and then the beauty for the thread actor is they can now add their own

device and when they add their own device into the Office 365 to now be an additional phone that could be used for two Factor authentication for the future whenever they want to use it they can add it in without Microsoft even requesting to reauthenticate or any type of additional information so the phone is just now added into it and so an instent respon team would go in um to to this organizations that are critical sector and they would go and they would check this because there was assumed breach all of a sudden they see some additional iPhones and phones that are not of the user right so that's a problem so safety moment here is if you

are using no 365 go to my signs. microsoft.com and make sure that if you are doing tofa which focally you are that is just your phone that's on there not other additional phones that you aware of and of course there's a QR code up here which I recommend nobody scan right because that's also not safe so let's jump into uh my bital real quick so I've kind of been in the game for about 20 plus years started in the late '90s uh for anyone that still remembers Circuit City I was you know fixing computers over there and then from Circuit City move up the ladder I end up going to Gateway again fixing computers and then I you guys remember when there

used to be like computer guys like in every single corner right the computer guys and changing out motherboard seemed like every city every state had a computerize at every corn so that's kind of where I really started you know jumping into I it was just like it administrative type stuff I mean we weren't really calling it cyber security uh back then but messing with Linux messing with the different operating systems and then from that point I went to a DOD contractor um they were doing a um missile simulations there I was not programming but I was doing their it admin stuff and again we weren't calling it cyber security this is really early 2000s setting up firewall setting up

vpns trying to segment trying to just make it secure right after a couple years of doing that I kind of accidentally fell into IC right industrial Control Systems infrastructure I gave you guys some examples and the way I accidentally fell into that was with my local power utility they were actually serving about 500,000 members over sever different counties they were serving power to my home and all of a sudden it was around 2007 they popped a post up saying we're looking for someone to do just cyber security the description was like super brief they just pretty much said this is a new position that they're starting had an existed before and so I said oh

that's pretty cool I mean it's the company that provides my power right now go help like do some stuff so I went there and of course I'm thinking okay I can spend two to three years here and I can do really well you know with security and and get them everything they need and then they're going to be good to go and those two to three years turned into a decade right because I I realized very quickly that it was a huge difference securing the operational technology right the the the IC the critical infrastructure and it was entirely different and it was extremely difficult to try to do both and you kind of had these um you know

competing outcomes OT care all about safety it care is all about you know you know something's wrong switch off the computer you know re you know do a reset do a reinstallation of the operating system so the mindset was completely different so I spent a decade there we had about 50 different substations again like I said about 500,000 members our employees was only about 500 individuals and so after a decade had an opportunity to go to Deutsche Bank went to Deutsche Bank started running the operations center over there and doing incident response the employees went from 500 to 80,000 and so now we were protecting 80,000 employees sometimes from themselves but nonetheless 80,000 employees right okay

this is been reported so that's how I'm going to say about that and then uh decided to go to 1898 which is part of burns McDonald right it's kind of that Consulting arm that cyber security arm and the reason why I went there they decided to start an IC soft IC specific focal soft which makes sense for an engineering company that's already building the substations and building all this stuff right doing things on oil rigs and so forth so Joy now been there for about 2 years and that's pretty much kind of 20 plus years summed up um in like two minutes so let's kind of jump into the presentation kind of the beat and potatoes of it high level Network

diagram so before anyone freaks out especially super technical people I put this slide on here only because I knew if I got too detailed everyone would start just looking at all the pieces and then they stop listening to them so I only put this up here to show that there's by stating the obvious there's a distinct separation between it Network and OT Network there is a huge difference there is a different way in how you have to manage it from a cyber and Security operation Center and yes there's pieces in here that are not showing but in the '90s and early 2000s you had your it Network right and everyone was focused on the perimeter

right put up your firewall put up the BBN bpn it's kind of like that M&M model right kind of hard on the outside super soft on the inside right and you assumed okay we're good we protecting then they started to realize the industry holy crap we got to actually put more stuff on on the inside assume compromise you know uh get quick to detect get quick to respond and then on it Network that's where you start seeing edrs in point detection and response crou string DLP data loss prevention you know antiviruses started getting a little bit better IDs is intrusion detection systems IPS is intrusion prevention systems start see all these things being put on the inside

your DMZ right all that stuff and sure the was part of it but now you got a lot more on the inside that you're using to prevent to detect to respond right and guess what the OT around that time was thinking well all this has happened on it OT Network and OT individual started thinking hey we just you know what we need to do we just need to have a strong perimeter have a strong perimeter let's just set up a firewall we'll be good so it's almost like you know we're we're you know we're trending like 10 15 years behind right but it does make sense because on the OT side of the house critical

infrastructure you put up devices that are supposed to last 15 20 25 years and now we're at a point where when these things start dying off these different devices these plc's programm logical controllers your hmis human machine interfaces all these things they're now getting replaced with things that are more interconnected right they got IP addresses which means they got a way you can get to them and then OT and the industry started thinking hey you know what maybe just perimeter protection is not enough anymore right it's almost kind of like dja Vu that's happening and so now the it network is basically like the internet to the OT Network and this is where you have to start trying to

protect the OT network but I'm going to talk about some of the difficulties and some of the problems that are happening there so with that let's talk a little bit about the traditional sock skill set set the Security operation set I made this super brief right because I'm not a a believer I'm just having an entire paragraph on a slide to read to you so this is not all inclusive so don't anybody get offended but your traditional skill set is going to be obviously a strong networking background right tcpip you're going to want to have potentially some good offensive security skills right some of the tools metlo uh nmap messes Blood Hound Maybe cats right

your M cats is going to go to the Elsas process is going to take your password right outside of the memory and you're going to do all this cool fun stuff then defensive security traditional skill sets on the it side right I named a bunch of them idrs IPS is ids's uh and so forth this is kind of your defensive security the skill sets that your Security operation Center needs to have and then there's different job functions again not all inclusive but you're going to have people who are specialized in incident response you are specialized in engineering right how you do the proper segmentation uh how you do the proper vling how do you set up you

know the the appropriate Network that you need thread Intel thread intelligence getting those feeds right um looking at virus total comparing ashes you know what's infected what's not Automation and then your your standard it analyst right that it analyst now you have this tiered system so that's where we start talking about that pyramid right that pyramid where the the biggest part of the bottom of the pyramid is your tier ones so you got tier ones and then you have escalation points based on the different playbooks that you have going into tier two tier three and then you have your skem and this is traditionally how your it stock is going to work and it does make sense in many

ways uh definitely makes sense to the to the executives that are having to um you know you keep cost in mind right to keep costed mind so makes sense just have you know maybe tier ones 1 to three years out of school um they're building their skill set and when you need someone with higher exp uh expertise you start moving up the ladder right and you have less individuals that have the the 10 years experience the 15 years experience right so that's your traditional uh soft pyramid if we look at this piece okay basically that's what I'm explaining your tier one two three and then you're SME your expert that's sitting at the top now I put here

customer customer could be different things if you're working let's say for um a bank or you're working for some other critical infrastructure power utility whatever might be the customer is that organization that you're protecting I mean even though you're still one of the employees you're treating the other part of the business as the customer sometimes if you're doing a manag security service right then it becomes more obvious who your customers are it's all the customers who you're on boarding and all the different clients that that you're protecting and you're feeding all those things into like a sin for example but the traditional model for it soft for informational technology is you typically again not all inclusive but

you're typically going to have let's say some type of engineer who can do uh thread detection writing who can do tuning for example on your in and is going to try to feed these appropriate alerts to the tier one so they're going to give your informational your low your medium your high all these things are being fed into your tier one analysts now there is a little bit of a of a delay that happens here and that delay is how fast can that engineer that thread detection engineer the guy who's who's working maybe on Spa on Arc site how fast can they do the tuning because sometimes that low gets adjusted to a high sometimes that high gets adjusted

to a medium sometimes 10 lows happening within a certain period of time can be considered some type of medium or higher so there all these different types of use cases and things that have to happen and your turn on time typically I mean you guys know how it is rep passion right when we're talking sometimes weeks and you got to test it so sometimes it's a weeks and it's months to feed that straight to the tier one you also have that situation when it comes to um instant and response at what point are you activating instant response so many times you're putting it in the hands of a tier one analyst to ensure based on

their playbooks that they know when to engage incent response and then we have threat intelligence same thing I mean we do try to automate you know the hell out of it you know threat Intel feeds connecting API doing all those pieces but you still have someone either thre Intel or in some type of role making sure that API works or making sure that the data is getting enriched so that analysts can look at right providing context to is this IP bad is this hash value bad right is this thing blacklisted all those pieces become important but again there's a certain amount of delay that's not necessarily like like a gigantic problem I would say I would say

it's an accepted risk it's kind of a way of uh of doing business and it works pretty well so so I'm not trying to change the it sock pyramid in the way that we know it right the problem that we're having and let me touch on a little bit on the IC skill set and again this is not necessarily all inclusive on the ICS skill set yes you still have all the networking and and traditionally most of the time you have everything that's going to be it and now you got to compound that with different things like mod bus right different protocols dmp3 right your Bob is going to be send sending things in clear uh

text for example your dmp3 is going to be uh encrypted you have uh hmis human machine interfaces you have to deal with you have things like a skada system where um a SK system actually looks like sometimes like you're being attacked and the reason why because I remember when I was working at the power utility you had 50 substations that were receiving this probing information back and forth from one that this one centralized system right it was actually more about what no ska works that way right skada sending the commands from a centralized place and it's sending commands out to all these different substations they have to talk back home right so it's almost like

a nice little bot net that's how a lot of the detection tools see it but it's not it's not a bot right so there's a different skill set and then offensive Security in ic talk so I did a straight copy and paste on this from the it slide because I want to have a little fun with this so I'm going to go a little bit slow on this piece because there's some members of the team that I'm part of that are specifically part of the OT I seen us pen testing and inst some response team and they are probably freaking out right about now and they want to come up here and tackle me because they know that

there's no way in hell you can do Meto nmap nesses Blood Hound mimic Pats that type of offensive Security on an IC Network I decided to leave it up there to to screw with them because I I knew they would be like freaking out and they're somewhere over years so we see them freaking out um that's that's the uh the OT pent testers Insurance long Ste but there's a big reason why you can't do this stuff the plc's the ska the HMI human machine interfaces these things start to fall over literally I mean metlo if you're going to try to run some type of exploitation on some device you actually might cause some physical consequence right you're

going to most likely bring something down so pen testing is extremely hard in OT it is very manual sometimes you actually are better off creating like a mockup environment or something that is as close to like a miniaturized basically version that's representing what that organization is running and then you can run you know all you want on that particular system right because there's not going to be a physical impact to whatever it is you're protecting I mean imagine being on an oil rig and then trying to run this kind of stuff I mean if none of you guys ever been on an oil rig first of all just real quick kind of a side tangent here

on the oil rig you have to go through what's called bosier safety training right that training is they'll put you in this mockup kind of pretend helicopter in a huge tank right you they're going to put your seat Bel on really tight and then they're going to flip you over underwater and you got to pause for a moment and then you got to make sure you're swimming out and that's part of the training you go on the oil rig and fly on the helicopter and normally when you're going to go do maybe one thing you're going to end up still spending the night there two days because you got to wait for the next helicopter coming to be all right and so

you don't want to be I there's nowhere to run right that point you're doing offensive security especially something that so you have to be really really sensitive to that and then defensive security defensive security also becomes really difficult as well because um you can't necessarily just put agents on anything even if you wanted to so some stuff uh the industry is getting better about creating agents that are OT specific right but there's some things like the PLC some of the hmis there's some pieces of Hardware it just agent's not going to go on it right you're not it's just not going to happen then you're also contending with regulations some of the regulations you start

thinking about ner SI when you're low medium high you don't know about ner SI God bless you um but you know there's a lot to it so you also don't want to mess up uh you don't want to you don't want to mess up their NS you know audit and and and the pieces that they're they're saying that they're attesting to right so that becomes really important so sometimes your defensive mechanism is hey we're going to add a sensor into the OT Network and now we're going to span off a switch and basically we're going to collect copies of traffic off the switch and then we're going to monitor this in a passive mode right and a lot

of the IC community and a lot of the critical infrastructure and people that are responsible for it like the plant managers and all this will get fairly comfortable with this because it's hey we're not doing anything active this all going to be passive right and we're just going to do a spam for it and because that spam for for the most part is not routable it does help with a lot of the the compliance and regulations that you have to um abide by depending on what you know sometimes it's TSA there's different regulations that are out there depending on what you're trying to protect but you are at a little bit of a disadvantage so you have to have that

right experience to know that you're only going to get potentially right if you're just seeing copies first of all you got to make sure you're spanning in the right spot right and now we're assuming like there's proper segmentation that's being done architecture that's a whole other presentation Inn of itself there's certain architecture segmentations the right place to put a sensor to get the right amount of traffic and then you have to have an individual that can look at that and say I have three pieces of a puzzle that is like 10 pieces that I can tell you with a high certainty that that is most likely an elephant right and you want to with some high probability be

right about it but that's going to take a lot of experience and knowhow on the engineering how did the different devices work then there's job functions incident response CIS for critical infrastructure while way engineering Intel Automation and analyst right A lot of the same roles but not nearly the same skill set right how do you forensically grab an image off a laptop traditionally 19 a lot of us know we can we can plug in devices we can FL hard drives we can do things like that in the OT I world that becomes a lot more difficult maybe it's more about pack capture or you have to have a different type of hardware and device to follow

side engineering is is also different I touched on that a little bit your Intel right your thread actors that you're contending with is going to be different that are going after OT right they might be going after safety um you know Safety Systems they're going to be a lot more covert many times and your automation your automation is great but is also limited automation is great to correlate data that you have but not always necessarily to again do something automatically on a physical device right and then you have your analysts so we have the same tiered system with the analyst Tier 1 2 3 and four and what we start realizing here is this is kind of what our IC pyramid

right that's the OT Network you didn't catch that that's your operational technology your IC Network and what your pyramid actually really looks like is flipped upside down okay and I would love to say that you know the team that I'm part of that we we had this Grand idea this is how we started but this was not the case this this really organically kind of happened and this organically happened because of the demand of the customers and the individuals the stakeholders that are responsible for things in the industrial control systems like power plants water utilities pipelines all that stuff they are not willing because the risk is too high they're not willing to put cyber

security operation centers as in the hands of someone that is a tear War right I mean that's that's the reality of it right so they want the front lines of the Operation Center to be people who have experience in those different areas is that know what OT is that it worked in OT and so that's kind of your your front and center and so now instead of your it uh model where the SME or the tier 3s are activated from the bottom up and instead your SME are at the top and they're using the threes twos and ones as a support mechanism right it's a support mechanism to help get either a particular uh project accomplish or a

particular task accomplish to support inant response to support some of the triology but not to spearheaded not to lead it right so the famous kind of convergence kind of that buzz word I did my best to stay away from it but I'm going to use it anyways so we have convergence so so this is what it ends up looking like you got your it sock and then you got your ICS sock right completely uh flipped C here so how do we how do we deal with this okay one way to deal with it and it's I'm openly going to admit a lot of times this is not necessarily obtainable or sometimes unrealistic from a financial standpoint

is you just get two entirely teams two entirely like dedicated teams for it two completely different experts and everything for uh for IC right with apprpriate skill and so you're saying oh great you basically just only just hire a bunch of sneeze and then weall get it but I understand that's not the reality of the situation right um this is cost prohibitive it's not the reality of many people in the critical infrastructure space right and so this is an option for some but not an option for many in the critical infrastructure space right depending on how you know we're talking about small medium large uh size uh critical infrastructure utilities again chemical plants so so what really starts to happen this is

now starting to stem more on the reality of how we start tackling kind of this let's say combined Security operation Center so the reality is many times it right the execs up top are going to say we already have a security Operation Center that it stock is going to discontinue Security operation Center for IC right I mean Sou sounds simple enough but the way you really have to start thinking about it and accomplishing this is you will have to ask individuals wearing many different hats now that's not foreign to the majority of us in cyber at all we probably say it like all the time you know um you know frequently people on my team even myself are asked what do you

do and it's like I don't know depends on the week right depends on the week depends on the month and and it's the same for them right people on the team they're like well this week I'm inent response this week I'm doing engineering this week I'm a you know threat hunting this week I'm Intel right depending on the different skill sets but what I'm proposing and what actually is the way we formed our icsa and we proved to be successful is we start really being intentional about how the different hats are worn right and again this is a high level if you think about an individual here they can have a very high skill set

on a tier four they could be a SME in that IC upside down pyramid right and yet they could just be a tier 2 security Analyst at the it level right you have another example here another kind you look at that middle line of someone that has a skill set right Ian we're looking at Peaks and valleys right we can really when you start getting in detail you start spelling out what those Peaks and valleys actually mean what are they really aing they're sming knowing what plcs are they're sming this type of sensor there being this kind of thing within the um within the IC world right so all these roles right this is high

level all these roles do get defined based on what type of critical infrastructure sector you're part of what if tier one tier 2 three and four and SME mean to you will be different for every for every single organization so that those have to get deped but regards of the fact this is what your team starts looking at looking like you have someone that again you look that middle line you have someone that's uh a tier three but yet from an IT standpoint they're literally put into a tier one and that's fine and then you have someone on the bottom line that is a tier two in the IC world but they're an expert when it comes to it protection so

it's really kind of this and that's why we call it convergence but there's a an intentional intentional way to do this type of converence right so that's where you kind of look at these pyramids unite right and and I'm not going to sit here and teach a math class right but this is kind of two pyramids coming together in this way it's called a octa hedron right prob OCTA because it's eight sides and all that jazz again promise no math classes you know we're not doing Ma B here so let's take a look at this in a different way what we did at I at the ICS do and to make sure we're supporting critical

infrastructure is we take one individual right an individual one by one and you start identifying what that skill set is now for the sake of just not making this super long um and and super detailed again you have to Define what incident response means in it what inant response means uh in the IC world but you have to have those levers and determine where they fall on on that particular period And so here we look at this 3D model but if you look here on the side this is now kind of put in in 2D right you can actually see how these pyramids for this individual pan out and you can quickly start to look at if you

look at this blue pyramid here on instant response they're kind of right in the middle you look at the next right this this orange upside down pyramid on instant response they're AE and they're also AE on Intel both IC and it right and so you start having to map this out for every single individual and then you have to start putting this on top of each other and you start finding out where that gap of that skill set is and depending on what situation you're in whether you're in a in a position to either hire right you're hiring based entirely on skill set not necessar saying hey I need a tier one or I need a tier two you're hiring on where

those gaps are okay now if you're not necessarily in the position to just hire certain individuals and you're working with individuals that are there you can actually have people that are part of IC right they've been they've been in forever and you can bridge the gap some of that skill set you can more easily bridge the gap to it or vice versa you can bridge the gap in it to that OT skill set so the simplest example here is if you look the two first pyramids here the the uh upright one in incident response and the upside down orange one right representing IC being see it is a much smaller bridge to unite someone that's an expert

in ir and I c s to become more of an expert in instant response on the it right and so it's more about the chemistry of the team and the balance of the team and not just saying hey I just wear multiple hats you're really starting to systematically kind of approach this in a more intentional way and then that's where you really start combining um your Security operation Center to make sure that it's appropriately supporting it and IC and then you can actually start having some of your automation actually not just doing all the things we know that it does already but your automation can actually start putting it in the hands of the right individual that has

the appropriate skill set for the the type of alert that it is based on the network or the area that it happens you put it in the hands of the right individual based on if they're you know currently logged in for example they're logged into the Sim they're on shift you're going to get it to the most appropriate in you know person and that is number one when it comes to an organization that's doing IC because too much is on the line right like I said safety is Paramount so too much is on the line and and you have to put um you have to put that in the hands of the right individuals right and so again

put the smeeze there you you build the different skill sets and you're you're not going to have that lag time right that lack time of tuning or identifying 10 lows that should really be a high and so forth so some of you might be thinking this and we've heard the term many of us so you're basically saying become a jack of all trades and a master of none but there's another piece of this whole thing that never gets said and this is where the challenge and all of us in cyber love challenges and so this is really where the phrase should continue right but often times better than a master of one so with that I'll take any

questions thank you

everyone yes the L look like for converg between I and op you have the both Sid on so ideally you try to you try to condense that management to be one that's respons just for the entire op or cyber security for both IC and it Ideal World right and many times it's actually two individuals that just have to come to an agreement and work together but it's a little bit easier right because when I when I show both the pyramids having two entire teams put into one that's very difficult but when it comes to like a leadership role when you're talking about management what we're seeing the trend is it's coming down to many times one individual right it's

very doable that they can overse see this that overarching operations your one slide a little Lin showing you know person who is a say yeah that one that one that is like our environment oh is this really currently right now but our problem is we don't have a man so we're small team kind at that spot right now we're trying to make it all in one and we're trying to decide if we want one manager yeah that was that was weird to look at yeah ideally I would go with one is ideal right one is ideal they just the key for that one person is just knowing where the strengths are and how to get it routed

right appropriately and how do you get your your sin that's ideally one sin because we have environments where hey I got maybe a sin in OT and maybe a sin in it you ideally want this one sin and it's more about attacking it and getting the right people to look at it awesome good question thank you yes question back there do you ever run into scenario is where say corporate has one set of policies and maybe the OT network is unionized and has a different set of policies and how do you navigate that all the time all the time yeah yeah all the time so you have competing interests is exactly like what you're alluding to with these different

policies right um the competing interests but so a couple things that answer that so first when I was at the utility that we found the most success was in inating ourselves into the safety culture and making cyber hey this is a safety problem it's not just some boring meeting you got to go to about fishing Mill so we made it you know part of the safety problem but that really kind of answer your question kind of that okay so as much is regulation sometimes is a pain it is sometimes very much your friend because a lot of times that trumps everything hey this regulation like we don't have a choice this is what we're mandated to do this is what we

have to do from the government and this is safety right and so a lot of times you can drive policies more from the OT and you can get that it side U to really kind of start abiding more by that right because of the bread and butter you know for example I'm at electric utility without the OT Network and the substations there is no power utility right I me it is important right you got to build people get paid right but it's it's really influencing the OT and working that policies into the the it you got to got to push for that change because of now you're consistently going to be competing right competing interests and that's that's

where um some of the executive levels from a cost pers perspective really want to keep that um it soft pyramid because if you notice it's financially it's a lot cheaper to keep a lot of tier ones and a smaller amount of tier 2 threes and S and it's sometimes either more costly or it's a entirely more complicated you don't have to drop a ton of money to to get kind of experts and get these combined but you have to invest a lot in the training you have to be very intentional about how your Chemistry works on the teams yeah great question thank you good question any other questions from anyone oh yes R so uh yeah thanks a lot of

times you know deal with OT spaces you already have people are worried about to buy an auditor doing this of Security Plus uh how do you recommend socializing with those folks to make sure that they're at ease and that the test isn't to cause issues so your question is more about how do you make sure like it's you're not going to cause problems with an auditor when it comes to like doing offensive Security in the OT space yeah uh yeah and and more you know the folks who have to deal with your a you know how do you you know communicate with them to let them know like Hey we're going to be careful with this test you

need to bring in certain tools that not you know this environment but it's still something yeah the biggest key is and we do this for ourselves is we audit we get like audits before the like the real quote unquote real Auditors come in right and then we use a lot of their own language everything spelled out in their language of you must do X and we and we provide that same verbage to say well this tool meets like you just draw that line This tool like an end map potentially and then map depend on how you do it very man just doing this one engineering workstation is accomplishing this exact thing so whatever that verbage is from

the Auditors or if you can use other supporting mechanisms like your cyber Insurance you can say hey my cyber insurance has this list of 50 things this is how I'm meeting it but you're also coming in and auditing me your audit says You must right buide by this and you just have to draw direct lines to their own verbage that's the only way you're going to explain it because if you try to explain the details of offensive security then you you BL them because many of the autors are not going to be experts in pent testing let alone when it comes to an ICS envirment so you have to do like this drastic amount of translation basically yeah

good question yes I'll yeah um the fragility of OT networks and the skill sets needed for them pretty different from the it side of the house I'm finding that you communicate those particular issues to to leaders more traditionally in the it space So it does take a lot of um education right um you really have to so there's no easy answer to what you're what you're asking right so you're you're basically your question is how do you communicate what has to be done in OT right and you communicate how something can effectiv be done how do you communicate into the it space right basically and so now I'm repeating the question for for the recording as well

so they can hear it um you have to get into the details there's really no way around it so you individually have to say here's how this endpoint detection like EDR let's say crow strike for example here's how Crow strike can work in OT right as well as work in it but but you have to use a lot of use cases you have to show where it's worked you have to show that you've done it in a lab and so beyond the discussions you have to go and and do the extra effort to actually prove it right you have to prove one by one the biggest win for us in cyber security was explaining the

sensor and how the sensor was passive what passive actually meant what spanning meant and then what spanning means that you're only getting a copy and you would have to explain it in a way that hey if this sensor completely gets compromised that does not mean it's an automatic compromise of the operational technology because it's a spamming right it's a copy the packet it's not routable so you're not going to Route it so you have to break it down that way and you have to make it super visual you have to make it very Visual and then you also have to use verbiage that's outside of your organization so half the time when we're making a new

space we trying to get something across I don't use my words I don't use the words of the team I use words that are if it's a regulation or if it's anything that's out there that some other study has been done I use the words of other organizations that say this is best practice this is what accepted right because I'm not it's looked at that oh maybe you just you just have the agenda to get this done right or not do it right because it can be vice versa like this is very unsafe to do and so they might look at it is it's you saying that it's unsafe but instead you take yourself out

of the picture and get you know 100 people that got together 100 really smart people that got together and said this is best practice that's the words you want to use to bring that into the conversation yes do you have much experience or um whenever you're want to bring like racking cloud service providers equipment in the no space because ith like that yeah lot of pain maybe notru but maybe like where there's manufacturers that want to know produce Goods it's not but it's only a matter of time I think it's so Cloud's getting really hard away from proof of that is like ner sit in January is now starting to addess Cloud they're finally coming around to

the realization that in critical infrastructure even in electric utilities and power utilities cloud like is a reality and so um yeah introducing it it's going to be really tough I mean you really have to lock down like if you think about the Purdue model right those five layers of the Purdue model you're really kind of that that 3.5 that DMC of the operational technology and so you really have to prove um what you're doing at that 3.5 or that DMZ level is very safe but there are critical infrastructure that are out there that are utilizing Cloud I don't I don't think we can get away from it and so um the hard part is there has to be people that publicly go

first like in a documenta to way because the only way other organ your organization for example the only way they're most likely going to be comfortable is by saying this huge place did it this mediumsized company did it this is how they did it and but someone's got to go first right what now yeah yeah yeah but I mean cloud is just another way I we have Plumbing that comes out of OT right from a networking perspective anyways so you just don't call it Cloud just say we're connecting to an infrastructure right that's what it is right you're just connecting out right antiviruses all these other things they have to they have to get out so air gap in case you

guys don't know this like air gap for the most part is not a reality right that just does not exist really in my opinion maybe at one point it did but not maybe not on purpose it was just because those massive amount of manual things ever happen right good question any thoughts on how we can learn more about systems for I'll give you an example I didn't know about this until I read Andy Greenberg's book sandworm okay and I learned about not Peta that's a great book for anyone who's lost but then I know you have your s Masters right they have a class it's 8 Grand you know what can some people yeah who maybe cuz like I learned about the

reality of the serious of this of this when I thought about the chaos that would happen to a city if the lights could turned off how do you drive in a city with no stop lights how many cops do we have that can you know turn all or something so any thoughts on resources research books You' read um so there happens to be I don't want to put Pascal necessarily on the spot so he's actually right in front of you Pascal just quickly raise your hand so Pascal is on our team he actually wrote a book on IC specifically in in Security in ic great book that is a great start his book does not cost $8,000 so that's

I wish yeah wish wish he cost that much so that's definitely a good start Idaho National Labs also has a lot of U free resources on that um not like this is a plug but like try hack me right they don't pay me on sponsor but try hack me is actually starting to add IC like Hands-On keyboard um type stuff so try hacking is a pretty good site I think hack the box is starting to do a couple things with IC um Pascal's B like I said ICS St so yes there's they're starting to become more free resources out there but I would say one of the top things is and I love this in general about all

cyber security and this is no different on the OT space is reach out like on connections like on LinkedIn or whatever it might be because they are you know you have a network of individuals that are more than willing to point you specifically in the right direction depending on what your interests are if it's like electric utility space or just what is a PLC like how does that actually work how does the Purdue model like all those pieces you can get pointed in the right direction um just with people here that are in that critical infrastructure space I'm going to point those three people out actually they're sitting right over there right right in front of me so yeah they're

within they're within talking distance so they so this is a team that does uh OT pent testing they do OT incident response um we're all on a team together so sorry guys I volunteered thank you great question though that's a huge thing so I see a lot more training coming so excellent question any other question a lot of good questions yes

EnV OT is more yeah so your question is more like see a lot of it could be outsourced and then there OT more in Source um yeah it depends so you're actually seeing some of the OT uh cyber security being outsourced that skill set as well if you can find this skill set because it does become very difficult to get someone inside like for example they're already Outsourcing the it component of it it's it's probably a high hurdle to then all of a sudden get someone that has the proficiency and and necessary knowledge in OT right it's actually a bigger and so we are starting to see that kind of IC s those things getting

also so yes because for some people it's going to be difficult to say hey I just need four or five employees and full-time employees and benefits and 401K in yards right as opposed to just what you what you actually see a lot of is um someone that is hyper focused on the monitoring piece right the cyber security monitoring piece and then they partner with people that are boots on the ground that are employees and they actually public doing like oh I got to set the password I'm going do active direct stuff and so you're seeing that and team yes around base on you I us they respond what are some things I said I assume you want that for sure

what are some additional things that you might want as a response for your team when you arrive so response like from a hardw work standpoint or just like in just like what are they going to do once their boots on ground IC incident where ransomware on a CNC machine yeah so an IC your biggest thing is going to be visibility and a lot of times the in responders are going to walk into something where there's not visibility set up in place and so unfortunately if they're not doing inant a response plan that's specific to OT and it then you're in a situation where this response can't work stra on it but your OT gu and so unfortunately the sa is more

of a discovery stage of let me look at your network diams let me look at your part picture so if the planning wasn't done you're going to have stakeholders and the executives going crazy because you're going to spend one entire day just figuring out what's what and what's where right how what are the choke points what are the segments where things that are going Central and then from that point what you really end up doing is you got to put some type of collector and you got start getting tons of packets get tons of packets in computer system you put it in something like let's say Malcolm and Malcolm is going to like now think of it just as a

a Sim on steroids it's going to start running all these threat WS on it and it's going to start trying to find um patient zero right whatever where the infection actually started or it start doing you start clearing through things of highight lengths analysis but at that point stop the bleeding right you're trying to stop the bleeding so you start doing things on your firewall to cut off uh a lot of the communication where possible you cut that off the might and also the internet because like I said OT might not be directly connected to the internet but it goes through gmz it then internet so inverting it can be if prod actor in there and there's ransomware

they probably created a ton of to Dev M so you're trying to stop the bleeding you're going to do something on the fire to cut off connections as long as it doesn't again you have to have the expertise to know that when I cut firewall connection this doesn't make all my Operator just completely block right so typically you do something on the firewall specific enough to say I only trust this IP going around that's where you start stopping bleeding and then you actually segmentation is not being done well guess what you now start doing segmentation so you can segment where that right own is actually spreading that's initially your first 24 hours sometimes hour that's a great

question

if you want like a detailed equipment that you need again I'm going to volunteer Brett's heel over there BR raise your hand real quick Brett raise your hand so I'm volunteering you so if they want a specific about Hardware they in response I told them to come talk to you I encourage a lot of my customers to remain keep those man process place in Cas you're you're 100% right yes yes but getting away from us some as well well away from p and paper so yeah so the manual process some of the gear that's being sold it's like sometimes are not manual when I was at the utility we were that was the back of plan we had Ukraine

had