BSidesCharm 2024 - Getting Started in ICS – Not just for engineers

[Music]

[Music]

all right uh gonna get started thanks everyone for being here um quick uh quick point of procedure um apologies this is not M ma deis for Defenders I wanted to attend that talk too so I I share your disappointment um in Li of that it if if you guys want to go check out another track that's totally fine I won't be offended um in L with that I'm going to be discussing getting started in ICS and ot not just for engineers so quick disclaimer the views and opinions expressed throughout this presentation are solely those of the presenter yours truly and do not necessarily represent those of burns mcdonal or 1898 and Co the company I

work for quick agenda we going to go through intro problem anecdotes you guys can see it for there um a little bit about myself my name is Tyler Jansen I work for 1898 in CO's managed Security Services um been working in cyber security for eight years got started in the Navy in 2016 working as a discover and counter infiltration analyst uh the Navy has changed the name of that role probably four times since I got out but essentially it's a cybercity analyst but discovering counter infiltration sounds cool so that's the one I tell people um I got scared and I emphasized scared into icot in 2018 um just the the it was in an exercise and they were going over just

the the risk the threats the capabilities and and just how exposed some of our critical infrastructure and environments are and it just it's kind of a crystalline moment where like oh I'm in danger and so that that uh crystallizing moment for me happened in 2018 um and 2022 started with 1898 and Co and my my experience in ICS andot has spanned uh quite a different quite a variety of Industries I've worked with power generation facilities uh oil refineries um fuel storage pipeline wa uh water Wastewater um building controls Maritime uh navigation and Engineering so quite quite a few different Industries some quick terminology just in case you're not familiar with IC or all the other

acronyms that are in there uh we got the big broad terms IC that's industrial Control Systems operational technology that's uh just a again another kind of catchall term for the the computers and systems that manage a physical process right and then with within those we have we have control systems so we have building automation systems think your the computers that control your HVAC your your building access uh various alarms that that's your building automation systems distributive Control Systems those are what's going to be running like your power plant your oil refinery what have you that's uh where where you you you imagine like operators sitting in a room looking at a bunch of screens saying okay like the

turbine spinning at the right speed it's the right temperature that that's your distributed control system uh programmable logic controller that's kind of what the it's it's in between your your embedded field devices and your highle DCS and so that that's the more localized computers that are controlling the various uh field devices um and then last one I I'll I'll touch on is scada so that's supervisory control and data acquisition so that's a a control system that's going to be managing like a a pipeline it it's meant for systems that are they're very far and disperate so imagine a pipeline spanning you know hundreds of miles maybe got like uh some product you're trying to move from North Dakota down to

New Orleans right and you need to make sure that you have the right um volume of product going through the pipeline you need to check the temperature the flow rate the pressure within there and you need measurements throughout the hundreds of miles of pipeline you have right and so the STA systems are speci specifically designed to uh gather those metrics and then manage whatever controls need to be in there in case you need to like flush the pipe or shut off a section of it that's where the control and data acquisition comes in um a few other terms on there in the uh field devices we got HMI so that's uh as the as a spelled out there human

machine interface think of like um your your thermostat at home where you you tap on it change your temp temperature in your house you have various kind of embedded screens to control whatever local process you're looking at um intelligent electronic devices and rtus those are other just kind of like lowlevel embedded field devices that are going to be controlling the other uh list of devices we have list out here in the net spots the actuators gauges pumps sensors valves so those are the actual um controls so imagine like a a Fiel tanker right or at a fuel Farm big fuel tank got tons of diesel or whatever choose your product you need to be able to track um how how

full it is you don't want it to over full be uh over overflow or to get too low so you need sensors in there to make sure that you you have control of of where it's at right so the problem the the whole point of this this talk I believe there's a shortage in icot focused cyber Security Professionals due to misconceptions and perceived uh preconceived notions about working in ICS preventing otherwise capable and I dare say sorely needed professionals from from getting involved um and this problem I think is exacerbated by by several other problems the first one being persistent threats to to icot you guys might have seen in the news just the other week uh director of

FBI Christopher Ray testified to Congress that uh Chinese hackers are already in US critical infrastructure that probably didn't just happen last week or the other week it's it's probably been going on for a while right [Music] and not just in our country other countries as well and there's historical examples of that we got Stu net in 2010 black the Ukraine power grid 2015 uh tric hit Saudi Arabia in 2017 Colonial pipeline that didn't even actually hit the ICS that just targeted the billing system for the pipeline and so it's it's causing effects to our critical infrastructure without actually getting to the the I the IC and OT itself right and then the the incentives kind of Behind these attacks money have

the meme there like why did you rans in the pipeline because money and then uh sub kinetic Warfare right so can nation states have some kind of effect fet cause some kind of uh disruption degradation to their adversaries without being bombed or having boots in into their country yes that's still possible and so the incentive remains all right the other exacerbating problem is Workforce gaps now this is uh a graphic from from nist published last June and there's a lot of numbers being thrown at you here uh one thing just to highlight is the energy utilities on there so just relevant to the IC um only 20% of energy utilities respondents think they have the right

have uh enough uh people and resources to to manage what they have um now this is describing the entirety of the cyber security Workforce not just the ACS or OT Workforce and then another study published last October from ISC squared that's the organization that does cssp right their cyber security Workforce studies showed that 11 that the US Workforce grew by 11% last last year from 2002 and the North American Workforce Gap increased by 20% so overall the overall industry of cyber security right that there's a growth in the gap outpacing ing the number of work uh professionals we're we're adding in and then uh Dean Parsons um a contributor and instructor for Sans IC he published uh some findings in

November that only 52% of OT and ICS facilities actually have an IC uh instant response plan so I mean that's you know pick your local utility flip a coin and that that's their chances of actually having a plan for or if they get hit with ransomware that they're going to have a plan that they can go ahead and execute right um another Finding from that in the graphic right there uh the organizations he he pulled deploying trained OT security Defenders to leverage their their Network visibility is their number one priority so just do we have the right there there's plenty of Technologies out there to deploy so you you can get your your IDs out there

your IPS your EDR but do we have the qualified people to manage and respond and and analyze the data that that's getting and so going through all this uh persistent kind of question remains what percentage of this total Workforce Gap is specific to icot I I haven't been able to find a good answer and not to Outsource my not to Outsource my homework here but if anybody has a good source for that or thinks like they they have they know that that please let me know I I would love to know or come find me afterwards but I have not been able to find a good answer to that so that kind of leaves us to assume

that there's there's a gap that that we don't know and so presumably if the entirety entire Workforce is lighing behind then it's probably the same for ICS and OT and then if we're missing these people what's preventing PE what's preventing those that are in the uh Community from getting involved into iot so here's a few anecdotes just uh reaching out to colleagues and and throughout the cyber security community that i' I've heard from people won't read this out to you guys but uh bit bit of a description of me coming to people and asking hey why are you interested in icot yeah all right so going through kind of my my anecdotal evidence of of chatting

with with uh the community uh these are the three barriers entry I identified people believe that there's a skill M skill mismatch between General it cyber security and ICS and OT just general unfamiliarity with the space and not really sure how to get started and then um a feeling of lack of opportunity to gain that knowledge if they did know how to get it and how to then you monetize it because I mean we're working professionals we need a job we can't just do this for as a hobby plenty of people do but you know you can monetize it all the better so here's a quick uh comparison of skills so kind T of answered it already

but the the core core competencies and skills that I think a general cybercity analyst uses and what you need as a security professional in icot I I believe it it overlaps the same it's it's it's the same skills and competencies so rting and switching do you understand vlans and network segmentation Port Maring and spans like hopefully your your analysts understand that and that translates to icot network protocol protocols in tcpip right you don't know every single protocol that's out there you don't know everything about it but you know the basics you know HTTP https DNS F right there there's core core pieces that you know and you can apply that Concepts to new protocols as you as you run into

them right firewall management do you know do you know the basics of how a firewall works you know like implicit deny it might be a different vendor in ic but it's still kind of the the core piece of how are you're configuring your firewall now work architecture how's the the overall now we're going to be laid out how are endpoints communicating how does that affect things the architecture is going to be different in ICS and OT but the the concept behind it like we have endpoints that users use and they reach out and communicate to some kind of server or service right we have there will be there'll be cases where you have um different types of

endpoints but the the core piece of it is your still monitoring network communication then uh operating system architectures how are your engineering work stations laid out how are your uh your historians and servers organized with their file directories where the startup files because attackers are targeting the same stuff in ic they're they're going after the same types of operating systems at least at a at um certain levels within the if you're familiar with the Purdue model um the the control level at uh three is is is still mostly windows and and units architecture so that's still relevant um security Technologies IDs IPS EDR the way you go about deploying them might be different but the Technologies

themselves themselves don't differ much you're still either monitoring Network traffic trying to block Network traffic or you're trying to deploy some kind of agent executable or service onto a host to to monitor for file or configuration changes and then threat me methodologies do you do you understand how an attacker might try and get into an environment say maybe there you have a VPN from New York to your power plant and it you're the Business Leaders are using that VPN to get data from the operating environment that's an attack vetor that an attacker might want to go after right so the the being able to understand how attacker might approach uh compromise in your network that skill that concept is

still applicable here so there's all these similarities so what's different there's there's industry specific knowledge so the there all all that terminology I just flash at you guys a few minutes ago that's different obviously and there there's quite a few more Concepts that I'm not going to go through here today because I mean you look at like Sans has I think four weeks of classes all dedicated to IC so I'm not going to be able to get through all of it in an hour so not even going to try but there's plenty of IND there's lots of Industry specific knowledge um the safety culture that that's that's one of the big things that's different with IC so these

environments are dangerous environments things can go boom things can release poisonous gases um it it can cause harm to the community you could like poison waterways all all all different ways things or people can get hurt and so the these environments already have a very well established safety culture and so as Security Professionals coming into these environments we have to be respectful of that we have to um we we have to adhere and kind of follow their lead in in terms of safety being number one and then that comes down into the priorities of of how we go about um our our our overall approach to security so you'll see the the triangle we have there the typically

it's the CIA Triad right you'll notice I've put availab availability up top here that's because in these environments they're they're all about up time you you don't want your power plant going down because you need to do Patch Tuesday right like you we're we're not going to tell the the the town hey sorry we're going to have a blackout for for a few hours while we while we update everything right you got to keep everything up and running number one CU because that's how the the business because their businesses that's how they make money is uptime and then it's it's not just oh we lost money while we're down it can take a lot of money to come

down it can take a lot of money just to restart because and when you make changes sometimes you have to recertify like that the environment is safe to run or there's some kind of regulatory regulatory uh check you have to do and it can cost money to bring out the regul Regulators to come out and verify yes you you guys are good to go back up and so availability is just hyper important and then the the data you're seeing in ic you don't have as much need for confidentiality because a lot of the data is hey what's what's the what's the temperature in on this valve or within that Pipeline and so it's just it's

pulling like every couple seconds or every couple minutes like hey what's the temperature and so we don't need to worry about the confidentiality of that cuz there's there's there there are ways to abuse it but it's less critical information because it's just hey register what what what temperature is that it's 50 degrees celius it's still 50 degrees Celsius it's right so it doesn't matter the Integrity that is important but availability is number one and then the the impacts of compromise and that kind of gets back to the our the uh safety culture and that when things go wrong when things get compromised the potential impact the risk is much higher so we already kind of touched on that

that things can go boom chemical chemicals can get released gas goes out it can cause environmental issues you can use your imagine to think of all the different ways of power plant or or pipeline misbehaving could could cause issues Okay so we've talked about that there's a not so much a skills mismatch mismatch but just the the environments that we employ our skills is different and so we need knowledge to know how we need to appropriately apply apply these skills in these environments so you need to know the IC OT like device specific terminology that we kind of went over and there's lots more I just we didn't need to do a a spelling in vocabulary quiz for an hour so save

to from that um there's different protocols in these environments and then uh the regulations in governance are are kind of the the top three things to be familiar with when you're first getting started in ic so just flashing that back um for industrial Pro industrial protocols here's a couple of just um basic ones for for when you're getting started mod bus if anyone's ever heard of that uh 502 TCP so that's um an open source protocol that is employed by by a lot of different vendors um used in a lot of different environments dmp3 usually in um the energy and po energy and power space uh bnet that's used for building automation control that's the BAC there

building automation control Network and then there there's just a plethora of proprietary vendor protocols so each of those vendors I have listed up there are going to have like their own proprietary like Seamans has S7 GE has srtp and a lot of these are going to be specific to the to the devices and environment they're they're they're deployed in um it can be difficult to get exposure to those unless you're working in those environments um and I I'll touch on ways you can you can try and get uh some information on those here in a little bit um one thing to note with these though is these IC and OT protocols these weren't developed as your standard

um TCP IP stack kind of uh thought process these These are old they've typically been around from when like the these industrial environments were built so I think like maybe the 70s the 80s they weren't built with uh Security in mind there's not a whole lot of authentication or or anything like that built into it so these these protocols were just they're originally serial Communications and so they're all built around hey we need to send accurate data as frequently as possible and so it's very lightweight just hey send the data send the data send the data and send the right data and so these protocols can be pretty easily abused for example like uh modbus has um

one of the functions and it it can do uh multiple rights so you'll have um various back here you you'll have like a PLC up top and then you maybe you'll have um some sensors deployed throughout your environment and usually you'll be running a read multiple registers so so that's be that would be hey go to these sensors and read get the value from from register 101 right and I'll go to each sensor and pull that information but you can also do a multiple write and know say hey go go to the same register write write a set value to register 101 on these sensors but there there's no limit to how you how you um structure that that

right command so you could say we have our normal go right to uh register 101 and do it for all these uh these sensors but then that you can kind of tack on on the back to rewrite to the same register over there's no way to verify that unless you have some kind of sensor or analy who knows oh yeah there there's there's nothing that's going to put a cap on whether or not you can write or read from the same register multiple times and so there's just a a lot of um aspects like that in these protocols that can be abused right um another big thing in these environments is the governance and regulations there's a lot and varies

quite a bit from from industry to Industry so the the first two items we have up there are broad guidelines and Frameworks so NIS special publication 800 82 rev3 that's just a guide to OT security um that borrows actually a lot from the second item there Isa I 62443 um that's put out by the uh International Society of automation global standard uh most widely recognized Authority on on IC and OT security um and then we we we've got DHS and siza they're they're doing most the kind of um protection providing resources to to various organizations and and occasionally providing response capabilities they and they actually just had a um a new draft rule put out for

instant response reporting uh if anyone's familiar with that cersa just outlining uh how quickly and which organizations need to respond if they uh if they get hit by ransomware and then we we've got EPA is in charge of cyber security standards for and regulations for water uh nerk for the uh electricity and power so power generation transmission distribution uh TSA for Rail and pipelines so it varies quite a bit and and it's it's changing still it it's not very well um settled probably the right term so some resources to get smart on this kind of stuff si si siza's virtual learning platform they have 13 free online courses about 16 hours go through they also have a couple instructor Le courses

that um that are pretty good they actually host those um instructor Le ones live if you want but you got to travel out to Idaho National Labs it's nice out there but uh you know it's can be a little chilly but they they also have an an online option for those um another great resource is Rob Le's Rob Le's blog for getting started an ICS data if you're not familiar with Rob Lee he's the CEO and one of the founders of dros uh local company just down the road a bit um that's got a lot of great uh reading resources um information on starting your own home lab um and all kinds of stuff and it's not just focused on the

IC he also has resources on the more engineering side of things too if you want to start getting a little more informed on that um some resources for just getting familiar with some of the protocols uh mod there's modbus samples on the Zephyr project so they they've got um PAB captures that you can download throw into wi shark and just start getting familiar with uh how how those those protocols work um if if you are more of a if if you want like the out kind of like that RFC essentially of modbus without actually having to read the RFC um fernell software has a pretty good explanation on that um PLC cables that's a kind

of commercial off the-shelf site that'll sell trainer kits for like PLC programming so if you want to learn how to uh write programs for Seaman S7 or for Mitsubishi or G any like their plc's you can you can purchase their uh their stuff all through like PLC cables and they'll they include um the software licensing and the the actual U like PLC itself some cabling and have some exercises for you to go through to like just get smart on hey how do how do I program ladder Logic for like these plcs um getting smart on the the regulations uh nist and all all the actual regulatory stuff for for EPA siza uh nerk TSA that that's all free online

on on the appropriate government sites um Isa IEC that uh it's you got to be a member to ISA for that that's $15 a year but if you're if you really want to get into it that that's usually the best place to get it You' probably find it online somewhere but don't want to encourage that necessarily um and then last resource wanted to share with you guys is uh the industrial security podcast that um covers a lot of different topics they they have a lot of good guess on from from various Industries they talk about different regulations different approaches people have taken to trying to secure their environments so that's a good resource um and then what

opportunities are there out there to get started and start kind of dipping your toes in so networking is always usually one of the best ways to get started the more people you know with the same interest the more ideas are going to flow the the the more opportunities you're just going to inherently find um conferences conferences and Summits are always good so bsides is always a great option for that um Sans has a couple uh IC specific uh Summits dros has has a conference in November um siza used to do a joint working group twice a year that has fallen off I think the last one they did was uh B 23 but if they pick those back up those

were always uh great and those were all online um and a lot of those conferences and Summits have ctfs those are always uh great opportunities to just get exposed to going through like different packet captures or or um or doing like host getting a chance to do some host forensics on uh IC specific devices and then the the last opportunity that um something I that helped me kind of get started was building automation systems so those are the systems that are most um most readily available essentially right you don't have to necessarily be working at an IC OT job to have access to building automation CU like this hotel is running some kind of building

automation the offices you work in the if you live in an apartment it might it's probably running some kind of Bas and so reach out talk to whoever like the building manager is and ask them about it and just try like try and talk talk somebody's ear off about building Automation and how it all runs and see if they'll let you kind of get your hands dirty a little bit maybe not and maybe they don't necessarily want you doing taking your first crack on like the HVAC for for your office everybody get angry if you you know crank up the the heat in the middle of July but just getting some kind of exposure and that's a that's all I got

thanks everyone for joining um if anyone has any anyone has any questions be happy to answer

yes yeah yeah there there there's um quite a few great contributors on on on LinkedIn um you said Mike hul

yep yeah networking anybody else no all right well thank you very much [Applause]