Defense-in-Depth for Industrial Controls

hey everyone I'm Brad Hamlet with cyber risk analysis group I do consulting mostly for water utilities and commercial real estate firms if if I'm not coming through clearly if the sounds not not very good can somebody just let me know in the chat really quickly here I'm not seeing anything so I'm assuming they're loud and clear loud and clear awesome awesome thanks so so the briefing the briefing today that I'm going to give is about cybersecurity for water and wastewater utilities so this is going to talk some about industrial controls information classification and just overall risks for water utilities and some of the regulatory frameworks that apply to water utilities so let's take a look

first of all why should you all listen to me well very simply my wife says I look like Matt Damon so that that joke works way better in person and video meeting but anyway so why should we really hey let me let me tell y'all I had my first I don't know about y'all but I'm tired of these like zoom meetings and zoom you know just video conferences at my first in-person meeting last week and I walked out of that meeting and I was like oh that was a great meeting I really loved that meeting which is not something I ever thought I would say about a meeting but take away that in-person factor and like

you just kind of really you know start to crave it after it's gone for a while but anyway now so like why why should you really listen to me so I've been working in the IT field for about 20 years got about eleven years experience in cybersecurity mostly with the army I manage a 34 million dollar portfolio of IT and intelligence projects for the army I was TSS CI cleared and one of the projects I ran was a counter idea min Afghanistan we were looking at hardware and software vulnerabilities of IEDs in my particular area of expertise and that was looking at wireless ie D triggers so a lot lot to do with cell phones I'm on the

American Water Works associations emergency preparedness and Security Committee and worked work cyber in a VA hospital and I ran their first tabletop exercise of a simulated network outage they never done they had never done that before so what happens when the computers go down and everyone was like we have no idea and so we worked through it and it was a good exercise so my my briefing here is sourced from major news sources major corporations and other other credible sources like that there are lots of numbers out there that are floating around and some of those numbers are not not produced very well so if you're not familiar with the terms IT versus ot here's kind of a breakdown

of it there are different terms for operations technology you might hear the term industrial controls or process controls that's those are two other terms for operations technology so operations technology runs equipment so it runs water pumps water filters it runs manufacturing equipment like at the the manufacturing building that Greenville tech has on the i-car campus you know all of that equipment that's that's down on the manufacturing floor that we kind of you know look at when we're having our monthly isset meetings you know all that's run all that's opera run by operations technology and the average age of the equipment is much older than IT the investment in cyber security is typically low the primary

concern is loss of production on the OT side versus loss of data on the IT side so and and patching patching is is more difficult and it's typically performed by the manufacturer although that is starting to change a bit so systems administrators [Music] systems administrators of utilities have a number of concerns only about a third reported being highly ready to respond to a breach about a quarter report being impacted by nation-state actors we'll talk later about the different hacking groups targeting industrial controls for utilities only about a quarter are typically targeted by nation-state actors those are usually going to be the largest utilities more than half more than half of all utilities reported a shutdown due to a cyber incident last

year 2019 so so big big deal with cyber security for utilities so let's let's look at some risks for utilities two primary categories of attackers nation-state actors and for-profit hacking groups nation-state actors as of right now are primarily doing intelligence gathering they are looking for weaknesses they can exploit later to switch off utilities if World War three breaks out that's what they're looking for they're mostly using spearfishing to get in there to steal Network credentials uit folks who were treating your utilities are do consulting for utilities you all are top targets you all are top targets to get spear fish so they can steal your network credentials on the for-profit hacking group side they are primarily in

it for the money they want to ransomware systems they want to steal data if you remember Greenville water got hacked earlier this year and it was a for-profit hacking group it was all over the news I was a for-profit hacking group that did it and they I'll go into that a little bit later got a little more detail on that attack they are using living off the land as their primary attack vector so they're just grabbing off the shelf tools using known exploits sometimes you're selling access they'll you know they'll set up a long-term you know persistent opening in a network and then they will sell that sell that access off to other groups that's other

other ways they can make money so two primary categories of attackers so nation-state actors again they are rarely detected they're rarely detected and I tell my utility clients like listen these guys especially folks coming out of Russia and China like they're so talented you're never you're never going to find them I mean you you will spend so much money trying to stop these people and they're so talented they like you you really can't out spend to their level of talent on defense and so you're better off mitigating other threats other than this threat and so in order the top threats to the US us utilities China Russia Iran and North Korea China and Russia are the top two top two most

talented and again they're mostly doing surveillance and they're mostly on the operations technology side they're not really interested in stealing data skewed the stealing customer data and stealing payment data for profit so let's contrast that with for-profit hacking groups they are interested in stealing data and ransomware so last year 2019 the average payout for ransomware was it's 40k this year 2020 it'll be up a little bit I'm guessing you'll probably be in the low 50s this year that's my best guess I don't know what the numbers gonna end up being data is also very valuable so Symantec did a very cool study they went on the dark web and pulled a bunch of data what is

personal data selling for in the dark web social securities you know about about $1 pretty cheap utility bills other scan documents they can be you know very wide price range they're depending on what type of document it is cloned IDs and complete identity packages now I don't know about you but the last time I signed up for water service they wanted a complete identity package from me name address phone email date of birth social and so water utilities a small water utility with 5,000 customers if those for 5,000 complete identity packages are stolen you can run the numbers that is a solid solid payout just even off of a small water utility so even those guys are at

risk and that information like customers information needs to be classified so that it is kept more private than other types of public information that the utility might have so let's take a look at some ot risks now so so over the last from 2018 through to 2019 the 2020 date is not in yet but from 2018 to 2019 there was a shift from IT duty attacks nation-states appeared to be ramping up their surveillance and some state sponsored groups were even trying to cause outages so again about half of sis admins that utilities reported an outage at least one outage and about a third were not detected I'm assuming that they were not detected at the time but they

were forensic ly detected later on because otherwise they were not detected how was that number anything other than zero so you know obviously they were forensic ly detected later on anyway another thing we're noticing on the OT side is increasing potency of attack so nation-state attackers are getting more sophisticated they are looking to damage things now they're not they're not just looking to do surveillance so again surveillance is still the majority of what they're doing the vast majority of what they're doing however there is an uptick in trying to damage infrastructure and and nation states have begun funding individuals and organizations that are not they're not government organizations or military organizations they're you know like a

private company that is simply just funded by the Chinese army or you know something along those lines and so that I mean I think that gives some plausible deniability so it's probably why they're trying to do it so here's an interesting fact so if you don't remember the wanna cry not Petya ransomware it's 2017 to 2018 made massive damage billions of dollars in damage worldwide more than a quarter of utilities were affected by those two ransomware and those those were actually military those were military technologies that were that escaped into the wild and were modified and turned into turned into malware so I don't remember I think that might be in this if it's not in this briefing that's it's

in another briefing that I have about want to cry that they were they were really cool I mean I don't know if you read about him but hey if you haven't read about those two those two ran somewheres you should you should read about and they're pretty cool actually but anyway okay so increasing consequences of attacks also on the OT side so so sis admins are concerned about damage to equipment about environmental incidents and about stealing high-value information I've got a case study later on in the briefing about a utility out in Colorado who got a lot of their high-value information ransom weird and and they they caused serious serious issues I don't think they responded to that attack very well

hopefully no one here is from Colorado I picked a utility like okay you know what's what's the wrong way to respond to this I picked the utility way out in Colorado nowhere near Upstate South Carolina so hopefully you know because my business is around here I don't want you know like say anything negative about anybody around here so anyway hopefully nobody from Colorado but anyway so that was ot risks let's move on to IT risks on the top you've probably all seen survivor awareness the numbers are very similar between for for utilities but these numbers right here are specific utilities so so about 20% utility employees receive malicious email the vast majority of it the vast

majority of it is either it's either office formats or executables and and you know I think I think the number of even malicious emails for the utility industry is is slightly higher than an average and it's probably because the utility industry is a big target they've got lots of money and they've got lots of customer data and so that just makes them a big target malicious email subjects the top two are either about a bill or about you those email delivery failure fake emails which those are like I I gotta tell you the email delivery failure those are those have got to be those are up there I can't decide if it's that one or the fake password reset

emails that are like my favorite phishing emails but like I keep I keep like a record I go through my spam folder in Google and to find like amazing you know malicious emails and there was one there was an extortion one I found last year that involved a major porn site and I mean just like they're getting so creative these days with with malicious emails that I kind of I keep like I keep an archive of them that's there they're really funny occasionally I use them I'm I teach computer science classes at Greenville tech and I really enjoy that a lot and so I like I use those emails as examples in my classes the students like them I leave them out

of my my corporate presentations though try to try to just focus more on a single topic so there are a lot of emerging malware risks were mobile ransomware that goes after office files so like one of my clients last week my endpoint detection response software that I had I had I had just installed it on their endpoints and it stopped and one of these attacks exactly a warm above war mobile ransomware that went after it went after Word and Excel files and it and so this this is real it is happening it happened last week too but it was blocked and never though the EDR just stopped it and it never never never caused any issues I've

just got an alert that it was blocked and it's like happy times so that's why this software is great and it works and the there's also there's also some more mobile malware out there that mimics Olympique destroyer so if you don't know a limpet destroyers another really cool piece of malware it targeted the South Korean Olympics and it was it was everyone initially thought it was made by North Korea but then upon further analysis the malware actually appeared to be kind of faked to make it look like it was made by the North Koreans to attack the South Koreans when in fact somebody else most likely made it I don't think anyone's really a hundred percent sure who did it

but best guesses are just Russians and Chinese just trying to create chaos and create problems for the United States but this this malware it again at targets office macros so just don't I really encourage like all of my clients I encourage them to stop using macros if you have any old Excel workbooks out there reformat them to get rid of macros you don't need them anymore you know Word documents get rid of them just don't you to just try and not use macros at all because they're they're really not good and really dangerous connected devices are another top danger because people usually forget to update them it's really easy for employees to just come in and you know if they don't have

good Wi-Fi in a room they just plug in a you know plug in a off-the-shelf consumer ap into the network and you know if you don't afford security it'll just it'll run internet through Wi-Fi won't rip anything so anyway so this can be a problem cameras can also be a problem a lot of these connected devices people don't update the default admin username and passwords on them like when I've moved into my house that I'm in now last year it had a camera system in it and and I wanted to run the king system myself instead of getting a company to run it I want to do it myself so I I've logged into the camera

controller and I just used the default admin username and password from the manufacturer and like the company that had install all this equipment never changed it so so good for me I get right in so but just kind of blew my mind that they didn't change it but that's just kind of a I kind of guess us in the cyber security field that's just how we think so it's it's very easy for malware to be to use a router or a camera or another connected device as a point of ingress and then spread into the rest of the network and it's they can jump from IT to ot systems particularly if the networks aren't segmented it's very very

easy for them to escape over into the OT network another risk is supply chain attack so you infecting devices before they leave the manufacturer a funny example of it well not very funny but just a unusual example of this was Schneider Schneider inverters last year shipped with infected USB media and they had to quit put out an alert recall it send out new media and so anyway that was that was a that to me that's that's pretty interesting so yeah the attackers will compromise the manufacturer and then and then and then infect their products before they leave the factory and then vendors vendors and third parties they're vulnerable to compromise Duke Energy's billing system was

compromised in 2018 and it was through they had subcontracted their billing to energy services group and it was actually the vendor that got that got compromised and lots of new customers data got stolen so there are a number of recommendations out there from industry and that is you know the number one the number one recommendation is know all the assets that are on your OT network take an inventory of everything that's on there so you know what's on their software and hardware look for cyber solutions that are specific to the devices and and boost your capabilities for rapid detection of security breaches so another another main recommendation is to look at organizational failures and correct

those so so systems system administrators that utilities reported about 42% little less than half reported that there's no clear ownership of IT versus OT systems like somebody needs to own the system and be responsible for securing it another major organizational failure is lack of human capital in cyber and lack of cyber training for personnel you know I mean I don't know I mean I've just seen it I've seen it so much you know don't click on the link in the email just all this basic cyber security training but how much of this training is specifically targeted to people running pcs or ICS systems on the OT side of the network you know how much of that I don't know not much from from

the utilities I've talked to you know most of them have basic cyber awareness but but very few have an OT specific cyber security training class for their employees which I have and if you want it I'll be happy to give it to you so okay so so another recommendation from industry is to instead of using a compliance based approach use a risk-based approach so there are a ton of regulations out there a ton of laws you can use compliance with those laws as just kind of a starting point to move to a risk-based approach start with your high-value assets prioritize your investments in cyber security your cyber spending on mitigating mitigating the biggest risk for the highest value

assets and that will get you the best bang for your buck in terms of cyber spending so there are there are basically an unlimited number of regulatory frameworks out there that apply to utilities you see even see HIPAA in there like yeah that does because lots of utilities collect medical information on their employees through their human resources department via health insurance I mean it's just it is a an alphabet soup of regulations that are out there however here's the top two America's water infrastructure Act which is a new law and PCI DSS which you all have probably all heard of that one because that one is super super widespread and popular so PCI there's four compliance levels and it requires

scans and vulnerability mitigation for a weeow America's water infrastructure Act there are 12 security domains and it is mapped to recommendations from the American Water Works Association and the primary focus is high-value assets so so it what what you can do what you can do is you use a we a compliance to focus on your high risk high value assets right and then you use PCI compliance to focus on protecting your customer data and my primary recommendation is use those two frameworks as a starting point to adopt a risk-based approach so first thing you need to do is do a vulnerability assessment that will check all the boxes for both a weeow and PCI compliance you

can do it in one risk assessment start a vulnerability management program you need to do quarterly network scans that's a PCI requirement and you mitigate your critical and high risks those network scans are going to spit out basically an unlimited number of things for you to mitigate there's no way to get to all of them so you just do your criticals and highs and for IT and ot network security look at cloud-based options there are there are cloud-based cloud-based network defense systems cloud-based endpoint defense there are lots of options out there to look at bring in a good third party who really knows how to look at your system look at what's out there and match a solution

specific to your system and multi-factor authentication I mean it's super easy fingerprint readers are cheap socially now with Windows 10 the Windows 10 you can use fingerprint and pin to get multi-factor it's super super easy now to do multi-factor authentication another big recommendation is to boost end-user training so again 61% more than half of all cyber breaches are caused by human error this is this is a pretty solid number I mean I've seen this number this is from Willis Towers Watson there hasn't seen a variety of numbers I've seen I've even seen I saw a number that was like it was like 93 percent of cyber breaches or from phishing emails but then like I dug into that number and

that number was produced by a company that's trying to sell anti-phishing software so so I don't know their precise methodology but I a number that's that 93% from a company trying to sell phishing software I would be skeptical of that number but from a major cyber risk insurer like Willis Towers Watson 61% is a solid solid number caused by human error and so Andy user training is very important it's important that it's customized specific to the business and that it's customized specific to what the employees are working on if they're working on ICS PCs systems on the OT network or whether they're working in the you know accounting software over on the IT network it needs to be customized it

needs to be role based training role based training also incident response plans are important new IT infrastructure is great new equipments great but when that new equipment is installed you need to plan for what happens when it goes down so what happens when you do this well here's what training does let's compare two water utilities Greenville water and Fort Collins Loveland water and again I hope nobody in here is from Colorado so but if you are sorry about it but hey you know I have to see what right looks like and have a contrast for that so Greenville water large water utility it's our water utility right here in Greenville they were breached by malware through a

phishing email earlier this year 2020 that breach was isolated to a single billing system on the IT side of the network the attack never traversed over to the OT side and it was isolated to a single billing system the company quickly switched to backup options as they had practiced and as they had planned for and the attack slowed down their billing operations but it failed to stop them completely they still process accepted payments they still process payments they move to alternate systems and I always tell people there's no such like no network is ever completely secure no matter how much you spend on it no matter how much you spend on it it isn't a network is

never completely secure so breaches are inevitable they will happen the question is what do you do to them when what do you what's your response when it happens and that's what table top exercises are for executing that incident response plan drinking water had one of those plans they executed it they had practiced it and it was it was a that I mean they responded like textbook response and that in my mind they are an example of what right looks like Greenville water now let's contrast that with a small water utility way out in Colorado nowhere near Upstate South Carolina so they were breached by ransomware in 2019 they had no backups they had no planning

for outages they yep no incident response plan no tabletop exercises none of that they lost all their technical drawings-- data and files for two weeks lost it all they tried many many options for recovering the data without paying the ransom and they were not able to recover their data without paying the ransom it cost him a hundred grand to pay the ransom and to make investments to recover from that outage and reduce their risk going forward so that is a lesson why you should train employees why you should invest in incident response plans do tabletop exercises do training it can either be just a minor slowdown in your billing system and not really that much of an impact at all or

it can be complete work stoppage for two weeks so so that that's kind of a contrast and it's a good example of why training and planning are important another thing I encourage people to do is engage with the cybersecurity community that's not the cybersecurity community so why these jokes work way better in person like because you know I put this slide up and and people are like oh my gosh I can't believe he's saying like and then I flashed not the cybersecurity community all right ok I got it so yeah so yeah we're the cybersecurity communities and actually been nice and friendly yeah and like I can't see anybody you know reactions like facial reactions so anyway like I'm

zoomed out I got to tell you I'm zoomed out I'm ready for you know kind of it to go away but anyway yes so I really encourage everyone to engage with the cybersecurity community we're actually nice and friendly people like us so yeah and really I mean you you really can mitigate your risks by engaging with us you know either contracting or hiring cyber folks getting training planning things like that I'm right on the money so I'm right on time so good ended at 11:45 so man that was like up 1146 I was one minute over my bad so you can feel free to contact me anytime about these issues we discussed here or if you need any consulting or training

or anything for your business feel free to reach out to me that's my personal cell there text me or call me on that and then you can reach me on my email there at any time I'll give you a minute for questions I'll try to pull up the chat window here and see if any questions come in

you

you you you mark Johnson what's up how you doing

nope just water utilities electric utilities are actually pretty solid I've found there there ahead of water utilities and we as a pretty new law that applies to water utilities so they're they're kind of complying with that law right now doing risk assessment spending up vulnerability management programs training program stuff like that cool doesn't look like any other question so I'm gonna go on mute I really appreciate everyone's time and yeah really appreciate oh I got another question here so yes yup yeah late to the game absolutely absolutely yeah so security it's like it's like yeah Security's an afterthought and I mean I guess that makes sense because when you're making a piece of equipment you want the piece of

equipment to be as reliable as possible and churn out as many widgets or as many gallons of water as possible and so really I mean yes security is just kind of you know kind of an afterthought but but one of the recommendations out of the American Water Works Association is is planning is planning for security at the beginning of the equipment life cycle during during the design phase of the equipment life cycles so this I think it'll change over time but it it's not it's not not quite there yet

oh there's a million yeah there's a million of them out there uh yeah different brands and stuff

cool well uh yep doesn't look like any other questions so yeah I'm gonna go on mute I really appreciate Mike Holcomb all the work he did to you know to put all this together there are no it was a huge pain having to shift this from in-person to virtual and anyway yeah I really appreciate Mike Holcomb and all the work you didn't make this happen this is a great up status is a fantastic group I really like it a lot and really appreciate you know my organizing and run it everything so it was and I really appreciate all y'all's time listening to my briefing and I really hope really help see you all again one of these meetings here pretty

soon so I'm gonna go on mute now thanks again bye