OT Security is Hard

But uh when you're doing multiple things in a con, you don't realize where you need to be. That's my fault. Anyway, our next speaker is Michael N. Sorry to keep you all waiting and make him almost introduce himself. Michael N is a loud and outspoken new cyber security professional. His career path started, as many do, studying some field in computing. Upon entering the workforce, he was given the opportunity to work on OT security. Wait, what is OT security? We'll get to that later. More importantly, he has two dogs and loves smoking meats. I agree. Thank you so much for putting up with me. Thank you. [Applause] All right, so let's get started. Um, so my name is Michael Nee. I'm here to talk

to you today about why OT security is hard. Um, who here knows what OT security is? Oh, that's more than I was hoping. So this talk is going to be a little bit more intro level to get people to know OT. We'll talk about some of the history. We'll talk about some present state and we'll talk about what I think next steps are for us as an industry. So before we get to that, I'm going to introduce myself a little bit more. Um professionally, uh I'm an alumni from RIT. I graduated in 2022. Um I originally started working in IT security. Um particularly doing software security and dev sec ops. Um gra following graduation I started

working full-time at a company called Zoetis. We manufacture animal pharmaceuticals. Um and when I got that job they offered me the only open position they had which was OT security which I didn't know a thing about. So, on to the important stuff about me. I have two dogs. Uh, over on the left here is Charlie and on the right is Bailey. Um, they're both a pain in my ass, but I love them dearly. Um, besides that, what I like to do outside of work, um, I love fishing, but I'm terrible at it. I love golfing, but I'm also bad at that. What I am good at is grilling and smoking meat. Uh, it's a hobby everyone can enjoy.

All right. So, I want to start off with a story. So, like I said, I got working in OT security without knowing almost anything. I was brought in to do vulnerability management for OT. We're not going to talk about the merits of vulnerability management for OT. That's a whole different talk. Um, but that was my goal when I came in. And I said that should be simple. I'm gonna Google a white paper. I'm going to start researching. I'll spend about a week and after that week, I'll just start implementing and everything's going to go great and fine and easy. Well, based on the chuckles, I'm sure some of you have maybe tried your hand at it. Um, it's not that easy. And when

I Googled for a white paper, there was none. Um, there were snippets here and there of like some ideas people had. at the last OT security conference S4. Uh it was a big topic that a lot of people are trying to figure out. So I said, "All right, I'm innovative. We're just going to borrow what we learned from starting working in IT and software security and background in vulnerability management and we're just going to shift that all into OT." And nothing went wrong. So I got started. I was like, I'm going to talk to the automation guys first. we're going to go to sites and have conversations. We'll get buy in on the ground floor. And when I told them my idea,

we're going to start, you know, scanning stuff. Um, they all made faces and the room went quiet and they said no. And that wasn't the buy in I was looking for. Um, so I was like, "All right, we'll we'll work on this." I took it back to my boss. He's been my mentor. He's a great friend. Um, and I pitched him the same idea. I was like, he'll kind of give me buy and little tweaks here or there and we'll get to work. And he made that same horrified face. So, I did some research to find out scanning in OT environments is generally frowned upon. Um, and if you try to get a site to do it, they might try to give

you a tour of their autoclave from the inside. Um, which was what this site right here, this was my very first site visit. Uh, they asked if I'd like to see the inside of the autoclave. I declined. So I didn't give up and I came up with a solution largely based around needing asset inventories and that's the whole thing. If we went to the end of my talk you'd have learned something but we're going to go through the whole thing. So I'm going to start with a little bit about OT just very general stuff. This is very beginner level. Um, so there's a lot of you in the room who know about it, but for those who don't,

I want to bring some awareness. So you might have heard OT before. Um, you've probably heard it by any of these other names. OT, operational technology, ICS, IAS SCADA SCADA you know, operational technology uh is generally considered just to be any hardware or software systems that interact with the real world. Um, and that's that's that's kind of the key piece is it interacts with the real world. There are real world consequences to this, not just a server crashing if you mess up. Um, some examples, CISA defined 16 critical infrastructure sectors. I've listed a few here for you guys to see, right? We have dams, manufacturing, energy. There's a ton. There's 16. So, there's 13 others. Um there's things

like chemical, energy, nuclear. Um they're all a little bit different but have plenty of similarities. Um couple things to note about OT. Um OT devices are sometimes maybe more often than not older than me. I'm not that old, but I'm talking devices. These things are running for 20, 30 years. And if the operators could get longer out of them, I'm sure they would. Um, another thing to note is OT devices are relatively low compute. This makes them sensitive, very sensitive. Um, and they're designed for longevity. Like I said, these things sit on factory floors with dust and whatever else is kicking around for my whole lifetime. So, moving on. Um, we're going to start talking about OT security and

I've got a historical reenactment for you guys. I'm going to do some acting. So, we have our plant floor operator says, "Wow, OT shore seems important. Should we do something to protect that?" I mean, it's our dams, it's our hospitals, it's our drinking water, our electricity. It's pretty important. And we got some managers, we got some upper level cyber folks, maybe some of you in this room.

No. So, this wasn't just a reenactment, and I don't think they're crazy either. When these systems came about all those years ago, they weren't connected. These things were actually airgapped. Not like not like the joke airgapped where everyone laughs after you say your server's airgapped, but a true and honest air gap. They're separated from everything else. only on the site. So, I'd ask all of you, if you had a truly airgapped network that stands alone and runs for 20 or 30 years, what's the actual risk of a cyber attack? Based on that risk that you've all determined, would you pay to add layers of security? Well, they didn't want to either. So, why do we need security? Why

is this an emerging field? Why do people care now? Well, in 2010, um, stuckset happened. I'm sure a lot of you have heard of it. 2010 is not when it started. That malware was kicking along around way a little before that. I think 2007 were the earliest dates I saw. I could be wrong. I'm not an OT historian. So, the 2010 stuckset attack, that's when it notably happened. um was against Iranian nuclear enrichment facilities, specifically the one in Nance. Um the Institute for Science and International Security has a dev devastating acronym, ISIS. It's tough. Um but their report in 2010 stated that about a thousand centrifuges or 10% of that site's capabilities were destroyed. real life nuclear enrichment facilities

being destroyed. There are real world consequences to this stuff. So, Stuckset caught a lot of people's attention. Um, some of the historical reports I read while preparing for this presentation called it the first cyber nuke, right? This was this was cyber warfare at a scale we hadn't seen. Um, it it shocked a lot of the world. That's one reason uh one attack alone is not enough to change an entire industry. One of the biggest factors to OT security coming about is OT IT convergence. I've got some cute little sparkles around that because it is quite the buzzword. It is maybe the biggest buzzword I've seen in OT security throughout my career. Um, and generally all it is is businesses are connecting

their OT environments to their IT environments, to their business networks, in some cases to the internet. That's safe. Sure, that's safe. And so they do it for good reason. These these people making business decisions are not stupid. Um they're looking to increase productivity, increase efficiency, um change systems, make improvements, make more money, um give people clean water, give people electricity for cheaper or more expensive, whatever they want to do. But there's reasons behind this. Okay, so where are we now with OT security? Well, OT security is still significantly lagging behind IT security. Um, there's some people uh at the last conference I was at were saying anywhere from we're only like 5 years behind to we're a full like 30 years

behind it security. Um, and honestly with what I've seen it it could be longer. I don't know. Um, it's kind of hard to pin down. Every site and every business is a little different. So, a little bit more about what we're doing. Um, I'm not going to explain the CIA triad to you all. I'm sure you passed your infosc 101 class, but we don't really use that. We don't use the CIA triad. Um, confidentiality and data is not something we necessarily care about. Sometimes in some sectors we do, but not all the time. So, a more effective way to measure risk is with the SRP triad. I find it incredibly funny. We just decide to make another

triad instead of do something else. Um, and the key points here are we care about safety, reliability, and performance. These are realworld devices. We have to care about safety. When you have manufacturing belts, manufacturing lines, people can be injured. There were reports of an attack in Iran against, I believe, an iron foundry. Um, and there's a pretty horrific video of iron spraying all over the floor because the site lost control of their um controllers. People can die. We have to care about safety. Imagine hearing that on your first day of your first full-time job. Reliability and performance. Um the whole goal here, right, especially in like manufacturing, which is the industry I'm in, uh is to make money. So

we need to make product. We need to do that reliably. So reliability and performance go hand in hand. We have to do those things. So what are we doing? How are we securing these things? There are two primary security tactics. The first one and by far the most important is network segmentation. Um remember how I told you we had air gap systems? The industry as a whole said like let's do that again. Let's go back as far as we can. Let's make sure that what the devices that should be isolated are isolated. Make sure we have good security zones. Make sure we broker through a DMZ all the good best networking practices. And I'm very thankful I'm not a network

architect. Um, another one, this is a little bit newer in the network segmentation. It's network detection tools. I'm talking as basic as they come. We're setting up a span port. We're looking at the mirrored traffic. Maybe we create alerts on that. This is old. This is old stuff. IT security has done network detection tools for as long as I can remember. or maybe as long as I've been alive. But the last reports I've read are companies in OT have roughly 70% implementation of these network detection tools. We're not even all doing it. So, what tools or techniques do you consider to be important for IT security? We're going to start to learn some lessons. We're going to move

forward past what we're at now. These are a few that I would imagine come to mind when you think of IT security controls. EDR, vone scanning, pentests, patching. Patching is a pretty simple one, right? Like let's update Windows. Let's go simpler. How about rotating passwords? Do you guys do that in it? Probably. I hopefully but these are not all currently being done in operational technology and for good reason. EDR if we put these on OT devices for one they're sensitive and they could break with another application running on them. Another reason is frankly the state of OT procurement is not what you might hope. Um when you buy a skid from a vendor likelihood is you can't touch it.

You'll get a Windows 7 box and they say don't patch that contractually. You'll break warranty if you do on your multi-million dollar system. So that's a pretty good reason. Uh pentests Well, I certainly don't want to let a pentester loose in my OT environment, and I don't think it takes a whole lot of explaining as to why. Vulnerability scanning pretty similar here. If we let a scan loose in an OT environment, OT network, uh, you're going to see stuff go down. I've seen pings bring devices down. So, a full OT scan is probably pretty rough. Patching Well, for part the vendors again that same contractual agreements, you can't patch sometimes. Besides that, some OT vendors just don't release firmware

patches. Besides that, why would we patch if it's not broken? If it's correctly isolated, why would we patch? And lastly, rotating passwords. I'm going to tell a quick story about this. Let's imagine a world where we update the password on HMI that controls some process. HMI stands for human machine interface. It's kind of how humans actually get to control the process. We update the password on a shared functional account that the HMI uses. And on Monday, Jim comes in and no one told Jim the password because he was on vacation. And now he doesn't know the password to get into the system and he can't change set points and suddenly we lose a batch. Rotating that password has almost

directly caused the loss of a batch, the loss of product, the loss of money for the business. So what can we do? We can borrow these lessons from it. we can carefully and I mean very carefully because the problem is we just don't know enough to do these things in a safe manner right now with patch management we usually don't know what patch we're on or if patches exist and if you do you usually got to go to a vendor website and find your device and try to figure it out. You got to have a guy walk the site floor and check the device. Vulnerability management. We don't even know what we have in what

VLANs. How are we going to scan them safely? EDR, we don't know if the vendor supports it or if it'll break the system. Pentesting, we don't know if we can test safely. Rotating passwords, we don't know what we have. We don't know what applications we have. We don't know what we need passwords for. So, we need asset inventories in OT. Not in Excel. You'd be surprised when I came onto the job and I walked around sites and I met the site guys. Uh, they threatened to throw me in autoclaves. I was asking about their asset inventories and more often than not I'd get an Excel spreadsheet emailed to me. Not in the same format as all the other sites, just

whatever they put together. And that's fine because that's what they needed at the time to do their job. I did also receive a picture of a paper asset inventory. So now I have a disclaimer about vulnerability management. If your resources are limited as a new OT security group, do not start here. This is wildly important to being able to move forward and implement these controls we learned from it. But if you don't get your stuff off the internet and segmented correctly, I don't think asset inventory is going to help you very much. So asset inventory and you I'm going to ask a few questions here to kind of help you figure out what you need to do to implement asset

inventory in your OT environments. What type of solution is right for you? What do you need? Try to really think about all your requirements. How many systems do you have? Like what do the licenses look like? That's important. Who needs access? Do all of your sites need access to all of your assets? Do they just need to see their own? Does the security team need it? Does the business need it? Who needs access? Can you tolerate the risk associated with active scanning? For current asset inventory solutions that are technical controls, not manual, there are two main solutions right now. Passive, which is more widely implemented. It's that same network detection just with a little layer of

fingerprinting on it. It's not always accurate. It's not always perfect, but it does give you something. And then there's active scanning which gives you greater accuracy, but now we're scanning networks. So, it's scary. It's really scary, but if you do it carefully and you work with your site teams, you can do it. Last, this is one of the last ones. Who owns this solution? If you're implementing asset inventories just because your security team wants them to move forward, you might be doing it for the wrong reason. Asset inventories do enable security, but they're also wildly important for the site teams. These are the guys who know their assets, who work on them dayto-day and are around them,

right? So, this is primarily most helpful for them. So work with your site teams, please. All right. So now you have an asset inventory. Couple things to think about. How will you update it? It's a little tricky. A lot of places when they do asset inventories for OT, they just go out, they get it. There's literally services where you just have guys come walk your site floor and write stuff down. Is that what you're looking for? If you're a small site, maybe. How will you verify accuracy? Passive scanning particularly is wildly inaccurate. How are you going to verify that? I would argue you should work with your site teams, but you might have your own

solutions. So, I'll ask you all what now. I'm not an expert. I started my career in OT security in 2022. I have ideas for sure, and we can talk about them after this if you'd like, but I'm not an expert. It's up to our field to decide what we're going to do and what we're going to implement. Separate from IT security and I would argue even separate from governance and policies pushed by any organizational body. We need to be creative and we need to do it the right way. So that's all I've got for you. I I want to leave you with something to think about. But in summary, this is my first ever talk.

Thank you. [Applause] Um, so please feel free to bully me. I just prefer you do it constructively. Okay. Any questions? Yeah. Um, I just want your opinion on something. I I heard of a startup that is using machine learning for physical anomaly detection. Basically, it just listens to the sound of it around it. And if a machine is doing something wrong, it makes a different noise. Yeah. Uh last year, uh there was some guy who worked with PepsiCo and gave a talk on that exact thing. Uh he's an AI expert. He was on the main stage talking about that physical anomaly detection. Uh my takeaway from it was like it's super sick that we actually We're mimicking

human operators because there's guys who stand on the floor and listen and they'll find problems and know what's wrong with what machine. Now, will AI replace those operators? Probably not. There's a lot that they do. There's a lot like kind of in their purview. They do a lot of maintenance and things besides just knowing what's wrong. Now that said, I do think AI is an incredibly useful tool and I think we could create solutions to help those operators. So, yeah, I think it's a great idea. I'm hesitant on implementation on floors until we have some more research behind it though. Yeah. So, just from a from a monitoring point of view, Sure. Are the people monitoring these OT

systems, you know, PLC's and all that part of the same team monitoring traditional IT assets or is it more? Yeah, that's actually a problem I talked about at lunch today with uh my intern. He he was very curious. Um, and we kind of talked through it and yeah, there's a gap. There's definitely a gap in the industry for OT specific socks. Um, some companies have them. I would say most don't. Like most don't. It's very rare to have an OT sock. Um, that being said, I think they're incredibly necessary. uh because an IT sock is just not gonna have the context behind OT. They're not necessarily going to know about the SRP triad or whatever. They're not going to

know how to handle that risk or what a day on the plant floor looks like. So, I think it's important to split it up, but I'd say unfortunately right now the way it is for most people is an IT sock that gets use cases, reads a playbook, and then usually just escalates straight to your OT security team, which is uh probably not the most efficient use of their time when OT security is this far behind, right? We need people doing more architecting, less incident response at an L1 level. Any other questions? You were talking about how you track things on the floor. No, it's a challenge for getting the people to do this. Yeah. But if you

could the determination part being replaced was done under a ticket. You could always just inventory all the spare parts as well and then just tell you which one they replaced with which one and that way you could track the change. Yeah. And then it becomes a change management problem which is relatively easy to implement if you can get the buy in for doing it. Yeah. That's sort of maintenance change management I would say. I've I've actually seen it implemented pretty well. Like when we're talking about just parts, you treat it totally as a part. Yeah. And since you can't you can't talk to it as anything other than a part. The hardest I would say the

hardest part of that though is initial gathering of information for those systems um for legacy stuff. But yeah, I agree. I think we need to get to that same model. I think that's a great idea. Anyone else? I think we're just about at time right here. Sure. Obviously, a lot of people still follow model. How do you feel about things like this where they're providing guidance on things like allowing interconnectivity with vendors? management. So, first comments about the Purdue system that uh Purdue model. I think Purdue has maybe outlived its life cycle. I'm not a big believer in using it for much of anything besides maybe a reference with some people who don't understand what OT is.

Um, when you talk about direct vendor support, yeah, I've had sites ask me, can we open up a port to this IP straight from our industrial land to the cloud sometimes and I'll say right SRP reliability and productivity if it is necessary we will find a way to do it securely. Now, ideally, we use brokerage systems, right? So, we can use like reverse proxies and stuff like that. There's ways to do it for vendor support. Uh, but unfortunately with that procurement model I mentioned, pesky procurement, uh, sometimes the vendor makes you sign the contract that says you'll use this software and this software will go to this link. And that's tough. I I don't have a

solution for the whole procurement problem, but it is something I'm working on. So, great question. Okay. All right. Thank you guys for listening.