There's No Place Like OT: Lessons from the Wizard of Oz for Securing ICS

Some would argue that I'm better. >> Yeah. >> So, thank you guys for having me. I'm Anthony George. Um, I'm the founder and the owner of GCE. And today's talk is AI and IoT manufacturing Ohio. This is actually a talk, this is two talks. The talk was originally supposed to be um uh a talk I gave for OC in Witchah last year um using analogies like the yellow brick road and flying monkeys and ransomware and such. But then I piled a talk together with the talk I gave in Houston 3 weeks ago. So you're going to get two talks put together with some um ideas mixed in there. I'll take questions throughout um and we'll just

go at it right here. So who am I? Um I told you I'm the founder of uh GCE. So we help small and medium manufacturers digitally transform industrial environments. I'm retired cyber warfare operator. I spent 14 years with the Kansas International Guard on the cyber protection team. Um that's reserve job. My full-time job is uh or was a senior systems engineer for Rathon, a defense contractor. 5 years ago I moved back from Italy and I launched GCDE. Um initially we did industrial cyber security um for which I came to small medium manufacturers but blocking those plants I realized the core problem they had was not that they didn't have industrial cyber security so they didn't

have anything connected they didn't have any need for industrial cyber security they were clearly lagging uh behind in adopting technology so um I taught as a project manager at WSU for a little bit at Builder Smart Factory I was the director of robotics for a short time um I spent six six years in the Navy as a cryptologologist. Uh I joined the Navy to be a Navy Seal. I got hurt and I got resigned cryptologology. Uh actually related to that this morning, uh my son is leaving for the Navy in a month to be a Navy Seal. Forgive me for whatever that's doing. Um and we did drown proofing all morning. So I did drown proofing this

morning. So if I pass out, that's because that's because of that. I'm a little bit out of shape there. Um all right. So the initial talk that I gave in Houston um OT set uh was basically the difference between how the military um secures industrial environments versus how civilians secure industrial environments and uh compare and contrast the two hourong talk. I'm going to condense two 1-hour long talks into 25 minutes. So I'm going to talk fast. So, if you're used to watching YouTube videos of 1.5 screen, this is you. So, what happened? This is my business. This was my business, you guys. So, this is my shop out of my home. This is my son's truck

right here. And the day before I talk in Houston, my shop burned down. And um I still have an office location in town, but all my prototyping, all my industrial equipment burned down. And I used that to to talk in Houston about the importance of mostly asset knowing what assets are in your environment, asset management and inventory, right? Because the first thing that I needed to know was not how much is this going to cost me is like what was in that, right? Customers projects and that my own projects in there. I did not have a proper asset inventory. And the first thing I do when I walk on site in most sites is figure out what they have. I

walk through I do what's called rapid plan assessments. I'll walk in manufacturing environment and I'll look at their problems and I'll be like you need a robot there, you need databases for all that. We need to deploy an IoT network here and that system needs to be segmented completely because that runs your entire operations. I was not practicing the same rules that I had my customers practice. So I gave a talk on that. So we're going to overlay some of this. This is the aftermath. This is what was left. Flag burned off my barn, but it's a t It was a total loss. 100% loss. My entire business was in that barn. 100% loss. So, how do we for one, we're going

to talk a little bit how I could have mitigated some of this. I'm going to talk about where the fire came from is actually inside job. And um how we can contain the spread of not only cyber events, but fire. So, we'll use those in parallel. Um that shop. Okay. So, here's what happened. my chicken coupe caught on fire spread to my barn spread to my fence spread down my fence spread to my home my home this is a corner of my home right so there was a little bit of I wasn't home at the time I I pulled up the next morning my phone had broken the day before I go to rep broken my phone

some way pulled up the next morning and this is when you pull up to your house you have no idea what's going on totally doing me for a loop right where's my where's my kids where's my family where's my chickens where's my stuff okay Look, we start with visibility. We need to know what's in there. >> Does anybody know who this is? >> Why can't Ray Charles see his friends? >> Anybody? >> Cuz he's married. >> I found you guys. Stat. Um so inventory location what is our assets doing so manufacturing environment what it's doing is probably more importantly than what kind of asset it is so we group them in you know criticality in the

environments and how we segment them called zones and conduits in manufacturing environments same thing as subnets vans um and IT um how does it act so OT um assets act in very traditional ways so there's a lot of um fingerprinting we you around those to understand how how they're acting and how they might deviate from that. You have any questions at this point? I'll take questions. Okay, I'll take any of them. Okay, four questions for asset visibility is what do we have? So, this is real world guys. This is what happens. I walk shops all the time. you know, I used to I used to be a little higher level guy where I didn't go down

the industrial environments. The first time um I was on a mission and I and I was essentially looking at a manufacturing environment from the internet. I ran into a PLC and when I got to that PLC, I did not understand how to write the code. I don't understand what the code was. If those who don't know, it's called matter logic or structured text on a PLC. So I went down to the local college and enrolled in Allen Bradley going into a brand of PLC's and I learned how to program PLC's because unlike the IT world, the OT world has very unique operating systems um and structures. And so when you get down in there to really have the effects

you want, you really need to know what the environment is doing. So if you take this for example, this is what's called a machine press. This machine comes down and just with 30 tons of pressure boom boom boom stamp steel. This is uh this is my one of my first customers largest RV air conditioning manufacturer in North America and they make the base plate for the AC right here. Comes through here. This machine press rock solid built in a 72. It works better than presses you would buy now higher quality machine. What we have here is called a die saver. This is this is where it all all comes together, guys. This is this is what's happening

in manufacturing. This is never connected to the internet. Never meant to be connected to the internet. You turn the switch on, it runs, right? This die saber essentially acts in a way that if the machine gets jammed, instead of continuing to smash the parts on top of each other, damaging the machine, damaging the tools, they have they add this system in line. So when the machine stretches in a way, it causes the machine to basically stop. So this can stop the machine. This is modern man. This is modern. This is old. If somebody was to hack this now, they could shut down this machine. So what happens? This machine built in 1973 is now being connected to the internet

using this using this. Right? So and and they have to do this because this saves them a lot of money. That tooling cost $15,000 every time they rebuild it. If they don't have this, they pay $15,000. Happens once a year. You add this. Okay. Now you also now they come to me and say okay well we know this tool knows this piece of equipment knows what product is what raw materials going in there how many parts it's making and the count of and uh the output like the part it's making the part the part count and the raw material. So we need data from this. So what how do we get it? We connect to it. Guess what we just did?

Connected this machine to the internet. So now this machine which runs this entire factory is vulnerable. This is this is real world. This isn't theoretical. This is me walking plants every single day. Um how am I going? >> You good? >> Okay. Uh so the factory floor then and now we know we're going through a digital transformation. I think everybody here's probably heard of industry 4.0. Um a lot of people would argue that we're in industry 5 right now. Um just just to be um okay. Yeah, here we are. Okay, so this has some words up here about the Wizard of Oz and stuff and you can read it and I wish I had some dancing stuff out here

and there's big plans but I got called late for this and you're getting the abbreviated version. >> We love you for it. >> Yeah. Well, I love being invited here. I love I love I love you guys' audience. Um the industrial revolution. So the first one we invented the machine cotton gin you know all sort of just doing it by hand we made the machine lots of machines doing lots of different things and then the second one electrification I think we really start to see this in the assembly line with u and the Ford assembly line the third industrial revolution is going to be computers this is the PLC that I talked about this is

where we start you hit one button on the wall and all the machines start the whole process starts this is where a lot of manufacturing shits today industry Industry 4 is connecting to the internet. The internet's for networking the internet for TCPIP in 1999 when they won the internet war. That's when industry 4 took off. Now why are we still talking about 25 years later? Because manufacturing adopts slow. The world adopts slow. Everybody adopts it slow. These things lag. These things lag. And every industrial revolution is getting shorter and shorter with the initial one being 100 years long, 50 years, 25 years. But now look, we got four to five. We got five going on. We

don't even know what six is right now. But we got we got question. >> Yep. I'm just curious. So, as someone who's not in the OT field at all, I wonder do in your opinion, do you think it is better for current manufacturers of like large machinery like that to begin to adopt that and integrate this capability internally? or do you think it's more beneficial for like third parties to be developing these PLC's that can then be sort of bootstrapped to any well like any compatible device shall we say or if there's any benefit to either of those one or the other so uh if you were to ask me a year ago um or let's say two

years ago I would say build the software for your company do all the things but now with the way AI it is we can put any device on there. We can use AI to write the C code or whatever and do that. So, there's a lot of different theories. Do you buy a system, an IoT system or network and overlay on this or do you build your own? Build or buy? Build or buy. That's the that's, you know, time tested. Build or buy your price point. But with the invention of AI, building seems to make more sense than buying. Just like in software, if I took an ESP32, which is just a simple simple microprocessor,

cost 10 bucks, I can I know how to write C code, my integrated computer programming, but somebody that didn't could go on there and write C code that pulls the same information that that die saver is pulling for 10 bucks. So now, as a manufacturer, you you have the ability to unlock it. So what I suggest to my manufacturers is pay me a bunch of money. But after that, after that, you know, what I really suggest is let me get in there. I go I go do the look. You know, my rate's 215 hours, so it's expensive. I find the person usually in the organization, a young person typically, but it's not always young. You know, he's tinker. He's the guy,

whatever. I sit he knows what's going on. He's running it. He knows their ERP. And I sit down with him and he works it through with me. And we decide if with a if in the organization there's somebody who can take on this digital transformation or if we need to carry it a little bit till we can identify somebody move in there. But long term the solution isn't for integrators like I'm an industry 4. Auto integrator if I haven't said that. This the long-term solution is not for a customer to pay for me to go in there and put those things on those devices. No. The long-term solution is me to continue to figure out what the future is doing.

what the future's doing and you running your manufacturing facility because you know how to run a manufacturing facility. I work in spices. I work in RV air conditioners. I work in aircraft parts. I work in military. I work in custom machine builders. I don't I never know your production as well as your production manager. I can stand next to production manager and honestly this is the most difficult part of my job. You try to get him. That's it's always a him, but it could be a herim him or her to be adopting the technology when they're trying to get things out of the back of the factory. If things aren't coming out of the back

of their factory, the owner is mad at them. The company is mad at them. So, their job is to make sure things come out of the back of the factory. And if you're a ransomware of the factory and you don't have things coming out of the back, you're in big time trouble because that's how you make your money. If you make beef like JBS hacked three or four years ago, they get hacked. They get ransomware. Eight million pounds of spoiled beef on their hands, you went from a cyber incident to an environmental incident. All kinds of negative effects flow when things don't start coming out of the back of the factory starting with 200 people

standing around making money that and you're not. So, it's very important that these factories make maintaining through misconfiguration, which is the number one thing I see in manufacturers. It isn't it isn't ransomware. It isn't some cyber attack. Isn't nation state you guys I'm I'm dealing with these companies. You got 200 people. They're not necessarily targeted by nation state. Although they could be if they're the lowhanging fruit. What they're trying to deal with is they're trying to stay competitive in this world, you know, and they need somebody who understands cyber security. They need somebody who understands engineering. They need somebody who understands their operations that can go in there and do that. Because I handed a tablet to a guy

not long ago and it's like sliding them up, man. Slide it. Do you got grandkids? What's your deal? These are the people that are working down these things. And you can't just say, "No, fire that guy because guess what? That guy runs the plant. He's a production manager. Like half of those half of those employees are felons, you guys. It's not it's not easy to go tell a felon who who's on work release to go do something sometimes, you know. So, that guy is very valuable. You have a question? I was just curious uh when you're working with these different size organizations uh sorry when you're working with these different size organizations uh I'm aware that there's like general

frameworks for how you would kind of set up you know an IC environment like a Purdue model or an IEC standard with some of these small organizations do you have to kind of choose a a light you know version of that where they're not going to get all the bells and whistles but there's like the 20% that's going to get them 80% person with the benefit that you really uh hammer on. >> Yeah. So, separate their IT and their OT. So, so look, I I land in the organization, I go to the IT provider, the managed service provider usually or sometimes in house. I sit down with them. We're in the same room. We're trying to talk the same language. I

translate between the IT and the OT. That's what I do really well. So, and I make sure they get the same page. The OT people, I already told you guys, their job is to make things come out of the back of the door. It's job is to make sure you guys can't do anything. Basically, is what it feels like, right? The INT guys are the ones that want to shut it down. No, that's not secure. No, I need this. No, you can't. No, I need this. You got to sit there and figure out how we solve their problems with making them comfortable. And it needs to go more from uh compliance and regulation, which we've felt for a long

time. It's like the no guys to enable them. You need to go to your IT guys, I need this dashboard. So the way I do it um and the way it's being done increasingly is I land in I land in the organization. I create a a Linux uh VM and I use that to pull data from the ERP and from spreadsheets. I package that up and send it out to the cloud, put in S3 buckets, store their data up there and I build software tools against it. That's meaning that means when and no inbound. So use MQTT use MQTT protocol outbound only. A little technical here, but you literally don't expose any inbound ports open. You only send it out and you only

use it to make decisions. Are we using AI in manufacturing? I'm using it to build tools. I'm using it to build logic. It's not built in real time. It's not it's not quite there yet. I have a food manufacturer. It's very difficult to schedule. You got five different blender sizes, different densities, uh, food recalls, allergens, colors mixed in here, salts, sugars. That blender can't run that. 5 minutes, guys. And we got 27 slides. Let's go. Um, very difficult what these guys are facing right now, but but my customers that are applying the technology are killing it compared to my compared to the people that are not. So, we're going to see a big a big turnover in

especially around machine shops. I think machine shops because that's there's more well structured data in CNC machines than there is in the original Chat GPT release. So when people start connecting those to the internet, if you don't find the efficiencies in your CNC machines, you won't be able to compete. Um >> yeah. So, uh uh it is CIA, you know, uh and uh OT is going to be availability, integrity confidentiality availability, right? That thing needs to run. That thing needs to run. I don't care if China looks at it running. It needs to run. Obviously, I don't want them to, but if it the most important thing is that when you hit the button and emergency shut down, it shuts down.

That's really important. This is why we don't put multifactor down on the factory floor. This is why we don't make it overly complicated for the guy that's a temp guy, right? You know, my users on the factory floor have user one, user two, user three, and some first names in there. Yeah, I know it people get them, walk out of this room right now when I say that. But if the tools aren't usable, they're not going to use them. If they're not going to use them, there's no point in even having them and the business is going to go under. You have to meet them where they are. these guys are making stamp steel because my my customer that

makes military aircraft part that's a different scenario right but know your audience you know there's got to be a level of trust in your organization uh okay this is be real quick here I open the machines I try to get data from the machines this is what we're looking at IoT makes way more sense than me trying to pull data out of this and figure out first of all if I touch this my son's actually on the site with me with this one we Open the cabinet. It's like, "Okay, we need to pull power." We open this cabinet, looks for power. There's power in here. Certainly, I ain't touching that because why? I'm taking on all this risk. That's running

that giant chainsaw in the background, right? So, this is this is what you're dealing with. You got to have an understanding. This is not this is not it world here. That's OT world. Um, okay. There we go. Is there is there any quick question before I finish?