What's the Big Deal with Assessing ICS/SCADA?

The B-Sides DC 2016 videos are brought to you by clearjobs.net and cybersecjobs.com, tools for your next career move, and Antietam Technologies, focusing on advanced cyber detection, analysis, and mitigation. Jim Gilson, I'm from a company called Connexus. We're a small consulting firm, and I'm talking about what's the big deal with assessing ICS and SCADA. So Joe just was doing a lot of talk about IoT and all the fun things you can do with that. I'm going to get into the dirtier, kind of nastier world of ICS and SCADA. So who am I? About four years ago, I joined a small consulting firm, Conexus. We do, we specialize in safety and security so the majority of our business actually

does the design safety systems. So going out to oil rigs and chemical plants and new plants and all that kind of stuff and actually building into the systems of SCADA, the actual safety instrumented systems in there. So the thing that you hit the big emergency stop button and the nuclear power plant goes down or the chemical plant, the oil and gas chemical plant will go and actually refinery system will shut down safely so you don't end up with things like Texas City and good things like that. Before that I actually spent 20 years at NIST in the engineering lab. I started out doing So actually I started out doing robotics and controls and then was told to

take over as a Unix sysadmin back on SunOS and Solaris when it first made that change and Irix and all those good SGI systems. Learned a lot of networking back then, learned a lot about Unix and went back to engineering for a while. And then they came around and oh, this, industrial control system cybersecurity stuff is kind of raising its head. And there's this new standard coming out to do that. And so back in 2000, 2001, I got started in doing ICS cybersecurity and networking. Since then, I've been doing that for the last 15 plus years now. And I got my first certification about a year ago. So before that, I never found the need to

do it. have a big thing about it. I've now gotten three certs in like a year or so, but mostly that's because they're the HR filter in the, the HR filter when you actually put in proposals and RFPs, you gotta get through that just to get through. So there's my Twitter and you can always email me. You can find me on the Connexus website. So why am I here?

Basically ICS SCADA systems are really a lot of it is an extension of IT. So the way they're built now. They will try and basically take the serial protocols and the specialized proprietary protocols that existed for multiple decades in the industrial world for a long time. And they will basically now try and layer them over IP and TCP, UDP, specialized versions, bastardized versions of the whole TCP IP stack. Some of the, like, actually some of the real-time automotive protocols, they actually, there's one called Profinet, which gets down to the point where it bastardized its overlaying their protocol all the way down to getting rid of the link layer for Ethernet. They literally lay over top of the layer one

and the phi portion of Ethernet because they felt they could do it better. So there are all sorts of really strange things that people have decided that they need to do in the industrial world. And the problem is that there's a business need to hook these into the business systems. They need to get data out of them because they need production flows, they need to know the Board level needs to know these kind of things, especially when you get into public companies, they need to know these things on a regular basis. They need to know them daily, hourly, especially now you get into power plants. They need 15 minute kind of like levels of power consumption versus

production and usage so they can do the proper billing when you come to doing AMI and the automated metering and things like that because the electrical market goes that fast and the actual billing prices for different systems change based on the commodities of how things all work. So they need the information and they need more information faster and

in better synchronization with the production that's going on down at the lower levels. Generally, they don't behave like IT systems. It's not the kind of thing where you've got a computer down at the lower levels that's accessing a database that might be higher up in your architecture or accessing an internet web facing client someplace out there in the cloud or anything like that. You've got lots of systems that talk at the lower level Then they aggregate information, throw it up to the higher level, and there may be some interaction between that and another higher level and not higher level and all. It's really fully treed architecture all the way down in most cases. Now, then they throw in all sorts of like ring protocols

and they throw in star topologies and bus topologies and all sorts of really strange things that have been sort of like moving away from the IT environment. They're still in ICS and SCADA. There are valid reasons for having them, especially the ring protocols are quite a bit used in redundant systems for substation communications. They'll use that sometimes where they've got, they need to have multiple paths between that substation and the control center. So they'll actually use a ring protocol to make sure that they always have at least one failure potential in the system for their ring paths. High speed manufacturing does the same thing, where they,

high speed manufacturing, especially like automotive and semiconductor manufacturing, packaging, packaging is like really fast. Pulp and paper is actually one of the fastest that there is out there. They get down to doing sub nanosecond or around 10 nanosecond timing on their motors for like a mile long run of paper. So the newspaper rolls that they big, huge like newspaper rolls that are about the size of like this from floor to ceiling here, those have to spin at like hundreds of miles an hour and they have to synchronize about a hundred motors and a quarter mile to get those to go right. Otherwise the paper will break and they actually take like two weeks to clear out all the paper

and restring the whole thing. So production down times can be pretty bad for that. So you have to understand the different ways that the systems work. in order to do this. And now they're actually being scrutinized, especially after some of the recent attacks, people are saying, wait a minute, these things can do weird things and you can aggregate them and this stuff is on the internet. If anyone doesn't know about showdown already, go look at it and do some searches. There's actually inside showdown, one of the additional things you can go to is their ICS world map

and you can see all the different protocols that exist out there and see some of the different ICS things. And searching around on that, you'll find that there's tens of thousands of easily identifiable ICS systems on the internet. There was a project a few years ago called Project Shine, which was done by some guys that I know where they actually went and searched the internet and found easily over a million devices using industrial protocols on the internet. And these are devices that may or may not have been patched since they were updated or since they were installed. So traditional ICS and SCADA environment, what are we dealing with? There's three main systems that are typically talked about

when you're talking about ICS SCADA. There's industrial control systems themselves, ICS. That is a general term used to basically describe everything. All different types of industrial control. So this is automotive manufacturing and chemical manufacturing, substations, power generation, nuclear power plants, diaper manufacturing, everything is pretty much all using what's called industrial control systems. In most cases, ICS are located in a particular, in one particular location. You're dealing with land connected systems that are all able to talk to each other fairly quickly on, in, general isolated or at least segmented systems. Maybe they have got a firewall, maybe they're using just VLANs because they think it's a real security thing. The next level up is what's called distributed control systems. And a lot of times what these

are, are a lot of the chemical manufacturers, petrochem, oil and gas refineries, this kind of system, what they do is they basically have this monster server that talks to a huge number of really semi-intelligent, anywhere from really dumb to semi-intelligent devices around the plant. And these, now you're talking something that may be a huge oil field, oil tank field,

different types of chemical manufacturing, so nylon, things like that. And you're talking about something that's miles across now. And then you get up to the next level up and you're dealing with what are called SCADA systems. SCADA systems a lot of times are a mix of the local ICS system that runs at a very high speed. So your substations, they have like 50 millisecond timing on their relays for if something goes wrong, it will trip a relay within a certain number of milliseconds. And a lot of times those have to do with the, literally the frequency of the power going through. It's got to go within a certain number of phases. And then on top of that,

there's a wide area network set up so that a control center at a particular location can monitor that and look at all the different stations that exist within a huge geographic region. Maybe it's a citywide, maybe now you're dealing with statewide or even multi-state kind of communications. So when people use the word SCADA, they don't always, a lot of IT people just hear SCADA and think that's like everything. SCADE is a particular type of industrial control system that's usually considered a wide area network like that. And if you walk into a plant and you call it the wrong thing, especially if you're coming in there to do an assessment, they will look at you funny and

automatically assume that you're IT and you don't know what you're talking about. So this is something to actually know when you go into plants or go into even if you walk in and you're doing an assessment on a building or a company and you walk in and you say, oh yeah, where's your SCADA systems? And they're maybe dealing with a building automation system. Their building automation system is a building automation system. It's not a SCADA system. They will kind of basically shoot you down right away because you automatically don't know what you're talking about. So.

And then, as I said, there's a lot of non-traditional ICS SCADA systems or other control systems that exist. Building automation systems are the easiest ones to kind of like start with because they hook into all sorts of things. So everyone's been complaining about the temperature in these rooms. And I'll get into HVAC in the next one. But basically, that is a control system. it's sitting there it's got sensors to figure out what the temperature in the room is uh... it's not like the home system that you have it actually is trying to regulate the whole building all at once because they don't have they've got huge chiller plants where they've got to turn on and on and off big big equipment not just your air conditioner that sits outside

your house they have to turn on whole chiller units and so these kind of systems only get turned on very slowly and it takes hours to cool down an entire building takes hours to heat up an entire building so they it they have to do these things at different timescales and you're used to dealing with building automation maybe things like the escalators the elevator systems the uh... the lighting controls all the other kind of stuff that goes on in the system that they can control. Even things like the audio inputs in here can be considered a control system because behind the scenes they have actually a computer that they can say, okay, I want this audio feed to go to this room over here and not this particular

speaker. And so that ends up actually being somewhat of a control system too. Energy monitoring and conservation systems. This is actually getting to be more

important and it's not just in industrial plants. This is actually a lot of businesses too have to deal with this now. The LEED standards and a lot of that kind of stuff where they get into EPA monitoring of sites and how much power they're using, how much power draw they're getting for different equipment turning on. Well, okay, in fact, when we were setting up here on Friday for the conference, Certain ones of these conference rooms didn't have the power, didn't have the air conditioning on simply because there was no one scheduled to be in giving a talk at that time. And they get charged by the company for the additional power they're using to turn on the air conditioning. So they turn everything off because

they know there's no event in there. Even though that we were setting up, their system knew that we had no event, so it actually automatically shut off. Again, that's another control system that you're dealing with. Fire suppression and monitoring. So all these fire sensors around here, they're actually networked. They use IP networking and they talk over, most of the time the fire suppression is actually a, our fire monitoring is a separate isolated network. A lot of times it's actually red cables. They do that on purpose just to make it obvious. But again, that's an IP based network that's attached to A PC server sitting someplace and they actually monitor that and there's communications between that and the fire department and the police department

locally and they they're all interconnected and again their IP based their Windows servers they're sitting there. They're also control systems. Physical security oh man. So physical security. All the badging systems, your door. The door in your room is not typically. That's actually a local one that's connected only locally at your door and it's a code number that's on the card that they reset every time. Oh, I forgot my key or my key doesn't work anymore so you go down and you get a new code. But badging systems for your company, the little HID badging system, that's actually connected to a network. A lot of times that's connected to the business network. It may or may not be segmented. And there's all sorts of

fun things you can do with that. People go to ShmooCon, go to all these places and they'll talk about, oh yeah, I got this really cool way of defeating that. Well, that's another entry point into your system. Monitoring systems, the man traps in a lot of the defense department stuff, that's all connected in. There's monitoring, okay, that man trap went off, so these guys get an alarm, they've got to come down with the big guns and all and figure out what's going on. That's all interconnected. Traffic monitoring, and control system so not the stuff you see in the movies the real traffic monitoring and stuff they actually do adjust for things like time of day whether it's rush hour not rush

hour the older systems are are analog or locally controlled and and it's like uh... there's some supervisory like at least monitoring that goes on some of the newer systems are actually getting into doing adaptive uh... control that goes and has actually be controlled by a server that exists back in the control center. And they're getting signals to change things in adaptive because they find, oh, there's an accident here, so these guys are now, they're rerouting, all the traffic's rerouting around that. So now they start to adjust from a top end saying, okay, well, let's start adjusting the timing on these things. And then sensor networks is another kind of newish one. that if anyone does military work, CBRNE, these are sensor networks that exist

around different areas to look for plumes of release for different chemical sites or nuclear or so chemical, biological, radiological, explosive and nuclear, I think is CBR, radiological, nuclear and explosive, CBRNE. And so there are these sensor networks. Things like the sensor networks on the lights that exist in the cities. They'll actually mount sensors that exist on the street lights to actually do sensing of

The weather, lots of different conditions. One of them in certain areas they actually will put CBRNE sensors on those that we, they've been experimenting with seeing what they can actually see for environmental monitoring and such. So there's lots of different things and it's all kind of like networked. So I'm going to do a little bit of some stuff that I like to throw in here. So if you live here and you're doing electric power generation or transmission distribution, you don't want to live here in the Northeast blackout. It's not really a place you want to be. You need to think about what's going to happen here. If you live here doing chemical plants or doing

oil and gas kind of things. You don't want to live here in Texas City where their safety system, its redundant backups basically were overlaid because they didn't think about the possibility of something happening. It just didn't occur to them into their risk model. If you live here and you're doing nukes, you don't want to live here. I know like some people in this room may actually recognize this. They may be old enough to recognize. Three Mile Island, yes. Some people are looking at me, what? And you don't want to live here. If you live here, doing manufacturing, car, auto manufacturing, stuff like that, you don't want to live here. This is a little bit old, but the Toyota Prius and their

manufacturing issues that they had. So, and if you live here, even if you're doing diapers or something like that, you don't want to live here, here, or here.

And it has to do with what things you as a business worry about. So it's the risks that your business has to deal with. So what happens when you're doing an assessment, pen test, whatever, and you go in and you find this stuff? It's because these things are being plugged into the network, you're now discovering them. And maybe they didn't tell you don't go to this IP range as you're doing your pen test. And you find some stuff that you probably shouldn't have found. What do you do now? Well, a lot of times you will knock things over if you're not careful. So what do most people do as pen testers? What IP ranges can't I touch? So they stay away from

those like the plague. They literally write everything in their tests to stay away from those IP ranges because they don't want to do anything to them. There actually are ways you can do it. You just got to be careful. And you approach it from a risk model of these systems need to stay up at this reliability. And while when we go in and do assessments on places, I'll get to it in a minute, but basically we have a whole set of rules of engagement written out. And it's like, okay, at the bottom of that, we have a whole series of indemnification of we will do our best not to knock anything over. However, there is the possibility with active

testing that we will screw something up. You will indemnify not to go after us.

for this if it happens. But, and actually at, I was just watching it, there's a really good talk by a guy at B-Sides Delaware a couple weeks ago. It hasn't been posted fully, but he's a lawyer. He actually talks about all of the stuff that you need to write into your contracts, indemnification and like the get out of jail free card and everything like this. I totally forget what his name is, but In the tracks where they literally recorded an entire room like they're doing here, it will be up later on, I know. You're going to have to look at the schedule for B-Sides Delaware, but there's a really good talk where he went through a whole series of stuff that you've got

to think about doing RFPs and doing contracts. So, understand the risks of what you're doing. Understand the risks of the places you're going into. Even if you're just doing a building assessment, understand that if they haven't done it right, their fire systems will be possibly affected by this. Their escalators, their elevators, their building control systems, everything can be affected if they haven't architected their system correctly. And I can tell you, most people don't architect their systems properly. They believe in VLAN segmentation, they believe in Oh, we're behind a firewall and we've got all the crunchy outside and the M&M model on the inside where it's the nice juicy stuff. They believe one firewall at the border

is all they need. This happens. This is the way of the world. Live with it. Understand the risks that you're going into as you're doing these things and work with people. So talk to the customer. You should never go in ready to actually hit things on purpose day one, minute one. When we go and actually do a vulnerability assessment on facilities, we don't actually usually touch the facility until at least lunchtime the first day, usually not even the first day at all. We spend most of the first day doing nothing but risk assessment and risk analysis. we go through and fully like look through where we can touch, where we can't touch, where we have to be

careful with, where we need to basically tell the people, okay, you need to be monitoring this when we do our job, just in case things do go wrong, we can back out, that kind of stuff. Very few things you will do in assessment will cause a crater, or cause bad, really bad things to happen. Usually when it comes to doing like, Stuxnet and all these kind of things, there's a huge series of things that have to go wrong for that to happen. And as long as you were prepared to stop what you're doing, the instant something starts to go wrong and back out or try and work with the customer to fix it, then you're okay. But if you're just kind of going along and doing your thing

and you don't pay attention to what's going on, that's when bad things can happen. So Pay attention to what you're doing, making sure the customer is also paying attention to what you're doing so that you can understand things and try and work things out. Most problems that you will come across doing an assessment of these type of systems is downtime of some sort. A system goes down because you ran a NMAP heavy scan on it or ran Nessus against it and shut this server down or shut this particular PLC controller down or things like that. That stuff's going to happen. It will happen. I've run into problems where you even send a ping to a

device and it goes down because it's just so overloaded that it can't handle this extra thing. As I said, people do strange things to the TCP IP stack. They decide that they want to roll their own TCP stacks. And this is not uncommon. It's actually not uncommon. They really think that they can do it better than everyone else that's been doing it for the last, what, 30, 40 years, because they know their systems. So they just get this in their head that they've got a better way to do it, and things go down. DHCP packets actually coming on a network, broadcast DHCP packets, do really weird things. First of all, okay, so DNS is one of the things that I know like

really bad. It's like all it does all sorts of really weird things to systems, but most ICS and SCADA and even building automation, all that kind of stuff, they don't use DCS. They do statically assigned IP addresses, or if they do DHCP, it's a statically assigned address at the server. They don't do DNS because they're not talking out to the internet. They're talking to a particular IP address. This controller is at this IP address. This IO block is at this IP address. They know that those two talk between each other. So they basically do that. Now the big thing is that unplanned downtime for these systems is real money. They are producing actual widgets. They're producing they're producing

chemicals, they're producing whatever. It's real money to them if their systems go down.

About 15 years ago when I was doing work with the auto industry, they were recording their final assembly line is $10,000 a minute if anything goes wrong on that one assembly line. And if you talk to, I mean, just look at like Amazon or those guys. If one of their servers goes down, okay, fine. But say the ISP that connects Amazon to the world goes down, okay, now you're starting to get to actual money things, where you can start to think this downtime relates to actual things going wrong. What was it, the DNS problem earlier this week where they were having problems with things? Dying DNS went, and so, that's real money when things like that go down.

So really determine what the customer really wants.

A lot of times we get asked to do pen tests. And then we really start going through what do you really want to get out of this? So in almost anything that we do, we do passive network scans. In fact, that's like where we start step one. We do passive network scans and we actually spend most of our time doing So I'll actually go into a site and we'll put, we have actually developed these little, not Raspberry Pis, because actually the TCP stack in them goes through the USB, so the timing on that's not real good. But we've got a little bit bigger devices, Atom based, that we go in and drop at different places around the network. and we can come back, we can just leave them

there for a day or like eight hours, 24 hours or something like that and they'll just capture traffic on boot up and it just sits there and then we can go back and do post processing on the data afterwards. Those, you can get a huge amount of data with industrial control systems, building automation systems, all these different types of systems because they're very deterministic, they do one thing, they keep doing it and that's all they do. So you'll see the same network traffic stream keeping, just doing what it's doing, and you can baseline systems really well that way. In fact, we basically tell companies, you should baseline your systems regularly. And if you see things that aren't in your baseline now, like anomalies,

that's something you should look into. That's just one of those things that we always do as part of our assessments. We give people the baseline that we capture and they say, OK, well, you should probably run this baseline next year too and compare it against it. And we've built some software to actually do some comparisons. Most people, when they say they want a pen test, really want a vulnerability assessment. They do not, and understanding the difference between a pen test and a vulnerability assessment is really important when you talk to customers. Pen tests, you are talking about actually trying to hurt the system you are trying to test. You are intentionally trying to do bad

things and unfortunately with pen tests, a lot of time you are leaving tracks behind. And these systems that you are dealing with are not like Windows servers that you can just load from the VM image that you had, that you took whatever day you took it. These are systems that you have to spend time reloading the firmware on them, then you have to go through and reload the program, and they may or may not have the program readily handled. These are systems that are not meant to do what you're normally used to doing as part of a pen test. So most people want a vulnerability assessment. They want to know where their systems are vulnerable, where they need to try and protect, where they need to put additional

monitoring around it, where they need to actually put additional protections in place. ACLs between these guys and these guys, all this kind of stuff that they can add on top of their system to protect the stuff that doesn't want to work securely. So. Figure out how to basically figure out how to tailor tools for use. Outside of one tool that I actually wrote myself, I don't use anything that's not a standard IT tool. So I use Nessus, I use Nmap, we load Security Onion, load some of those tools, Squeal, Suricata, BroIDS, we'll go through. We do, when we want to do demonstrations, we'll load up Metasploit and do some funny things for like VNC injection to like take over an HMI to just show that we

can do it. but we don't actually go and like proceed farther. That's when we actually want to do a pen test and show them, it's typically, okay, we're going to just take control over this. We're not going to do anything to it other than to show you it happens. When we do anything with like Nmap or Nessus and stuff, use the features that allow you to slow down their attacks. These systems are not capable of handling the amount of traffic that a normal laptop can generate to deal with the types of attacks you're dealing with. They're not used to that kind of attacks. They're used to maybe 2,000 packets a second or really low bandwidth kind of stuff

that a normal PC wouldn't care about. It could handle it. The TCP stack's pretty stable and all this kind of stuff. These are not able to handle that. So they're not used to seeing that kind of traffic. When they do see that kind of traffic, they go, so slow things down. You can still test all the same stuff, just it may take you longer than you're normally used to. Don't be as aggressive. So don't go into doing a full 65, 536

like port sweep on these devices. They're not used to handling that. They have, I've run across a few devices that don't have it, but almost everything has some sort of proprietary vendor protocol and their little configuration utility that talks to it. And usually those protocols are really, really dumb and they're meant because the vendor just said, I want a way to do this and I don't want to use somebody else's protocol. I don't want to use soap or JSON or like this kind of stuff that's out there already. I've got a better way to do it and I know how to do it better because I know my systems. So they go and write their own stuff. It happens. Another thing too

is yeah yeah. I haven't found a single one of the vendor developed tools that does that kind of stuff to configure their devices that is more secure, more stable or anything like that than the standard protocols that exist out there. So just, oh, by the way, if anyone wants to ask any questions during it, just Raise your hand, speak up. I'm not too, yeah. It depends on the protocol. So typically when we put on the, when we put those kind of sniffers on the network, we're hooking into a monitor port off that switch. Some protocols are talkative over the network though. One specifically called Ethernet IP, which is used in a lot of the auto manufacturing.

Yes, Ethernet slash industrial protocol, not internet protocol. Terrible name. It's a particular company's trademark name, so they can't get rid of it. Yes, it was a really bad decision by this particular company. It is totally IP based. In fact, they are really good. They actually made a point from the very beginning to say they were actually going to use the entire TCP stack. So they actually made a point of saying, we're going to do it using standards. We're not going to overwrite things. When we want to do a particular feature, we're going to use standard ways to do it. So they're now, they're actually in the process right now to incorporate security into their standards. So they're using TLS, they're using SSL, TLS,

all this kind of stuff. They're using the actual TCP IP defined protocols to do it because they said, we don't want to reinvent the wheel. this has been done so we're just going to overlay our stuff on top of the good protocols that exist. So when they want to do high speed control they use the precision time protocol, the IEEE 1588. When they want to do security they layer it over TLS because they didn't want to reinvent the wheel. They saw what WEP did, they saw what all this kind of other stuff did and realized that these are industrial guys wanting to do industrial things, not this. And that particular protocol uses multicast for its communications. It's a

publish-subscribe kind of thing. And so since it's a publish-subscribe, they assumed that one device was going to publish their information out to multiple devices. So they used multicast. The problem is that that meant that the industrial vendors couldn't use their layer two dumb switches anymore. They had to go with a higher power layer, two and a half layer three switches in their networks. So, of course, certain switch manufacturers, large scale switch manufacturers that everyone knows that may or may not have actually come to the conference today, they are selling lots of really cool equipment because this protocol really likes level three switches, layer three switches.

But yeah, most of the other protocols though are unicast. They want to be unicast because they just say it's a master talking to a child, kind of a master or slave kind of relationship. I want this information from you. Give me that information. Client server kind of stuff. Most protocols work that way. Yeah? So most of the stuff that I deal with in serial, We deal with Ethernet to serial converters. So we're not getting, I, my company doesn't do a huge amount with lower level stuff because we're finding that the IP networks are designed so poorly that we can't even like really think about getting to the lower level because they're not even doing the basics right.

So typically we go in and there may be a corporate firewall. There may or may not be a business to process control network firewall. They believe in dual-homed NIC kind of control stuff. So business NIC, process control NIC, all sorts of really fun things. Yeah, there's a lot of different things that since we haven't, when we're brought in, we generally haven't bothered to go down to that lower level yet. But I do know people are doing that. And there are, one of the good things is Wireshark has a lot of those things built in. And so there are people that have built cards to do serial capture. And basically you can feed that into Wireshark. And

like you can do your own If you don't have the protocol, you can define your own protocols in Lua or whatever, or actually write your own decryptor or decoder in there. But most of them actually have stuff in there. It's really kind of cool. Another thing, too, is second guess the results you get from the tools. We actually had a customer that

We captured data, we actually ran our stuff through data. We found this really weird protocol that was going on. It was definitely a vendor proprietary protocol in the end, but what happened is when we actually started researching it, it popped up in the system as a high, basically Nessus labeled it a high. And so we started investigating it. It looked like it was malware on their systems because the protocol was, the IP,

TCP port was right, the types of data that looked like it was coming across was being analyzed in Wireshark in a particular way, so it looked like it was all this malware traffic coming out. It turned out it was a vendor proprietary protocol that the plant forgot that they had taken half of this system out. So what it was is it was at the top of a smokestack They had this environmental monitoring system. And then they had a system down lower and it would use wireless to talk between them. And down at their plant level in the building where they could actually get into the main network, they had this receiver in the system and it was going out and communicating over into the network. They had pulled the

one from, they pulled the one from the actual network down lower, but they didn't pull the one from the top of the smokestack because it was too hard to get up there. It turned out it had another receiver that was in range of it that was receiving that information, and it was just a vendor proprietary protocol that ended up getting blasted onto the system and looking like malware. So don't always trust what you get, especially when you're dealing with industrial protocols. they decide to use ports that may look like malware. If you throw it up into VirusTotal or if you run it through Nessus or whatever, it'll come back with all sorts of really weird things.

You have to actually go and look at those and say, is this really something that's bad? Or is it just something that these tools think is something else? So most PLCs don't come up properly when you run an NMAP scan on them. NMAP tries to do its analysis of like what operating system is this. And it will come up with all sorts of things and just really strange kind of like ideas of what it is because its idea of what these things is based on the responses back during the TCP, like the different attacks it's running. And so you don't always want to trust what it's telling you. Sometimes you have to actually, turn off the recognition that Wireshark does, turn off the protocols and actually just start

looking at the actual hex or looking at the packet decodings and seeing what's going on. When you actually go in to do an assessment someplace, whether it's doing something in a building system, doing something at an actual ICS SCADA kind of stuff, There's a bunch of questions you got to ask before you even go into this. So what kind of personal protective equipment do you need? And yes, this is a picture of me. So do I need a hard hat? Do I need safety glasses? Do I need a vest? Do I need fire retardant clothing? Do I need safety shoes? Do I have to worry about steel toes versus Fiberglass safety shoes in, so that's an issue in manufacturing where they deal with crush injuries to your toes. If

you're in steel toes and that gets crushed, you'll lose your toes whereas if they lift that off, the steel doesn't actually release and so it bends and actually like bites into your toes whereas fiberglass will release once the weight's released. So they have different regulations for different industries. Do you need chemo, the ones that I have are chemical resistant. They're steel toes, safety shoes, but they're chemical resistant. So I can go into chemical plants and not have to worry about slipping or having the actual rubber on the bottom deteriorate because of the material that I happen to be stepping on. Things like that. A few years ago, I was in a ammonium nitrate plant. And this one makes military grade ammonium nitrate, so the stuff that goes into

C4, TNT, all that kind of good stuff. And I made the good decision to take my boots off before I went to the airport. My partner did not.

This is back before they had the TSA pre and all this kind of stuff, so they had just started putting up the Smith machines that have the puffers in them. And he set off all sorts of bells and whistles because his boots had imodium nitrate residue on them. And we were coming through with a whole crap ton of equipment and wires and laptops and everything else. Yes. Needless to say, he had the love-love treatment that day.

Yes, in the back. Explosion proof.

Yes, that's another issue. Because of the network stuff that we do, a lot of times we're not allowed into those areas, but it is something to consider. A couple of years ago I was doing a corn processing plant, and corn dust is explosive. So, of course, the actual the racks that they have in that plant are actually positive pressure racks. So they actually force air into the rack so that any dust that is in the area does not get into the rack. What they found is that corn dust is not just explosive, it's also corrosive. It's corrosive to copper and corrosive to things like electrical. So it was corroding the electrical connections in the UPSs and it was corroding the

network switches because they were using RJ45 connectors on their network switches. They weren't going with like M12s that are environmentally sealed or even the big environmentally sealed RJ45s or anything like that. They were just using regular like Cisco 1U switches. And so they ended up basically every six months they'd have to go and replace a layer three network switch from Cisco, which they're not cheap. Because they found that it literally was like corroding the connections on the RJ45 pins and all kind of weird stuff. So yeah, there's all sorts of really weird things you gotta deal with industrial plants. And then, oh, actually, cool. Safety training. Another thing you got to deal with in a lot of places is

maybe you have to take an hour's worth of safety training when you first show up. So you got to plan for that when you actually show up for your engagement. If you got to be there and meet with these people at 8 o'clock in the morning, well, you got to take your safety training at 7 a.m. So that means you got to get up and get out the door that much earlier. This is actually something we're dealing with right now. Can you actually plug in and use your equipment on their network? Regulatory requirements in some plants will not let you actually touch their equipment. You are allowed to tell someone else what to do and they can collect the data, they can actually go and run the tools, but

they have to use their equipment because it's regulatory mandated that they have to use that vendor's, or not that vendor's, but that end user's equipment. And only end users that are qualified and usually it's like four months worth of qualification to go through to actually get that. Logistics of communication, this is another one. Because they're dealing with a lot of times proprietary stuff where they don't want it going out all over the world, do I have to encrypt the data going over, and a lot of the people that deal with military, government, all this kind of stuff, they're used to this. But the industrial guys, they don't. A lot of times they don't know how to do this properly. So we had to explain how to use PGP and

things like that to some customers. They were regulatory mandated, they had to do this. In order to send data back and forth, they had to actually use different types of systems. Where are you allowed to actually store the data? Can you store it on your systems? Or does it have to go on one of theirs and you access it? All kinds of weird things like that. During the engagement. This is the big one. What are the risks during the engagement? When you go on site, where you spend the first few hours at least going through and figuring out where are the risky parts of the plant? Where are the risky types of systems? Am I

going to go in and knock over the fire system and then have to deal with that? Am I going to be connecting up and hitting the... hitting...

the control system for this particular really bad part of the plant, ethylmethyl death or whatever, just all sorts of different things. We joke around. That's one of those things. Always do a walk down before you do anything. What I mean by walk down is literally walk the network, walk the actual plant to see what stuff is, get a feel for what's there. That's one of the reasons why we go in and we actually like wear the full gear. We'll go out and I mean I'm going in wearing radiation badges and my most recent customer just because I'm doing a nuke plant recently. Just go and literally walk down around the system to see what's there, see how it is, get a feel for

the environment you're dealing with. come at this from sort of like IT ivory tower. They really hate that. If you can actually go into a control room and walk in with a box of donuts and talk to the guys that are actually like control operators, they will love you. They want you to treat them like real people. And a lot of times IT comes in and they enforce all these policies. They don't want, they just They just are used to IT forcing themselves in and doing things to them and making them have to go through stuff and oh god I got to remember passwords and this and that. You're seen as IT in a lot of cases so you want to go in and actually

talk to them as real people and learn what they do. Spend 15 minutes or something like, spend five minutes even just talking to them, find out what they do, how they do their job from day to day. If they need to move something from this computer that's on the protected network to this computer, that's their email system, how do they do that? How do they do their job? One of the big things that we always insist is, will someone be monitoring the system when we do our job? Yep. Will someone actually be monitoring the system? So is somebody going to actually be an operator sitting there to look for weird things happening if we're starting to hit NMAP and starting to do things? And then how are we

going to report things? So this is, reporting is one thing of, okay, we're going to just issue a report and all this kind of stuff, but what if we find weird things? It has been known to happen to find porn, to find games, to find malware, to find all sorts of things. On one engagement that my partner was on, he found child porn on the industrial network. The problem with that is he has to go to the FBI regardless he cannot ever stop and not tell the FBI about that. Which means that the FBI now comes in, stomps all over them and shuts them down. Certain cases you have to deal with that kind of situation. In most other cases

you report it to this guy over here and only this guy if you find bad things. Don't go to this guy You have to go to this guy over here. Maybe it's the CFO, maybe it's the IT manager, something like that. But you have to go to a particular person to report your results. So that's actually the end of my spiel. If you got, yes, please.

So one thing I didn't say is that I'm also the co-chair of the ISA 99 committee. So if you've heard of ISA 62443 or the ISA IEC 62443 series, I'm the co-chair of that committee. Generally, we actually find a lot of people using the ISA 95 Purdue model, at least thinking like that, because it's a manufacturing kind of way of looking at the system. So when we go in and talk about, okay, you're dealing with your level zeros or your level ones, they actually understand that because that's been around for a while and it's very industrial control system manufacturing based. Now that works great for manufacturing systems. When you start going into SCADA, actual SCADA systems where you're dealing, it

breaks the Purdue model a little bit. But generally most people can at least describe their systems in some form of Purdue model. Yes. Yeah. Is any infrastructure using wireless and Wi-Fi? More than you would think, yes. And a lot of times they don't even know they've got Wi-Fi in the system. It's the way war driving used to work. It's now wireless in some shape or form because the operator doesn't want to move his computer from this side of the plant to that side of the plant. or other little things. A lot of times they'll actually use point to point wireless links in places. Tank farms are like that a lot. They'll use wireless links between things because it's just this huge long distance to

actually get from 10 miles away from the other side of the plant. Nuclear plants are actually like this in certain areas where they'll use wireless because it's $5,000 to $10,000 a foot to lay cable. when you're dealing with anywhere around the actual reactor itself because you have to go through protected cement, protected kind of systems and things like that. So wireless is being used quite a bit more. And I've been told to wrap it up here. So I will go ahead and end it here. And if anyone wants to talk to me afterwards, feel free to do that. But right now, we're actually going to go over to the room next door for the closing ceremonies. So please

visit us next door for the closing ceremonies and I'll be around.