Network Reliability Monitoring for ICS – Going beyond NSM and SIEM

So good afternoon. Uh my name is uh Jim Gillson and I'm going to be talking about network reliability monitoring for industrial control systems. So ICS SCADA. Uh and this is really going beyond what NSM and SEAM can do uh into areas that they don't really work out so well. So a little bit about me. Um I'm a senior investigator at Kexus Consulting. We're a small consulting firm that does uh reliability uh systems for uh industrial controls and SCADA. So uh a large portion of our business does safety systems. So fire and gas and safety uh and uh process hazard analysis, things like that. I work on the ICS networking and security side. Um so we do network assessments,

uh security assessments, designs, things like that. Uh, and I'm also in the process of developing some software to actually automate some of what's in here. So, I will not mention my product anymore, but basically we're doing this kind of stuff as we go along. Uh, previously I was uh in the NIST engineering lab for 20 plus years uh doing cyber security and uh network performance for ICS systems. I've also worked in uh control systems, automated vehicles, wireless sensors and systems, things like that. Um, and uh, for those that know anything about the ICS cyber security space, I'm the co-chair of the ISA 999 committee uh, which develops the 62443 series and I'm also developing the

security program for uh, ICS SCADA as well. So, so what is network security monitoring? So, NSM is I'll read this. This is the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. And it's a way to find intruders on your network and do something about them before they damage your enterprise. Um, so it uses a lot of the existing information that that you ex that you have on your system. So CIS log and a lot of the different uh uh logging capabilities and tracking capabilities that you have on your network already. to monitor for things large scale over your whole system. Um, now a lot of times it involves adding

sensors to your system. So you've got to add uh sensors around your networks. A lot of times they're at your the points that cross over zones. Um but uh so they they try and take all that kind of stuff aggregate it into one location and then look for security incidents. Um now what's seam? And seam is something it's sort of similar um but it uses a little bit different technology um and so seam products are available for real-time analysis of security alerts generated by network hardware and applications. So they do data aggregation correlation alerting dashboards a lot of the a lot of similar kinds of things uh uh but they use different technology. A lot of times

seam systems are a dedicated product uh and seam fits into a larger scale NSM system but uh seam systems are something generally it's a product you buy from a company that does these things. Um so this is not a talk about NSM and seam. I am not an NSM expert. I'm not a scene expert. Uh I know a lot about the lower level protocols out in the IC space. Uh if you want to find out more about uh NSM, uh there's a rich Richard uh and I always mess up his name, Bich, I think. Um he's got some really good books on NSM. Uh and then uh one of the other guys in the IC space, Chris Cyrunk, he's

got some really good presentations. Uh he did the one at Defcon this past year on NSM for uh ICS systems. Uh, and then of course you can pick your own search engine and find all sorts of stuff out on the internet. Uh, as always, take it with a grain of salt. Especially if you're going to vendor sites, they're going to claim they have the end all beall of everything. So when does NSM or when NSM won't work? If you can't observe the traffic that you're that you care about, NSM really won't work. And then node to node activity in your network though is largely unobserved by uh at the network level. So when you have a lot of systems

that talk to each other and they're not actually going up through the architecture uh and crossing over zone boundaries, you don't have a lot of visibility into what they can do and what what security indications there are going on. So here's an example IC SCADA network architecture. Uh I've this is the the upper level architecture. So this is dealing with a lot of the server communications. So we've got uh business level systems and they talk down to systems in the the DMZ uh and then they talk down to the plant control servers. Uh and generally you've got most of the traffic crossing across the boundaries. So you've got the majority of traffic at the servers are are doing information

passing between the DMZ and then the DMZ is doing information pack uh down into the plant servers and vice versa. So you're getting a lots of circular communication going through these zone boundary points where you can actually put NSN NSM sensors and get a lot of information out of your uh system. So you can do a lot of security uh monitoring at that. you're you're not dealing with a lot of the ICS protocols at this point either. So you've got a lot of uh um more common protocols being used. So you've got XML information, you've got uh OSI PI databases, you've got different kind of uh higher level protocols that are more well-known and there's uh um stuff that's been written

to actually analyze those uh even in the IT space. Uh and then you've got more common platforms. So you you have normal server kind of hardware. It sits in a rack and it it gets all the normal data uh data server kind of systems, dataccom centers. So they're you can install things on them. You can actually put software on these whether they're Linux, whether they're PCbased. Most in the industrial world, almost everything is is Windows Serverbased. Uh it's rare that you will find things that are not Windows Serverbased. Uh but things like Active Directory are run on here. Things like uh um uh SQL databases and a lot of this kind of stuff that already exists

and the IT world knows about they can they can analyze in NSM systems uh and and get a lot of information out of that. So, as I said, um most of the network traffic crosses carbon boundaries. Um you can actually observe it with uh your actual uh NSM sensors that you can put in line. You can put taps in there and actually pull things off. Uh you're not dealing with as many ICS protocols. Uh so there's more automated traffic analysis, so you don't have to do as much by hand. Uh you don't have to have an engineer actually sitting there looking at it. you can get Splunk or or whatever whatever product you choose to

use can do a lot of that analysis for you. Uh and then you can actually get things that have security logs. So you can find uh you can install agents on them. Uh maybe you can actually get access to uh uh SNMP information. You can do your uh CIS logs. You can have a lot of that information already existing on your platforms and you don't have to do anything special to get that out. you just have to pull it at at particular frequencies. So now what happens down at the lower level of industrial control systems. So in these kind of cases you have a lot of communications that runs locally in these sub networks that exist down here.

So this controller is going to talk to some devices here on this field bus and it's going to talk to this little HMI server human machine interface. Um, and maybe that's a PC or maybe it's uh some form of specialized panel PC or things like that, but almost all of that communication resides inside this switch and never leaves that switch. Uh, and a lot of times these are uh layer 2 switches as well. So, you're not talking about really smart kind of devices. Same thing. Uh, sometimes you have uh it's not uncommon to have radio links between remote sites. So this may be uh an actual pointto-point wireless uh dedicated link um using directional antennas and everything like that or uh

it's not uncommon to have radio links uh or sorry uh um satellite links between remote facilities. Um generally you won't see them it's not it's not uncommon but it's not really as common to see them using the public telephone system anymore. most of the time they're doing more higher bandwidth uh radio links uh than trying to go over the POTS network anymore. So uh but the majority of all that traffic resides and stays in these small remote sensors. Uh you may have say an engineering workstation that can talk to these over the network but it isn't constantly communicating. This is when they need to make a change. They go out, communicate with that controller and they they do a particular task and

then they log off because you don't want to leave that session up and running especially if these are low bandwidth links that are running over radio or satellite things like that. Uh you may have uh environmental sensors that are in your system. So the EPA comes into certain facilities and they've got to monitor 24/7. So they've got their own set of uh sensors that have to go through and a lot of times there has to be a very specialized link through VPNs and or like their own separate line that they even bring into the facility. Only a very small amount of information actually makes it up into the plant servers. So the majority of traffic

doesn't even really leave this network zone at all. And a lot of it, I would say 80 plus percent only stays inside those local switches. So you're not really going to have access to a lot of that information. Um, and you're also dealing with very dedicated ICS protocols. These are protocols that are designed to do controls at that level. So um there's Ethernet IP, there's Modbus, there's Profet, there's a whole list of different uh control protocols that are out there that use Ethernet as a physical media. And some of them use different various levels of the TCP stack. Uh some of them use the full TCP stack. Some of them have found that they wanted to do more determinism. So

they've actually like eliminated portions. So some actually will reside right on top of 8023. Some actually even get rid of the link layer in 802 and run right over the hardware layer of 802.1 or 8023. Uh so they they get down into really trying to do sub microscond uh real-time communications with their systems. Uh and then you're dealing with IC specific platforms. So these are small computers that are about the size some of them are about the size of your phone. Uh, but they've got to run 247 for the next 5 to 10 years and never be shut down. Think about that for a second. How many times do you reboot your phone? So, as I said, the traffic generally

doesn't flow through any of these actual NSM sensors. In a system, you would have to literally put an NSM sensor off of here, hanging off of here, hanging off of both sides of this radio link, hanging off of uh this plant network control. And this is just a small one. In a typical plant, you're going to have anywhere from 20 to 30 of these ice or segregated networks inside the plant control network. So, and that's a that's a normalsiz facility that I deal with, which is considered actually mediumsiz, small to medium when it comes to actual industrial control plants. When you deal with large scale SCADA systems, uh like water utilities, wastewater, uh power plant utilities, pipelines, things like

that, then you're dealing with um sometimes uh you're dealing with like a thousand uh independent like isolated networks because you've got that many substations in your network or whatever. So, and each substation gets its own separate segment of the network so that it doesn't transmit over back uh to the main network. So, yeah. Quick question. Do these switches communicate with each other? Like if one switch went down, you're telling me that whole subnet would go down or so they don't spend anything. Yeah. Typically these and so typically these are not even Oh, yeah. repeat. Um so yeah the question was um uh about basically um uh load balancing on the on the switches and having uh

like spanning tree running and and if these switches go down do um what would happen if for that subnet going down? uh in in reality most of the time unless these are truly critical systems there's one switch that's it um you may have dual fiber coming in so in some of the high availability systems they will be dual fiber coming in to that switch so they have two independent paths coming there but that one switch if that fails you're done so but again for critical systems yes you can actually have two switches and meshed and things like that. But you're most the time talking about only two lines coming into there because in some of these facilities like in a nuclear

power plant um it's $5,000 a foot to run wire um minimum and that it can go up from there. uh in like remote uh oil and gas pump fields, you've got um fields that cover and like 25 square miles. So, you're not going to be running wired communications to a lot of those. So, um it just it's really difficult. You can run it to maybe a uh like a centralized control center and then you can plan for having dual path and redundant uh systems and and yes in in those kind of cases like uh um you will see a lot of times uh having um independent paths so that when the forklift runs over uh or pulls down the

fiber that's hanging on the poles, he doesn't shut down the whole plant. You've got another path over there. uh and redundant ser redundant uh switches on both sides. But that's very rare. That's only for critical sites typically.

The problem is you got to remember most of these systems um have to actually sit in place for um decades. It's uh I just got through a plant uh last year that had um deck alphas and running x11 traffic over their network uh which is back mid80s and they had a plan to replace it by 2018. Um these systems are older than you are and they have to stay up and running 24/7 365. So basically, you have to deal with what exists because also like you can't get parts for a deck alpha anymore. It's not like they went and actually like had to go and and get someone to actually build replacement parts for them

specially because they couldn't get them anymore. It things like that happen and that's not uncommon. Um that one was a more extreme example because they were they were just really really bad. Um, but it's not uncommon to see things in in service for 15 years. So, another network, couldn't you just set up one VLAN trunk on all those switches and then send all log traffic through it and then you can monitor it all instead of setting up individual sensors just have one or multiple main sensors, you know, load balancing. So the question was, couldn't you set up uh load balancing VLANs and things like that? Um in in the way it's typically set up, the

way we we typically do, each one of these is a separate VLAN for performance reasons. No, it's it's it's specifically for perform performance reasons because this these systems have to stay up and running and they have to run continuously all the time and determinism is much more important than security. So they were they would much rather deal with the VLAN stuff and and yeah, you can do trunking and yeah, you can do monitor things and stuff like that, but also you're dealing with a lot of times these are layer 2 switches, so they're not smart enough to do any of the VLAN trunking that you can do up at layer three routing kind of layer

switches. So

um we actually have concentrated on assuming the hosts cannot be touched. So, um, so we have gotten to the point, uh, I'll talk about it a little bit later, but basically we actually go in and as long as these have mirror ports, we can basically go in and do, uh, assessments uh, periodically and and actually do signaturebased uh, detection, but it's it's developing a signature of what good looks like and then comparing that signature over time. So, so that actually leads into some of the other stuff. So again, you're you're not dealing with standard OSS. A lot of time, these are real-time offices. If you've heard of VX Works, that's like that's a high-end one for a lot of

systems. And so that's actually really good. A lot of these guys actually decide to write their own real-time operating systems because they can actually get down to sub microscond uh timings and determinism on a lot of these things. sometimes down into nanconds of of timing uh determinism kind of uh uh jitter on their communications. So very low determinism uh on their systems. So again, and then also even when you have standard OSS uh in your systems, you can't always trust that the vendor will allow you to install something on there because they've got it sort of locked down a as a warranty thing of we won't service this if you install anti virus on it

because anti virus slows it down to the point where you've lost your determinism. So um as I said the devices really aren't capable. A lot of times the information doesn't exist. Uh and then the hardware designs they're min minimalistic designs. So as I said assume a cell phone from 10 years ago is what you're actually powering the substation on. That's the power you're dealing with typically. So what do you do? um looking at uh NSM a lot of times looks for negative cases. It looks for something bad happened, so I'm going to send an alert uh and someone should go and look at it. Well, in an industrial plant, looking at the negative is not always

the best thing. What you want to do is actually look for the positive. look for what it should look like and then anything outside of that is actually what you want to look for. So looking for anomalies uh developing a known good set of traffic. I have not walked into a plant yet that everyone or like anyone knew what their traffic looked like on the network. They knew what it should kind of look like, but they never had done full traffic captures and actually gotten to the point of isolating each one of the traffic streams and looking to see what it should look like. Um, and so these systems generally go through multiple levels of testing

before they're actually up and running. There's usually some sort of factory automate or factory acceptance testing of the systems before they even get to the site. Then there's site acceptance testing before there's actually like startup. A lot of times there's some sort of commissioning testing of the whole system. Uh then there's uh when you got your system changes when any like major system uh changes a lot of times they will have to go through and reertify it. Uh and then almost always there's some form of periodic testing. A lot of times it's uh yearly uh sometimes it may be like 3 years or 5 years but again you're dealing with there is some form of periodic testing that goes on

these systems. Uh and and so what you can start to do is compare what the traffic looks like now to what it looked like during those known good sets. And if you see things that are out of the norm, if you see new traffic streams that shouldn't be there, if you see traffic streams behaving differently than they should, if you see anomalies in the traffic that are peering that you didn't see before, that's something to alert. These systems don't change all the time. You don't have or okay, you shouldn't have people down on the control networks looking at web pages. They shouldn't be going out to name your malicious actor country. They shouldn't go out to those countries. If you start

seeing traffic going there, there's probably something wrong. Egress filtering uh for firewalls is one of the biggest things that should throw up red flags. If you start seeing egress traffic out of your out of your uh company from the industrial control system directly, then you should be like running to figure out what's going on. Um network reliability monitoring really doesn't have to be difficult. A lot of the tools exist already. I mean, wire I I live in Wireshark and uh XML like text files or basically uh Excel spreadsheets. You can do a lot of what you need to do with Wireshark and Wireshark will export PSML or PDML files which are their packet data markup

language. you can actually like look at the individual fields from there and then you can process that pretty easily inside uh either Microsoft Excel or a lot of different tools uh and get some data out of there. I won't say it's the best data, but it's at least something to look at. And the metrics you're looking at aren't very difficult either. So, you're looking at jitter on a deterministic signal. You should be getting a straight line across there and all of a sudden I have some wiggle in there. Why? And so you go investigate that and then latency. These systems as a controller I talk to some device and that device responds back. How long does that latency take? What's

the latency there? Almost invariably it's the latency in the actual devices. If you're having network latency problems, then you need to talk to your ISP or you need to go to your network guys. But almost all the time, 95 plus% of the time, it's the devices that are actually having the problems. Um, unless you're dealing with manufacturing and then it's usually the guys with the forklifts that run over network cables or do things like that. Um, as I said, Wire Shark really has a lot of capabilities. um you're not going to find things uh like I don't know if there was a wire sharkark class here before but basically like the um the TCP streams you're not going to be able to

use those really to follow uh industrial control system traffic. Generally Wireshark doesn't know how to follow those TCP streams very well. What you need to do is actually get in and just learn how to do your filters really well in Wireshark and you can filter down to getting all the individual streams. Did you write your own decoder for that? Yes, you can. Um, yeah. So, you can write your own decoder. I actually, what I personally do is I actually script T-shark uh and use that as a way to uh um do my filtering because I can actually automate the process of scripting the T-shark and then getting my data and then automatically pumping that into something else. So, um but

yeah, in in Wireshark you can you can do that graphically or whatever. As I said, my company is developing a process to automate this automate this. But I'm telling you that what I'm doing to automate the process is no different than what you can hando with Wireshark and Excel. It's faster, it's cheaper, it's easier. So my company actually is doing it because when we go in and do assessments, I want to be able to go in and I don't want to have to be the one to go to every single facility to look at the graphs and say, "Yep, that's good." good or nope, there's something weird here. I want to send a junior guy

out there to do the to do the basic stuff. And maybe I if he finds something really weird, he'll send me that. But I don't want to have to be the one to go out there to do it. Plus, we want to be able to actually teach vendor or teach end users how to do this stuff so that when they find the really bad stuff, they can come ask us, but we want to be able to sell them products and and services associated with that. So really it's really not that hard to generate basic graphs uh and charts. Um now as I said detailed analysis of these things can take a lot more experience. I have

been doing ICS network performance now for 15 or more years. I've seen enough graphs to kind of get a feel for what might be going wrong in some of these devices. And what we're trying to do is actually build some of that intelligence into our analysis tools so that it gets easier for end users to actually use. And then root cause analysis is one of those really hard things to do uh with this because then the vendors actually have to get involved in a lot of cases if you've got devices that are showing weird things even like right from the factory. Um, and I've I've done some prototype testing on on vendor devices and uh they they have to then go back

and sort of analyze all that traffic data and figure out, oh, it was something weird going on. Okay, so what do what do I actually look like? So, this is a graph of what's called measured packet interval. Um, and you'll see this nice sort of clear line at around 10 milliseconds. What you're actually seeing is the deterministic frequency of every 10 milliseconds, this device was transmitting a packet on the network and it should happen at approximately 10 milliseconds. These variations in here are the sort of like it's sort of around 10 milliseconds and as long as you're within a certain percentage you can consider it a good device. Um in this particular case you had 10

milliseconds and a 400 mil or 400 microcond jitter. So that's a wellperforming device that's considered something that I don't have to worry about. Um this is another device similar 10 millisecond but it was slightly shifted off and it had a wider distribution. Uh so if I did standard deviation and all the kind of math to go along with that it would be a wider distribution here. But again 10 milliseconds 40 microcond jitter not that bad. So now you get some weird stuff happening. So this is this is actually a time scale. I've I've blown it up here so it actually shows up nice here on on uh PowerPoint. This is about a 60-second um performance test that I was running.

And you'll see there's actually some sort of beat pattern that happens every 30 seconds in this case. It's interesting. Again, this was one of these prototype devices where I had to like give it to the vendor and say, "We saw something interesting. Go figure it out in your system." Uh I believe in this case it was actually a garbage collection routine they had running in their real-time operating system. They weren't using a standard one. They had written their own and their garbage collection would go and actually slow down their communications. Is this a good when we did the root cause analysis we found a vendor was the actually causing this problem. If I saw this in a in an IC plant itself would I

know whether that's good or bad? If you've done the signature analysis or at least signature collection earlier in your process, factory acceptance testing, SAT commissioning, things like that, you would know that this device looks like this. And if I see this down the road, it's nothing special. So, this is where you get into the okay, what does it really look like? Not what it should look like, but this is actually what it really looks like. Another thing here, too. Here's another one. This actually had a beat pattern that was about 50 seconds long in this case. But again, lots of sort of wiggle traffic around there. And if I looked at this at an IC plant, I would question

them, well, what's going on to cause this? I would have to go back and actually deep dig deep into that vendor device to find out if that vendor device was the cause or if it was some sort of network attack or things like that. So a lot of these are are more uh individual testing. Um I said so what's causing some of these? So it may be garbage collection. Uh anti virus uh checks and updates are like terrible. Uh human machine interface HMIs they will actually kind of like lock up when anti virus kicks in. Some AV vendors are are worse than others at it too. Um, the largecale AV vendors are better, but even then

they get they get kind of really uh hard on the system once they actually kick in. And again, you're dealing with systems that are are process limited to begin with and now you're loading it down with something that has to go out and download a 3meg file every whatever and oh, I've got to run my process and check to make sure everything's going wrong. But the nuclear power plant can't shut down right now because I've got to actually hit the button on the screen and make it do something. Onscreen operations are actually another one of those that that causes weird things to happen sometimes, especially Windows will just go off into La La Land while it's waiting for some process to

happen in on the screen. You click the button and it kind of just decides not to react. So that's another reason why a lot of times the vendors will not allow you to install things on there that they don't know about because it could interfere with their HMI software or their whatever software they've got and then you run into problems of okay well now who do I sue and effectively that's what you're talking about in this case the end user couldn't do something they had a fault they had an incident who do they sue to blame for that incident going on um There's network anomalies. Uh a lot of times in these in these plants,

you've got uh large scale EMI kind of interference. Uh large motors about the size of this room generating uh lot a lot of EMI interference or welders or toxic uh chemical areas where it actually corrods the contacts and things. I was in a uh corn processing plant where the corn dust was actually eating away at the gold contacts on the RJ45 plugs. So, they had to replace their switches every 6 months to a year because they were actually getting eaten away until they figured out, oh yeah, well, let's put some positive pressure environmentally sealed cases in here and then work uh with those. But that was only in critical locations, not every location. signal degradation. Um, you're dealing

with places where there's a lot of metal. Uh, a lot almost every machine is is made of metal. And if you've got all if you're actually dealing with wireless networking or something like that, you've got a lot of refraction. There's a lot of uh reflections you'll get and you'll end up with a real mess in in some cases. So, they have to be really careful. Literally, like you could be here and get 10 reflections and then here and you'd be fine. So, it happens a lot. uh flaky connections, uh stupid human tricks with forklifts and dump trucks uh and people hauling stuff around. You will find anything that could go wrong with a network has

happened in anal plant. Uh and then maybe it is a security related incident and then what happens? What do you do? So I actually decided to test this and see what happens when you actually run a man-in-the-middle attack against an industrial control system. I didn't bring it in here, but I have a little suitcase with a controller and an IO block and some buttons. Uh, and it literally is is just a it's a it's a switch. It's the Netgear like one uh GS 108. So, it's got the it's got a mirror port on it, an eight port, so I can run that. I've got a a PLC controller, a pro, it's a programmable logic controller, uh, a 24volt IO block. Uh

and then what I did is I hooked up a Cali VM uh and Wireshark on two separate machines. So the the Wireshark was on the mirror port and Cali was on the on the network. Um I am not a man-in-the-middle attack specialist. There's other people if you want to know. I'm a script kitty. So I said edercap. Uh and so I ran the first one which is art poisoning and found some really weird stuff happened. Um I captured traffic on the mirror port. I was using a protocol called Ethernet IP. This is not Ethernet IP that we know. This is Ethernet industrial protocol. This is a trademark named very bad name but it it is what it

is. Uh and running at 10 millisecond frequency. And I was running the man-mill attack against the controller itself. So I was actually trying to mess with the communic the commands coming from the controller to the IO block itself. So this is the communication actually from the IO block back to the controller. And again this was I think this is baseline but again the IO block didn't notice that the man-in-the-middle attack was going on for the entire time I was running. Didn't have any problems. Now this is when I actually ran the man-in-the-middle attack against the I the the PLC controller itself. So again, I have this nice relatively stable connection at 10 milliseconds. And then

this is where I ran the man-in-the-middle attack. And if you see there's something happened there. Well, what happened? This is actually the PLC itself did not even know that the attack was going on. It was too stupid to know. It just said, "I'm sending my data. I'm sending my data." It never even knew that the man middle attack was going. And this is likely due to the fact that um this protocol actually uses a multiccast signal for this. Uh this guy actually transmits over multiccast back to the controller. Um and that's actually for a it's actually for performance reasons because this guy could actually be talking to a controller, could be talking to an HMI,

could be talking to a historian. And you don't want to send three individual packets out there. there. You want to send one from this IO block and have everyone just listen in on that just like you would for video conferencing or things like that. So the protocol is designed to actually transmit multiccast out. This guy listens to it and because this guy continued to transmit over multiccast. He had already set up and said, "I'm going to listen to your multiccast." He didn't even know anything happened with the man-in-the-middle attack. So with this uh there actually was I think three additional packets in here because of the way I had had the VM set up. I don't think I set it up properly,

but uh there was um yeah, so there was a there was a a large there was a couple different uh streams of extra traffic in here. Um and I've actually signed up for Delaware to talk in more in depth about this whole man-in-the-middle attack. So, if you're in the area, I'm going to try and talk at that uh besides Delaware about that. Um, and then when I actually isolated the PLC traffic from the man-in-the-middle attack itself in in Wireshark, I was able to go and strip down into just looking at them the PLC traffic itself. It didn't notice anything even happened. It didn't blip it. Nothing happened at all. It was a nice straight line across

the whole thing. Um, and then what I did from there is I actually wrote some editorcap scripts to change the actual command signals to the PLC or to the to the IAB block. So when issued uh when it actually went and was trying to turn these lights on and off, I basically switched it to the other light. So uh you'd push the green one and the red button would light up. Uh things like that. So, but and and I did this all in the man-in-the-middle attack. Again, the PLC didn't even notice anything was going on. Um, there's a sequence number in in the traffic stream to actually go and and basically identify the the packets. So,

that if you get one that's out of uh because it's multiccast, it's not TC, it's not using TCP, it's using UDP. So, again, you don't have that session information to actually make sure you've got this your session coming in the right order. So that's actually handled by the industrial protocol. And the industrial protocol can be spoofed by basically just advancing the sequence number out of out of range. Um so you can actually get it to basically forget about what's going on. And I I just to try it out and see what happened, I actually ran my capture files through virus total network miner and bro. Oh, cool. Thank you. So um yeah, I I did

bring it with me. I just didn't bring it in the room. Um the So I ran it through Virus Total. Virus Total noticed the ICMP ping messages, but it didn't actually trigger on an alert. Uh network miner and bro both did not trigger on an alert either when I was doing that. I didn't run ARP watch against this, but uh I'm not sure what that would have shown. I just didn't have didn't have enough time to actually run it before I came here.

Well, It's it's an issue to deal with that it wouldn't it didn't even notice the um it didn't even notice the uh the ARP uh the rearping of the system and all. So um so NSM's good if you're doing it great. It doesn't cover everything though. Um, and especially when you're down at the lower levels of ICS, uh, SCADA networking, um, it really it can be it can be difficult to deal with. Um, and then there are relatively easy ways to measure, uh, network reliability for industrial control systems. Uh, the tools exist, they're freely available. Uh and as I said um there there are some people that are actually trying to do some investigation down at the lower level.

So that's my information. Um yes.

The the question was have we actually found anything malicious in industrial control system that we've gone in and done assessments? Yes, we found active malware on some systems and then we also found things that alerted as active malware but were actually false positives too. We found actually you'll find a lot more false positives than you were than you will active malware. But I have we have found active malware on systems and a lot of times it's it's um where they did not protect or they didn't prevent the the operators from loading web pages and plugging in their devices into the sta the operator stations. So if they didn't have their firewall locked down to not allow web

traffic through, then what happens a lot of times is they will introduce uh stuff by visiting whatever they go to. And and it's sad to say, but it's not uncommon to see porn on HMIs in industrial control systems or other weird places that people have visited. So

Um, it is actually on some we haven't noticed like malware down at that lower level, but we've noticed strange things happening. Let's just put it that way. Um, some of the industrial controllers have USB ports on them. Uh, and when the operator gets to has to sit in front of that system for 20 or like eight hours a day and they want to charge their cell phone, but they haven't been told not to plug that into the HM or into the controller itself, they will do stupid human tricks. Uh, and you will sometimes see weird things happening to the network. Um, at that point, a lot of times it's easier to actually just reflash it than it is to try and fix

whatever's going on. So, if they start seeing weird things happening with the network, they will actually most times just re-upload the latest firmware of that device and the latest program and that's it. So it's rare that you will see ICS forensics down at the controller and lower levels at for the HMIs and PCbased systems. Yes, but you're not going you're not going to see the FBI knocking on your door and saying, "Let me see your PLC." They just don't know what to do with that. Another thing too is that that PLC has to stay powered uh in order to get the forensics off of it properly. And there there don't there isn't a really good way to do memory mapping on

devices that don't have standard memory and and standard interfaces. So you can't do a lot of that. Um and as it is the firmware for these devices is getting loaded into NV or into NV RAM anyway. So they'll just like reflash with what the vendor gave them to begin with and and be done with it. So IR is not they don't do a lot of IR and forensics in the IC space simply because the devices just don't have that capability.

Um so the question was about targeted threats. uh do they consider themselves to be targeted threats? Some do. Um so I know Anonymous has OP oil and they've got some of those kind of things where they will attack they will actively go after those kind of companies. Um and there's always pick your malicious actor country and you'll see all sorts of warnings from ICSert and DHS and oh my god cyers and they're coming to get us. Uh there there's been a lot of that talk um outside of stuckset which was perpetrated by certain countries that we happen to maybe reside in. Um there aren't too many of those going on. it. You'll you'll hear a lot of stuff

and you'll hear a lot of FUD about um everyone's coming to get us and and this country and and other countries are coming and ISIS is coming and everyone's coming to get us. There is some truth to that. Um they did find controllers in uh I mean they found like manuals for controllers. They found actual hardware in Afghanistan when they like did the whole bombing of that mountain. They found stuff in there. They found stuff when they've gone in to investigate a lot of things. You can get a lot of stuff on eBay. I mean, it's not that hard to get. And it's the same equipment for decades. So, if you find something in there, it's not uncommon to

find that same vulnerability in there for years. Again, it doesn't benefit the vendors in a lot of cases in these things to actually go out there and try and fix problems too. It's not like the vendor shaming that happens in the IT world. A lot of this stuff actually gets hidden behind closed doors because it does affect real things and there is a real boom factor and big clouds of ethylmethyl death and stuff like that. Um, so that stuff a lot of times is actually held closer to the chest uh by these companies because they don't want to release it uh out. So, any more questions? Thank you very much. [Applause]