Shining a Light into the Security Blackhole of IoT and OT

uh right good morning SL afternoon welcome to bsid Las Vegas track name um which is proving ground right now this talk is not on my phone hang on a second it is shining a light into the security black hole of iot and OT this is Huxley please give a big round of applause real quick I got to thank the sponsors because they give us money right all right quick thank the sponsors Diamond sponsor Adobe woo gold sponsors choose three Prisma Cloud sem gra blue cat prra Toyota conductor one it's their support that helps will happen thank you if you're a sponsor a donor or a volunteer you're amazing Huxley it's over to you my friend let's go all right

thank you very much all right so hi everybody my name is Huxley Barbie I'm the only Huxley Barbie ever going to meet and I am the lead organizer for beiz New York City really happy to be here with the mothership beiz Las Vegas really excited about that and I'm also the security evangelist at run zero uh but more relevant to this talk I've had a long career as a security consultant and most of my clients had just it environments but many of them more than a few have had OT environments as well and specifically have had customers in manufacturing transportation and higher education all of which often have OT environments so these environments include a lot of

factory devices or bus or rail station equipment uh campus facil ities and so on and so forth and so much of what we're going to discuss here today falls under the category of critical infrastructure and key resources in the United States by the end of this talk I hope that you will one know more about OT than you did before you have a few pointers on where to go to do your own security research on critical infrastructure you have understanding of uh the challenges of scanning OT networks when you're trying to satisfy CIS control number one and have a few ideas on on how to overcome these challenges also I'm giving out cash as part of this very exciting all right so

when we think of compute we often uh think about laptops servers and databases and this is what we would call it devices but this only represents a small percentage of devices uh chips that are manufactured I saw this one statistic that says 90% of chips that are manufactured go into embedded devices iot and OT devices so it's a much larger landscape uh outside of it some of these invited uh embedded devices are iot devices which have a huge variety right uh we often like to joke that these days even like a coffee mug is on the internet right so that would be an iot device um but it also includes like run mill stuff like printers IP cameras home automation

like your nest and whatever as well as even power supplies some of these eded devices are are known as OT devices operational technology devices which operate our factories our water treatment facilities recycling plants oil refineries gas pipelines all sorts of areas that are called cikr critical infrastructure and Key Resources now two points of clarification here in this talk I'm using the term OT you can think of it as synonymous with IC industrial Control Systems uh the second point of clarification is you'll notice there on the lower right I have a medical device often times people call this iomt internet of medical things but with the role that IMT plays and its usage pattern it actually fits under the OT

category a lot better all right so obviously in this talk I'm focusing on OT and some on iot um most of us are not familiar with OT so I'm going to go over that first and then I will talk about how uh really it's not a challenge to uh go on the offense against OT environments I'll briefly touch on what other people do for uh scanning uh defensive scanning and then I will talk about the novel idea in this presentation which is active scanning in OT environments and then finally we'll take a look at what this means for the iot side so even though many of our OT environments are considered critical infrastructure they are

shockingly unprotected but what do I mean by an OT device so let's take a look at that first there's actually a huge Variety in OT environments it's not like it where most folks have a stock PC uh either or a Mac with you know modular components rather uh there are a lot of devices that are specially designed for a specific purpose only can be used in an electrical plant only can be used in this Factory and so on and so forth I'm going to walk through an example with you uh to show you what I mean by OT but just keep in mind there's a lot of variety in OT environments okay so what we have here is a water treatment tank

the contaminated water comes up from the lower left and then goes into the uh the tank and then comes out the the right pipe over there well you see here are two different sensors that are attached to the right side of the tank so when the water is below the lower one uh the right valve closes the left valve U the left pump pumps and so the the dirty water comes in and then when the water reaches higher than the higher sensor the left pump stops and then the right valve opens up and then drains out um drains out the the cleaned water after like an hour of just like sitting there for the treatment uh the pumps and the valves

are called actuators uh although out in the field you might find these to be integrated so an integrated actuator and sensor again there's a lot of variety in OT environments the brains of this operation is known as a PLC program logical contr programmable logic controller and again lots of variety in OT environments in an electric plant electrical plant you might have an ie Ed instead which stands for integrated uh sorry intelligent elect electrical device uh there's an HMI a human uh management interface uh this is the panel that a technician would use to adjust the behavior of that PLC uh think of the HMI not as a computer but more like a keypad of sorts like a thermostat

in your house that that's really an HMI it's meant for like that level of technician to to operate it's definitely very locked down uh plcs are programmed with an Engineers workstation this actually is a PC although what you often find is they will be running things like uh Windows XP old operating systems uh apparently I was talking to somebody where he found like um Windows 3.1 on on a particular like engineer station so um this is one system okay and at a site you might find multiple of these systems all coordinating with each other through what is known as a DCS distributed control system on the other hand you might also have a deployment where these OT systems are spread out

over a large Geographic uh region and that is organized into what is known as a scada supervisory control and data access with skas you might actually have an rtu which allows you to relay between the PLC back to some sort of centralized control center at the headquarters right so this is a quick tour of how a small part of an OT environment might look like so now let's take a look at what it means to secure OT environments and how that's different from securing it environments in it we are uh we are concerned with the movement of data but in OT you're concerned with the movement of widgets and gears and cogs and and things like that moving of

stuff Machinery uh it vendors will uh release products with planned obsolescence right so all of you have phones and laptops which are probably no more than 3 to 5 years old on the OT side these devices just sit there pretty much forever in Internet time yep all right here's the first question where I'm giving away money what is commonly thought of as the Triad of major concerns for any security program there you go that's yours all right so conventional wisdom in our industry says that these are the pillars of um of that of concern and now arguably on the it side confidentiality tends to rise to the four right um most companies on the it like an e-commerce

site they don't get sued if their e-commerce site goes down they just lose a little bit of money uh but if they lose pii then then of course you know they're going to be investigated they're going to be sued and there's reputation loss and so on and so forth on the the OT side though availability is absolutely Paramount they will do everything and anything to avoid an outage right now let's talk about why this is important for a commercial organization uh loss of availability means loss of Revenue right your your cars are not being built in your factory the the gas is not flowing and so on and so forth uh but also for many of these

critical infrastructure and Key Resources they're regulated by the government right so for example Colonial pipeline was fined a million dollars by fsma because of an outage right so when they are not able to deliver that availability they can be fined by the government right uh on if it's not a commercial organization but like a governmental or quasa governmental organization what you have is a politician that doesn't want the the bad press of you know that particular service municiple service uh going down and so on and so forth uh so you know for one of one of these other reasons you know availability is far more important with that CIA Triad uh nearly all of it devices run on

one of these operating systems and they're all time sharing operating systems but on the OT side there's far more variety of operating systems than are often real-time operating systems inste which you know has real implications or ramifications for what happens when you contact it over the network and what it might do uh OT de OT devices are programmed in languages that I had never heard of before I started digging into this right um ladder diagram I have to read this out function block diagram sequential function chart and so on so forth OT devices are almost never updated or patched so this goes back to what I saying earlier availability is the most important thing for these

organizations they will do everything and anything to avoid an outage so they don't want to shut down for patching they also want to avoid any potential extended downtime due to a bad upgrade or a bad update now you might laugh here that I'm saying like it secure by Design but relative to O the OT side yes it is is you know secure by Design with on the OT side though like there's pretty much nothing right uh the moment you have access to a PLC you will have access to actuators and sensors and be able to modify Its Behavior right uh many of these OT devices do not require authentication many of them are talking plain text over the network there's no

encryption so on so forth and frequently there's no governance in these organizations to remediate default users default passwords as well as default settings basically once you get access you you you own everything uh it devices these days tend to have a lot of endpoint protection or there's you know um they're they're being scanned by vul scanners and even the network itself has some sort of network level production in OT some of the industries have started introducing controls and making great improvements but in many Industries there just still are no security controls at all traditionally uh it devices have some have had some connection to the internet but often times OT environments were air gapped they were air gapped

which explains a lot of what I've said so far right uh um security through isolation right not security through obse security but security through isolation uh so often times people thought it was okay for OT uh devices to not have so much protection because in order for you to compromise that OT device you kind of had to walk up to it to do something to it now traditionally OT networks have had their own protocols But but so this is the kicker here starting around 2005 in order to be more operationally efficient a lot of these organizations started connecting their OT networks to the Internet so they could do Remote Management and things like so think

about it this way if I have a valve on a pipeline out in the middle of Wyoming that needs to be adjusted it's so much easier if I could just you know do that over the over the network as opposed to Flying somebody out there in order to just you know to to to to turn the valve right so for operational efficiency this started happening but up to now up to 2005 security through isolation was the thing and so that sort of curtain of air gness sort of came down and then that sort of exposed all these other problems that I mentioned uh so far all right here's the next question what is the name of this model

shown in this picture yes that's it somebody pass that over okay so this is the Purdue model uh it shows an ideal model of an OT Network or what it should look like where there's different levels of uh risk and controls that are stratified for the sake of security and you'll see here sensors and actuators are down at the bottom field devices as they're called right in uh layer one you got the PLC layer two and three you have the HMI and so on so forth now what's supposed to happen here is between each layer the there's supposed to be some sort of security control that adjudicates communication between those layers the other thing that's supposed to happen is you should

only be communicating with your adjacent layer you should not be going from one to three or 1 to four you can only go from one to zero or one to two right um the other thing that that you want to note here is the higher layers are very iish and the further down you go the more OT it gets okay um the other thing to note is the lower you go the fewer security protections you have you're going to find more security protections at at the top so what this means is if you're going to make your way in infiltrate into the top it gets easier and easier for you to get down to the bottom right

so once you're in layer five it's pretty much a foregone conclusion it's just a matter of time before the adversary can get down to layer zero so sounds easy right well what if I told you it's actually a lot easier than that right uh so in the scenario I said you get into layer five you work your way down to layer one and zero right but what if you could skip through all those layers and just go right to layer one over the Internet wouldn't that be easier right yeah answer is yes it is easier so remember what I said um I I don't know if I mentioned this but no organization actually TR truly

implements Perdue that is an ideal model that most organizations do not uh live up to and so we have an example here of what is supposed to be layer one in Purdue a PLC that is directly connected to the internet so you can skip 5 through two and just go right to layer one so now you might be thinking well Huxley okay fine it's on the internet but like how do I log into this device right to the web interface on that device well you know what what you might want to do uh aside from using showan of course is uh you could just go on Google to find this PLC as I mentioned um but if you want to log into

it what would you do well you just go on the internet as well and you go and find the usernames and passwords remember what I mentioned before most organizations don't have the security governance to remediate default users default passwords and default settings which means what you're going to find on GitHub is probably going to work at least some of the time okay it's all there uh here's another organization SC Strange Love uh they're an independent group of circity researchers they also publish these this type of information so it's all there and you don't have to try very hard to find them now what if you have a situation where that particular device uh has been remediated

such that it doesn't have default passwords right so maybe this organization did a little bit of of governance on these things and you can't just like go find the the default password and just log in well remember how I said earlier that OT devices are almost never updated or patched so what you would do then in this case if you can't just log in with a doal password is head over to ca website and find a vulnerability to exploit right they've basically given you a pro map road map of things that you can try out because most of the devices are not patched more than likely some of these vulnerabilities are going to work even if that device was deployed 20

or 30 years ago and so now you might be thinking okay I see that you know I can probably have some success with default usernames and passwords and if not I have like vulnerability that I can probably exploit but you might be think okay OT devices are so different from it like do I need different tooling to exploit uh these vulnerabilities well the answer is no because with current tooling there are modules that you can use to go ahead and exploit those those vulnerabilities I hope that I have I have impressed upon you that there's a real problem here with our critical infrastructure and Key Resources some of you might be thinking you know I'm going

to go back back and take a look at my plan for going off the grid cuz cuz who knows right um but the fact of the matter is any organization with an OT environment should have a hard look at it and I was being a little cynical earlier by saying like oh the only reason that people care about availability is because you know they're going to lose money or they're going to be fined or or it's going to be bad press but um that in no way discounts the importance of availability with OT devices because many of our Lives depend on what water and electricity we depend on electricity here in the city uh for

sure um Pharmaceuticals and and so on and so forth like so this is really important that all of our organizations that have OT environments to to take a look at this because our lives depend on it um and arguably one of the first things you would do if you're protecting OG environment is to figure out what you have right U many organizations in the past have tried uh using uh scanners like nessus or an nmap to figure out what they have which unfortunately gener uh resulted in uh major Financial loss or major outages these tend to not be published for obvious reasons but I'm sure you speaking with your friends or others uh might have heard about such a things for

this reason security teams tend to use a passive Network monitor sniffing for traffic uh with a with a tap or a Spam port to figure out what what's on the network which is fine as long as you can access enough choke points on the network but let's take a look at this more closely right suppose here you have a SCA system that is spread across multiple sites and all the communication at these various branches are backhauled uh to the headquarters well in this case you know you will have access to most of the core distribution switches at the headquarters and so therefore you can see all the traffic um for the device that communicate inter site

right but now consider this scenario right where the site toite communication is not back hauled through a central location but instead uh they can have peer-to-peer communication among these sites well this is operationally efficient right in for these sites to be able to talk to each other they can be more operationally efficient so obviously organizations are going to do this but it makes it much much harder for you to get enough choke points to figure out what are all the things that are talking on the network and if you have hundreds of sites I guarantee you that it's impossible for you to get some sort of comprehensive asset inventory so by sticking with a passive

Network monitor solution instead of active scanning security teams are uh inviting these these issues right not only do they get an inventory that has uh gaps but it's complicated to deploy if you ever tried setting up you know spam ports at scale I'm I'm sure you know what I'm talking about and uh in order for you to collect all that Network traffic you need to have these big beefy Hardware Appliances right so just deployment wise it is complicated and uh you get poor performance unless you spend the money and what do you get for all that you get an inventory that has poor fingerprinting and has a lot of gaps in it it's not getting all the

devices um and the fingerprinting tends to be poor simply because the only thing that you have for fingerprinting is what's going over the wire and so if a particular device is not very chatty it's not you know giving itself up then uh the fingerprinting tends to be vague or even incorrect so let's dig into the reasons why actress SC scanning has failed in the past like so why why is it that there are all these outages when people try to actively scan the network and based on this sort of look in at that uh we will come up with the five principles of active scanning in OT and here is the next question uh packet 20

53 what do we call it like what does it it has a name that's named after a holiday the interesting thing about 2053 is that the fin bit is on the push bit is on the um no no he he raises his hand Christmas all right there you go you got it um can you all right so scanners like uh nmap and NIS they intentionally use non-standard packets or unexpected payloads for fingerprinting purposes right so they'll jostle that device so to speak to see what the response is and based on that response it's like aha like you're you're that thing right um this this Christmas packet here is a non-standard packet right it's not something you

would normally see and depending on the network stack of that device it might handle it it might not handle it what is often the case with OT devices is they will not handle it and they will crash or they will reboot or they will freeze up right causing that that that outage now the same is true when you move up uh above the network stack you're looking at the application itself like application payloads also have this issue where uh they will freeze up or crash and so on and so forth the thing is programming in it has has benefited from Decades of innovation in terms of like release engineering sdlc you know quality insurance and things like this

um but you don't have that in an OT with OT they the quality insurance is like oh this thing properly response to somebody pushing a button or flipping a switch not oh we can handle arbitrary Network traffic that is completely unexpected right um also like input checking I think is not a thing in OT like it's like an IT thing but like it applications yes OT I I really don't think they do any of that all right number two here is vulnerability scanners will send security probes to detect vulnerabilities and by its very nature this is unexpected traffic for an OT device for so for the same reason as previous previous slide uh OT devices

will oftentimes behave very erratically when it sees a security probe all right let's talk about um heavy traffic so Legacy vulnerability and network scanners um they can potentially send lots and lots of traffic on on the Network to a particular endpoint some OG equipment cannot handle uh that much scan traffic all at once uh and so when they do and it's partly due to the fact that they're they're real-time operating systems um when they when they see receive that much traffic they will get slow downs or they'll freeze up and and things like this but but there's also the issue of the network itself right lot of these network devices are not going to be able

to handle heavy scan traffic as well so in this example here we have a Mission Control in the middle you have that Pipeline and then there's like the pump over there so that pipeline is not in an urban environment it's actually in a remote location where they can't get FiOS they can't get DSL all they can do is get a modem right and then that pump you can't even get a phone line out there and so what they'll do is they'll use radio or satellite to go back to the pipeline and then piggy back over the modem in order to get back to Mission Control this is not the type of network that can handle a lot of heavy scan

traffic right and so uh what you want to do is uh you want to be able to tune your scanner in two ways right first is by being able to dial down the total number of packets that you're going to send out there but the other one is to be able to distribute that traffic smartly so that you are sending the least amount of traffic you need to to each endpoint without having an extended scan time so fast overall scan time but least amount of traffic uh on those each individual device and number four here this one is extremely important there are some devices that will crash even if you send standards compliant traffic they're just poorly built poorly

written and and things like this so in this case what you have to is do something called incremental fingerprinting so what you do is you first you send a super benign query to that device and you sort of like understand the shape of it you know just like high level like what this thing is and then based on what you've learned what you fingerprinted so far about that device then you go down this code path or that code path for the next query that you send to it and so successively through these iterative queries uh where you're being very careful about what you're sending and not sending you could ultimately get to a point where you're

actually have um this device fingerprinted pretty well but also avoiding any sort of situation where you have sent a particular query that might have crashed it right um and then last principle here it's right at the bottom don't be stupid go slow start small and then and then expand out all right move on to iot here go WRA up oh really okay we're gonna we're going to skip right over rot sorry about that I'm so sorry yeah no worries no worries um I'll be out there if you have any questions please connect with me find me right all right thank you very much sorry we we're already over y