Your security black hole

it's 11 30 on the dot here so I will be getting started if you came to listen to a talk about iot and OT scanning you're in the right place the title looks a little different from what you see on the schedule it's the same content I just really wanted a cool picture to start with so I changed the title of the talk so good it's still warning so good morning everybody my name is Huxley Barbie I am the security evangelist at run zero you might know us from our previous name Rumble network discovery everything that I speak here is my own opinions not those of my company but that's just so happens to be where I

work I am also the lead organizer for B size New York city so I I'm very happy that I was accepted to come speak here because I don't get to speak my at my own conference I have to fly around to to be able to share my own ideas if you want to connect with me uh those are the links down there so when we think of compute our minds often go to laptops and servers and databases and these are what we call it devices but this only represents a very small percentage of chips that are manufactured in fact I saw a statistic the other day that says that 90 of chips go into embedded devices

some of these embedded devices are iot devices which range in a huge huge variety we often like to joke that these days even coffee mugs and lava lamps are on the network somehow but iot devices also include run-of-the-mill stuff like your printers home automation like your doorbells power supplies as well as IP cameras and some other of these embedded devices are what we call OG devices operational technology devices that operate our factories our oil refineries our gas pipelines and water treatment plants and all sorts of areas that our government considers to be critical infrastructure and key resources so two points of clarification here throughout this talk I'm going to use the term OT as a synonym for ICS industrial Control

Systems so if you hear me say OT you can also automatically translate to ICS if you like and the second one here is that you'll notice I have a medical device medical devices are often called iomt internet of medical things so you might think it falls into the iot bucket many of us would think that but due to the way they are used their usage patterns and their roles and how long they are kept in operations iomt devices medical devices look a lot more like OT devices and so they're in the OT bucket right in this talk obviously we're going to focus on iot and OT most of us are not familiar with OT right if you're like me you started now

an I.T and so I'm going to go over what that is first then we'll talk about how it's a very easy challenge for offense to compromise our OT environments then I'll briefly talk about what people are doing right now in terms of defensive scanning and then I'll talk about the novel idea of this talk which is you know actively scanning in an OT environments and then finally we'll jump over and talk about iot so even though many of our OT environments are considered critical infrastructure and Key Resources they are shockingly unprotected but when I say OT device what do I really mean so let's have a look there's a huge huge Variety in OT environments so it's not like it

where people have stock PCS and Macs and all the the parts are sort of modular you have modular components so rather there are a lot of devices out there that are specially designed for specific purposes and so a particular OT device in an electrical electrical plant would look very different than an OG device in a manufacturing plant for example what we're going to go through here is a simple example but just keep in mind there could be quite a lot of Divergence from the diagram that I'm showing here right and what we have here is a water treatment tank so contaminated water comes up from the left pipe and then goes out the right pipe

you see two sensors attached to the right side of that of that tank so what happens here is when the water is below the lower sensor the right valve closes and the left pump starts pumping water pumping in dirty water and when the water is reached higher than that higher sensor the left pump stops and after an hour the right valve opens up and drains the tank so you know after an hour of treating that water and cleaning that water it gets drained out the pumps and the valves are what are known as actuators in the OT world out in the field you might even find actuators you might find sensors and actuators that are integrated into a

single device again there's a lot of variety in OT environments but just be aware that there are very different functions one serves as an input sensors serves as an input into the OT system whereas actuators serves as output from the OT system the brains of the operations is a PLC programmable logic controller and again there's a lot of variety here in OT environments so in an electrical plant you might have an IED instead of PLC IED standing for intelligent electronic device or electrical device and an HMI is a panel that a technician uses to adjust the behavior of the system through the PLC so think of like a thermostat in your house that sort of

lockdown functionality it's not a full computer just locked in about functionality where you're pushing buttons something that a technician would use plcs are programmed with an engineer's workstation this is an actual PC so an I.T device in the OT environment and it's often running an older version of Windows even Windows XP right so this is one system and at a particular site you may have multiple of these systems all collaborating with each other in one is known as a DC as a distributed control system or on the other hand you might have these systems multiple of these systems spread out over a large geographic area organized into what is known as a scada supervisory control and data access now

if you do have a scada you might have what is known as an rtu that allows you to relay between a PLC and some sort of centralized control center back at headquarters so this is a quick tour of how a small part of an OT environment might look like you know basically we just went over the the terminology so now let's look have a look and see how OT environments different from different from I.T environments in terms of security in it we are concerned about the movement of data or restricting the movement of data in OT it's about moving widgets in Gears in IG it device vendors they produce products with plan obsolescence they expect you to replace your computers and

your tablets every three to five years but OT devices tend to be deployed and remain for a much much longer time conventional wisdom in our industry says that the three pillars of concern for security are confidentiality integrity and availability and arguably personal opinion here with it environments confidentiality tends to be more important so if you think about it an e-commerce site if it goes down it's unavailable the company might lose money but they're not getting sued on the other hand if these Commerce company loses customer pii they're going to get sued right so confidentiality is relatively more important in my opinion now on the OT side availability is absolutely the most important thing right and there's a couple of reasons

for this let's say this is a commercial organization oil refinery every minute that the plant is down they are losing money so their interest and availability is based on the impact of the business in material and in real dollars also many of these critical infrastructures can face fines for an outage so when Colonial pipeline was hacked and they were not able to move gas around or natural gas yeah they were fined by fisma for one million dollars for an outage it wasn't their fault well maybe it was their fault but they got hacked right um and they were fine because of the outage so availability is so much more important for commercial organizations for that

reason now if this is a government organization or a quasi-governmental organization like a Municipal Water Treatment Facility there are politicians in power who want to avoid the bad press of having an outage so depending on the type of organization there is concern for availability but at the end of the day that is the most important concern in OT security more so than confidentiality nearly all it devices have one of these operating systems which are all time sharing operating systems in OT you find a lot of real-time operating systems instead and there are so many more of them OT devices are programmed with languages like ladder diagram functional block diagram and so on and so forth

and OT devices are almost never updated or patched go back to the idea of availability these organizations do not want to shut down these devices for any reason they don't want to shut down for updates or patches and they definitely don't want they definitely want to avoid the possibility of a longer outage because an update going bad now here I say it is secure by Design and some of you might laugh at that and I don't totally disagree with you but when compared to OT devices are secure by Design with OT field devices actuators and sensors there's just absolutely nothing zero the moment you have access to a PLC you own those field devices another concern here is that OT traffic

is almost never encrypted so traffic interception and modification person in the middle that is totally a thing with OT many OT devices do not require authentication even and even if they do there's usually no governance around remediating default passwords default users defaults default settings and so on so forth so basically once you get access to the PLC you own it you own all the field devices that are attached to it it devices these days tend to have endpoint protection EDR hospace firewalls where they're being scanned by a vault scanner in OT some of the industries have started introducing endpoint controls security controls but in many of the industries there's just nothing whatsoever traditionally iot devices are the ones that are

somehow connected to the internet right through your cable through your wireless somehow connected to the internet but OT devices were kept physically separate and this sort of explains why people thought it was okay to not have endpoint protection and to not have authentication and how like um for for traffic to be unencrypted and so on and so forth because in order to compromise an OT device you have to walk up to it traditionally OT networks had their own protocols but starting around 2005 he started seeing more and more of these OT networks to get getting overlaid on top of Ip Networks thereby reachable from the internet so you have a whole history of lack of

security no T and it was okay because everything was air gapped at one point but starting 2005 that sort of protection of isolation just sort of went away and we have a new reality to deal with what you're looking at here is called the Purdue model and it shows an ideal model of what an OT Network should look like where there are different levels of risk and controls that are stratified for the sake of security you'll see here that sensors and actuators since those Inn actuators are down at the bottom in level zero plc's also known as field controllers over there are in level one and you see hmis in layers three two and three

in Purdue each layer is only supposed to communicate or devices in each layer it's supposed to only communicate with devices in the adjacent layer you're not supposed to be able to go from one to three you go from one to two one to zero you don't go one two three one to four between each layer there's supposed to be one of two things one physical separation air gapping which is good but keep in mind stuxnet was a very good example of how an attacker was able to bridge that air gap through a USB key so not foolproof the second one here is some sort of network security control that adjudicates access between layers or communication going across boundaries

that could be a firewall it could be an IPS or what have you in reality though very few organizations ever fully achieve Purdue in fact notice how um in fact if you were able to infiltrate the higher layers of Purdue a secondary attack into the lower layers is shockingly easily notice how in the upper layers it's very i.t-ish right we're looking at you know web servers and so on so with things that we would see in it while the lower layers are more more OT compromising devices gets easier and easier as you move down this stack so basically once you get in from the Top If You Can infiltrate to the top it's pretty much a no-brainer for you to

get to the bottom and once you do you move swiftly to layer one to the plc and then you have access to the crown jewels in layer zero sounds easy right what if I told you it's often easier than that wouldn't it be easier if you could just skip through all those top layers and go right to layer one over the internet the only answer is yes it would be a lot easier of course remember no organization truly implements Purdue some organizations are very far from Purdue and as you can see in this example this is layer one there's a PLC directly connected to the internet you can skip through layer six through two go right

to layer one and once you get to layer one you only are zero right I didn't even have to log into shotan for this search results but why bother with Showdown why not just Google for the with the proper dorks these days plcs have web interfaces for Remote Management why don't you just log in so you might be wondering well what's the username and password remember what I said earlier plcs have default users default passwords default settings and there's no governance around changing that or mitigating that so it's all there here's a CSV of usernames and passwords from a scada strange love they're an independent group of enforcement infosec researchers focused on ICS and scada

systems and they just publish it all out there for you for those default usernames and passwords all right so now you're maybe wondering okay what if this PLC does not have a default password then what do I do remember what I said earlier about how OT devices are never updated or patched just head on over to sisa and search for volunteer exploit more than likely that volume's not been patched so now you might be thinking well OT devices are so different from it devices would I need a different set of tools to exploit this volume the answer is no just go with your current tooling Metasploit has modules for exploiting bones and OT devices as

well

I think at this point some of you might be thinking I should really think rethink my plan to go off the grid or at least be prepared but I I'm hoping to also impress upon all of you that there's a problem with our critical infrastructure and key key resources any organization that has an OT environment needs to have a hard look at it I was being very cynical earlier when I said that organizations are focused on availability because you know they care about making money or their reputation but that in no way discounts the importance of availability so much of our Lives depends on a steady flow of electricity of gasoline of pharmaceuticals and clean water

it behooves every security organization to dig into their OT environments if they have one now arguably getting a full inventory of all of your OT devices is the first step in making sense of all of this so how do we find out about all of our OT devices some organizations in the past have tried scanning OT environments with active scanners like nmap nessus qualis and these usually resulted in Mass outages and financial losses which are not reported so for this reason security teams usually use a passive Network monitor to listen for traffic on switches for span traffic from switches or or by a tap which is okay as long as you're able to access the necessary choke points to see

all the traffic but let's take a look at this more more closely suppose you have this scada system over here where obviously you have multiple sites and any sort of site to site communication is back hauled through the headquarters well in this case all of your core and distribution switches are in the headquarters so at least you can see all the traffic from the devices that are talking outside of their own individual sites now consider this scenario where all the site to site communication is not backhauld through the control center well this is obviously operationally more efficient right better for the business but it also makes it much harder to set up a passive Network monitor if you have

hundreds of sites it's going to be impossible to get a full asset inventory there's no way you're going to be able to configure that many switches and those many sites to gather all the traffic that you need to get a full view of all the OT devices within this scada so by sticking with a passive Network monitor solution instead of active scanning security teams are inviting a lot of these issues passive Network monitors are very hard to deploy connecting to spam ports or taps at enough truck points is difficult to get done passive Network monitors Collective collect a lot of data so you get to so you tend to get really poor performance unless you have some really

high power for high-powered Hardware Appliance so as you can imagine they're also going to be very expensive and what you get for your money and your effort is not a full asset inventory the details that you have on those devices are minimal and the fingerprinting can often be wrong because you're only fingerprinting based on what you see on the network what's on The Wire so let's dig into the reasons why active OT scanning has failed in the past right why there was were those all those outages and the reasons why are also the basis for the five principles for active OT scanning scanners like mmap and esses use intentionally non-standard packets or unexpected payloads for

fingerprinting so have a look here at packet 2053 this is what is known as a Christmas packet notice how the thin the thin bit the push bit and the Urgent bit are all set to true this is not a standard packet nobody sends this unless you are some sort of volt scanner or a network scanner and depending on the network stack of the device that's being being scanned it might handle this but what is often the case with OT devices is they crash or they reboot or they freeze up and the same is true higher up in the stack with application payloads as well programming for it has had Decades of innovation in the way we engineer our

software release our software test our software handling arbitrary input coming over the network through input checking or balance checking is very standard practice these days for it applications for OT applications the code was written to respond to somebody pressing a button not arbitrary strange input coming over the network so unexpected payloads can also lead to reboots and freezes the second one here vulnerability scanners send security probes to detect vulnerabilities by its very nature it is sending unexpected traffic that's what it does and for the same reasons as the previous slides OT devices May erect May react erratically Legacy phone scanners and network scanners can send large amounts of traffic to endpoints in a short amount

of time OT equipment may not have the processing power to handle heavy scan traffic which causes its slow down or freeze as well now sometimes it's not just the endpoints but also the networking devices that can't handle the scan traffic so in this example we have a pipeline in the middle of nowhere some rural location where you can't get fiber and you can't get DSL the best you can do is a phone line with a modem now imagine there's another site with a pump in an even more look more remote location you can't even get a phone line so what they do in that case is they set up radios for the the pump to communicate back to

the the pipe and then piggyback over the phone line to get back to headquarters this is not the type of network that can handle a lot of scan traffic right plain and simple the solution is to make sure that you are tuning your scanner in at least two ways first is by dialing down the number of total packets per second from your scanner and the second one is to spread out that scan traffic as much as you can across multiple devices so rather than hitting each device sequentially with a ton of packets instead you're actively scanning multiple devices at once but with less traffic to each device at any given time fourth principle here there are some devices where even

standards compliant traffic can crash them and to handle these it's really important to avoid the type of fingerprinting that you might find with Mapp or synfp where you're basically just trying to capture all the information at once and doing fingerprinting on that rather what you want to do is start out with some really super benign query for the device get some idea of what that thing is could be and then you iteratively follow up with more in-depth queries when you know it's safe to proceed and the last principle here is to go slow start small until you can scan larger and larger parts of your OT environment until you can get everything all right so we're going to move on to

iot here so one major problem with iot scanning is that they tend to have these iot devices tend to have very few listening ports and they tend to be encrypted so when you all all you see is an SSH port and an https port it's pretty tough to figure out what that thing is right and so for that reason naive scanners often identify iot devices as Linux machines can't tell the difference they can't tell you if that's that's truly a coffee mug or if it's if it's an IP camera instead it all looks like Linux if you start digging a little bit deeper right maybe looking at Mac addresses you might be able to see the iot platform

that the device was built on right Raspberry Pi or espressive these are two common iot platforms but you're still not really getting a really good sense of of what that thing is fingerprinting iot often involves this painstaking process of looking into the payload to find model specific information that tells you what the thing is and oftentimes this means content contacting the device via https parsing the payload and looking for telltale signs that give you some sort of indication aside from being really hard to fingerprint menu iot devices are also prone to disruption just like OT I mean they're coded just like OT devices they're coded to respond to somebody pressing a button right not arbitrary traffic coming over the

network and again like OT devices these some of these devices some of these iot devices will behave strangely even with standards compliant traffic right examples of this include serials to ethernet adapters or print servers in fact there are some printers out there where if you send a certain certain characters over a port 9001 it starts spewing paper out of uh of of the printer it doesn't print anything but it just starts pewing paper stewing paper until like you run out of paper so you might be thinking well maybe this isn't such a big deal right disrupting the your network enabled coffee mug just means you get a bad cup of coffee right so who cares well

I think it's important to keep in mind that many of these OT devices are definitely valid attack vectors into our homes and our businesses with real real impact to to privacy right it's no surprise that many of those of these devices are restricted under the National Defense authorization authorization act section 889 right Chinese devices if you will and they're very much a real threat so and I I personally know of one organization that found an iot camera that was making a network connection uh to an IP address in China even though that organization did no business with anybody in China and I'm sure there are plenty of instances of that you know this is just

the one that I know about so it's important to get out there and know what is on your network the good news is the same five principles that we discussed for OT also work for iot I have time for one question if anybody has yes please

the question is how seriously are owners of scada systems taking um security right do they think security is important it depends it depends on the organization some more mature private organizations where um where either they've already been hacked or um there's there's adult real dollars that matter they are taking steps towards it but for many organizations not at all and I'm I'm sorry to say that many of our public organizations governmental organizations are not really on the ball in terms of that yeah it is scary

yeah things things are changing obviously but you know it's it's not where you want it to be not where you want it to be all right so I'm going to leave with you all with a parting thought here there was a time where you wouldn't do any of these things but now you get into Ubers you might have a Tesla that drives itself you may have bought crypto and and you all work from home at some point so maybe it's time to consider actively scanning your OT environments if you want to contact with me or connect with me this is how you can do it thank you everybody foreign