There and Back Again: Discovering OT Devices Across Protocol Gateways

Good afternoon everyone. Thank you so much for attending Bsides. Please welcome Rob King with his talk there and back again. Discovering OT devices across protocol gateways. Hello Mike. Mike mic live. Yes. Perfect. Hi everybody. Thank you for coming out and uh and thank you to the Bides staff and volunteers. This is awesome. This is a cool venue too. So, thank you so much uh for coming. So, how did uh how did we get here? Said in the most uh talking heads way possible. Well, I'm sure when you uh when you read the title for the talk, you were like uh just like me not so long ago, you were probably like OT and protocol and

gateways and across what does all this mean? And it's like well uh let's start with some definitions because one of the great things about this part of our industry is that these terms are very uh fluid and overlapping and uh interesting. So operational technology is technology that affects things in the real world, right? The pumps and the the uh valves and things like that. Um, and includes things like programmable logic controllers, which are the ruggedized computers that work in high vibration environments with heat and molten metal surrounding them, not touching them usually, hopefully. And computer numerical control systems, which are uh even more ambiguously named because they're actually the machining things that spin and do cuts and stuff, but it

just sounds like math. I guess it is math. Um, and a lot of people will include things like medical technology in operational technology, but a lot of people don't. It's a very fluid kind of term. Um, and industrial control systems are then specifically stuff for doing industrial control. And then SCADA or supervisory control and data acquisition, uh, which I actually have a colleague, um, she has a shirt that says SCADA, but it's written in like the Sega logo. So every time I I say the word SCADA, I'm always like SCADA, but I never never quite do it. Um I want to though. Uh SCADA is the computers that do the actual control. And one of the

sort of the hallmarks of SCADA is a having a a similacrim, sort of a a diagram of how things work and a model of how things work. Now, what's interesting is I've had plenty of people way smarter than me, that's most people way smarter than me, who have used these terms differently and other smart people who have used these terms differently as well. So, I know that some of you out there are maybe looking at these definitions and you're saying, "Well, actually, and I've heard all the well actually. So, just we're going to stick with OT for this talk and and if it uh if you disagree with me, then you know, that's okay. Come come tell me and we

can uh we can nerd fight about it. It'll be great." So when people think of of OT nowadays, right, operational technology factories, they they think of this sort of cool mission control thing that looks like it could be coming out of the Apollo missions. And it this is part of it. It looks super duper cool. Um this is from a a manufacturing plant in Germany, I think. And I don't know what they're making, but I love the photo. They even look like they're wearing like Star Trek uniforms. It looks super cool and high-tech. And this is definitely part of it. This is the technology part of OT. But what people kind of forget is that at the other end there's the actual

operational devices, the old school things with gears and valves and pumps and you know they make noise and they spin and they vibrate and it's crazy and there's a lot of history there that maybe you know we kind of need to talk a little bit about how we got from where we started to here. And so operational technology, I mean, it if you want to get down to it, it's like, you know, the first time somebody hit a rock with another rock, right? But it's we've we've come a long way since then. Um, but it used to be that operational technology was it started out with maybe the conveyor belts that you would get on

assembly lines to keep things in sync. Or you might have a single large uh steam engine or water wheel and then ropes and pulleys uh to convey the force and the energy around to keep things moving. Um, if you ever been to one of those restaurants where all the fans are on like the same belt, you know, it's kind of like that. It was pretty cool. Um, not that I saw it personally, obviously, not that old. Um, but then it kept moving on until you got to wanting to have computerized control, more intelligence behind this, not just gears and clockwork, not just off and on. And so we started getting these sorts of things, these little panels that could

control a specific part of the control loop, a specific part of the factory floor, you know, one machine or whatever. And this was, of course, a big advance, but if you wanted to control it, you had to physically be there. You had to walk over and push a button or take out a fuse or a relay and move it around and, you know, do all sorts of crazy things. But you could still do things like, I want to make this go faster, push, push, push. I want to make this go slower, push, push, push. Much easier, much better, much finer grained control. But then we wanted to, you know, what if we got even more control?

What if we made it even better? Well, then we introduced the concept of programmable logic controllers. And so PLC's. And PLC's are these ruggedized computers like I was talking about that are designed to bring, you know, the full paniply of touring completeness to the concept of industrial control. And it's super great and they work well and they're universally everywhere. Um, they're called programmable logic controllers not computers. I I don't know why. Uh, for the for those of you out there, uh, I I'm going to digress for a second because I'm super history nerd. It's like when Deck called the digital equipment called the PDP the PDP and not a computer because computers were expensive and big and the PDPs were by

today's standards expensive and big but you know they were much smaller and cheaper than IBM's computers. So so programmable logic controllers then let you start controlling things at a distance and monitoring things at a distance and doing more complicated things. And right about 1978ish, Modicon introduced the Modicon field bus. So, it used to be that you'd have these devices if you wanted to monitor them or control them remotely. You would have to basically run a a set of wires to each one, right? And Modicon wasn't the first to come up with the idea of a bus or or even the first one to do it in in the industrial control setting, but they were kind of the first

really big standard player. So um a bus, you know, it's just a a shared medium across which you can send messages and multipplex this medium between potentially many senders and potentially many receivers, right? So now you can get away with just having one set of wires. And so Modicon, the modicon field bus would let you uh even, you know, just daisy chain serial connections from one device to another. But when you did that, you would need to have some way of disambiguating which device you were talking to because now you're not talking across one set of wires to one specific device. You're talking to potentially many devices. So with buses comes the concept of uh addressing and

uh con uh uh collision detection and you know reserving the bus and making sure no two things talk at the same time because then it's chaos, right? And with that comes the concept of protocol adaptation. So now you've got this Modbus device, Modicon field bus, Modbus. You've got this Modbus device that's talking to a controller which is being controlled by a human or monitored by a human or whatever. But then this device is potentially talking to some other device that doesn't speak Modbus, right? It's talking to a pump or a valve or it's maybe talking to another uh intelligent device, but it just doesn't speak Modbus. So now you have to adapt the protocol from Modbus to something else.

And then these protocols they there every vendor had one um and they were all different and they ran over different media and they use different you know communications protocols and they use different connectors and of course you know it was just chaos right so what we're seeing now is this big push towards uh convergence and so convergence in this case is it used to mean again if you're super old it used to mean having a a cable box that was also a web browser. Anybody remember that set top convergence? Um yeah, I'm dating myself there. But but uh we're now talking about the convergence of IT and OT. So information technology and operational technology. Uh and it used to be that OT

was considered to be the uh the IT in the non-carpeted parts of the building, right? It it used to be that it was it wearing a hard hat, but now that's really true. It's now falling under the purview of CISOs. It's falling under the purview of CIOS of the IP world because things are converging on IP protocols. So IP, remember IP wasn't always the standard. Like Modbus came out in 1978 and TCP was not standardized until 1982. And even after it was standardized, it wasn't necessarily going to, you know, nobody was sure it was going to take over the world, right? Um it was the internet protocol designed to communicate between multiple networks. Those multiple networks might be

speaking IPX or DeckNet or the bag of Eldrich horrors that was IBM SNA which token ring was nice but so it was so the but now IP has taken over the world IP and Ethernet for the most part taken over the world and it's become a commodity and so that's great because now all of these industrial protocols are starting to migrate on top of these commodity technologies. Um, and that's great and that has benefits and detriments, but one of the main benefits is that these devices become more accessible. You can reach them more easily. They can be integrated with existing networks more easily. You can communicate with them remotely. That's all great. The biggest downside is you

can in fact communicate with them remotely. So uh we've done some studies and we a lot of people have and it turns out that there are thousands of OT devices on the public internet. You should not have your OT devices reachable on the public internet. This is a really really bad idea. A lot of these devices security was an afterthought. Security wasn't even a wasn't even an afterthought for a lot of these devices. They were designed to work in a factory with a grade separation. And the only way to control them was from the factory floor or from the control room at the factory. There was no concept of you know please authenticate me let's have

cryptographic device identity at hestation right these systems were not designed to even have that kind of CPU power they they couldn't do it so it's getting there we are getting better which is great but for a lot of these devices there's no concept of login there's no concept of password there's or if there is it's very rarely enabled and even if it is enabled there's they're trivial to bypass. So um we've seen quite a few for example there was uh automatic tank gauges which is a protocol used uh not surprisingly to read tank gauges like you know big industrial tanks of fluids right super common uh at like gas stations. So this is uh one of the protocols that's used

to determine how much gas is in uh you know like the big tanks underground. And it was it was also super frustrating because so this heat map shows just how many automatic tank gauges are accessible on the public internet and it's huge and it was super frustrating because I was doing this research and I found all this great stuff and uh I work with uh one of my colleagues who is absolutely brilliant and it's always frustrating because I'll do something and I'll like do be doing this research and I'll come across this perfect piece of code or this little article or something and I'm like oh this is awesome. this is exactly what I wanted

and oh it looks like they already kind of started it 10 years ago and then I find out that it's exactly what it was and it's the guy I work with and you know who you are um HD but uh he did something like this quite a while ago um but super interesting and yes you can go and find out all sorts of gas station information publicly but it's not just gas stations we found and of course I'm not going to show you the IP addresses because I'm not a monster but we found all sorts of things publicly exposed on the internet PLC's uh power devices. We could shut things down remotely. Not that we would because

we're nice people, but these things are exposed on the public internet. Not a good idea. But this is all due to this convergence of OT and IT. So, how do we discover these devices? Right? So once you're on an IP network, you're now having to communicate through uh the you know, you're communicating with this IP network, but you're now having to do this protocol adaptation because you're still speaking these older uh industrial protocols that have just been sort of brought up to the uh the the IP world. Um which is great and I love it and it's awesome. But so the easiest way is to work with um Modbus. This is kind of the first

easiest example to give. So Modbus was originally designed to run over serial connections. Um RS232, RS488, uh a lot of the other RS standards. Um and so you would just they would even be daisy chained along and you you would pass a message, you say, "Hey, you know, I want to talk to device 14." And so you talk to you send the message down the serial line and the first device would say, "Oh, I'm not device 14. I'm going to forward it on." You eventually get to device 14. And device 14 would come back. And there wasn't any need for identifying these things because you already knew what they were. You had to go physically like

set jumpers and stuff to set their addresses. So you you should know what they are. So the earliest versions of Modbus did support only on serial lines only in the serial version of the protocol they did support like tell me your address. So that if you had a direct connection on a serial line you could say hey Modbus device at the other end I'm just going to send it to address zero which is you know nothing. please tell me what your address is and it would tell you back. That doesn't work for Modbus TCP. Modbus TCP is uh a much nicer protocol though in that it was they added the concept of the Modbus encapsulated interface um which is

always fun to say. And so what the mod exactly what it sounds like, it lets you encapsulate a more complicated message inside a single Modbus frame, single Modbus command because the original Modbus uh protocol super simple, right? It's designed to be parsed by a system from 1978. So it lets you as part of the payload just give another command with a richer set of, you know, messages or objects or whatever. And then if a device doesn't understand it, it'll just ignore it because it's an unknown command in the command field and you know the payload goes away. But this um Modbus encapsulated interface supports the report device identity command. So if you want to discover what a Modbus

device is, you can say, "Hey Modbus device address 15 or whatever, please tell me, you know, here's a modbus encapsulated message. Give me back your report device identity." And it'll say, "Okay, sure. Here you go." And that reports it back as a set of objects which is pretty simple. Uh you can say object one is you know product code object two is product name but it's very it's stateless. So it'll say um yes you know here's the first object object zero next object is object one. And so you request object one. It says okay there's object one. Next object is object two or or there are no more objects. Um, we did have one that was really fun and that

there was a bug where it would say, "Okay, next object is like four and then you get next object four and then say, okay, I'm object four and next object is three." And you go back, it's like the DNS compression pointer bug, which if you've never dealt with that, that's a lot of fun, too. Um, but the point is Modbus TCP, there is a modern way to discover these devices and you can discover things across the the protocol gateway by simply specifying an address that you, you know, can determine is on the other side. The address space is not huge, so you can even pull it if you want to, although you know, maybe not. But, so let's move on to something

that's a little bit more complicated. This is the distributed network protocol 3. Um, which apparently there were there was one and two. I've never used them. DMP3 is the sort of de facto protocol standard for electrical distribution networks in North America. and it when they ported it to TCP to run on top of TCP, they literally just took the entire stack of DMP3 and set it on top of TCP. So DMP3 has its own transport layer and its own addressing layer and its own connection layer and its own application layer and it's all just kind of shoved in there and it's even still uh blocked and checksummed inside the TCP stream. Super fun to work

with. Really, really complicated. And this is a diagram just so you can kind of get an idea of how complicated this protocol is. Uh but I do love writing protocol parsers and it was a lot of fun to write this one. But the problem with DMP3 discovery is that because oh and DMP3 does support um from the get-go it does support a collection of you know please tell me who your manufacturer is. There's these numbered attributes and you say hey please tell me the value of numbered attribute 57 which is your hardware version. You'll get it back. That's all well and good. The problem is that you have to know the DMP3 address, not just the IP address. So if you know

the IP address of the DMP3 gateway, that's only half the battle. That's not even half the battle, in fact, if you if you do the math. So DMP3, like I said, literally just put the entire DMP3 stack on top of TCP. So they they didn't even bother to like add a little bit of other stuff. So you have to when you connect via IP, you still have to know the DMP3 address and there is no way to discover it except by the couple ways I'm about to tell you. So a DMP3 address is 16 bits, but some of them are reserved and you still roughly have 65,000 uh addresses to work with. Secondary devices and primary

devices. The secondary a secondary device will only talk to its preconfigured primary device. you can it will not respond if it's if a message comes at the DMP3 layer from another device even if it's coming from the correct IP address. So it's a completely it's a grade separation inside the protocol itself. And this is usually configured at you know boot time or whatever. There's no DHCP for DNP3 um although what an acronym that would be. Uh so you have to guess both if you want to communicate with it and get information about it you have to guess both the primary and the secondary addresses. And with 65,000 possible addresses, that comes out to like 4.2

billion possible combinations, which you should not do. That's a bad idea. So, the good news is that usually, at least in all the DMP3 networks I've seen, the addressing they keep it towards the front of the range. So, usually the addresses start at one and usually the addresses don't exceed like 512, right? Obviously, different networks could do it different ways, you know, who knows, right? But even with just 512 addresses, you're still going to end up with 262,44 addresses, which is more than address pairs, which is more than you're going to want to deal with. So, what we have discovered is that you can kind of lean on a it's like a safety, it's not a

safety feature, but it's an interesting sort of hello that you get from about 70% of DNP3 stuff that we've tested. So, DNP3 has this concept of an unsolicited response, which means it's responding to a question that wasn't asked, uh, which I do all the time. Um, so with DMP3, when you connect, it will immediately send out a response. If you connect on IP, it will immediately send out a response saying, "Oh, I am uh I'm in this error condition or no error condition. My clock is synchronized with the primary or it isn't." And but the benefit there is that when it sends out that message, it's going to send it from its DMP3 address. The the DMP3 DNP3

packet is completely just sent on the wire. So you'll see the DMP3 framing with the DMP3 address. The source is the device you're talking to and the destination is going to be its designated primary device. So you can simply reverse that and you can say, "Aha, I now know what I'm supposed to do. I can now spoof this primary." And it also lets you narrow down the range of addresses you have to look through because you know if you're getting a response from device 7 then there's probably devices 6543 and there maybe a device 8, right? So super super useful there. Um but the big the big one and my favorite is uh Ethernet IP and SIP and

this is um the worst. So we were talking about like bad terminology earlier or conflated and ambiguous terminology earlier. This to me personally takes the cake. So Ethernet IP is is barely Ethernet and is not IP and SIP is uh there's the critical infrastructure protection which is a common set of regulations in OT and then there's also SIP the common industrial protocol and I'm talking about that one not the other one but I have to ask every time when somebody's like do you support SIP or are you SIP certified I'm like I don't know which one. Um, so SIP is an abstract protocol built around objects and interfaces and it's it's the the height of like late 90s early 2000s when

everything had to be object-oriented and it's quite pretty but it's this abstract protocol that is then concretized adapted to different transport and physical layers and Ethernet IP is the adaptation of SIP to internet protocol and Ethernet So you literally have Ethernet internet protocol Ethernet IP and it's you got IP Ethernet Ethernet. It's it's really really confusing when when you're talking about it. So usually it's a it's abbreviated as ENIP which which helps a little bit. But there's other applications uh or I'm sorry adaptations of SIP. You have device net componet control net all these different ones. But what's cool about it is that makes it a little bit abstract which we which we'll see. But at the lowest level, so

the Ethernet IP level, you don't even have to talk SIP to discover something that's directly connected via Ethernet IP. It's still going to speak SIP. It's a SIP device, right? But you can uh it supports discovery at the Ethernet IP level, the actual IP addressing level. So you can send what's called an Ethernet IP list identity command. And you can see, you know, a wire shark dump here. And you say, okay, you know, here's an Ethernet IP list identity command. please tell me who you are and the device will come back and say oh yes of course I am a Rockwell automation Allen Bradley programmable logic controller and my version is this it's super great you can even which is really

really nice send this as a broadcast UDP packet and just ask everybody on the network please you know come to me my children and tell me what you are and you'll find it all out and that's great but Ethernet IP that's only getting you what the protocol gateway is which is great don't get me wrong but that's where a lot of this stuff stops s and I take it as a personal uh a device on a network and I can't find it, right? Like I uh I really really want to see everything that's out there. So the way Oh, and and there's a Ethernet IP can work with lots of different devices. This is a one we have in our lab. It's

an OLink device and it speaks Ethernet IP and you can just say, "Hey, Ethernet IP, tell me who you are." And you get it right back, which is great. And this is another example of a protocol gateway that is talking to something completely different, right? This is actually talking to um Olink, which is another industrial protocol that isn't actually an application of SIP, but this adapts Olink to Ethernet IP and SIP, and it's all super great. But what you'll often get is these devices that have um multiple, you know, you'll have the Ethernet IP adapter, right, the protocol gateway, but then you'll have another bus or uh even a whole other network speaking maybe even Ethernet IP behind that

device, but it's a whole private network that you can't necessarily see. And when you're talking to the Ethernet IP protocol adapter, you'll say, "Hey, tell me who you are." And it'll say, "I am this." But then it's not going to necessarily tell you about the, you know, eight devices in a trench coat that it's got behind it. And I want to know about those guys, right? So what you can do, Ethernet IP supports this concept of an I of well of a of a connection manager object. And the connection manager object uh you know, not to keep you in suspense, it manages connections. But you can say okay I would like to you know talk to this Ethernet IP device

this gateway say okay please give me your connection manager object it says okay you say all right please tell me about your connections it says oh sure it says okay please build a connection for me from you know my management station to this device and then I want to send I want to connect to and you then you send a message that says okay now I want to connect to the identity object on that device and what'll happen is the connection manager It's almost like a VPN right in this industrial network and you say okay sure I will forward I'll forward your connection to the identity object through myself um and then forward the response back and

the identity object is so doing requesting all the attributes of an identity object in SIP is almost equivalent to doing a list identity in Ethernet IP you'll get pretty much the same information back but what's neat about it is that this is this could potentially be going across like four different media right like this could be talking over a back plane and then talking over a uh a device net or even a whole other there might be an industrial firewall in between and then you're doing NAT. It's it's super super cool. And in fact, that's kind of what you can see in this example here is when you're talking on the back plane, uh when

you're talking to the the connection manager object, you say, "Hey, please, you know, give me this connection to the back plane." And so you'll talk about the back plane and then you'll say, "Oh, I'm going to keep building the connection and say, now I want to go across device to this device." Um if uh anyone out there who's old enough to have had an exclamation point in their email address will know that you you have to build the path yourself, right? You have to say, "Okay, now I want to go to the next device. Tell me about yourself. Go to the next device. Tell me about yourself. Go to the next device. Tell me about yourself." And this could

it's it's like you're a puppeteer because you're talking over a serial connection over here and you're talking over a device net connection over here and you're you're you're spreading your your your fingers. It sounds nefarious. You're spreading your your device discovery throughout this this uh this hidden this hidden network. And uh so but what you have to do is you have to build that uh path in this this um route path specification in SIP and so the route path specification is extremely rich um like there's it's super super complicated but the route path each segment is identified as okay you can say I want to talk to the back plane slot four and then maybe the device in

slot four is another Ethernet IP adapter and it's got its own IP address and it's talking to an IP address over there and so you have to say okay I want to go first step is back plane slot 11. Next step is Ethernet IP adapter. I want to talk to IP address 1926 1687.4. And you just keep building out your path until you discovered every single thing in that hidden network. And so this is an example of when I was uh walking the back plane, which um sounds like uh life in the fast lane if you say it fast enough. So it's stuck in my head now. Um, but you can see that this was

talking to the uh protocol adapter, the general purpose discrete IO, and then it's speaking to the 1734 PLC. And you see there's an RS232 device in there. So, this is actually I'm having to transit from Ethernet all the way to RS232 classic serial to talk to another device and get information about it. And I love that. I that just makes me so happy, which it does. It makes me happy. I think it's super super cool. So the idea is that you can in fact discover devices on these networks even through these legacy protocol gateways, right? These protocol gateways are not where you should stop. If you're trying to really find everything on your network, don't stop at the protocol

gateway. Keep going. Um now the fun part is um the standards are not really super available. Like some of the like the Modbus standard just go to modbus.org, download it. It's a short small PDF. Read it. It's printed out. It's on my desk. It's only like six pages. Do it. Um, DNP3 also publicly available though there is no hard and fast specification as to what attributes correspond to what like vendor name and stuff, but everybody kind of standardized on the Schneider Electric numbering. So, yay. Um, but the SIP stuff is uh you have to about half of it is publicly available, which is great. And about half of it you have to be a member of ODVA,

ODVNA, which as far as I can tell stands for nothing. Um, I think the A is association, but I literally think the rest of it is used to mean something and now means nothing. And if you're not a member of that, there's a lot of cross referencing that you have to do to things that you might not have. And so you have to do a lot of reverse engineering. But the point is, you can do it. I can do it. I did it. And you're all smarter than me. So go forth and discover things on your OT network. Bass Protocol gateways. Thank you. Hey, I made it through time. All right, we have two Rob

questions. Yeah. Any questions? We got two minutes. We We have one question if you want to go off the slido. Uh we have a We have a few. Oh, perfect. Okay. Okay. Um they're anonymous, so I'm not sure if it's yours. How does the pump know that the signal to turn on which it is receiving is authentic and appropriate? ah that so uh the answer is it probably doesn't um there is especially in SIP there is the concept of authenticated SIP and SIPs and and things like that but once it gets down to that level um it it's and and that often is not even enabled in a lot of networks and once it gets down to

that level um it's just kind of implicitly trusted that it should do what it should do. This is why you should not have your OT stuff exposed to the public internet. Please please don't do that. So all right one more. How does the system know that the signal from a sensor is authentic and accurate? Again, it probably doesn't. And in fact, going the other way, it's even less. A lot of times, those devices are the authentication usually is go, if it's enabled at all, usually goes from the top down. Coming from the bottom up, there's considerably less. Um, but I mean, you can do it. There are, especially in SIP, there are ways you can, but it's often not enabled. I think

we can squeeze this last one into about how many of these internetex exposed OT devices are actually just honeypotss. Um, we did check and so what's funny uh for like the ATG ones, you can tell if they're honeypotss or not because the honeypotss will respond faster because these devices are so old and slow that uh if you connect to them and they respond immediately, you're like, "Oh, that's a honeypot." So, uh, there's probably around, at least in my in my data set, we had around 10,000 that I would consider legit. So, yeah. Thank you so much, Rob. Thank you, everybody. Thanks for attending, folks.