GT -Defensive Counting: How to quantify ICS exposure on the Internet when the data is out to get you

hey everyone uh thanks so much for coming my session this afternoon um my name is Emily Austin I am a security researcher at census um where I study weird unusual or otherwise interesting things on the internet um and today I want to talk to you specifically about what I'm going to call defensive counting um or how to quantify industrial control system exposure on the internet when the data is less than friendly um before I actually get into this though I do want to say that this I while I'm up here talking to you about this this was a huge team effort by the entire census research team and so I just want to acknowledge the efforts of

Aiden Ariana Hima and Mark on this um because this was truly again a team project so here's what I want to talk to you about today kind of a rough overview um I will spend a bit of time talking about research motivations and some context for this just because it is a rather applied problem um but then we'll get into some kind of talking about some existing work before we talk about the actual quantification piece uh and we'll wrap up with some takeaways talk about kind of where do we go from here and get a little philosophical um so let's just get right into it so I imagine like maybe a lot of you in this room um my career has been in

varying degrees at the intersection of security and analytics or data science and a couple years ago I was having a conversation with someone uh with a similar background ground although more much more on the data science side and they said to me you know security people you'll really like to count things um but why don't you move Beyond counting where's all the like really interesting analysis where's the cool modeling and and all the cool fancy stuff and I I sort of took umbrage with this statement for for a couple of reasons um the first is that you know I think this track at this conference is ample evidence that work like that exists very much in the

wild you just have to know where to look for it um but the second piece that I that I really didn't love about this statement is that counting is actually hard well let me back up so maybe counting is easy but counting the correct things is actually challenging particularly when you want to do this at internet scale so for a slightly less philosophical motivation for this presentation um there's a lot going on on the slide but um this represents sort of a a highlevel overview of a string of threat activity against critical infrastructure particularly in the US um with some focus on water and wastewater um I'll call your attention to the screenshot here in the upper right um

this is an HMI or a human machine interface which we'll talk about in a minute um that was uh defaced last Fall by an Iranian actor this they went after these Israeli manufactured devices um after some local tensions in the region and uh actually defaced these panels This one um you might have heard the story about a water facility in alipa Pennsylvania uh that was hit with this um there are also increasing concerns around People's Republic of China based actors gaining and maintaining access to critical infrastructure networks um and the the final big piece I'll talk about here um is the screenshot on the bottom right um the Cyber Army of Russia reborn um which

is an actor that's maybe potentially affiliated with the Russian military um in early uh 2024 gained access to several water system control panels for small cities in Texas um this is actually a screenshot from one of the videos that they posted on telegram showing their access and sort of messing around with the control panel so I want to go over just a little bit of terminology because there is a lot of jargon in the IC space um IC when I talk about that I'm talking about industrial control system systems um and these are really any systems that are used in manufacturing and automation processes um a lot of them also fall into the category of critical

infrastructure but these are not mutually inclusive and we'll talk about that in a minute um another thing to be aware of here are automation protocols so these are used for communication between industrial devices um so things like building automation or power system automation meter reading Etc um they're really kind of low-level protocols many of them have been around for a lot of years uh and they also typically don't have any form of authentication on them some other Concepts to be aware of as we talk about this um are human machine interfaces or hmis um so these are the you might Imagine by the name these are the interfaces that operators use to Monitor and interact with these systems um while

they are on site at these facilities many times they also offer remote access and the final thing I'll mention here is web admin interfaces which also might be self-explanatory um but they go a step further and provide HTTP based management interfaces so this is literally something you can look at in your browser if you know the IP and the port um they a lot of times ship with default credentials so what I'll I'll leave you with in regards to these kinds of systems is um they're not necessarily Paragons of security engineering so critical infrastructure um I have a couple screenshots here from cisa in the US and the npsa in the UK but when we

talk about critical infrastructure just to get a clear definition we're talking about infrastructure that is considered so essential or critical uh by governments or nations for the functioning of their society and their economy um different nations will have like slightly varying definitions of what is actually critical infrastructure but the important thing to know here is right these are things that commonly include you know power or energy uh Emergency Services water Health Care um and for This research kind of given those previous attacks that we talked about we focused on water and wastewater specifically so our goal for This research was really to develop a quality data set a high fidelity data set of industrial Control Systems devices that

would be granular and accurate enough for us to be able to notify the owners of these devices that they had a problem this is our goal this is our undertaking now I do want to acknowledge we are far from the first folks to try to quantify industrial Control Systems devices on the internet um this is just a sampling of of many many many pieces of research from both Academia and Industry um a lot of these Works focus on several different automation protocols specifically um and maybe a subset of the ones that We examined um in some cases it's not clear whether the researchers excluded known honeypots or deceptive services from these counts um and in some cases it's

also not really clear what the implications of the exposure numbers are so for instance if I tell you there are you know 7,000 modbus Services Exposed on the internet but I kind of leave it at that I'm not really painting a picture of the actual threat landscape right like there needs to be a little more context there what might those be connected to what could someone do with those things um and so we wanted to keep building on this body of research we felt like there was still something we could dig into and and really figure out about this and uh so we took all this knowledge decided we wanted to build on it and we we got

to work so let's talk briefly about the base data set um so at census we scan the entire ipv4 space some of IPv6 uh all the time 65k Port scanning um we have about 250 million ipv4 hosts in our data set right now um with about 5 billion services within that um we have coverage of 22 different IC or automation protocols and over 200 different types of IC software so this is sort of our base data set what we're what we're dealing with all right so let's get into the quantification that's perfect like halfway through all right so first as sort of a Step Zero um we kind of we knew we wanted to shore up the data um

before we really dug in so this is kind of our data enrichment phase um there were two pieces to this so we knew from the outset that we wanted to improve our collection and detection of various IC protocols and software um so some of this was discovering different software on HTTP like in browser interfaces some of it was discovering interesting things over VNC um and as our researchers on this team team started finding the software they also started noticing other interesting protocols running on these same hosts these were protocols that we maybe didn't have uh detailed scanning Logic for and so some folks on our team actually wrote some um so this is the second piece you see here this

collecting additional data um I'll call your attention to peom here um peom is actually a proprietary unitronics protocol um you might remember that screenshot with the red hacked message earlier um that's also unitronics device so we felt like it was particularly relevant to to scan for for peom and add that to our data set so this gives us about 57,000 industrial Control Systems exposed to the internet in the US now because we work in this internet measurement internet analytic space because we're data people we knew that there would be false positives in this data um and this context when we say false positives we're talking about honeypots so uh in this case right like these are things that are pretending to

be something they're not they're they're duplicitous um and there are a couple of really common well-known ICS honey Poots out there um a couple I'll talk about today are gas pot and con pot um both of them are actually available on GitHub so you can audit the code you can look at them you can run them yourselves um go home today and and spin one up um but in detecting these you know analyzing the code some folks on our team were able to figure out uh so in the screenshot on the far left you can see a gas pot uh instance in our data this is a screenshot from census search um and there's some interesting differences in

the date format in Real uh atg or automated tank gauge uh systems versus the Honeypot um automated tank gauge is a is a computerized system that collects and displays information about underground tanks so like your local gas station fuel station will sometimes run the um but that's one way we can detect those um another that we knew we wanted to like pull out of the data is called compot um and compot allows you to emulate a variety of different services including like uh modbus and S7 but um Teenage Mutant Ninja Turtle enthusiasts might notice um in the these screenshots on the in the middle and on the right um for instance on the right you can see

the system is Technodrome and you can see that the plant ID is Mouser Factory so just some fun little for those there so we subtracted these and other similar Services out of our our data set so this leaves us with about 42,000 honeypots or dup or 42,000 Services rather with those removed um and so now that we have this reasonably comprehensive set of data we realized we wanted to filter it even further because we we wanted to again focus on those things that were particularly important for water and wastewater um and so we wanted to filter out protocols that are most commonly associated with building control so like running the lights in an office building

um the you know security system or door system not that that isn't important and not that someone could not cause harm with those but again they felt a little bit on the edge of relevance for us given our very specific focus on water and wastewater um so Fox and backnet are those two protocols that we opted to remove from this data to sort of filter out this leaves us with about 18,000 IC protocols in the US and so we've gone from like 57,000 to 18,000 and now we start to ask you know what metadata can we glean by you know looking at maybe the network where these devices run um maybe there's useful DNS or who is information maybe there's

other interesting tells that will help us figure out you know maybe who owns them so spoiler no that's not at all what happened um so this is the top 10 networks or autonomous systems where we see IC protocols in the US and there is a long Tale on this it is truncated to 10 you might notice a lot of consumer um or business isps here um things like Comcast AT&T you might also notice T-Mobile so a mobile network um I'm actually curious uh is anybody here familiar with celco part Gabe you cannot answer ready familiar cart okay CCO part is actually Verizon um so uh when we start to look at some of the metadata of these hosts

running on these networks there's not really a lot that's very useful there you know when we look at DNS when we look at the who is it all points back to the Telo um and these are often running these you know low-level automation protocols that don't really give you a lot of information about who owns them or where they might be or any other details um so again considering that original goal that we you know had of wanting to um identify owners for as many of these we went back to the user interfaces so these user interfaces are hmis um we identified around 430 internet accessible hmis um you can see the variety of Industries here oil and

gas is a whole other story that's we'll talk about another time um but for water we found just under a hundred and there's some really helpful details about uh things in the in these hmis and that they'll often like just present you with stuff like this it's like city of X plant or city of X water treatment station um and you're able to you know go and look at the geolocation of the host you do a little Googling and you can actually figure out like oh yeah this is probably this water facility here's a contact I'm going to email them right um and in some cases we actually even find the you know a picture of the

tank itself which we can then uh find on Google Maps and verify that that's actually what it is um so the HMI were actually pretty useful in identifying identifying ownership so ultimately of these roughly hundred water related hmis we were able to confidently identify owners for about half of them and so I just want to let that sink in for a minute we started with around 57,000 devices in the US and we identified owners for 50 of them we'll leave that there all right so let's talk conclusions in this last few minutes so first I think one thing we learned from this is that you know looking at the protocol exposure those you know things like mod bus and backnet

and those things um that's one part of the puzzle to understanding this story I think it's also really important to consider those internet accessible control panels because those are things where you don't have to have a lot of specialized knowledge you can access it in your browser and go start clicking around if there's no authentication which many of them don't have authentication it's also not necessarily the number of devices themselves I sort of was trying to tease this on that that last slide um there's not the number of devices that's so concerning um but I think what's really really concerning in the point to drive home here is the real ones we do find in particular the ones

that we we identify owners for they're often they're cities they're actual like municipalities water plants or drinking water facilities um and those are particularly worrisome with they're not protected by any kind of authentication a VPN any sort of measures like that and finally I will leave you with this I will zoom out and be a little philosophical for a moment um and I'll just say you know simple tasks sometimes can be deceptively challenging um and Counting is actually hard to do correctly that's all I have thank you so much [Applause]

we can

yeah yeah and if anyone has questions I'd be happy I think we have a little time be happy to to to take

some I'm also very happy to chat afterwards you can find me uh with the chat how you doing uh any attempts to contact Verizon or any of these providers and try to you know work out attribution yes that's an excellent question so I think we probably need more than just our resources to get all of these Telos in a room and say hey help us figure out who owns these have you tried the isacs no not specifically but that is a good lead thank you um hello thank you for presentation uh question uh you mentioned the modbus protocol and similar protocols that are unsafe um currently there is no incentive or benefit for the uh

companies that are using these protocols to migrate to a safer one and therefore uh no incentive for the vendors to stop implementing those in their products because therefore uh if they do customers won't acquire the product so how do you see that moving forward and do you expect a mandate to come out on that thank yeah so this yeah thank you this is a really good question so I think this kind of gets to the point that there are issues sort of at all the levels with this right like there's issues in the manufacturer space because there isn't really pressure to improve security for these devices at least in the US right now from kind of a

regulatory perspective um I don't know what the future of that looks like I don't know that things will change vastly without some type of enforcement or regulation um so yeah I think I think there's potentially a path forward there I know um I believe in the UK they've enacted some uh manufacturing kind of putting the burden on the manufacturer and so I'm very curious to to see how that goes over the next few years and then maybe maybe that's something we adopt here we can do this the last question yeah and I'm happy like I said please find me after I'd love to chat more about this um so with water I know that they're pretty like cash trapped

it's really a thin business so do you think that there's like something that should be done because they just don't have the money to do any of this stuff like it's not kind of not their fault in a way there's no money and no they're all thin operations so like what do you think think the solution is to actually bring these uh utilities up to speed to what you know the threat landscape actually is yeah so that's an excellent question so one of the things I know uh I think the EPA is now responsible for drinking water facilities in the US and I know with some of the recent kind of attacks and things like that they've um

stepped up their inspections and enforcement actions and they're I think also offering resources if you reach out if you're a water facility and you reach out to the EPA they will help you uh make some of these assessments so I think you trying to find ways to offer those resources because yeah a lot of these especially these small kind of municipalities are resource strapped um so I think finding ways that that the regulatory bodies can step in and offer assistance is probably going to be um going to be key I think that's all the time we have but thank you all so much I appreciate it