The ICS Attack Surface You May Be Missing

all right so what do you think should we get started go ahead and get going all right well thanks thanks for coming to hear me talk about ICS attack surface I am Jason Holcomb and what I want to talk to you today talk with you today about are some things that end up in places that you don't expect or things that end up showing up in strange way is kind of like the the picture that you see here all right as anybody use this site called the wire cutter to read reviews anybody seen this okay I was hoping for more response from that well they one of the things they do is they they do all

these product reviews and they always start off the product review with asking this question how do you know that you can trust us or why trust us and so I thought I'd do the same thing here so why why trust me to talk about some of these things related to industrial control system security and part of it is that big big number one reason is that I actually was even though I don't live here now I grew up here in Oklahoma over in the little northeast corner there in the grove in Grand Lake area so it's great to be back here with a bunch of fellow Oklahomans although I've talked to people from all over today so

it's not just an Oklahoma crowd but I was hoping that you know playing that card would would help me out here and building some trust but in addition to that I have spent the last twelve or fourteen years working in the control system security world crawling around in places like this power plant that you see here and oil platforms and and that sort of thing so along the way I've collected some war stories I'm gonna call them and I'm gonna share those with you today specifically around finding some attack surface in places that you might not expect it in this kind of environment and what is this environment that we're talking about exactly you know a few years ago

we were talking about industrial control systems and these sort of heavy industry applications like electric utilities and oil and gas and manufacturing and then this thing with embedded device security started happening and then the Internet of Things and so we ended up with this sort of mix of terms and terminology and and I don't think it's real clear and in everyone's mind and I think it helps to kind of level set where I'm coming from so you can kind of help understand the things that I'm going to share with you today and I'm hoping that what I what I'm able to share with you will help you either from the perspective of sort of the defender and understanding the kinds

of things that you can do to better protect your environments and and even if even if you're not someone who has that kind of responsibility for actually defending an industrial control system environment I think there's some things you can take away from here as well and if you happen to be in the position like I am where you're doing testing and assessments of these systems I'm hoping you can take away so maybe new rocks to turnover and some some things to look at when you're doing evaluations in these environments all that to say to help us level set here on the industrial control system side that you see in the the top circle of our Venn diagram there you

know this is sort of the historic you know the way that we think about control systems but like I said things are evolving a little bit so on the bottom right there you see this Internet of Things concept and and there's some similarities right there's some overlap there you know just the way that these systems work and I try to highlight a few of those in some of these intersections here and and really the the space that we're in this industrial control system world is changing so you see the era that I've got there sort of trending into this industrial Internet of Things and it's that's affecting some industries more than others but what it

means is that we're bringing in a lot of different ways little different parts of the technology that make it harder to sort of isolate those systems and just keep them separate over here to the side right this is a trend that started years ago as they started moving out of the world of you know serial communications and into IP communications and you know increased need from data from the business networks that was you know started pushing more connectivity and soon all of these systems that were really designed to be isolated and islanded weren't that way anymore and now you start layering on some of the data analytics and cloud aspects of this and it you know the picture gets even a

little more convoluted and you see that we're kind of evolving from even what we you know used to consider these traditional ICS systems into more of this industrial Internet of Things so that's so I guess wanted to use this to kind of level set and say what I'm talking about are the industrial applications of these things here not necessarily the consumer side IOT there are fascinating topics there you know with what happened last year with the Mirai botnet and they're just there's a lot of things going on in that world and some of it intersects here but really the perspective I'm coming from is this ICS and industrial Internet of Things side and kind of again here to share

some stories with you about that so there was a talk earlier today that was kind of the scary wake-up call here's how bad things are you know in in these kind of environments it kind of laid a good foundation if you happen to catch that one I'll hit on just a couple of you know things here if you haven't been exposed to this world you know again it was a lot of these systems were not designed with security in mind so they're sort of open by default and so they had these insecure by design issues in them so that the the software configurations the even the protocols you know didn't don't require authentication lots of issues that

you've seen with PLC's the embedded device side of of these industrial control system applications so those are known issues those are things that we've been talking about for for quite a while in you know for those of us that have been foe just on this problem as well as all of the software vulnerabilities that go along with it so you know communicating out to all these you know connected devices in a substation electric substation or in an oil and gas application there's software right so we've got software we've got operating systems sitting there in an environment that perhaps doesn't get as much attention as they should from a patching and you know doing all the good hygiene

sorts of things and and again what I'm talking what I'm sharing here are some of the known issues and then what we're going to pivot to is some things that that I've observed just encountering all these systems that are sort of outside of this normal box right so fairly common to see these in secure protocols fairly common to see unpatched windows boxes for example you know how much MSO 867 do you you know do you want because it's out there but in some ways you know and in some applications in some places it's progressed beyond that and so our assessment techniques and our defensive techniques also have to move beyond some of those fundamentals and I'm hoping to

what I share with you today will show you a few of the areas that I've seen that if you you know if you've got everything else kind of locked down that fits into this known issue category then we've got some other things still to look at so you might think that with all of the all of the challenges that were really just you know have a problem here with with security on these systems and in some places that's true but again I think we've progressed in a lot of industries have started making significant investments including electric utilities and oil and gas and and starting to and some of the heavy manufacturing as well so we're moving

out of this world where everything is you know not that great when it comes to to configuration and part of the way that we're you know the historically that's been done is kind of like the fence that's around this electric substation right here we've sort of walled off you know and fenced again the the isolation aspect of that that's one way that you know historically we've tried to protect these systems that are fragile you know like it was mentioned today earlier and there are you know cases where a single packet can cause some of these devices with fragile IP stacks to fall over and so you know the natural thing to want to do in addition to start that long path

of building better systems and building better components is to kind of wall that off and like I said in a lot of places this is this is starting to you know this this is happening and things are getting a little better but there are still a few things out there that I want to talk to you about today that present an attack surface opportunity for ICS that may be kind of like the the dog sitting on the cat pedestal aren't what you would expect to see in these systems so the first one I want to talk about and it's it's it's afternoon time here and I it's like sleepy time I can tell so we've had lunch we've had one of

the other talks so we'll try to get some interaction going here like who can tell me what is this little symbol mean what am I going to talk about in this number once a database is bingo thank you we have audience interaction success so databases are probably more common than you think in these systems and and the reason I say that is because you know you think of some of these specialized control system SCADA and DCs applications and you just think of it all as being very proprietary and that's a pretty common perspective but what has been you know enlightening to me over the course of the last 10 to 12 years of doing assessments on on these systems is

how many sort of off-the-shelf DBMS tools there are out there you know the Oracle and I'm a sequel and so they're out there and they exist so what you see on the left and I'm gonna in case you've never seen this sort of representation of a control system environment before this is based off of the Purdue reference architecture on the top level there you see sort of your enterprise IT zone and then kind of in between there you've got this DMZ world that is sort of sits in between the the plan or the control system environment you have the control zone in that layer-3 where you might find some things that look a little bit

like traditional IT and then the further you go down the the further you get away from the look and feel of traditional IT you get more into the controllers and instrumentation and the things that are actually down you know actuated a you know a breaker or taking some physical action and the kinetic world you know or majoring something in in the physical world down at that lower level so that that's the concept we're setting up there and I put a little database icon to show that often there are at multiple levels different database applications that are running and why do you think that is what are some things that the on the up on the IT the top level of this

that we might want to know about from a control system environment you know a lots of things to inform business decisions you know need to happen up at that level that are coming from the control system environment so there's a lot of database communications that happen there so a few things about the database that you see in the bullet points here the first one is sometimes these things are buried into these applications and that well that's part of what makes them look like it's proprietary so you go talk to a control system engineer or an instrument instrumentation and automation technician they may not know that their application has an Oracle database embedded into it that's very common

because it the the vendor has made it to look and feel like it's all part of you know that particular SCADA application and you're gonna find these in various places so IO servers historians configuration repositories just in a variety of places right and again they may not even be aware that they're there part of the reason is sometimes the the file structure you know is you know for you know for those of us worked with Oracle in the past you know you you're used to looking for certain directory structure that may be buried tin directory levels under whatever the application is and unless you've actually you know looked at the services running on the Box done some

kind of interrogation you know of that you may not even be aware of what's running there minor point here is that sometimes this stuff ends up because it's kind of wrapped in the control system application shell it sometimes looks a little bit more like it's embedded part that application sometimes that means that it's running on a non-standard port even so why it you know when we look at this what's the problem here what is the security issue that that comes to the forefront and I've seen just over assessment after assessment here is the sort of trifecta that causes the problem of databases and control system applications it's it's this starting with default creds on the

on the left there it's it's a it's a rampant issue again if if you're an administrator for one of these systems you don't even know that there's a database there and you know it was kind of designed to be open anyway they're gonna be default credentials in there very it's there very few that I've ran across that have actually taken care of this fully and if they're not default credentials like the glist that you might go pull down or you know for Oracle or what's embedded in some of the scanning tools inside of Kali if you know the names of the application or you can go do a little google hacking you probably can find some of the default

accounts very easily and I've seen that played out in assessment efforts but just getting into the database is kind of interesting so you might be able to log in you might give you some more information about the system you can dump some tables and take a look at things but it's this additional level a week configuration that becomes even more interesting things like being able to execute XP or you know windows level shell commands out of your database right that's a configuration problem you know in in most of the modern systems this is something that's disabled that may be an overstatement but in the control system world it's not something that people have paid attention to so

very often you can move from having access to a database elevating your privileges exploiting the weak configuration to run whatever it is that you want to run and then you gain a foothold out of some you know untrusted zone that you're coming from into a more trusted zone and then you see the attack path where you know I'm able to progress you know maybe I've got you know a lot of controls around a lot of other parts of the network environment that would keep me from moving from the enterprise you know level four and five down to level 3.5 but the database opens this tiny little window that allows me to jump in and again something I've seen in

numerous places examples of of where I've seen this you know Oracle example for an oil and gas environment where it was doing exactly so the that model of the the Purdue reference architecture it was it was exposing the control system up to a higher level of the network and you know was creating a vulnerability from from that perspective another example electric IO and system configuration exposed through misconfigured in this height this time it was a Microsoft SQL database server same kind of some kind of issue where it was creating that opportunity of creating that attack surface that someone could use to gain access to the control system environment and then the last example here was one

in a recent pen test where we had an objective to move from the IT environment into the industrial control system environment in a very controlled manner that's tightly controlled with rules of engagement in case anyone is concerned about that one of the one of the ways that we found in was through a database that was used for regulatory compliance again very similar situation these other two it was what created the opportunity for us to gain a foothold and gain a pivot point into the ICS environment from the corporate IT side so in summary and for for the database we're gonna go through a couple more of these and then we're gonna follow kind of the same format so we'll give the

introduction to what what this attack surface is will cover some stories about you know where it was a problem and then we'll kind of do this summary slide of of what the issue is so again it's database management software creating this attack surface in ICS that you may not be aware of and again we're trying to focus here on things that might kind of be fall outside the radar if it's not something you're specifically looking for it doesn't fall under than normal you know sort of Windows and you know network vulnerability assessment it might be something you would miss and I think we've I think we've hit all of these points except let me let me talk

just a little bit about you know how do you test for this so you know I have already mentioned that scanning in some environments can be problematic just because of the fragile nature of the devices that live in those environments if you know if you know what's there well enough that can be done in a controlled manner from particularly and you know the further you are in that Purdue reference architecture you know we've again done that under under careful control and usually it's a you're doing things like you know saying saying to the control system engineer I'm going to scan this particular device now let's understand what happens if it goes offline let's understand if it impacts the system and

you know you walk through that and they say okay well that's it's a historians historian server sitting in a DMZ the impact if something happens to it as minimal okay go ahead and scan it and and scanning here we can go in as much depth as we want and I'm happy to talk more about this offline as well but this scanning here I mean this could be everything from just network discovery within map it could also be targeted scanning once you identify that you have a database let's say it's Oracle sort of one of my go-to tools for that is oh scanner and you can pop up oh scanner and it will you know do some of the

default credential techs in an Oracle database you're gonna need to know the SID or be able to you know use some tools to guess the SID that's usually not that hard and that's a kind of first hurdle so there are some tools you can use to help with this in addition to that there's main your configuration inspections so there's actually just digging into these boxes figuring out what's running on them looking at what services are running and shockingly a lot of the times no one has taken the time to do that so if you're in the position where you are the defender for this kind of environment this is you know the the call to action

for you is to really understand how that system is working both at a host level as well as the network level so that you can understand if you've got this kind of attack surface issue on the how do I fix it column that this is going to be kind of a theme as we talk through the various topics here obviously we want to reduce that attack surface if we can do that with an were control and not impact the operation of the system that's a starting point if we can then lock further lock that down at the host and database configuration level that's great sometimes that means that you're getting on the phone or working with

your control system vendor or integrator to actually make that change happen in a way that doesn't affect production operations and I can't stress that enough in these environments it's it's not a place where you want to be testing things in production right you want to understand what the impact of the change is gonna be so any any change that you introduce could be detrimental to the environment therefore you know know what you're doing and a lot of times that means working with a control system vendor to do it alright so number two and we only have three of these so I promise I won't bore you to tears Ford well I might bore you to tears but it'll

be short so the second one that I want to talk about is who wants to guess the icon Wireless why would you have Wireless in a control system well it's all over the place in various applications you know various types of systems it is used in in the industrial control environment so we're gonna start by talking about industrial wireless just a couple of you know stories here you know early days of ZigBee there were a lot of challenges with different vendors implementations some of the early smart grid you know home area network applications had some serious challenges and you know there were issues there that we're exposing from the home area network side of things

back into the SCADA environment you know depending on how the architecture was was set up so that's that's one one place where we've seen it another just you know example of something we've seen is a wireless heart so wireless heart is another industrial protocol and in this instance it was for an oil and gas since her network and the way that the controller was configured it had a vulnerability that allowed you to act gain access to that environment you'll see over on the far right there a little snippet from a show dan search and you can see that a lot of these things end up getting connected directly to the internet and so you've got you know the

case where you've got vulnerable industrial wireless connected the internet just not a great situation for for the control system and great for convenience for people that might need to to access it but not great from a security perspective when there are remote access vulnerabilities in that as I was putting this together over the past couple of weeks I noticed one of my friends in industry here was also doing some research and wireless heart and put out a tweet about you know building this sniffer and found hard-coded backdoor in the vendors non upgradable gateway so there there are challenges for wireless in the industrial side and again this is one that I find that people one either

they're aware of wireless applications in the control system side and but believe that its proprietary enough that there aren't vulnerabilities in it or or they're just not aware that there's that much Wireless happening in these environments and part of that depends on kind of which industry maybe you spent the most time in for me I you know kind of grew up in the electric utility world spent the first part of my career in electric utility world and there were certain applications where you know I would just be soap right well why would you even think about you know using wireless here if you've got some other option but the game change right the you know between the massive distribution

and the you know smart metering you know advanced metering infrastructure 's you know the game changed and you needed those kind of the reach that wireless communication provides and what we're not covering by any means a comprehensive list there are a lot of different industrial wireless protocols and different you know stories to go along with us but that's a couple examples the wait that's not all right it's not just industrial wireless that exists in these systems there's there's also you know Wi-Fi 802 you know Wi-Fi out in these environments and I would say level just take an example by examples so in in the manufacturing world we've come across this several times where you've got you

know an application for barcode scanners you know they need to roam around the the plant to do something and they're communicating over Wi-Fi because it was the technology that work at the time that this system was implemented and hey they might have even turned on security in 2005 or whenever it was that it was implemented and that's actually fairly recent history and in control system life cycle sometimes you know but it's just using web and in the the challenge we've run into here is on the face of it it's like okay well these are scanners they don't really do much but what they do what that does is it provides an entry point for someone sitting outside

of that plant to you know break into you know to break web was a shocker there that would be possible right I mean but so but it provides an entry point into the network and then it becomes a question of what is it connected to you know beyond just that barcode scanning application okay well it's connected to this broader manufacturing network which is also connected to the business network and by the way the egress controls don't really prevent you from getting out of the manufacturing world back into the business network so it becomes another entry point and maybe you know I tend to focus a lot on protecting the critical infrastructure protecting the operations the production

process in the industrial control system but sometimes the industrial control system also is the the entry point that could become an attack back into the business environment as well and that's exactly what had happened in this this manufacturing example this was an interesting one there was a guest wireless network for an electric utility client and didn't realize just due to some the way that this thing was configured that it was actually possible to enumerate all of their control system IP space you know not exploit level stuff where I'm gaining access but I could map out all of the control system space from the guest wireless that's probably not ideal not you know not not an immediate thing like someone's gonna

exploit this today but but not really what you want to have in place simple configuration fix they are fortunately and then another fun one you know I ran across this for someone and you gotta hand it to these guys right for you know building a solution to their problem right so they bought a couple of Linksys routers through DD word on I made a little point-to-point environment so they could extend the control system in this power generation environment so many challenges there right it's like okay how do you know I understand that in that environment you got to just make things work sometimes but this created a vulnerability again that you know was allowing access into a sensitive area of

the network probably a better way that we can do that I will say you know so again I'm giving you a lot of stories about you know things that are way less than ideal but I don't want to paint such a negative picture because I will say that an electric utility world especially a lot of organizations have gotten better about you know change control and things that would prevent something like this you know point-to-point wireless thing from happening not everywhere not all but some there is hope alright so following the format again here this is you know sort of the summary slide I will say so on the wireless side it's actually a little bit

easier to test for that just because you can do it passively to get an idea of what's you know at a particular site and you know there's a lot of configuration inspection that goes there a mission showed in here just also because you know we look at that wireless controller that was sitting out there so that's certainly you know something that's of interest and someone else had that up on slides earlier to you if you're not familiar with that this is another one that that can be difficult when it comes time for remediation like how do you actually strengthen these things make it you know make it better in in this case you know it sometimes it

does like let's take the manufacturing example either you're gonna segment that off which you know would be one option or you're gonna see if that technology is upgradable unfortunately sometimes through the age or you know just the way that those systems are we're designed from the beginning they're not upgradable and it makes it more difficult to secure so you do some things to try to manage it you know segmentation fortunately you know not this is not specific to Wireless but on the embedded device security side of things the industry is moving forward a lot of vendors now are offering you know stronger better defendable if that's a proper grammar embedded devices you know out it out in the control system space

we're not not perfect we're not you know all the way there yet but it is getting better and as things you know as we move forward in this world of industrial IOT and a lot more connected things there are you know challenges around that that we'll continue to have to solve all right this one is not quite as clear as database and Wireless because it just looks like a laptop right any guesses as to what I'm going to talk about here what was that that would be a good one we're not too far that's close but I heard somebody else unsecured terminal yeah that's that kind of fits into that what I'm going to talk about under this little icon here

is remote access mechanisms and it could kind of fall under you know both of those those sort of things you know there it's not too hard to imagine why someone would need or want to to have remote access into the industrial control system environment you know there just a number of different scenarios you can think of not the least of which is you know having a need for vendor support right so I can't have always the vendor representative right here ready to go setting out the terminal so I need to find a way to provide that access for them and so people do find a way to get from here to there and sometimes very interesting in

clever ways and this first example is it's just an unfortunate combination of configuration problems the concept of auto log-in our auto logon in the control system environments that while you know if you come from an IT background and you see that you'd be like why win the world would you ever have a machine that you know was actually configured to auto log on and any kind of you know enterprise of our operational environment there are some times where you want that you know maybe the application depends on that box starting up and this is you know this is this is the way that the vendor has chosen to to implement that but then you run into situations where it's like okay

you've got this really bad combination of auto log-in of administrator with a blank password and RDP is enabled you know not a great situation the other interesting one here I ran into the situation where the IP KVM for all of the control system servers were was connected to the IT network again we're talking about attack surface here so now I have now I'm able to access from the IT network the KVM switch that's connected to the control system computers and multiple challenges with that KVM controller including default credentials remotely exploitable vulnerabilities they're just a number of different things that were creating a problem there that again created an attack surface that if you weren't looking for this stuff on the IT

side of the network or you weren't crawling around in the server room for that industrial control system application you might not ever know and I guess the other thing to add to that list or if you weren't actively talking to the people that were using that you know control system engineers or administrators for that environment so that's another example and then all of the remote access software everything that you can that you know that you can think of has you know has been used in these environments for exactly the the reason that we talked about some of the you know vendor support issues internal remote support you know I can VPN into my corporate environment now I want to

be able to jump to the industrial control system environment so I don't have to drive in to do something and there are valid cases you know that the answer can't be you know no you can't do that because I just from experience that won't and in most environments that won't work very long unless you're you know in something nuclear power and it has sort of a special set of circumstances around it but in most environments you need to have some kind of emergency remote access capability available and if you're not proactive about setting that up in a secure way guess what somebody will find a way to do it and work around you so so what do

you do about that so obviously there's a lot of different testing methods and and testing opportunities that you can do to identify you know these kinds of remote access solutions we'll call them but then when it comes time to to fixing it again I think the best thing here is to be proactive and have something that's actually set up and approved to use for that remote access ahead of time and I'm a big fan of having some things that are set up with like a connect and disconnect mechanism and I've seen this implemented in some pretty creative ways where even a control system operator maybe can toggle a switch that that turns on access so maybe it turns on at

one of them I did one time was it turned on a little media converter that then enabled access from the business Network temporarily you know over a restricted set of ports in a sort of an emergency remote access feature that could be toggled on and off there were you know you have some other controls around that like reports that come in if if that's left open for you no longer than a certain threshold so that this is you know the the remote access attack surface again being proactive I think about finding ways to just to configure this ahead of time is probably the best bet here but then also you know regular assessment and testing for this sort of

thing is critical as well all right so we're getting close to the end here a few conclusions that I want to leave you with again I really wanted to just share some more stories and maybe get you thinking a little bit differently about where some of the exposure might be coming from from the industrial control system side of things really that thorough assessment is is important for understanding the posture of your ICS and and you can't look at it even just in isolation for some of the reasons we saw there's there's this concept of inherited risk and inherited vulnerabilities you know to all of the networks that the ICS is connected to and that could be your business

networking environment it could be third party providers that have a connection it could be business partners you know lots of different you know examples of things where you've got interconnection and you really need to understand if that if that connection is also creating risk and in mind so that's first thing thorough assessment the second thing I guess is a takeaway here is some of these small details like whether or not you check for default passwords and disabled those in your Oracle database that's buried in your control system application makes a big difference I can have you know an a fantastic you know cybersecurity program that puts me at the top of the maturity level across all of the NIST cybersecurity

framework categories and I can feel really good about that but some of the details of understanding down to you know that level of you know database accounts and what you know what what is the attack servers they're creating those things become really important and then lastly and you've heard me echo this probably several times throughout we are making this better this situation is you know is not hopeless and I don't want to paint that picture we have real challenges in all these networks and there are challenges that are specific to manufacturing and specific to electric and specific to oil and gas and even you know different areas within those sort of industry verticals but but

there are in a lot of places there are things that are getting better we're you know we're doing a lot of these assessments where we're discovering these things I'm hoping that by passing on some of these things that we've seen to you it gives you some inspiration for what you know how you can check how you can make things better in your own environments if you happen to have any responsibility for for these industrial control system networks so that's really it for me I think we do have a little bit of time here for questions and discussion I didn't want to make sure I think besides Oklahoma again I live over in Missouri now but it was great to come

back to to Oklahoma here and be here for the day any questions out there

has anyone still awake all right well thank you guys very much it's been a pleasure being here and I hope to talk to you some more out in the hall and if you if you think of any other question you know any question for me I'll be around Thanks [Applause]