Coordinated Disclosure of ICS Products: Who's Got Time for That?

all right welcome everybody welcome - good afternoon welcome - I am the cavalry this is coordinated disclosure of ICS products who's got time for that and our speaker is J Angus before we get started just want to do a couple of quick announcements number one we'd like to thank our sponsors especially our inner circle sponsors critical stack and Vala mail and our stellar sponsors Amazon blackberry the NSA silence Microsoft Robin Hood secure code warrior and paranoids it's their support along with our other sponsors known as a volunteers that make this event possible all these talks are being recorded and streamed online so if you can please silence their cell phones and that's it I'll hand it over to our

speaker thank you alright so I've de Angus with the organization formerly known as ics-cert also formally known as the in kick and currently known as the cyber struck cyber security infrastructure security agency but they were twice the security now but we are still doing vulnerability disclosure and kind of talk about that a little bit here so quick rundown of me there's been a little bit of confusion but yes that's me I've been a federal employee for about 15 years now and I've been doing cyber security for the most of the currently I'm the Federal League for the IPS vulnerability disclosure program within sysm and if we get into that a little more details later but I'll be here for

the rest of be sides and also I'm going to be at Def Con I'm going to be in the aviation village so I really often came by and I chatted me up a little bit there's some cool things going in going on over there also another way to encourage folks to come see me if you have ever disclosed an ICS vulnerability through the in kick or ics-cert I have in kick coin if we don't have Sousa coined yet but I have coins for you because it's the least I can do to appreciate the good work that you've done coordinating disclosure through us and if all else fails and you just want to stop by and chat in general give you

a couple of cues of yes this is me and I'll tell you about my Jeep and I have a dog that actually used to work for DHS so that's a good story there but that's fun stuff all right so so what is coordinated disclosure I I just went straight to the cert Guide to try to you know wrangle up a few key things that I think that's important to this the number one item is just reduce harm right so working with industrial control systems and recognizing the function that they have in our day-to-day lives this is should always be the most important thing when you're really okay

alright that better all right okay so when working with the industrial control systems we want to reduce harm by reducing risk here right so this should always be the most important element of your disclosure with an ICS L I I do recognize that I cannot personally reduced risk of vulnerabilities but I can give asset owners the the information about risk we have a pretty wide audience that we can advertise this to can't force them to read it right but I want to make it public knowledge and out there for them to make decision point on there most importantly during this process I want to presume benevolence right I don't care how or where you found this vulnerability I

want you to know that my goal is not to call the feds on anybody I want you to know that I believe you're bringing this information to light with the best of intentions you may get pushed back here and there but you want the vendor to understand the risk and to take action to fix the vulnerability for their customer base right if you have something and you feel like you can't trust me just get a throwaway email address come on just we can get this information to be public even it through an anonymous beans I still want to talk to you and I still want to convey risk to the the larger crowd and to the vendors

both large and small I don't really think that this is the crowd that but um stop calling your lawyers you know that they don't scare me and you shouldn't be intimidating researchers that are out there that are doing this work on behalf of you really whether you like it or not it the vendors know this and but and don't seem to stop them right DHS would that has excellent legal counsel and when you bring as a vendor bring your lawyer to the table I'm obligated to bring my lawyers to the table so let's just let's just not go there let's not deal with that right I do caveat that with if I have a suspicion that there's some kind of

criminal activity I get it to provide that took the law enforcement at some point but that's that's not my primary function I want to publicly disclose zero days that impact industrial control systems to as wide an audience as possible so one of the key tenants that can be difficult for us to manage at times is avoiding surprises as a researcher you know that we're gonna have to talk to the vendor at some point they're only those are the only one that can fix this right no matter how terrible you have proven their product to be we want to put this information in their hand before it's public knowledge and things can get contentious at times

and that's that's okay but that does not mean that you should go silent and then go out and go public and just kind of you know run these organizations these vendors through the for the mud so yeah I and I take a lot of heartburn with folks when they leak this information in advance so that doesn't help the process right it doesn't help the vendor when they're doing this again and again later on because if you can find one vulnerability and a product I guarantee a lot more in there so think think about yourself understand what your goals are but don't don't throw the rest of the research community under the bus by by leaking the

information in advance so what's what's not up here I summed up the Fudd's here we have what's not here is fear vulnerabilities can be scary to the uninitiated you know don't get mad if you get a low cbss score you know not all vulnerabilities are scored that same but it doesn't make it less valuable in my eyes I want to get you a CV I want to get it a public knowledge but just don't overstate the impact simply so that you can get more press coverage that's not helping anybody I want you to be a reliable voice for the ICS community for for all of vulnerability disclosure don't don't call us uncertainty this is hard to work around at times both the

vendors and the researchers kind of love to play this card but but you know well don't write it this is a zero day we've not seen this before to a certain extent and so nobody knew and here we are but if there are more vulnerabilities to be don't go find them don't don't make these broad strokes of you know this could be and should might be to the facts right so doubt if you question them vendors intention when you are disclosing to them don't think they don't think they're taking you seriously and you start giving them grief they're on them shade right don't expect them to suddenly be nice right these are human beings to let the facts of the

vulnerability speak for itself and stick to the facts I can't say that enough if you can show potential of cascading effects for these ICS vulnerabilities do it you know that's great we'll take you serious either way and I go back and forth on this last point but the self-promotion aspect of it I always want to attribute a researcher to a vulnerability that's that that's the only currency I have is my good name or the the good name of DHS so I want to do that it doesn't matter the scenario even if you've done everything wrong for the disclosure process we're still going to attribute the discovery of the vulnerability to you just don't make it

so hard for us to do that if you can't follow there we have at times not attributed disclosure to a researcher when they haven't followed the process very well and have been difficult to work with but that's that's very rare and it's not something I take a lot of pride in doing because it is the your good work that you're doing doing on behalf of these vendors whether they like it or not but um yeah so so some things to think about is what is as a researcher what is your disclosure policy you get to make that call right so from my perspective no to disclosures are really ever the same and my boss likes to

describe the disclosure process as everybody holds guns on each other and nobody publishes till everybody publishes right and sometimes folks jump the gun on on us there or kind of in our our standoff so what kind of helps prevent folks from jumping the gun in this process is your opportunity to signal what your intentions are going into a disclosure if you have a policy that you can communicate upfront with the vendor they kind of get a sense of what your goal is and how serious you are and from the ICS perspective things are definitely maturing quite a bit I know a lot of software companies have products certs but these are becoming more common in the ICS realm this wasn't

necessarily the story in recent history and and you have to recognize these all of these P certs operate at different levels within the organization so that can be a challenge at times when you show up with a vulnerability and you're contacting helpdesk and they're trying to convey it to the P cert that's completely in the wrong part of the organization and they're conveying it up to the siz o who may not actually be a part of the executive suites so there's there's challenges there and you got to know your target if you're doing a lot of research leading up to that and these scare things that you should know so first is our policy within the ICS realm

for public disclosure is that we will release after 45 days and that's not exactly what it is right we release 45 days from the last time well we had contact would say the vendor so if you're a vendor it moves you in this process to be communicating on a regular basis and really showing what your the work and the process that you're going through to remediate these findings it is I don't I don't take a lot of pride in making a publication that has no statement from the vendor and we send it out to probably about 30,000 subscribers and and no one has any kind of remediation to a vulnerability but suddenly after that happens a vendor

comes to the table shortly thereafter and is very interested in trying to give us an effective way to remediate the findings it's just kind of funny how that works sometimes so I kind of tried to pull some of our ICS vendors that we work with to see what their disclosure policies look like and it just varies for every vendor there was no consistency there it's a 60 to 90 day disclosure window for a lot of them the more mature vendors in the IT room like kind of like an IT room are focused publicly disclosing once there's a patch many recognize this isn't always the case and will publicly disclose a vulnerability and reference compensating controls that's always scary when a

vendor does that for an ICS system but if you're doing your ICS networks correctly they should be segmented off in advance to you know prevent the risk from some of these vulnerabilities and sometimes is a research you can't always expect a patch to be ready to go there is sometimes we encounter a end of life products so that's like a huge issue in the ICS room because these systems will be put in place for 20 and 30 years right we they're just not supported by the manufacturer anymore so what do we do we're still naming given CVS to these vulnerabilities we're helping helping and hoping that the assignment of these CVS to these at-risk systems even though their end of

life can still maybe influence an asset owner how important it is for them to go go out and try to replace these systems so and then for researchers policies that always comes back to you right this is where I kind of hope you take into consideration some of the factors that went into your discovery right how long did it even take you to discover the vulnerability would it be unfair for you to discover a vulnerability and say it took you a year to workup and discover it to expect a vendor to have it fixed in a few weeks I don't know it just depends on the vulnerability right and you probably by this point have some

pretty good insight into the product I mean how hard is it do you think to remediate the vulnerability and and what are the impacts of your findings is this a medical device is this something that could have a life safety risk and some of the next couple slides I kind of present kind of give you some ideas of how you should pick what your policy is for the for the disclosure process all right so this is the Purdue model here and I was curious I this is pretty common knowledge to me but who in here is familiar with the Purdue model okay okay awesome awesome so this is kind of something that's real common in the operational technology

area the ICS realm it's a reference model that's been through a few iterations it started out I think it came to fruition in the 90s it's also known as is a 95 but essentially this model presents the ideal interconnection of discrete enterprise components between the IT and the ICS network so we start at the top we've got our basic kind of boundary systems you know our usual IT stuff that we're all familiar with on the level for these are our more enterprise IT services that we deal with kind of as your your desktop your day to day work and once we kind of moved down to level 3 we move away from a typical CIA triad and it kind of gets flipped

around and our availability becomes a priority as we enter into this kind of cross a DMZ into the OT environment and so level 3 is where we start seeing operational components of the environment some of our you know historians hm eyes these are just traditional high level systems that we have in the OT world and down to we move on to level 2 and this is where we start seeing proprietary ICS type protocols here you know Modbus DNP and and more HMIS that are directly connected to systems down to level one we got a couple things going on here this is you know what we see the hardware that we think of is that the PLC's and you know

fill devices you know the fun stuff that you can go buy on ebay that's you know 50 around 20 years old for 500 bucks that was originally $100,000 hasn't been making advanced polymers until they dr mode the product right so and this is another thing that's real key here is where we start finding our safety systems these are of what I kind of refer to as the the control systems control system this is what shuts down a control system that has gone out of control that has gone out of bounds and risk to up to a life safety issue and these these are kind of strange systems that a lot of times get put in place they're certified

and no one touches them they get left alone they're usually pretty good about them being air-gapped but we see very deep maintenance cycles for these type of systems just because of the amount of effort and the risk that comes with a maintenance of them so here we are again so you kind of want to be a researcher in the ICS field now and I think that's awesome and I definitely wanting to hear from everyone you know their opinions on this but kind of be prepared for the long haul I don't want to cover this these models right here so you got the Purdue model there on the right and on the left is that the DHS

defense-in-depth model that you should ideally be modeling in your ot environments but I bet no one does any essentially this network topology is the Purdue model laying laid over on your your network and as you kind of guess the deeper you go into the network we get more critical applications and pieces of hardware so as a researcher or an asset owner even becoming a little more it becomes a little more difficult to to work with these devices as you descend these models the the level of expertise that's needed increases significantly and they had the potential expense for these systems go up as well so that should kind of be being factored into your disclosure process and I tried

looking back in at some of our FY 18 data and we had a you know we have that 45-day disclosure process policy but we averaged about 190 days disclosure and I to try to strip out some of the outliers I did the median on that and we were closer to a hundred and twenty days so I'm guessing from an IT perspective that disclosure is probably a lifetime but in the operational technology realm it seems pretty reasonable I know some folks would definitely argue me on that one but you know and as I mentioned before we we did find that most of our vendors have about a 60 day disclosure policy and want to remediate as quickly

as possible but still you know that's 120 days on average to get this get these findings straightened out and something to keep in mind when you're looking for system to focus your research on is whether or not it's a regulated device we do see devices in the ICS realm that are regulated specifically a great example is the Schneider Electric system the the sis system that was impacted by the Hat man Tritan tries to smell where there was actually a delay and getting the patch out because of the the need to have it recertified by the Safety Board there and so I kind of wanted to see if I could correlate if the disclosure process was impacted by where the system

resides in the purdue model and it wasn't something that we're explicitly tracking but I did find a subset that I was able to identify that existed in three discreet areas all in one package and it was medical devices right and the good news is is the medical devices on average followed the same trend as the ICS realm on the median though I found it was about five to ten days longer for the disclosure process but that was because we allow the FDA in this distinguished disclosures to review the the risk because we're not doctors at DHS we give them the opportunity to provide the input on the patient safety impact so I think I'm running out of

time pretty quick here let me wrap this up so if you want to start researching ICS systems ginger ale or your ginger beer mix them together but you can't have your day thank you thank you thank you there was a rider in there and I went for some for diet ginger beer but apparently that was that was too much all right all right so awesome so you want to be an ICS researcher that's great and you think you found something that may be significant even better please bring it to me I want I want to get you a CDE I want to get this information in front of vendors but ultimately I want to make this

information available to asset owners because those are the individuals that are responsible for remediating these findings this is whole process is is no good if the the asset owner doesn't find out about it in the in to fix these issues so I think I think I'm out of time but a quick story I've been on three assessments with DHS and they were all ICS focused they round up pretty easy there's one it was terrible it was scary the second one was what was that okay was I [Music] oh man enjoy thank you so so this one acetone we went to they were doing great they were doing everything right they were checking logs they had their

network segment it but we start to walk the plant and we find a PLC box unlocked hmm not great no big deal we find the program key in the program mode in the box that's bad that's bad but it happens a lot and we were like well you know this is probably gonna be a finding and they're like okay we appreciate it but we haven't done maintenance in five years on this anyway so we'll get around to it eventually so that's to the tune of $30,000 an hour for these guys so so this maintenance and patching effort is a significant process for these asset owners and individuals so you try to factor that into your disclosure process and I'm

getting the no really stop now message so so I guess I can have take questions but no really stuff now we can continue this conversation over at the Platinum there we're having a coordinated disclosure discussion over there my boss is there and if you tell her how terrible I am at this maybe she won't make me do it again so thank you [Applause]