BSidesSF 2012 - SCADA Security: Why is it so hard? (Amol Sarwate)

so first of all I would like to thank all of you for being at this awesome be site could really a community-driven event and it's really my it's really my pleasure to be here I know all of you guys are busy so I'll try to make the most of the next hour that I have that I have here so let's get started oh well by registering today for this event and when I got this badge I was talking with a couple of folks about their expectation from b-sides and not just from b-sides but from my talk some folks were really excited to hear that there will be like packet snips and Wireshark dumps in the presentation while some

folks said well we could do that more in the privacy of their offices that cubes their garages and they really want to focus more on what's happening in the bigger picture on the industry and what are some of the trends and so on and so forth so what I'll try to do is keep sort of a balance between both of these so that both the audience are if you may entertain a bit so my name is mo and I work for Wallace in the vulnerability management group this Scylla thing started for me about two five years ago when one day all of a sudden my boss comes to my cube and says hey do we have

signatures to identify a scale of our abilities so to give you a little bit of a background what my team does is we write signatures that identify vulnerabilities misconfigurations possible threats any sort of loophole by witch and hacker or an attacker could get into your network and compromised your network and so on and so forth and remember this is five years ago I knew what SCADA was a little bit at the time but was in no position to start writing vulnerability signatures for these so I said well okay we we don't know exactly what SCADA is at this point but we'll start investigating for the final thing which is to write signatures for those so as I started

talking with a lot of people in the industry started attending various SCADA conferences what hit me was that what I refer to as SCADA may not be or may be entirely different than what someone else refers to a SCADA some people and and there are a lot of differences some people say okay this is not scaler system this is really industrial control system this is really distributed control system so on and so forth and there are differences but for the purpose of this presentation I would sort of use the word SCADA loosely and interchangeably for all the existence so I said fine let's let's start investigating what if I may ask you a question what what

what are some of the pictures that come to your mind when when I say SCADA and these were some of the pictures that came to my mind at that time when the word SCADA was said electrical power lines pipelines carrying either water natural gas petroleum so on and so forth and these areas these are all examples of SCADA systems the sort of rule of thumb that I have is if you sort of hug a system and if you end up hugging a country or half of a continent then yeah that's that's probably a SCADA system and the top three systems are kind of the examples of those the systems on the bottom are more or less can refer to an

industrial control systems where they are big they are huge but they are still somewhat confined to a plant big plant or a set of plants so for example food industry donut manufacturing automobile industry car manufacturing so on and so forth so I was wondering why was this interest in SCADA I started investigating and there have been some accidents in this kada space the links that I have here they are pointing to some of the root cause analysis done by the US government on some of the liquid pipeline failures some of the power failures and other accidents and again at the end of this presentation I'll point you to a link you can find this so you don't have to

write down anything so it's a plane okay there have been accidents but that's that was not a pressing reason enough to start writing vulnerability signatures for SCADA systems so I started looking into more there have been cases of vandalism and the example that I have here is from Portland some of the Randalls did is they destroyed some of the rubber gaskets on electrical electrical poles did cause an outage or anything but still the electrical company had to send down crew place the gasket spewed out power and it caused them a little bit of a headache a lot a lot of money to do that it's fine there have been cases of vandalism which which which may justify writing vulnerability

signatures and scanning for those there had been insider threats and the example that I have here is from from a disgruntled employee this is an example from Australia where an employee was fired from water treatment for a sewage plant and the very next day what he does is because of his knowledge of the system it goes in and opens up the sewage system into rivers and parks so I said okay that's that's something pretty serious and that is one of the reasons that one would try to secure their SCADA systems but what I failed to notice at that time was the threads or the advanced persistent threats that we are seeing today these are threads from

states to other states or from terrorist groups and the link that I have here is for the dooku worm which which has been said to be a son of Stuxnet and the folks at semantic your thing that people who created Stuxnet also created Dooku it doesn't really do anything for now it just goes and finds information but these are some of the things that we could see in the future so I said like you find there have been credible threats for securing those how do I go about understanding the system writing signatures for these what we usually do when we have such a task is we get the database the device or any system in

concern in our labs we bang on it till it breaks we find wailmer abilities and then we write signatures or if there are no more abilities we write signatures for those but if you recall all the pictures that I showed you it was well impossible to get all those systems in our lab to start to start off pen testing them or banging on them to find vulnerabilities so what we did so that our engineers could understand these SCADA systems in a much better way was created a sort of a framework that could be applied to almost any industrial control system or SCADA systems and then they can be studied so this is what we came up with what do you see there on

the left hand side is the field so these are the components that would be maybe thousands of miles away in the field and on the right hand side or is your control center where you control this system so if you--if we take a more detailed look at what it looks like is on the left-hand side our data acquisition components so these are components like sensors meters feel devices and what they essentially do is convert parameters like for example here I have a photo sensor which converts light into an analog signals or temperature or pressure into analog signals these components may be would be mounted on for example a gas pipeline or an electrical poles or whatever wherever

we need to sense those those attributes this data acquisition component has sort of a two-way communication with the next component in our system is the data conversion component these are this component basically consists of PLC's IEDs or RT use product programmable logic controllers intelligent electronic devices remote terminal units and they convert your analog or discrete measures to digital information and examples from starting from left to right what I have is PLC ID and RT U and B just to give you the how big these components are physically I have another picture of a man with a PLC in his hand so okay what we have is we have on the left hand side is IO sensors meters which acquire these

characteristics then we have this data conversion components and then these this converted data is passed to here to the SCADA master using one of the many scale of communication protocols now this communication could be over radio over Ethernet over various channels and using any of the SCADA protocols that some of the examples of which I have mentioned here they say that there are hundreds of speed of protocols proprietary non proprietary open source close source I haven't been able to count up to hundred but there are really a lot of SCADA protocols out there and we'll see in subsequent slides a little bit more in depth about these protocols so fine data is acquired it's converted

using one of the communication means it gets to this presentation and control layer which is essentially where a human being or operator continuously monitor alarms and acts on it what you see on the left hand side is a pretty traditional control center where you have certain monitors or human-machine interfaces where you see what's happening in the plant and the operator basically monitors usually there are 24 by 7 operators during these alarms and reacting on those on the right-hand side what you see is a little bit more modern HMI actually these diagrams are from the HMI designers but a lot of modern HMI systems do look like the ones on the right so this is sort of a framework

that we came up with to classify these systems so that we essentially we ourselves could understand weeds kaida and mystical control systems so we said fine okay this is what sort of most of them look like where are the threads so threads could be at the i/o and remote level so between your sensors and PLC's but a lot of these threads require physical access so a thread could be that you go and change a sensor or a meter or tamper with them in the field so the example of vandalism that we saw could be categorized here where someone just goes and destroy it something which requires physical axis but field equipment generally does not contain

process knowledge so if if you as an attacker just go there without having any background about the system what how it works and just remove something mandalite something you generally would not know what would be the effect of this because field field information it's something like valve 16 or breaker 9b unless you have a full picture you don't know what that does and without process knowledge this leads to basically nuisance disruption like vandalism example that we saw so we said okay there are some good credible threats here but at least at that time and even today I don't think or it is my knowledge there have not been too many threads of this source so we said okay

let's try to focus or attention more on communication threads and communication basically imply you manipulate FTP directly and which would change the HMI output so what comes from this side you change that so the operator there at the HMI station gets to see something different than what is really happening in the field and there were a lot of protocols a lot of scale of protocols so we said okay this is something that typically as computer science engineers as signature writers we have been more accustomed to these protocols so we decided to take in-depth look in some of the SCADA protocols we looked at Modbus which is a pretty old old-style protocol pretty old protocol but nevertheless

used in a lot of control systems it's a very simple client-server protocol and all of you could have you know no problem in basically writing anything for more business if you have if you know telnet if you know if you know any client-server protocol it's exactly not exactly but it's it's sort of the same what you have is you have a model this client and more the server there is a request that comes from the Modbus client goes to the Marcus server and then there is a response that comes back the only difference that you have is the Modbus client is the master while the Modbus server is the slave so that's a little bit different than what we have in our

typical tcp/ip sort of protocols so Modbus was not really designed initially to run on tcp/ip and it but it was later ported to tcp/ip so mark bus consists of its Adu or application data unit and through PDU which is the program data unit and the way it started running over time on tcp/ip was just by putting the entire protocol inside your TCP and IP header so now I think it becomes more clear for all folks here who are very well-versed with tcp/ip is you just put the Modbus packet inside your TCP packet which is inside your IP packet and just send it over so let's take a little bit detail look removing all the tcp/ip

thing from the Marcus protocol what you have is a Modbus header which has a transaction ID protocol ID length and unit ID so transaction ID is basically very simple it's a transaction so for example if there is a photo sensor there and I'm asking the photo sensor hey what is how is the light there or if there is a temperature sensor there what is the temperature I say this is I or basically the client says that this is transaction ID hundred give me that give me the answer what is the temperature right now and then the response that I get is for the transaction ID 100 by which I know that oh I had requested temperature for

transaction ID hundred and this is the response so it's a simple transaction ID protocol ID is fixed for model length is basically the length of your packet and there is a one bite unit ID now this is let me go back a little bit for this is so what you have is you have a PLC and there could be one or more PLC's or one or more sensors connected to that PLC so when I request a temperature there by specifying a unit ID i specify that okay this is the unit that I want the temperature for or pressure for that is specified in the unit ID so what what what which destination PLC's or sensors are my

querying for so pretty simple so far if you look at function codes they are also pretty simple they are function codes like read discrete input write discrete input I want to read maybe input off or I want to read temperature for these certain coils or I want to read temperature or pressure or whatever that is connected to for ten of them for hundreds of them and there is a opcode associated with every request so pretty pretty simple here as well and data finally is the data as per the opcode which is in the function header so pretty pretty clear so far of everyone with me you know okay I see few heads nodding so well if you can easily write

a Perl client to create a Modbus request as as we saw you have a transaction ID here I've put it as 256 protocol ID is always 0 for Modbus you basically create the length for the entire packet you put the unit ID which I've put as 1 here you put all the data as per your function code which is I have specified 3 hit 3 here which is basically read a lot of data you create a socket and you send away your request pretty simple if you copy paste this example in in Perl and send and run it against a Modbus device it will send a more bus request and the response that so yeah and when you send

that request you will see something like this in your Wireshark now Wireshark and almost all sniffers already have very good support for Modbus we didn't have we didn't have to write any packet decoders for these and we were sort of fortunate in that request in that sense and what you see here is your transaction ID your protocol identifier zero your length your unit identifier and what you want to do is I want to read multiple registers and I want to read five registers so let's say if that PLC was connected to a temperature sensor I want to read like five temperature readings and the response that you would get from such a request in this case what I

received was okay this was the transaction ID that I had sent and it copied the same transaction ID in its response mmm length was 13 and it gave me a byte count of 10 so I asked for five temperature readings and it gave me five temperature readings two bytes each so I got 10 bytes back again pretty simple I think everyone is following with me till here but the main question that we have to ask using this simple example of a model with client-server communication is that what this mod was provide this is the this is a sort of a triangle that every protocol or every security protocol is measured against availability confidentiality integrity by availability is used it's usually

authentication and authorization authentication is basically are you are you really who you claim you who you are and authorization is are you really authorized to do the action that you are asking me to do and we saw in the previous example there we I didn't like this the court didn't really have or didn't really provide any authentication or authorization in it it just said that okay give me this reading it didn't have to say Who I am it didn't have to say I am authorized to do this because of so on and so it just said give me this reading and you got a response okay these are the readings so pretty much there is no confidentiality here

there is again no pretty much no no integrity here I could be a man-in-the-middle change anything in the middle and instead of a right request I could change it to a different type of right request or a different type of read request and I would get get the results back so pretty much there is no confidentiality or integrity so we said okay this itself is a vulnerability so what do we do we try to find Modbus installations at our mmm at at our customers network put that in our signatures and sort of notify them that hey you are running Modbus which is fine but make sure that all your other security measures are there around this

protocol since inherently this protocol is not secured and again I'm not here to bash morbus or something it's an excellent protocol it just was not designed for security so we said okay let's take a little bit of a more recent protocol and we chose dnp3 which was designed sort of from the ground sub for the electrical system and try to see how dnp3 works so dnp3 and Modbus fundamentally are the same which is basically you ask for certain readings it gives you a certain readings you write to it or you sort of change certain settings and those settings are changed so fundamentally it's the same it has although a lot of differences with Modbus differences like the np3

it's strong data types which Modbus doesn't have or Modbus it's strictly polling so the you have to always constantly poll give me the temperature give me the temperature give me the pressure you always have to pull but in dnp3 you have the ability to say that okay give me the temperature only if it changes so it reduces network traffic also there is time stamping so in Modbus as you saw there was no time stamp so the client sends the request and receives a response now I have to sort of the client has to assume that okay I sent a request at this time and so the response that I got must be for the at that time but meanwhile the

temperature or pressure could have changed so that was not really acceptable for the electrical system so there is a concept of time stamping there that you say okay this is the time stamp on my request and when you get data you get a time stamp saying that okay this was the parameter at this time stamp so it's it's a little bit it's a little modern protocol then then mod this so Oh SCADA sense so before we're going a little bit into dnp3 this is the output of the pro request and off the open source tool that we are updating today and what it can do is it can scan your network or scan other people's network

for 404 for the presence of Modbus protocols where multiple servers are present and can sort of brute force the morbus IDs and now I'll go into more detail about this little tool or utility that we are releasing in in a bit so coming back to dnp3 dnp3 as i mentioned it's a layered protocol it has a link layer a pseudo transport layer an application layer it's a little complicated than Modbus and we won't go into a lot of details into dnp3 but take a quick just a quick overview so this is the dnp3 application layer headers as you could see it has a similar sort of thing which has request headers response headers the request headers are

basically read write select so logically it's doing the same thing as Modbus is getting parameters getting from from the sensors this is the transport layer header and it sort of tries to provide the same functionality as the TCP transport layer not not quite but similar and this is the link layer where you have the start of the packet which is always this 0 X 0 5 6 for the length of the packet control bits source address destination address CRC whoa now you have we have CRC and various different function codes so what we did was okay let's create another Perl example of dnp3 link layer frame and send it away so this is exactly as we

saw in the header is we we created a packet with with the start byte with length with control with source destination the CRC and well created a simple tcp/ip socket and send it over the request looks something like this on the wire you can see all and again even in this case most of the packet sniffers are pretty good at dnp3 and we didn't have to write any custom plugins for Wireshark or ethereal so we were really lucky in this case you see all the you see the request that it's sent basically which is a request for link status I'm trying to find what is the status of the link and with various parameters and I

got a response from dnp3 saying that okay these are the status bits of the link and well one is basically on and zero is basically off so I sent a request I received the response same question again that we asked with Modbus what does dnp3 provide did it provide confidentiality authentication authorization not not really integrity maybe a little bit because it had some CRC but again not really any confidentiality so we created again signatures to identify dnp3 protocol and say that ok if you are using this protocol then be careful make sure all your other security measures are in place if you are using this protocol but well not ok so and this is an example of

the open source tool again that we are really that we are using running in the - T or in the dnp3 more of which I'll come back later again but all is not lost in these two protocols soon in 2007 the dnp3 the next version of the dnp3 was released which had a lot of good things including hashing including key management functions including a lot of good things that are needed for twos basically satisfied this pyramid so if you are using the mp3 in your installation make sure if you have the secure version or the 2007 implementation of the mp3 and it's configured to use these different new security features that have been added so we we looked at a lot of

communication threats threats and protocols and the common theme that we had was basically a lot of these protocols are we're not designed really to run under tcp/ip now they do and they're under tcp/ip but they have been not really designed to with security in mind so you have to have a lot of other controls present if you are using these protocols coming to scale our masters as I said threads on SCADA master these are the places where this is a place where all the control happens so usually what happens is when you walk into a SCADA master room you have some sort of a good authentication mechanism to go inside the room physical access there are as I

showed you in a couple of pictures ago a lot of flat screens or some screens that give you status and operators continuously monitoring these alarms and basically ready to act on them the number one thread that we saw for SCADA masters was that the control system network or the this SCADA master network was inadvertently connected to your corporate network or sometimes even to the Internet now it used to be that these networks were completely different from your corporate network so when you go in to work every day you go log in you check your email you do your certain tasks and that network was completely different than the SCADA master and on paper in many

organizations is this network is different than your SCADA master network but what we found was that inadvertently your SCADA Network it was connected to to your corporate network or in some cases even to internet when Stuxnet was a pretty hot thing I'm not sure if you have heard of a website called showdown how many of you have heard of showdown so a couple of couple of fans what it does is it basically it's a crawler it basically scans all the web sites all the servers on the net and stores their IP addresses and their banners so if it is FTP server it will store its banner if it's HTTP server to store its banner if it's a telnet server to store its

banner so after Stuxnet and it had search and it shows what were the top searches so after Stuxnet was released the top search is there where for PLC's where for the vulnerable PLC's that Stuxnet that Stuxnet targeted so I went in I put my little search for let's say GE or some ABB or various manufacturers for SCADA systems and even you when you when when you even you go outside this presentation you can go to showdown and try to find if there are any control systems on the internet accessible to you and well I as of yesterday I was still able to find some systems that gave that that we're on the internet so that'sthat's the number one I think

problem with SCADA says with the SCADA master systems secondly there is no authentication or poor user authentication so as I mentioned usually the SCADA master is a room a physical room and it has physical when you go in there could be like a fingerprint scanner or maybe a guard there to make sure that on an unauthorized people cannot get in but once you get in this is let's say this is a SCADA master terminal I can just go and tinker in with anything there is no real authentication for that terminal and that was fine when your master network was totally segregated from your corporate network but now since that's not the case all the viruses and worms they don't really need

to physically go in the room and do anything they can just hop on from your corporate network into the SCADA master and if there is no authentication on let's say the HMI the HMI software application they can directly manipulate these applications to give let's say wrong instructions to the PLC's to give wrong instructions to to do something or maybe to retrieve data from these PLC's and again if there is no authentication there is very there are very few systems with poor user authentication so so as to log what you what operator did what there are share passwords or default passwords and in one of the conferences that I attended with Joe and by the way

there are two really SCADA experts here with us today so if you want to join if you want to talk with them feel free and talk with me or talk with Joe and hear who who are really who have really dedicated their entire life in control system security and in just control systems so in one of the conferences that I went to the most the number one reason given for not having passwords on the my system was that because an operator has to respond to an emergency it's an arm maybe something is burning something has exploded something is leaking there is no time for that operator to key in a long password and then know what the alarm is or to react

to it but at the same conference there was a military some some guy from the US Navy I don't know there was both Air Force and Navy and they said that will we have 25 or 18 year-old cadets typing in 25 character passwords which are changed every two weeks and they have no problem in doing that now in the military you could do that but I think it's time that we start we start thinking about these control systems as really systems that are important for our critical infrastructure because they are critical infrastructure systems and they are very critical know patching so this is this was also something that has been discussed in the control system areas

that often these systems are not patched and there are various reasons around it for not being patch it's not just that the control system operators are lazy or something and that they do not want the patch but it's that around 2000 or maybe in the 1990s in the late 90s when Windows started to become the predominant or the most or a pretty famous operating system a lot of control system owners asked their vendors to port a lot of their software on Windows because it was cheaper instead of having a proprietary system from n to n if you could sort of migrate some of the HMI components on Windows and have the operator log in there have the graphics

there it's it's much more cost effective but what happened was the the entire patching concept of Windows came to control systems so now when Microsoft releases a patch how many times a month I'm pretty sure everyone knows once every Patch Tuesday you have to patch this critical control system which is and which is not possible many times on top of that control system vendors not all the time there is no real guidance from these control system vendors on what would happen if you apply a patch so let's say I am owner of a control system I have a system from some vendor and my vendor doesn't tell me if I apply this Microsoft patch what would happen

to the system I would be really scared to apply that patch because this is the control system which operates maybe some really important temperature or dam or I don't know something and important and I don't want to apply a patch for which I don't know what would happen if I apply that patch so there is no patching guidance from control system vendors or things have changed a little bit but for the most part there is not much guidance from the control system vendors so lack of patching is definitely an issue for control systems not restarted in vendors so we saw systems that we're running Windows 2000 Windows NT and they have not been even restarted in many years

there were a lot of precautions been taken like UPS and a lot of other things around those systems so that the system doesn't go down and when asked why have you not restarted the system the simple answer was that well we are not sure if the system will come up again so if you are in that state where you are not really sure if the system will come up again if you restart it then I think you are really in deep trouble as a lot of systems are migrated to standard windows or Solaris systems the same issues that apply to these windows or Solaris systems or any any off-the-shelf operating systems like unnecessary services hardening of the operating

systems and so on and so forth they apply now to control systems as well so some of the challenges that we saw and the most daunting challenge is the long life cycle of a SCADA system these systems are meant to last for decades or multiple decades thirty years 40 years 50 years they are not like your laptops which you can replace or which are replaced every three to four years and the challenge is in upgrading partially some of the components in this systems and the long life cycle of SCADA systems I think is is is a machan we could we could overcome and I'm gonna talk about that in a bit there is no testing or guidance from

operating system vendor patches some systems are directly managed by SCADA vendors so all although you own the system you do not have administrative level privileges on that system so we went into an instance where it was a windowed system we were asked to sort of scan it and said oh we have seen lots of Windows system easily done but then men asked for credentials they didn't have admin level credentials which were with the vendor who provides that system so we could so the owner couldn't really do a good assessment of that system if there are unauthorized services or not because they they really don't manage these systems other things like on the SCADA master network there are a lot of

other systems like data historians which are basically databases of the Shelf databases Oracle mice equals one and so forth which also needs patching and which are also affected by the same vulnerabilities that as other of the shelf software and in more than one instance we saw a lot of internal differences between the security scheme and the team which really managed its SCADA networks and because of these differences or differences of opinion what ended up happening was there was no real scanning or auditing that happened one of the instance was that well someone ran from one from the IT just ran blindly a scan of the SCADA system and a lot of PLC's and Artie use they just were knocked off

because it was like a standard since scan that he would run against operating systems and that caused a lot of havoc and then they said ok no scanning no auditing we'll deal with it later so if you are someone in the idea organization who is responsible for the security of the SCADA system make then you scan them with certain settings that are that are suitable for for scanning deed systems so some of the proposals that I have is you could have some sort of password policy access control rules on these SCADA systems have some strategy for software upgrades and patching as the software that you are getting gets older and older it becomes difficult and more difficult to

upgrade and patch SCADA test environment this is easier said that done if your SCADA system cost like a million dollars then it will almost cost you half to have a working test environment and that's why there are no test environments at least in the instances that we have seen but you could have simulators and SCADA software or vendors giving a really small test systems and you could have you could test for patching and so on and so forth they could demand for from SCADA vendors to expedite on operating system patches give you guidance on what will happen if you apply the patch and basically since a lot of SCADA networks are now starting to look as your normal IT networks with

Windows and Oracle and so on and so forth try applying your usual IT network management security skills to the SCADA network as well which would which would take you a long way so finally SCADA scan this is a open source tool that we are releasing an update we are releasing an updated version today what it does today is it scans your Network for Modbus slaves for dnp3 slaves it is really a alpha version in the beta version we are adding support for SCADA master vulnerability scanning vulnerabilities for HMI and so on and so forth SNMP support and in the one overage and finally we what we want is we want user configurable signatures so we we have not seen all the SCADA

systems there are various different types of SCADA systems so what we want is the owners of these systems if possible to run this tool and Lord they're the signatures if this tool has never seen a system like their upload that signature to to that to this open-source tool so that others can really benefit from this community effort we are not there yet we are at the Alpha origin today but we will get there and I will have updates on my Twitter feed about where the tool is and what what are what sort of updates we are having it on a day to day basis so that's the location of where the two leads today that's my Twitter handle if

you want to if you are interested in updates on the tool and generally updates on various other security vulnerabilities that come on a day to day basis its I'm almost out of time I think maybe we have time for one or two questions go ahead Joe

it's called protecting industrial control systems from electronic threats and it's there so people can understand what the differences are between control systems and IT systems there's been more than 200 actual control system cyber incidents to date in the US alone four have killed people we've already had three major cyber related electric outages we've had two nuclear plants shut down from full power we've had a water pump from a Superfund site into the drinking water system those are just some of the things that have already happened you don't have to try and figure out what might be I happen to be the managing director of is a 99 which is the International Society of automation 99 is process control

security standards one of the group working groups is ninety nine oh six and I would encourage anybody here to join that's patch management for industrial control systems and the reason is that you may use Windows on a control system but it isn't Windows its Honeywell windows or its ABB Windows you know where it's Emerson Windows if you take a standard Microsoft patch and apply it against these systems you will crash the system not maybe so one of the things I would I would heartily suggest if you want go to is a org go to is a ninety nine but ninety nine oh six is patch management the other thing I wanted to point out actually a couple of real

quick ones the biggest issues with control systems are that we've got systems that have nine issues with security they are not patchable they are not fixable that is what's tucks in that one after you cannot patch the vulnerability that went after because it is part of the design of the controller and we have a number of controllers with those types of issues so there's two things going on one is what I'm always talking about make sure that the if you will network security part is taken care of the other is the actual control system piece and what you've got to have is a team of people like me or others who know the control systems as well as the security

people who know the network and security piece because you won't get there from here without it the last point I wanted to make out is or point out is that one of the reasons you don't restart systems is because bad things happen when you restart systems especially if a facility is operating we just had a case literally two months ago where two units were operating and somebody restarted a system that hadn't been restarted it shut down the logic in every single one of the processors in the distributed control system if you can imagine 214 processors all lost logic with a plant operating never did we think that was possible and it happened you know total

loss of control loss of view loss of everything with a plant operating thank you so I think we are out of time but feel free to talk with me and to Joe and everyone here who are from the control systems and have any of your questions answered thank you [Applause]