CyPSA Cyber Physical Situational Awareness - Kate Davis, Edmond Rogers

so okay next up uh we have Edmund Rogers from the University of Illinois uh give him a round of applause all right so I do cyber security research at the University of Illinois with a bunch of different people on this project uh from Illinois and Oregon State and ruter and power world here's some pictures of some of the key people on the project somehow I got involved with those guys Kate was supposed to be here with me this would have been her first hacker convention talk she's done many uh i e type speaking speaking engagement before she was really excited about being here her husband was much more exciting excited when he crashed his bicycle and he needs his shoulder

reconstructed or something so Kate sends her regrets and I'm going to be responsible for her part too which is all the power stuff so it's going to be pretty cool but the research project that we were on it was funded by arpa we at the end of 3 years and the at the end one of the goals that we had was we were going to ensure that our software was going to have impact and in order to do that um I really pushed for open- sourcing the engine that we use for cyber physical um impact analysis um and um primarily this was the velop for use in the power grid so the idea being how can you ensure

operational reliability and uh understand the impact of you know the seword or you know the the network attack surface around uh a large system that provides power to many people and you have more um cyber security things then you have time to take care of and how can you effectively prior prioritize those things so your effort can be directed where it would have the greatest impact on reliability of operation because the problem is very complicated when you think about you know there's internet connectivity coming into the corporate Network and then we've got all these little field devices and then you got you know nasty hackers trying to go into substations and and or they're oh we're going to

come in through the internet and attack the land so it's it's a very large complex problem when you look at um how do you put everything together so that um we can look at adding on to traditional contingency analysis because planners and electric operations manually Define contingencies like oh we're going to lose this substation or we're going to lose this generator and they um like ecosystems are big things within the power grid uh but we want to try and then they automatically insert these things and they Sim they simulate the impact of what would happen if you lost that substation or that generator or something big but it was always meant to be this like

n minus one criteria where we lose one substation what's going to happen and then traditional contingency analysis has always been that way where we're going to we're going to plan everything based around that one thing but the but what happens when you add this uh cyber on top where maybe it would be very you could simultaneously bring down or provide um a den service condition to multiple locations and how would you plan for that um short of yeah we can make a big giant blackout but then there's like a middle ground between that like I get five things happen at once what is the contingency for that and previously we couldn't do any um computation with that but newer more

Modern Hardware has giving up given us the capability of providing for much more fine grained information about impact and that's what the research project was all about and because of the Advent of more modern Communications in things like the power grid you have this idea about the size of the list can just grow very large if you look at a th000 a th000 lines and doing an N minus one means you have a thousand different criteria but you go down to n minus two and just take two things out then you're you're just getting really big and so what do you do because uh when you look at Game Theory and there's a lot of game

theory people on our research project um they wanted to compute everything and it got really really large fast and we're like what we want to do is come up with a way to reduce the where we're going to be calculating we we do that and I'm going to get to it a little bit about which is my part about we add network attack surface to reduce the space of computation so we can come up with a rank list of contingencies so if you look at like things like the this is the beginning of the 2003 blackout um what the tool can do is take all of the different physical components that are inside of a substation and then

we map them um M map the Cyber and the network attach surface around it and we can provide a mapping so this is like normal and um so when something happens there's a little bit of a disturbance and then depending on what the piece of equipment is how long it takes to clear the fault determines whether or that the thing would Cascade beyond the bound of whatever it might be and then if something bad happens we all know that this was probably bad so um modern Computing has allowed us instead of just oh we lose this substation something bad happens you can actually go into uh an industrial facility after doing some of this inventory work and I

would I always like to to I want to go on the sub station and point at something and go I want to know you know does this thing have an idea address how is it connected on a network what would be the impact if this thing were to be misused and um so the project tries to answer that question and um this is kind of like a preview of what's what you're getting ready to see this is what the front end looks like and what you're able to download now was released 4 hours ago um but CES so streamlines the utilities ability to inventory and analyze cyber physical assets so it brings out um first we have we have to do work to

collect the data from the individual devices that make the power grid and then provide a mapping between the physical device and then its manifestation in the planning model and then and it's if it exists it's IP addresses and uh network attack surface and as we did the research we came up with use cases and like the first one I kind of touched on a little bit was asset ranking so the idea is like I've got a fairly large or a medium siiz whatever you want to call it I have a bunch of substations I have 10,000 100,000 assets do you know which ones are the top 50 individual Computing devices that are in your network likely not because planning

has always been I always viewed it as being from the top down so we're going to look at this substation and we like to think we do cyberphysical from the bottom up so we look at all the individual components in the in the power grid and then rank those assets and they can actually be ranked in uh within like um the threshold for um voltage uh stability which is like 15 minutes because uh utility will calculate how they're doing every 15 minutes minutes typically so control panel kind of looks like this I'll do a demo later I'm not going to hold on this too long unless the demo screws up but you can get a length list of what your

top assets are and then we calculated a cyber cost based on network availability and vulnerability scores from the icert database so we can ingest nmap data to come up with this score and then there's also performance index which is related to what does the what does what's the impact of of the equipment in the on the power side and then some of the other use cases that came along as we were doing this at utilities because we actually have deployed this software at certain utilities is this idea about patching so need to patch a lot of things and I want to know if I only have a limited amount of time to patch things which

things would get the best benefit out of being patched first so if you're able to provide a ranked list of all your assets you you know which of the assets you should patch first and there's actually a little box in the tool that says I just patch this asset then the tool will recalculate the list of um top assets based on the fact that you've patched certain assets you can also keep track of which ones are patched and not patched so hopefully that'll work in the demo not there's a screenshot of course so the idea is we have a network attack surface over here and we can we can map the attacker and I don't know if I can use

the mouse for the people online but on the left over here we have the attack path and this is like a standard network attack surface kind of graph but we also can graph graph that into a physical oneline diagram so if the electrical engineer was here she'd be able to tell you a lot more but I know even as a cyber security guy 126% is bad so the guy gets in on that device that line goes to 126% and in the animation that that we actually can run and I'm probably not going to do it today it causes a little blackout so um the other thing that's really cool if you think about it is aggregate

exposure so how many you know Willies 385s do I have in all my stuff because if there's a vulnerability and a certain revision level or model has a vulnerability what happens if they all go out at the same time we can actually model that now so um there is a way to just uh select or show how many of each different thing you have and what the impact would be if those things were to go down and then um this then leads into another case we got from actual utility feedback is they wanted to be able to do cyber incident planning where they wanted to be able to come up with more realistic cases of a

cyber incident and then what could they do to plan to make for a more reliable system so the idea would be uh in the blackout you know there were there were C this is the actual 2003 blackout in uh Power flow diagram and then if you had a way to map all the individual components um you could show the where the most important things were and then the impact of that uh event like we we showed before and this piece probably would have been a lot better if Kate was here um but this is the transient stability problem where within 5 milliseconds if the fault doesn't clear there's a problem so we want to take all

the individual components and determine which ones have this 5 millisecond threshold exceeded which would mean that it would impact devices around it and that could perhaps get larger and then and again of course another this is what you're going to get in a download and we'll see if the uh demo actually works later but so the aggregate exposure like I said before so if I wanted to come in and there were three different um things in common so I can have a multiple Pathways coming and we can then simulate that and know now again not even being an electrical engineer 114 142% bad so and then this is uh just a power world flow diagram

and line on line an attacker it's a threat model it's an attacker I don't like the word attack but the academics I wor I work with love the word attack so I really try to remove the attack as much as possible instead hm how would you call it instead what would I call it instead a day that ends in y so the the question was how how would I how would I uh call the instead of the word attack what would I use I would say it's just a day that ends in why so the things happen and if there's a of multiple if you have something in common amongst things there's not really a a

until now there was a really way to effectively model this and so one of the things that we've been really working on is how do you visualize these things because there's a lot of junk in in this and it's a very hard problem just in visualization so actually the first thing is if you go to this IP address this is actually live right now you can see what we did was we took the it 8 bus model and then modeled all of the protective relays and everything the distance relays and you can see a cyber physical representation of the I8 bus model it's actually online this is using uh Gabe Weaver cptl um and then uh we did a talk I did

a talk with him about this a couple of years ago cptl is cyber physical topology language and it's in the plumbing of the engine so it provides us with ontology so that we can describe the relationship both from a cyber perspective and a physical perspective of individual devices that make up this system and that kind of like all works together with with you have you can use your network software of your choice because there's a there's a Json format and I'll get into this in a little while and then we have planning software and the engine is what's been open sourced today you can put your own impacts and network attack service and make your own

ranked lists and um with the whole idea the goal of the project to be grid reliability and uh it's been I released it uh I think it was 11:00 this morning after we got the license attached to the software which I noticed wasn't there was no license I'm like on the phone like oh wait a minute maybe we should put a license on this so it's open source under the NCSA uiu licensed license and um what we did was we worked to remove remove the dependency of any particular piece of commercial software so um I have a slide about that later the idea is and I think I've already kind of talked about this a little bit

the physical connections and the impact together are so like I've got in the top here the host may be compromised or line k then line K is at risk and then this is all manifested back into what is the protection scheme at the substation and then which individual device is that risk and then what would the impact be if that device were compromised and then with this critical clearing time we can determine whether or not it's going to be an issue because the planner will tell you if it clears within 5 milliseconds I don't care about it so now let's see what happens and if I remember to turn the porn off on my because I was you know I was waiting

in the speaker room so this is that IP address thing that I showed and I will provide the slide so that you know what the IP address was but it's 72. whatever that is um it's uh and um this is actually hopefully online so this is actually out on the internet somewhere so you can see that the visual representations of the 8B model you can go online and look at those um by going to the IP address you can get me afterwards and then the other thing is the actual tool which is right here so when you download cypa or armadillo which is what it's called the release is called armadillo you're going to get uh HTML 5 front end you need

tornado and python to run it on your on your machine and um I could go through and start it but I actually went ahead and loaded it I could show you later we'll see how I'm doing on time I'm I think I'm talking really fast fast right now so um I don't want to attempt starting it right now because it's all running and it might screw itself up because it does open up a port inside your computer because we connect to like 5555 so when I click on cyes analysis so if you have multiple projects you can select those uh and there's a folder structure for that and um so I can go in there and select that and then the

little um visualization will appear and I can view the Cyber assets and then we can also view the physical asset we get the oneline diagrams and then um if I start this it kind of looks like this so now here we have a a rank list of Assets in the 8 bus model so if I click on different IP addresses I'll get the corresponding pack pass if it's still working um it was working cuz the pass were up there or I can view the physical that's still working and then the the Cyber connectivity in that specific location there's also a couple of other features in the tool that's been released today and that's um let me go

back

here there we go so there's also the other visualization so we use D3 for visualization and um that's coming up but this is the um this is going to look really really small on this video so this is the 8B model but if I click on individual substations I can then get we have a tree view on the left and then we can show the both the physical they're big from a planning perspective or oneline diagram perspective assets and then or click on individual things and the network would come up or the network elements or the cyber security the Cyber elements the network attack surface of individual substations and then there's also um this is another tree view of the same

cptl based um things once we get the mapping of this relay is this IP address it serves this function we can tie all those things together via ontologies and then look at all of the different components that make up the different substation so I can select any substation at random um or if you can see that somebody can pick from the audience and I can show not if I do that there we go let's see so how about you pick one and we'll see if it actually works still somebody how about H Brook okay there you go so this is a h Brook substation and then I can look at um the different cyber and physical components that are

in that um substation looks better on a if you have a bigger screen obviously and then um get back here and then it kind of can get big and here's where we hit the D3 wall so we actually have a model that has 300 substations in it and it's about 5,000 devices so this is 5,000 devices and 3 kind of like starts to on itself so like if you wanted to zoom in it takes like 30 seconds so I could get all the way in here but I don't have that much time even as fast as I'm talking right now I don't have that much time but uh so it's it's kind of there but going back and some of the

other trees that we used in the visualization really help us put more information visually on this screen because um one of the things we learned in the research going into utilities is that it's it seemed like we were more interested in the visualization than the utility actually is because all they really wanted was either a CSV file or a list we want the list we want to see a table and that's really where they always go back to the the you know the home interface where we just go into not the right one we go into here and they want a nice table so they can see their top assets and this to be

working here we go they want to see their list of assets and I think if I come down here and with the video kind of like think maybe see that now I can mark two two things I can do with the engine that are kind of cool is that you can Mark a host as being patched so that you know which ones haven't been patched yet just utility assets for this but the other thing I thought was really cool is I could actually Mark a host as being compromised and then recalculate the rank path based on that and then you can also go through and see the vulnerabilities based on cve score for each of the individual components uh

that have IP addresses and we tie all that together and then we can also patch so Mark the vulnerabilities as being patched and then recalculate the um the pass and then did I forget anything I'll find out when I go back to the slides um I think there was one

more and yes there's one more so this is actually D3 so this is the eight substation model and one of the things it's like okay that looks like uh some kind of Blended salad so what are you going to do when you have all of these different devices and one of the things that we're working on and we we built the trees is that I could show you all the different devices but then how does this make sense you add names do you not put names in these are all traditional visualization problems which is why we have a blend between you want a visualization you have a smaller Network that's going to be fun especially from an open source

perspective because maybe you don't have 5,000 devices uh one of the things the bides folks ask us to do is to come up with a way for I'm not a power guy how can I use this so the visualizations are are very mature and stable we've had we had professional programmers working on some of this some of the visualizations and then you can always go back to the tables because uh you can get a lot of uh from just tables because people understand either the name of the of the device or its IP address cyber guys are always going to look at cyber security people look at the IP address and know what you're talking about and then like

operators always ask for the name or they look to see the name and you can change these kind of things once you have everything into a database so I'm going to go and see if I forgot anything here I'll need to pick the right slides okay so here's how you start the tool

so let me see not bad um so when you go to the GitHub and download armadillo you're going to need to be running tornado it's a some guy shaking his head yeah yeah so then all you're going to do is and it's not start tornado it's I think it's not it doesn't say that it's something something else now so that that's that's wrong it says start cypa it doesn't say start tornado anymore but then it comes up just like I showed you in the demo you click on cypes analysis maybe I should have looked at these slides before I tried to demo because I probably would have done better um click select and then eight bu analysis comes

up a bunch of stuff starts to go in the background and this is the engine doing its thing and then you can go in and look at your rank list of um contingencies and then here's the the list of assets and then we have this like I said the security index is determined by the Cyber cost and the performance index and um you want to contact any of us we have contact information there's been Papers written on some of the formulas behind the performance index and the Cyber cost and there's more papers being published towards the end of the year uh regarding this research and um did that showed you that there's the tree view maybe it looks a little

bit better in the slide where you can select out on a different yard and it shows the connection between the distance relays this is our attempt at really better organizing than we can get out of D3 because it's so jumbled when you put a few hundred devices on a small screen but um it's more than just power the cpes of engine and and the web front in there can be used on things other than the power grid and like I said before there is a performance index Json template so you can make your own performance index for individual devices and then build your own network attack surface via the the Json provided in the open source and use the engine and uh

something that's called uh offline mode and then you're like well how do I do that so well um the this is like literally something that was done last week because I was very insistent about having a standalone product that would run without any proprietary information from commercial products so Olivier calls it offline mode so you actually have to type offline in the command line he wrote These up um we had Olivier was uh he was a we had him on loan from a Marine Institute in Fr France he's a Frenchman and uh he calls it offline mode so now it's offline mode right now I don't know if it's going to stay that way but it's

actually in the command line so you would do run SES of bat eight bus Boop offline or and you can also change the output in CP gen. CSV so that you can pull your own information in and uh Olivier is not in America anymore he left last week and uh but but I thought that I would want him to be in the talk so that's Olivia he really indoctrinated himself into America while he was here and he was amazed at the size of the uh of the soft drinks so CA is available GitHub uh as of 11: this morning there is the papers involved with some of the formalization path analysis that we did at the Illinois website under I cypa and

then kadeo is a company that has been formed to uh work on servicing the open source kind of like with snort or something like that where you want uh somebody to come in and do this and part of the arpa project was um path to commercialization they wanted they want to see the research have impact so um kadeo can kind of help with that and then um Kate would have been happy to be here to talk to you about it this is going to this is her company um and um my role at the university is to help researchers have see their research have impact and to uh perhaps have them start companies and this is the second one I've done at the

University of Illinois so I think that's it and I have plenty of time for questions people have questions come I was going ahead Chris TR M um so your model I'm an electrical engineer I don't know if you know okay so be gentle I'm I'm a hacker I'm not an electrical engineer so I used to work for entg and um so I actually know Matt Davis that's uh Kate's husband we went to school yeah he's he's okay yeah U but he's not he's not he's very happy right now yeah yeah yeah so we went to school together at Tech anyway my question is is I I know it focuses a lot on the relays and

things like that does this project also include things like rtu substation rtus as a matter of fact it does but the idea is you know we can map impact of anything inside the substation because with the ontologies that way the the sexy things are the relays and because they have the greatest uh precision need to be I need to understand what's going to happen 5 milliseconds within 5 milliseconds after something happens in a relay rtu goes out oh no we don't have readings from this site the Grid's not going to go down it's different different performance index but it can be included but if you have 10,000 things they're going to be towards the bottom of the L

yeah we had to do risk ratings for every type of piece of equipment for relays I had to do them for rtus yeah and I wanted to include cyber at the time but we didn't have very many rtus on the network uh so uh if you have a an older substation with a lot of electromechanical relays but you have a modern uh substation Gateway or rtu with IP that would be really good to yeah the other thing is is the idea behind we see this a lot out in the field too cuz we've been to a lot of utilities is that we can model um cereal or like non rotable protocols because it's just a relational

map to us a can get to B to get to C and it's much easier to model then when we're looking at a larger utility that has maybe multiple Pathways and you can still map impact on that but you lose like so I won't have an end map there's no IP address so you're you have to adjust your cyber score because it's different CU like your cyber score probably one then because if I can get access to that I own it right exactly all right thanks good question and would you be so kind to share with us uh if you could the future direct like you know I guess what's next I I mean I see

that a company has been founded I know that at Illinois there's going to be a lot you know you're expecting a lot more paper to get out you know to a lot of more publication but you know um is this going to be used I mean is it going to be is this going to be shopped around to utilities and power uh industrial uh companies I mean could you share a little bit more about how you're going to what the approach is sure I can do that so I didn't want this to be a vendor pitch yeah so you know we are out in W we've been at w user World meeting we're working closely with

planners um we're with the big utility out in wet we've actually modeled the Northwest with this and um we can do transing stability studies and critical clearing times for some of w I don't want to misspeak because I think this is live but for some of w but it's possible to do all of w we just would need to model it and and um so we're actively involved in that kind of thing on the cadeo side of the fence but this is not a cadeo talk this is a talk about the engine we use this engine and um and we want other people to use it too because some of the continued research that's going to be

happening is in cyber because we can formally describe what's going to happen in physical impact cyber is a little behind we're about maybe a year to two years behind in cyber we can't formally describe the way we can physical you know you know this and there's continued research on this with some of the folks that I flashed their pictures up earlier I didn't looked at this as a as a sales Pit by any no way what I looked no no no but I'm I'm trying to be very careful about that I think there's something that you do bring up that's pretty valuable and that you're also demonstrating the the entrepreneurial aspect of it right because when when we were

talking about what we were going to do with the software and there was like you know and you have to be very careful about what I say now because this this is being recorded but you know we wanted it to be open source because we wanted to get it out there and it makes it it makes a commercial pathway easier because anybody else can go out and jump and grab the software now but we have a three-year head start on you and we know how to do this and we've broken this thing a thousand times before the web front end is very stable you build your own impact models you can do this stuff now and what we're really this is also a

call for volunteers this is a new open- source project let's let's make this for the community the engine is very nice and um I'd like to see people get involved and so if I can ask a question about how you're cascading um impact through the graph um first are you maintain two graphs one for you know reachability know if you compromise this compromise this in one for something like flowing you know increases in power as you remove or add um yeah that's like or is it a single uh graph structure no it's multiple graphs so you have the graph for the power which is very formalized in many different commercial applications planning software whichever one you use and we built an interface so

that if you don't use power world you use something else yeah you can write to the XML and we can take in that impact data and then on the other side like for Network you need to just tell us we need to see firewall rules or in the format that the CSV sits in so you can push the two things together and I've completely forgotten your question at this point oh the graphs the graphs so um so yeah um the it's more than just graphs but the online mode for example what we do is we can recalculate what this scores based on both the P the power is always changing what's important on the power

grid is going to change depending on what time of day it is because load moves all around and then you're corresponding where is the hot spot is going to move on the Cyber side too because these are the things that are more important and I don't think I answered the question no it explains how you're um flowing the cas game failures through to get impact by yeah Power has you know exceeded threshold if if you had power world there's a socket that's open in the engine it talks directly with power world and then when there's changes power world does transia stability um analysis and it also does a critical clearing time on the individual devices and this is something that we

literally just finished two weeks ago so we can we can provide a CL critical clearing time in power world for individual relays now are the attack paths through the um are you modeling attack paths on the attacker side um as in you know they got into this and with this they can get into this or you more modeling just um if they you know we know that this is connected to something an attacker could potentially touch and therefore if it goes down we see this and are you modeling um failure modes other than just availability yes so it's a combination we look at the firewall configuration so we get a reduced set of network attack

surface from that availability and then through nmap you can if you have nmap data and we've done actual nmap scanning and we've artificially produced nmap scanning to interrogate the user database so that these are the vulnerabilities that might be on this system so if there is a vulnerability that matches the software that's in the system it's going to percolate up on the list and this is also something that can be calculated over a period of time or iterative iteratively is that iteratively yes and so and then when you're determining when you're prioritizing assets by likelihood and impact um are you just kind of using some kind of 5x5 Matrix for Impact versus likelihood how are you weighing

those two together to get your single this is the most important um how do you combine those two orthogonal vectors in the tool there's a paper you can read it's it's it's the answer is longer than I can sure get to right now but there and it it's also a research topic you know we make some decisions about what formulas we use to come up with these indexes we test them against the utilities that we're going out within the real world and we're getting close and one of the things that I'm really I'm really happy to release the software out to the general public because we want to know how close we are and how

we're how we're calculating both performance index on the network side and then the impact and then blending them together because we started out first using Game Theory and that was very MP very MP problematic and then we went to depth first search you know we've got a lot of phds on this project lots of papers were written about this and it but it all came down to practically how do you reduce the attack surface so that it's big enough to calculate but then it's still interesting because I can reduce the attack surface down to zero then we can calculate everything it's easy so you got to find a there's a middle ground in there somewhere and then the more people

that use this we can find how far can we take today's Electronics because I think we can get much further now than we could 20 years ago but no work has really been done on modernizing this kind of planning okay thank you that's a good question thank you so anybody else so thanks I think that maybe the electrical engineering part would have been a little longer if Kate had actually come and um appreciate everybody's attention and just go out and get the software and tell us what we did wrong thank

you