BSidesCharm - 2017 - Rob M. Lee - Keynote

yeah it seems good all right last couple stragglers uh yeah microphones yeah that that probably helps a little bit

all right well we'll just deal with it uh so welcome everybody to b-sides charm 2017.

so 327 b-sides event overall 216th global so a whole lot of b-sides um 1077 tickets this year and we sold out faster and faster and faster as some of you found out as you try to get tickets and twitter exploded with tickets in the last few weeks so that was kind of fun how many people actually got their tickets via twitter only a couple yeah crazy so um yeah so sponsors-wise uh we had one platinum this year with lockheed martin uh several gold uh with nsa sans hack head raytheon booz allen point three talos tenable ceiling tech and capital kali or capital technology university trimark cyber and aspect security so a lot of awesome gold sponsors

yup they're all over in the sponsor room please go visit them uh for silver sponsors dragos carbon black fun times tapestry anchor technologies uh first silver sponsors also traces grim howard county economic development authority uh net craftsman g2 iron vine security and noveta then for bronze cobalt strike dynaxis and symantec we actually had a lot of other contributors who jumped in isc with coffee and cdw with some lovely tablets uh point three with lanyards no starch hacker boxes and hack five with some lovely giveaways nobody likes giveaways though right and then our lovely charities this year that we're supporting an allocated space digital harbor foundation hackers for charity and security besides global so later on when

we have t-shirts available you can choose where your donation goes with those lovely poker chips hfc has a table as well as unallocated in digital harbor so please go visit them as well they are also in the sponsor village uh badges so uh some lovely electronic badges for our speakers or rather keynote speakers and staff and those of you who elected to do the electronic badge option and then we have blue for our attendees white for speakers trainers and charities red for our staff and then green is the sponsors please visit them they have tokens for coffee and later they will have drink tokens for the party so you want to go talk to them if you want drinks

some lovely notes some all the red badges and electronic staff badges grain red shirts please listen to them they make this event easier after the con the link will be tweeted out if you're interested in helping as an organizer next year uh pictures and social media please ask everybody in the picture if they're cool with it some people are don't like that be courteous all the talks will be recorded and online we have iron geek here helping us this year so they'll be up a little faster because aaron geek is awesome uh please use the hashtag uh the wi-fi is available uh this is the lovely uh password uh those are l's not ones that was fun

because they got it wrong in the email when they read it and set it up which was fun so fun mornings uh the challenges over in the next room we have the wireless ctf and point three um escalate ctf so a jeopardy style instead of uh the round style they had last year so hopefully a little more open for folks lockpick village and then the tree training rooms no alcohol smoking vaping in the building there's a patio outside for that it's rather nice feedback forms these links will be on the website using them may lead to prizes at closing so uh yeah if you want the prizes closing um please use the feedback forms they're

really helpful for us to be able to get your feedback to improve for next year uh party location across the street pratt street dale house again at 8 pm um soda drinks upstairs there will be crabs adjust humidity for those who aren't interested in drinking or just want to have fun heckling each other and being horrible people because it's fun uh you will need to bring your badge to actually get into the party so please bring your badge to the party otherwise you will have to go back and get it and that kind of sucks again tag us so that we can retweet all your awesome feedback and comments that is it so i'm going to turn it over

to our introductory keynote speaker rob lee

thank you all right cool all right well thank you for having me gonna kind of walk around a little bit so uh greatly appreciate everybody coming today if we look at the uh agenda we've got a lot of really interesting talks i don't know how many of you have been to b size before any any besides all right so that's a good returning number means we've also got a lot of folks who are first timers so if you are a first timer really at b-sides you can just grab on to anybody ask them what's cool to see what's cool to do they'll take care of you so my name's rob lee and i was asked to

talk about industrial control systems and specifically some research i've been doing into industrial control system attacks recently a lot of interesting things out there a lot of hype actually a lot of fear when it comes to the discussion of industrial systems so i'll walk you through why it's such a challenge for the community to actually understand what's going on in this space and talk a little bit about the things we've seen before some things we need to be aware of and then the research that's going on right now but really as a keynote i only have a couple jobs the first of course is to run over my time slot and get everybody off track

that's that's the first thing i'll try to do well uh the second thing is to try to bring a tone to what we're doing uh for the rest of the day so jim chrissy as keynote tomorrow he's going to tackle it very very well and set the tone for tomorrow my tone for today and a lot of what my focus is on is trying to push back on some of the hype we see with good knowledge making sure that people that are speaking about events in the media that are speaking about uh things to their friends their peers their their parents whoever it might be that we do so from an informed position and so that's what i'll be mainly

focusing on here so getting us going here uh my background uh do a lot of different things uh teach at sans uh run a company called dragos one of the sponsors and uh my background though and the reason i'll highlight my background for the purpose of the talk is i started out on the air force side of the house but spent my entire career in the intelligence community and while they're working for one of the intelligence agencies that seems to be less popular these days i set up a mission looking at nation states breaking into industrial control systems so the entire purpose of what we did was can we find different targeted threats breaking into

industrial sites and what can we do about that from understanding and patterning out that activity to do better in the future for defense and then of course i also uh write a little comic called little bobby every sunday is anybody actually familiar with this couple of you yeah awesome this all started out really because i found in the military that it was really uh interesting to try to explain technical topics to leadership and if i could break things down into little three pane comics it was just way more entertaining to brief generals with that so that's where it originated from but i try to keep a little levity uh with going on so uh here's our agenda

today i actually started out today walking through the wrong side of the conference and i saw the audience i was like wow besides charm you finally solve the diversity challenge in infosec like this is this is awesome uh unfortunately it turns out the sorority is not part of what we're doing today but i do i do like looking around and seeing the vast sort of different ages and diversity that we have we should do better though it's a focus area for everybody but here's what i'll talk about a little bit more pc of topics first of all how are these things these industrial control systems these ics attacks how are these even done there's a lot of misconceptions

around what an attack is a scan on your honeypot that you set up port 502 on to mimic modbus is not an attack and it is not a control system so we've got to do a little bit better in the community uh ics cyber attacks fact works fiction this is where i'll try to give you a historical perspective of what we've seen to date and what matters and really a lot of the hype we've seen and then lastly i'll talk to you about this project mimics that one of my one of my folks ben miller has really been leading up on our team looking at what we can try to return to the community in terms of some base metrics can can we

understand a little bit better around what are good metrics around things like ics incidents around the community and of course little bobby down here just saying hey matt we need more scalable cloud-based threat intelligence and analytics and automated big data endpoint security solutions matt says you have no idea what you just said do he's like nope but i read it out loud and i got invited to speak at a conference i figured if i uh told three more people i'd be a security expert matt says that sounds about right so that's that's why i'm here i end up saying the word cyber a lot and got invited to speak so how are ics attacks conducted well

uh this one seems to be interesting of late we've seen all sorts of things going on from shadow brokers everything else so little bobby down at the bottom calling up what looks maybe like microsoft depending on if they want to sue me or not then it's not um but little bobby calling up saying why don't you do more to protect me from nation state espionage and the lady kindly replies well we want to but while espionage gets a lot of media attention it impacts less than 0.1 of our customers and little bobby says so you're saying i'm special in general what i like to highlight here before getting into how ics attacks are conducted is there are a couple who've gotten

most of the attention uh in the community there are the the big ones that like i can't get through an ics presentation without saying the word stuxnet somewhere and these things have controlled a lot of the narrative but as i get through later in the presentation and pull out what's actually affecting the community you'll see that you're far more likely to be impacted by some virus on a usb from the early 2000s than you are from nation state espionage it's cool and it's exciting and it's interesting sounding but not necessarily what's actually impacting the community all right so how many of you have seen a kill chain before be honest all right how many of

you have seen lockheed's marketing team here all right good so uh kill chains are actually really really important uh very much like mike klopper and the guys that came up with the first one from a digital perspective when looking at an ics cyber attack i want to set the bar here right up front of what's different i authored a paper at sands with mike asante titled the ics cyber kill chain and it the whole point of it was to say look nit and enterprise and anything related to it that whole kill chain thing is really useful it's not the preventative whatever let's do predictive anything kind of model it is just let's put data into

buckets and observe patterns and i can do intrusion analysis by putting data into buckets that's the whole point of the model but in i.t it kind of stops down on that actions on objectives one of the things i would note for industrial control systems is that's when things get interesting and in an industrial control system context all of that stuff up front is just the first phase of an ics attack the next phase is after they get all of the information that might be useful this is where we see them have an opportunity to pivot to stage two this is where you would have to develop something it could be knowledge on the systems and

it could be a malicious capability or both you're going to have to test it because any any smart adversary and not all your adversaries are smart i've met some of them uh but not uh but they're going to any smart adversary is going to test out their capability because you're not going to run six months of operations multi-million dollar campaign and then go all right hope this works click like that's not very likely to occur but testing actually doesn't have to be in a lab it could be in other people's utilities i've seen that where very small electric and water utilities who say i'm not really interesting no way anybody would ever target me they can get compromised and just be

used as a lab for somebody that wants to set up their environment to test out capabilities instead of setting up an expensive ics environment then there's going to be that delivery and installation and executing of the actual ics attack i'm going to flip back and forth here real quick this to me demonstrates that there's an extended kill chain when it comes to ics there are more things an adversary has to do to achieve the types of attacks we're most concerned with and that's one of the first key points i want to highlight there is a lot of rhetoric in the community today and i say the larger community not even just information security that the grid is going to go down or

oh my gosh uh the pipeline is going to blow up and kill people it's really really difficult like way more difficult than people give credit for it like well i could use metasploit and pop the human machine interface and i can get remote access cool now make the lights blink there's a big gap between those two things so there is an aspect that goes into these type of attacks that's really really important to understand for any type of real defense so let's use one case study before we're going so for those of you that aren't super weird like my team and i that hang around with things that beep and and spin large turbines and things on a

day-to-day basis a power grid uh the easiest way to think of a power grid make it real simple there's generally three components to it generation transmission distribution gonna generate lots of power either coal nuclear wind uh water all sorts of different opportunities i'm going to transmit that electricity over the big big wires that people that live to it they're like the nsa spying on me and it's like no that's just a transmission line calm down and then there's the distribution side of the house distribution is your local sort of neighborhood um for the last uh 30 plus years the security's been prioritized on generation and transmission because if you wanted to do the most damage that's where you target

makes the most sense um so back in 2015 there was an attack in ukraine that was the first attack that actually took down portions of a power grid uh and it was done entirely against the distribution network so it was interesting because that's kind of the soft underbelly around the world of what's not been as protected so let's break that down along that model real quick and then we'll get into some of the fact and fiction and what we can do about it so what i've done on the left hand side is structured that stage 1 and stage 2 kill chain on the right hand side i structured out what happened in ukraine and i put the full report up at the top

i was fortunate to be one of the investigators and got to uh write the industry report on that attack and so i'll walk you through really quick how that went down to bring some context to what we're really talking about here so about six months before the attack actually occurred which was december 23rd 2015. so about six months before um the adversaries targeted a couple obligos which is their word for energy company around ukraine fairly simplistic normal things that you would expect let me send you an email you know tell you that you should open up the attachment when you open up the attachment for the exploitation it was just social engineering they said hey

if you want extra features click enable on the macros and everyone's like i totally want more features and when they clicked enable on the macros it dropped a piece of malware called black energy 3 to the systems which then got access gave remote access the adversaries come in do some lateral movement all the kind of things you would normally expect that's what it would look like traditionally in an it environment that's run in the mill nothing too crazy there where it got interesting is that second stage so what the adversaries ended up doing was they found vpn access into the ics so they found remote access into the industrial control system networks themselves and for the next six months because all

of that it stuff took a matter of like three days for the next six months they operated inside that environment doing what a good defender would do they mapped out the environment tried to understand it profiled the devices figured out what they were dealing with they learned the systems and it took them about six months to learn these three different distribution management systems across three different regions of ukraine when it came time for the actual attack when we see it finally uh on december 23rd uh there was a lot of other stuff they had done beforehand so if you're just waiting for that attack you've missed all the actual activity right so if we look at what they

actually did a couple cool things a little bit of hat tip to the adversary first they found that there was these devices called serial to ethernet devices so if you ever play around with industrial control systems a lot of serial protocols still out there one of the devices that you need to be able to communicate from something like control center that's communicating over the internet or ethernet would be ethernet packets but down at the substation level where they're doing distribution probably got a lot of serial so there's a device called a serial the ethernet converter we're not very creative in the ics community but anyways so serial ethernet converter and they found these devices there's

about 30 of them and if you've ever done a firmware update to a control system you just fail sometimes it happens it's not a big deal just reset the system and stores it in memory flush it out not a big deal so if you ever try to upload you're going to fail sometimes these guys found the 30 devices and develop specific malicious firmware for those devices the reason i feel pretty confident in saying a specific malicious firmware is because and the reason i'm confident that they tested is because when it came time for the attack they had a 100 success ratio of bricking all the devices in exactly the same controlled manner that's pretty good remotely right that's

that's pretty good too so anyways developed out tested they also had to develop knowledge on the distribution management systems themselves because when the real portion the attack all they did was remote desktop out the human machine interface the little graphical thing that shows you how to open breakers and closed circuits and things like that this remote desktop it out and use the systems against themselves they use the naturals ics systems to actually disconnect the environment they also modified the ups so they had a backup power system and they said hey when the power goes down we want you to maintain power but then i want it to set about 30 minutes into it where it drops power

then reboots all the systems and then drops it again and drops the network interface card which seems kind of weird until you realize that a piece of malware they put all over the windows environment between 500 600 window systems a little piece of mower called kill disk that it was set to activate upon restart and it deleted the master boot record and all the systems so if you're the defender around 3 30 in the afternoon because nothing ever happens in the morning around 3 30 in the afternoon your mouth starts moving in front of you you're like oh this is not good one of the guys actually took out a cell phone to record it was like my boss is not going to

believe this so you see the little mouse moving around and start seeing breakers starting to open up those breakers and starting to de-energize those substations tries to fight it back and if again if you're the defender try to fight for control get locked out power you just start seeing it go out across the region then the power goes down then it comes back up and like and then all your systems delete themselves it's like uh oh uh in in that moment that is a tough position to be in and i was actually pretty proud of the ukrainians because they were able to get back up their operations in about six hours which is amazing but they did it by

going to manual operations it means they had engineers that went back out to the field and knew how to manually control the substations they left the automation the windows stuff the control systems and went to manual operations the reason i highlight that particularly because i've heard a lot of wrong narratives come out of this attack if you're at all interested in you go and do your research you will find some camps of people who will say oh well it was pirated software or ukraine's just insecure or as tim conway says that's that's the other side of the internet like that couldn't happen here uh and while i'll note that the american grid has done fantastic job of actually doing

security uh and they've definitely upped their game over the last decade nothing about that attack is specific to ukraine you could repeat the exact same thing the difference is it would be harder to take us down but in my opinion keynote opinion it'd be easier to keep us down because a lot of that expertise and manual operations you know that's leaving the workforce we've got a lot of people getting a little salt and peppery in the beard that are not sticking around for ipad hmis and we're seeing a lot of young people come in the workforce which is great that are heavily reliant on automation and what their screen tells them so in those type of scenarios the

ability to go back to manual operations is a skill set we're losing over time so this is why defense matters to me this is a great example of quit focusing on the malware it wasn't malware that took down the grid it was human operators using the systems against themselves so cool little case study let's get into the fact and fiction and some of the research that we're doing so a little bobby here talking to cso the cso says i need you to ensure no attacks occur little bobby says well you know what's our budget he goes oh no no we had to cut all that budget stuff out little bobby says well i can assure you we will never see

any attacks and to me that is one of the interesting things about the ics community i continually hear hey if it was so you know important for threats to be able to compromise the grid why don't we see more outages well sometimes things happen that we just blame on stupid activity that might be cyber related but also there's a geopolitical context to doing those type of attacks and it's not really in the best interest of some states to be able to do that all the time that being said there's a huge lack of research in this community which drives a lot of the lack of visibility into the threats so let's look at what that does to the

community when there's a huge interest by a populist so a lot of people are interested in ics maybe not you all i see a couple glazed overlooks we'll see we'll get there don't worry good there all right so um there's a lot of interest and not a lot being reported which leaves this chasm this chasm is naturally filled by hype people just make up stuff happens all the time like i'm an ics expert and there's 30 000 attacks around the industry look in the background they're like the chief marketing officer for some startup you're like okay so you gotta be very careful in the community one of the first cases we saw where this occurred was 2011 at the

illinois water utility so those of you in the ics community i see like jim and a couple people like smiling this was a fun one so what had happened was uh it was the year after everybody learned that ics was a thing because stuxnet happened in 2010 and everyone's like ooh scada and people were really really interested in the illinois state fusion center was like let's watch for attacks because that's what they should do and what ended up happening is a pump failed like a water pump out like now like a little small station failed and by epa regulations by state regulations they had to report up that failure so they said hey we had a pump

it failed and the state fusion center the people getting involved were like i bet it's a cyber attack and there's a lot of reasons pumps fail cyber attack is not really likely uh anyways they're freaking out and they go and they drive out they're like show us your logs and they actually have logs which i think is pretty awesome anyways but they had logs and they were like look in the state fusion center people and some of the folks that got involved looked at the logs and they saw that there was a russian ip address three months before the attack which you would think like maybe that's just not the right time correlation anyways three months before the attack

russian ip address logged in with credentials that he obviously stole and accessed the pump and three months later it failed that's a correlation causation fallacy but either way they're all freaking out they end up leaking it to the washington post and a couple others because you know that's why we can't have nice things uh and so you know leaked it out to the media the media is going wild and bloomberg publishes an article russian cyber attack or maybe it's the iranians pretending to be russia then they're attacking because they're pissed off about stocks in that and this is just war and it's like whoa calm down guys uh the guy who did it the guy who actually remoted in he was

just on vacation in st petersburg a couple months before and so he logged in because they asked him to and he learned about it in bloomberg on online like if anybody would have just asked him he'd been like yeah that's totally me and um turns out there was just a lot of build up and residue on the pump and that's why it's failed if you looked at it and saw the old build up and i was like 20 years old you're like that bum's going to fail so a little bit of hype uh the chattanooga apt so this was fun so this is steven this is steven hilt uh so bloomberg again it's not always

bloomberg it's just always bloomberg anyway so bloomberg comes out and uh they said look we're gonna do this research to find ics cyber attacks so they started anchoring themselves and how they were going to look in the community and they said we're going to look for this stuff and we're going to partner with this uh honeypot vendor this threat stream now they're anomaly they do better stuff now that they're anomaly but started off and they said we're going to partner with a honeypot vendor to build ics honeypots and we're going to look for nation state attacks which is like correlation you just shouldn't do out loud but anyways so stephen hill they published the article saying

look at this the number one at the apt in the world is the united states with over 6 000 attacks on control systems coming out of chattanooga tennessee because that's where we put the big nsa headquarters anyway so um everyone is freaking out they published a national article about this and stephen hilt finally pipes up and says no i i think that's me and they did like research and that's what it was it was stephen he was doing research ahead of a talk to look for ics connected to the internet and he found what in his words a really crappy honeypot that was trying to pretend to be ics so he just pelted it with scans

and and they started indexing his every scan against the honeypot as an attack and reported this out so you know maybe some qualifications are needed can't get through a hype talk without talking about norse all right so uh norse had put out this article one time i get this phone call like three o'clock in the morning like i wasn't still in the government at the time it came out and i get this phone call like ring ring ring three o'clock in the morning like hello like is iran coming i'm like what like is iran coming i'm like this is a really weird phone call man like i don't know what to do right now and they said well this is so-and-so on

the national security council and i don't have to tell you much about military ranks and say captain lee national security council reporting the president there's a gap there's like a huge gap i'm like why are you calling me um they're like well we got this report about ics attacks you do the ics stuff as if the government only has one person and they're like we're getting ready to brief the big guy off this report we got to know if iran's preparing a cyber offensive and i i'm like this is not a secure line it'll take me like 15 minutes to get to work but i can tell you like no it's coming from a vendor report i'm like

go ahead and just send it on over and i get this report from norris that it was bad uh basically honeypot data again and they didn't know what ics was they were like all these scans against this specific range and this port is ics and it's a television system i'm like well first of all port allocation isn't what a control system is i was like but second of all um tcp or udp you guys what the cto of norse is like why does it matter i'm like oh oh no no this is going to be embarrassing for you i'm like is it the tcp or udp like i was like it's tcp like uh udp is the port for televis the tcp port

on the same thing as the symantec updater that's not a good report and the interesting thing is they ended up putting out this article uh later on as well they decided to run with it because why not you know don't listen to me and they ran with it with a think tank called aei they fred kaplan those folks authored part of the bush search strategy very influential folks in dc very right-wing conservative think tank and uh they made policy decisions they made policy recommendations saying hey the nuclear negotiations with iran which were happening at the time should fail because norse's data proves that iran is going to use the relief to attack control systems the united states

so the hype has impacts it's it's hilarious but it has impacts policy makers look to our community ics is a topic that everyone's interested in and almost no one seems to have experience in um so why does this matter again well we see another case where bloomberg said it's not always bloomberg but it kind of is um bloomberg puts out this article saying hey this pipeline that exploded in 2008 which it did it did explode in 2008. they said we found out in 2014 that's a long instant response case by the way i would love to have that hourly rate anyway so uh in 2014 we figured out that it was a cyber attack by russia like okay dig into it

i'll skip to the punch they're wrong and uh turns out they found like a piece of malware in a control center and just assumed that the explosion that happened must be related to the malware because there couldn't possibly just be malware there and it beaconed out to a russian ip address they're like it's russia like that's not how we do attribution and the other interesting thing about this though and i try to kill this story at every conference i go to because turkey came out at the time and said it was the kurds they did this and the kurdish extremists came out and they were like it was totally us and like seven years later somehow

russia's getting blamed so the reason i bring this up though is i've seen congressional members reference this story as a reason to regulate the gas pipeline industry now if you want to have a regulation discussion let's chat but we don't need to be seeing fake stories causing changes in our community and again why does it matter well uh i'll give you i'll give you a couple more quick case studies to move on but in turkey uh power outage occurred back in 2016 2015 and it was a 10 hour outage across a couple different plants and people are freaking out like surely multiple plants can't just go down this happened recently like san francisco and new york and they're like

it's gotta be a cyber attack i'm like you have no evidence like it could be a fish i don't care just don't make it a cyber attack um turned out to be a circuit breaker anyway so uh in in turkey folks major major news outlets cnn um bloomberg was there to be like cnn and stuff other people are coming out and saying power outage in turkey and based on the norse data about iran and based on the bloomberg report about the btc pipeline this might be a cyber attack from iran to turkey i want you to let's let that sink in for a second for the first couple hours maybe days if you're the host country you have no

idea what's going on it takes time to do investigative work and for that first day or more you're getting told by major news outlets that it's probably a cyber attack from a country you're not very friendly with that's a very interesting geopolitical tense time over nothing but hype um israeli cyber attack that never happened basically the they were having a conference in tel aviv being like tel aviv is pretty sweet at cyber security because that's what tel aviv says and uh the guy one of the ministers for the government gets up and he's the minister of in charge of energy and water resources and says right now as this conference is taking place the israeli electric authority is under

the most severe cyber attack israel has ever had oh my gosh so again cnn and everybody else ran this screenshot ran this view israeli power grid suffers massive cyber attacks what he forgot to leave out in the details was the israeli electric authority is a regulatory body that doesn't even touch the grid it's just a little office with like 30 computers and it was a ransomware infection because they opened up an email wrong so maybe again maybe a little bit better and then this one last one this one was recently um we started seeing this and luckily ron fabela and some folks jumped on it very quickly we saw this article come out saying clear energy malware is taking advantage

of these clear energy vulnerabilities by the way don't name your vulnerabilities and if you do don't name it the same thing as the malware everybody hates that just please all right so uh clear energy malware what it does is it locks down control systems infects them erases the logic and then shuts off everything that's crazy for an ics person that's like that's not good um what they didn't tell everybody is there was no malware and it did nothing seems to be a gap um what they were trying to say well i mean they're trying to say that but what they should have said was hey we found some vulnerabilities we weren't getting enough press coverage

so we made the malware ourselves and ran it in our lab to show that it still doesn't work in the lab um so again a little bit of hype there i feel bad for the company because i reached out to them and they were totally friendly about it it was just marketing gone wrong but but get a get a hold of your marketing team all right so let's start transitioning out of the hype stuff here's an abbreviated history of some ics threats first of all insiders people with actual knowledge they've done some stuff before right in the merushi water case if you're interested to go look it up basically reversing the flow of sewage by a disgruntled employee

um not awesome lots of stuff can go on but if i asked any general audience and i said point out the one that's been most impactful to the community everyone will be like stuck snap or you know like start whispering to each other and rocking back and forth stuck and uh we're talking about any of another one like well dragonfly was far-reaching or sand worm they took down the ukraine grid um the number one impact to the community is probably been like configure like incidental incidental malware infections i love to explore nation state stuff because it's awesome to ruin bad people's day it just is fun you spent two years developing this and i'm just going to write a snort rule for

it that's awesome like this it's fun but what is the actual impact in the community it's the incidental stuff it's the operators you're bringing in usbs it's the laptops that are still dirty it's the vendor connections with direct vpn access that get infected people like why don't you just patch your system it's not that easy don't start there or i always get this stuff like ics is totally insecure they have default passwords it's like yeah but what's more likely the operator forgets his password and accidentally kills somebody or russia gets down into the control system network and goes damn a password like you know it's a risk perspective um but but that's where the mimics research

this is stuff that i'll talk about for the last half of the presentation basically we wanted to figure out what are real metrics around this what are what are we really seeing in the community that's not getting reported and the big question is why so let me let me give you an example of why so you already heard the hype piece but what about the real numbers so whether you like the metrics or not and there's a good debate to be had the most authoritative metrics previously that have been put out were from the department of homeland security's ics cert so the ics cert they do a lot of cool things in the community they're very

very proactive good folks but every year they put out their metrics of the incidents they're seeing and they've said you know last year we saw 300 instances the year before that 260 so forth so forth energy is the number one targeted sector no it's not you're just actually like focusing on energy so you see a spike in metrics but either way they put out this every year and it's a little bit hard to see on the screen but every single year the number one attack vector like how do these things occur the number one attack vector every single year in those metrics is unknown they have no clue number two which makes up the big green spot is

spear phishing and everyone's like yeah totally makes sense no it doesn't we don't have email and scada environments so what it's saying what the metrics actually say is when we see something it's because i.t caught it going through the business networks otherwise we have no clue what's going on and that is a problem because without that visibility without understanding how the attacks are taking place i understand how the incidents are taking place you leave that chasm for the hype we just talked about but you also just copy and paste it security solutions and i t security best practices and the ics because that's what you know and it's not necessarily the stuff that actually fixes what's going on

um so this is these metrics are what really started this idea of let's do some research here and i and i know every every time i present these numbers i always get at least one person to go well dell secure work said this or fireeye said this or these big vendors that are trustworthy said this let me i mean i like those vendors too but let me draw a big disparity when your av company puts out or your endpoint security solution whatever quality is the ai it's not a thing but whatever anyway so when people put out and they say here's what we're seeing and here's the intrusion analysis and here's our big threat intelligence

report good threat intelligence reports are all based off intrusion analysis you see the attack taking place you patterned it out and you have observables that has existed in it because we've had those vendors that have endpoint security solutions and ids's and firewalls and av and whatever else and it reports back and it says here's what we're seeing and then analysts can use that data we don't have that stuff nor have we ever had that stuff in ics so your big vendor who is trustworthy on the i.t security solution side of the house and the threats that it's seeing don't have the visibility there's not giant reams of instant response data for what's going on inside the ics so that's

that's part of the problem where we don't have that level of visibility so let's get into what we try to do to find this out so i had metrics on one hand which were the ics cert saying there's 260 to 300 incidents that occur around the community each year and i used to be in the government so i generally don't like government metrics and i was like well you probably need to work on those and on the right hand side i had vendors we'll just discount norse but we've had like dell secure works come out and say we saw 614 000 ics cyber attacks that was their 2015 data and i'm looking at it going

no you didn't like you just lights are on water is working something's wrong with these numbers like you're counting port scans on a firewall i guarantee it um so we've got this chasm between 600 000 on the upper end and that's about where the metrics all sound because nobody likes to report small metrics they like to say we saw super advanced stuff nobody ever gets compromised and like dude it was basic like i don't even know why the security team's still here like it's just nobody does that right so it's always super advanced and really jacked up numbers um so that's what we were working with that's what we were trying to combat um we got little bobby here saying

um alice walks in al says what are you doing little bobby says well i'm installing next-gen software to protect me against foreign governments and alice says why is there a pop-up saying you still need to patch your system little bobby says i'll get to that later i i do this again ahead of the sort of incidental infections and i'll say this once and i'll say it very bluntly your adversary has powerpoint in management too right they're human threats no adversary like i and i'm i'm allowed to say that i was on offense not like pen testing red team so i was on offense uh in the government uh after i did intel and defense and at

no point in my day did i think to myself hmm how do i develop malware or do this campaign in a way that's really going to impress the defender like that's not my consideration like it was not to show up in the kaspersky report and look good like that was not a consideration operations right it's get on with your day you've got a lot of other things to do so basic works and if you're smart you stick to the basics because if somebody gets compromised with a basic piece of malware they don't call the big instant response firms and they look the other way and if you're an adversary that's good so let's talk about some of the basic

stuff and trying to return the metrics so what we did with this mimics project conveniently named mimix basically off of malware and modern ics is we try to have some census data because i talk about the hype a lot and trying to combat the height but i want you to have real numbers so people ask you questions you have fact driven numbers that you can talk about we looked only in virus total and only in public databases because everybody uses virustotal as basically their free you know sandbox which is bad please stop it unless you're in ics keep doing it because i like watching but like you know i'm kidding don't do that but anyway so use the virus total data

and your first question is probably well how many ics locations are actually using virustotal a lot and even if you don't think you are at the end of the presentation i'll show you some of the biggest offenders the biggest offenders are your av companies and your outsourced i.t security teams to be an av company in that virus total list you have to submit tons of malware back so they sanitize your data and then submit it and the it security companies that are outsourced a lot of the smaller ones i've seen just do a bulk api submit off every file they find the virustotal which means they're exfiltrating your data for you so go back and look at your service

level agreements to make sure this isn't going on anyway so that's what we worked with and like all good hunting we started with a hypothesis he said what do we want to do so he said we specifically for hypothesis one want to find non-targeted intrusions just incidental malware infections in industrial control system locations not honey pots not enterprise networks of an energy company specifically in industrial control systems themselves to do that we developed euro rules there's some classes and stuff going on for you are as well this week or this this two days yara ends up being a great tool basically just pattern matching uh kind of like grep on steroids but we looked and developed jr rules

specifically for the right ics file paths software versions taking some of that knowledge of what ics equipment is and when it's actually installed in the right file path and not by a researcher and looked in those locations across the data and then mined out what we could find we ended up looking at 15 000 samples over three months so we found 15 legit 15 000 legitimately infected ics software so already installed in the location already running properly already properly set up in in the different environments 15 000 infections over three months but that's not a fair metric and it needs refined because first of all we can have duplicates if a virus gets in a

you know an environment infects it and you submit it you might have a hundred submissions for one infected site so we then started going down the list and looking and grouping them into different industries and the different vendors around those different industries and looking at different locations and trying to de-duplicate the data on a conservative estimate we found 3 000 unique sites across the three months that we did the analysis so we extend that out to say in any given year there are likely three to four thousands we're very very conservative on the numbers likely in any given year three to four thousand industrial control system sites that are getting infected just with incidental malware so there's

stuff going on it's not as low as the 200 numbers but it's also not the 600 000 numbers so a nice base census metrics to work on um i like word maps because they look crazy but this is the only thing i wanted to highlight when we looked at the word maps of all the different infections and what they were that giant arrow that maybe you can barely see that giant arrow is pointing to the word stuxnet it's not well represented in the data you are more likely to get infected by civis and configure than you are by stuxnet so quit focusing all of your resources on i wanted to defend against the nsa fsb 8200 and gru and start thinking

about the operators bringing in usbs and what you do about that because it turns out it's the same when you do defense you're raising the bar against everything anyways so these were the most common infections just for those that are grabbing the slides after the conference they want to look through them all basically just looking at some of the highest hitting samples of malware and go figure a lot of it is virus type malware stuff that's just spreading and propagating interestingly enough though there was lots of trojans as well that weren't just spreading around so you would have to have a little bit better infection routine instead of just usbs and things like that we ended up finding some things like

trojanized versions of ics installers that were being targeted against environments we'll talk about a little bit later so base metrics for everybody to use going forward three to four thousand in any given year incidental infections not non-targeted so about what about some of the targeted stuff can we can we try to figure out if there's any ics themed stuff that shows that the adversaries are maybe uh investing some time in this and are specifically targeting but not using crazy level malware so we looked and said you know what there's very little known about this if you look into the ics community today and you look for ics themed malware infections they don't get a lot of press right it's

not like the big threat intelligence report coming out um so there's really only three that you can find that are referenced at all operation electric powder was an israeli company that put it out good good really good work by them looking at some specific targeting of an energy company didn't affect the ics didn't get to the ics but an adversary who was very very much targeting this energy company in israel we saw ransomware masked as a rockwell update um so that was interesting but it was just the zip file was named that it wasn't anything crazy and then we had irongate from the fireeye crew which was a cool discovery but it was a point of it was a proof of concept

it looked like maybe a researcher made it they submitted it themselves the virus total was never in the wild never infected anybody so two days before the project we haven't seen ics themed malware that was actually infecting ics sites oh that's a problem can we can we return some of those metrics to the community so we looked for theming around ics like if we were an adversary and we were going to target an ics how would we do it and how would we do that theming one of the first ones we found was back from 2011 that no one had reported on and it specifically relates to the nuclear materials management and safety organization so if you're in the

government you deal with nuclear equipment you deal with nuclear power generation stations nmss is a group that you're very familiar with and we saw back in 2011 that a phishing campaign went out against select nuclear sites themed specifically for them didn't end up impacting that we could see anywhere else and it was just a regular trojan nothing crazy again it's not ics tailored malware it's just ics themed malware but by all accounts showing a level of uh understanding the community and targeting the next one we found we ended up finding about 13 so there's about 13 out there that we found that were specifically theming towards ics and targeting them the one that we found that was pretty

interesting though is the siemens themed installer i want to be very clear this has nothing to do with the siemens folks like is siemens systems insecure it has nothing to do with this right this is just theming like i can theme it for anything but the adversaries decided the theme for siemens themed installers specifically for the applications that work on the control systems and interact with the control system themselves because if you want to bypass all the defenses why not just trojanize legitimate software that they're going to install anyways and what was interesting about this to me is we found 10 different uh binaries 10 different samples and they were different locations starting back from 2013 to 2017.

right we actually thought we stopped the research in march and we found a sample still in march so for four years an adversary has been theming their malware specifically for siemens installers and hitting different locations around the world so basic stuff it's nothing crazy it's just uh execute on the environment reach out to the internet grab the right payload and be able to execute and run on the system does it take down the power grid no it just gives you remote access right it's it's a remote access tool um for adversary so all that stage one kill chain stuff nothing stage two so my god the power grid is coming down none of that where in the world were we seeing it

based on the uploads and based on the environments that are getting affected we saw a majority of which six of them were in the united states we saw one in a european country and we saw two in china the interesting thing about this as well is it might just be a selection bias off collection because how many chinese-based companies are using virus total probably not as many as american-based companies so there's there's a selection bias in there but i just want to note again from a moving from hype down to we can have base metrics back to actually seeing that targeting is going on so do we need to be taking care and watching and researching and thinking

about ics security yes the adversaries are interested is it because the power grid is going to go down at any moment no there's there's the whole sliding scale in there of just responsibility around this um as i close it out we'll look at the user behavior and poor operational security we have some good talks coming up about virtualization we've talked about opsec later today um some good stuff with oset and micah we've got some great stuff that'll help you with this next section and this is why i wanted to end it as i looked at um the the different talks today i feel this aligned very closely with a lot of the talks that are going on

so our third and final hypothesis was why is this data getting up there right like how is this happening and our hypothesis was that non-ics security trained teams were just submitting all this stuff that they didn't understand and it turns out that was what it was also your antivirus is not really good at picking up what legitimate and bad ics stuff looks like there is a lot of false positives so we found that completely legitimate ics software was routinely very very uh routinely getting popped as malicious software by av it's another reason a lot of people don't install av all over the ics because av companies haven't shown that they understand what ics stuff is

and they just just destroy all of our equipment so that's one of the reasons anyways so from nothing but the public data sets of the legitimate stuff not malicious now we saw over 120 different project files in those three in those three months so people that were submitting the actual logic that goes on to the control systems with their internal very sensitive data directly onto these public databases we saw stuff that definitely shouldn't be going up there that was getting flagged by antivirus so this document got flagged as antivirus as being malicious and so it got submitted i don't know why it didn't get sanitized correctly surely av companies would never do anything wrong but uh got submitted and it was a very

very sensitive document around incidents in the nuclear community and we fuzzed out the details but it was identifying incidents that were going around the u.s community and specifically i think two sites yeah two specific violations that occurred in the nuclear sector that's not information we want on a public database right that's not that's not typically a good thing um we saw a lot of substation layouts and things that were getting submitted as flagged by as malicious by antivirus for whatever reason uh and it was like here's exactly what our network looks like or here's exactly how our systems are set up or here's all the internal ip ranges that we use to manage these systems with all these

passwords it's like oh my gosh don't put that up there we saw a bunch of installers if you were an adversary wanting to transition to that stage 2 of the kill chain you would need to test right i told you like one substation in maryland's baltimore region completely different than the bg e substations that are just down the road all these things are set up differently even if they have same vendors the physical process of engineering is different so you'd want to be able to set up a test lab we found all the different very very expensive installers and key generators for all the major vendors so you just download their software with the key generator and have your own

environment how expensive is some of this uh as an example i wanted to do some research into ge simplicity equipment one time and i asked because dragonfly was targeting ge simplicity i'm this threat group but i said hey ge i'm a researcher i promise i'll behave can you just like give me a copy of simplicity and they're like oh we have a student discount so it's like awesome i want that how much is it and it was more expensive than my house and so i was like no i don't think you and i have the same definition of students um never mind but now i can just go download it for free with the key

generator so we don't want our adversaries have access to this stuff we had one utility that was indexing their entire public website since 2012 and indexed 130 different file directories and paths onto virustotal for the last five years so that's that's good again what i would note for some practices here number one use vt as a data source if you're a researcher in this community you're trying to do some fun stuff i get you excited about ics you're like i want to do some ics stuff virus totals a great place to start also i have a blog robertinly.org there's a blog like four down that's just a whole resource section of so you want to get

started in ics it's like here's all the youtube videos there's packet captures here's everything you can get going on we need more people in this community i want you to come to this community well informed if you find something and you're like oh my gosh no one has ever found this before and it's a major nation say attack and it's going to take down the grid hold up all right like no one's ever found it before and it's super super important maybe let's let's put some thought processes into it right i don't want people going out uh talking to media about their latest discovery to find that it's a honeypot at steven hill up in your honey box um so use vt as a

data source grab some installers and things off there if you want grab some test environments to go up on i don't know if that violates a dozen terms of service but technically you're not clicking through the terms of service so i think you're okay i'm not a lawyer and don't submit your stuff to it because again what we found was it's all the outsourced teams all the av vendors not all of them are bad by the way i have a lot of love and respect for a lot of av teams you look at a lot of threat research it's those guys that are doing some awesome stuff um that being said though it's just not made for ics uh so be very careful

there so some key takeaways as i end out uh what we're doing here first of all industrial cyber attacks they're worth understanding it's understandable that a lot of people are interested in them and you're probably a smart technical person by being here statistics would say not everybody in the room is smart but let's give it to you like everybody's here is probably smart um i imagine that your peers your loved ones the people that you know look to you when stuff happens like can you explain this so ics is going to be one of those things that's probably useful to be smart on because questions get asked about that a lot security in ics contributes to

reliability we generally see that even simple things like infections can cause issues in environments shutting down operations i've seen uh two major news headlines come out around a piece of malware was found in a german nuclear facility and shut down the facility like the malware didn't they shut it down for safety reasons but yeah that's one of like three thousand that's happening each year was our point um so there's some some base metrics for you to work from some some opportunities to do some good things and have some fun uh and generally a little bit better understanding i hope of ics attacks and sort of how they occur so with that uh if you want to be in

touch uh my twitter handle is up there my email's there dragos is actually a maryland-based company so if you ever want to come in and play the control system stop by we've got little bobby here that says you look matt says you look like you haven't eaten in days little bobby's like i haven't but then matt says why not he goes well my fridge toaster and oven are all infected with ransomware but at least my artificial intelligence is protecting me from china so have a good understanding of all the fun stuff you're going to do these next two days but definitely have a good understanding of risk as well and trying to say maybe i don't need to stop every single

nation-state attack maybe i need to start addressing some of the issues that are actually causing problems in my organization today so with that i think we have time for like two questions at max and uh thank you so much yes [Applause] all right yeah no that's good yeah i'm kidding all right thanks yeah what's up yeah and the data set and everything else as well just hit us up yeah yeah no problem we've been sharing it up to the different uh researchers that wanted to work on this so no problem the what oh the new york dam that was a funny one so since jim's being a troll up here um there was a new york dam that got hacked

you've probably heard about that one as well maybe they were like oh my god the iranians and the doj actually did an indictment on it and made a big deal out of it and said these iranian hackers hack this damn so here's the real story i had to go on fox news for this i never go on fox news if you i don't care about your political biases and leanings and stuff but it's not a news source i'm super excited about but they were like we're going to talk about this you can come on i'm like all right and so they're hyping it up they're like a nation-state cyber attack on a dam in new york and uh the iranians

are attacking this is war i'm like oh my gosh so i get on there i'm like yeah it's not how it happened and and you just see his face like he just had like the senior member of the dhs telling him that it was a big deal and here i am being like nah that's not how it happened um it was actually a case that i worked in the government was previously classified but you know everything gets leaked these days they declassify the details so i'm allowed to talk about it uh one day i get called up i'm sitting over in the intelligence community i get past some data and i get told iranian actors are targeting

infrastructure in the united states can you look into it like oh okay and it validated and really was the iranian actors and so the attribution was solid in the case at least from our standpoint it really was iranian actors targeting some infrastructure sites but i looked at it and it was just what looked like a basic hmi human machine interface with no elements of control you couldn't do anything i mean the iranians could stare at it really intensely but it wouldn't do anything and so i passed back to the dhs i said yeah it's this place it's called bowman dam but there's like 53 bowman dams in the united states or something like don't go

like don't freak out whatever you do don't freak out what is the exact opposite of freaking out briefing the president yeah feels so bad for president obama you know like i'm sure he just got tired of cyber people uh but so the dhs and in kik and everybody else briefs up to him being like there's an iranian cyber driver against a hydroelectric facility united states so this is not what i said at all and uh they call me back and they're like hey we need you to investigate this pretty quickly and i'm like no it's really not a big deal guys and they're like no the president wants updates every 15 minutes i'm like what

like what did you what did you do um so we usually joke that there's no black helicopters that people fly around in the government this was one of those times there were because we called up atm t and said hey because it was a cell provider that was hanging off the hmi i called up atm t and said hey um can you tell us where this ip address is because the way you do allocation i have no idea and they're like no i was like please they're like no they're like damn it and everyone's always like the intelligence community spies on you like we can't even do who is so anyways uh like we're looking at it and we're like

so finally got at t to say like what quadrant of the united states it was in and while the fbi and everybody else is flying out to bowman dam and i think oregon or nevada region because that's what they told the president and they get there and it's and it's a damn that if you had managed to obstruct like six seven thousand people downstream from it the problem is it's an earthen dam like you're not you're not hacking dirt right it's like exactly not gonna go well for you uh so anyways long story short find out it's this bowman dam in new york they fly out they talk to the guy on the phone the

the operator of it and say hey this is the dhs i didn't work for the dhs by the way this is the dhs you're being attacked by iranians when you get your facility which is like not the way to start a conversation because the guy was getting haircut at the time was like this is fishing and hung up like user training 101 right there anyways so uh long story short they go out there it's a dam that's maybe like knee-high if you manage to hack it you might flood some rich people's like carpet but it was not worth anything and it yeah made it all the way to fox news doj still went forward with

it but it was not a big deal anyways thanks everybody awesome oh nice i like that