Red Teaming a Manufacturing Network (Without Crashing It)

so a little bit about me I'm a red team lead at General Motors responsible for logical physical assessments country's testing and red teaming globally I've got a few certs there that you all are probably familiar with and that's my twitter handle everybody cares to follow and my name is Kyle QC I go by Kyle QC Kyle koosy Kyle coochi the last one is not the suggested version I'm a security researcher at a large unnamed automotive supplier and my responsibilities are basically security research for making automotive systems safer and more secure and yeah thanks everybody for being here and we hope you enjoy this this talk alright so for our agenda we're gonna start by getting level setting with some

terms definitions acronyms so if people are not familiar with the red team inside or if you aren't familiar with the manufacturing side we kind of level set before starting then we'll go into why red teaming a manufacturing environment is important and the little introduction into what a manufacturing environment and network look like some potential methodologies you could use we'll go into a step by step with a couple of attack vectors of a red teaming assessment some lessons learned common findings and mitigation measures so our goal for this talk really is two goals here to motivate organizations to do red teaming manufacturing assessments if there have them within their organization and also shed some light on

some common issues maybe for better awareness for defenders and for management on what these common issues are and how they can be mitigated so a couple of terms here retching it's not penetration testing I know there's a lot of talks on penetration testing and red teaming so we thought we'd give our terms here so a penetration test really the objective is to find all of the vulnerabilities so let's say it's penetration tests against a website you know find all the vulnerabilities cross-site scripting sequel injection on that one website you know write up a report exploit them and show you know here here everything wrong with your system on the other side red teaming is more of objective goal driven

so maybe the objective is to break in to set box and that's that's the main objective how you get there how many vulnerabilities you exploit and find along the way is less important a couple of the industrial cyber security and industrial terms so ICS industrial control system here we're going to use that as more of a generic term that includes all of those industrial devices embedded devices if you go out and read the manufacturing you know environment networks and resources that we have listed at the end there's a lot of other acronyms but right now we're grouping them for simplicity into ICS we have PLC's program logic controllers those are what execute on the manufacturing

Network control network which we'll show in a bit and HMI which is a human man human interface so a visual representation of how or how to control and configure some of those systems on the control network

so why red team manufacturer facilities attacks on manufacturing can inflict you know quite a lot of damage there's been a few more than a few recent vulnerabilities that have been exploited in manufacturing along with specific malware like wanna cry havoc stri in SATs neck especially sex an especially complicated and well large impact from what psychics had and shed a lot of light into the manufacturing environment red teaming can help an organization you know get a real assessment on how their organization security posture actually is so let's say you run a vulnerability scanner you're doing some of those other you know layers of security before going out to red team you know penetration testing you're gonna find

vulnerabilities you may not find how they're connected or what the true impact on the organization and network is to do that you would need something like penetration testing and then even more so red teaming right so how do these vulnerabilities work what is their true risk to the organization and how can they be exploited and used by an attacker and adversary to get a foothold into your network and complete their objectives so instead of you have cross-site scripting or you have sequel I you know it's more of a chain of events that would lead to what could happen if those were exploited quick introduction into manufacturing networks so the top layer we have here is the

internet right everybody familiar with that we got some users on the outside the firewall coming into the network the next layer we have here is the office IT network that's going to look like any other organizations network you know you have printers maybe you have normal servers clients people are doing work in the office these this is the network which they reside next we have a supervisory network this is more manufacturing industrial security specific there's different acronyms that you may see going across different types of references here it's pretty standard with supervisory you may hear DMZ as in DMC from office to control network there's the control network that's where the magic happens that's where you know

machines computers meet physical the physical world and our action creating processes or actions that are producing something Manufacturing something so we have the machines maybe some hm eyes on the left those robots physical robots acting on a process completing a task

so to define what kind of devices are in this network as I covered we have the the business servers the client servers they're in the off-site t network we're all familiar with that some supervisory systems and what's called historians on the supervisory network configuration databases for the control network devices how are those configured what are the specifications all those details that keep those machines running and doing their job effectively we have PLC's HMIS and then we have the actual manufacturing machinery so potential methodology for red teaming and manufacturing like penetration testing or red to mean anywhere else you have a scope a scope we think is very important within manufacturing if you take a look

at you know your environment within your organization are you a single are you going to take a look at a single manufacturing facility are you gonna go broader out look at the entire infrastructure of the network entire physical infrastructure of a specific facility you know logical physical all those determine the scope how broad are you gonna look what is your focus what do you want to focus on so constraints how do you communicate the testing to the facility the people within the facility are they going to be aware of it you know typically a red teaming they're not aware its testing active defenses testing for instance your monitoring capability and maybe your physical security guards are they following the

processes that you believe are in place in a manufacturing environment availability is key also with of course the other CIA triad but availability is important these devices are making something making typically a product for the company so you want don't want to interrupt that product being made you want to do your assessment but don't have any negative side effects on your organization also we put here no mass scanning tools if you're new to this it's best to do it learn the manual methods before moving on to the scanning tools a lot of the scanning tools are very aggressive and some of this devices within the control network may not play so well with some of these automated scanners

that look for everything you know against against devices here so you want to really understand what are those scanners doing if you are running them and it's best to really start off with not running those scanners or really running the basic low-level version for example let's say and map most people are familiar with that map you don't want to go t4 on these with scripts and identification and everything maybe you want to do an ICMP maybe nothing at all and we'll talk about some other methods of using without using tools of discovering the network and vulnerabilities and exploits once you're on systems and then objectives like we mentioned before the red teaming is mainly objective driven so is what are

your main goals what are your main objectives that you're trying to accomplish with this assessment gain authorize access in the network gain unauthorized access to sensitive data maybe find specific data within your network that your company holds as the holy grail and then gain maybe into the manufacturing process so influence the process that's being used to produce something some of the attack vectors we're actually take a look at here shortly are from the internet so anybody on the internet what can they see with your organization what can they do fishing as well as Wi-Fi and physical

so yeah there's lost ones quick note about scoping scoping is critical in manufacturing as I mentioned before there's a lot of systems a lot of sensitive systems producing a product or doing something that needs to be done it decides what systems are in scope and out of scope what kind of attacks and methodologies are in an out of scope right you don't want to do a do s probably hurt your own organization you want to come up with this area scenario and most importantly you want to keep safety in mind unlike other networks maybe their penetration testing or red teaming these can have real physical effects you know in the real world that robot or that

machine may do something physically and in manufactured environment there's a lot going on right there's a lot of moving pieces there's a lot of things moving and you want to be careful and keep safety in mind so with that I'll pass it off to Kyle yeah Thank You Johnny so Johnny already mentioned the four attack vectors were going for so this is just an example scenario that we came up with from prior research prior testing but this this scenario is it's kind of a good overview scenario for for what a typical Red Team assessment could look like okay so we're on the internet and we're trying to get into the into the manufacturing facility so the first

step we do is okay there we go the first step we do is we always do internet reconnaissance right this is this a comment in every penetration test in every Red Team assessment but it's even more important in manufacturing which will find out why in a minute but this is in a screenshot of a show dan have you guys you show Dan who's you show Dan show of hands okay cool so everybody's familiar most people familiar with show dan the show Dan is basically a tool to find devices on the internet that maybe shouldn't be on the internet or maybe they should be on the Internet in this case with industrial control systems sometimes they shouldn't

be on the internet so here we have use this little magic and wanna press anybody's I want to break it okay anyway so here we have a short and search of a oh nice okay no sorry guys oh my god I knew I broke it yeah oh yeah okay well anyway yeah so fantastic is my experience not not tech support so you know the short answer of port 102 which is a seamen sport siemens s7 usually uses it which is a certain kind of programming programmable logic controller and manufacturing and here on the right we see a whole bunch of different Siemens products on the on the Internet and if you notice something on the left

here some weird operating systems are in place here we have Windows XP old versions of Linux things like this so these are types of things you want to look for when you're doing your recon on the internet for manufacturing sometimes you can find a quick win just from basic internet reconnaissance with showed in you can see here on the right we have some interesting little values like this on my screen but yeah we have a lot of a Siemens information we have even a public key right there which obviously public key it's on the internet anyway but it's just information that you can use to to to further infiltrate this environment another awesome repository is paste bin and github also Google

Dorking which we can get into later but paste bin basically is is a public paste repository site where sometimes hackers will paste user names and credentials on this site a github is another one code repository things like this are really critical for for finding publicly available information that can be used in an attack in this case we actually found something kind of cool this was basically an admin credentials of one of the manufacturing systems engineers on the internet on paceman and we got lucky here and we were able to log in to the manufacturing remote ax appliance just by plugging in these credentials so that was pretty cool and so now essentially we have access to

the office IT network so we can't really do much with this though because we just have access to the the VPN appliance we're on the network technically but we only have access to some of the some of the web servers and things like this so we want to take it a step further by doing a little bit of fishing all right so what we have here is a it's a website for I can't I'm not going to say the company but it's a website for a common manufacturer of PLC's and hm eyes for manufacturing and on the in the middle portion here you can actually publicly a sex access the the firmware updates for this product so essentially what we

could do is we could clone this webpage and set up a fake fake repository for for updates and firmware updates patches things like this so that's what we did we cloned the website and we sent it to a one of the systems administrators that was that worked at the manufacturing facility and the same one that we found the credentials for online so we already had a lot of this information as well as an email address so we sent this information to this as a Systems Administrator and inside his email with a link to our our fake website with a fake update for a fake PLC so when the systems administrator clicked on our link and download our software we

created a sort of a malware exploit kit to to model what a an actual adversary that attacks manufacturing networks would do so we had a recon component and exploitation component and a persistence component no good talk going on there you guys should be over there I guess yeah so basically all ICS malware will follow this kind of format here usually once malware execute they will do some reconnaissance on the environment this is exactly what Stuxnet did it looks for operating systems operating system patches software that's on the system things like this once we have done some reconnaissance on the system and identify that it's running Windows whatever we can find an exploitable hole on that system to to escalate our

privileges and do some other things so in this case we we were looking for common Windows SMB exploit we found it on the system through our reconnaissance methods and we were able to exploit this this Windows X and B hole and establish or escalator privileges to that of administrator on Windows after this we establish a persistence method and this can be done in many ways some of the common ways our registry task scheduler cron jobs if you're on linux things like this we basically established a persistent backdoor that we could connect to via our command control systems on the on the internet so now we have we have internal access to this office IT Network so Johnny already

mentioned you don't want to once you're on the network you don't want to just start scanning and nmf scanning and necess cannoning and expose scanning and everything basically you want to be as quiet as possible not because there's no one I think it's people are watching you necessarily but because you don't want to crash anything it's very important and map is Johnny explained and map will crash in network Nessus will sometimes crash manufacturing network so you just want to do passive techniques such as Wireshark TCP dumb things like this in our case we just use net stat like the most basic Windows tool ever and we found that this this Systems Administrator was actually had an RDP

section session open to a system on the on the Supervisory network which was one layer down and that's one of our target networks so we were able to log in to this person's RDP session and access this HMI so this is what a common HMI looks like basically I'll just come over here so I can do this little thing here basically you have this is a so this would be for some kind of a some kind of a water pump or dam or something like this we didn't actually include a something manufacturing specific this is more ICS specific here but this is some of the common things you can do with an HMI right you can control the pump you can

control processes you can you can shut down processes start processes basically just just really screw with with the manufacturing process so that's a bad thing so now because of this we essentially own all the network we were able to get down to the control network this HMI connected to a robot on the manufacturing plant and we were able to control the manufacturing process I will pass it over to Johnny for the next stage all right so the next example attack vector we have is Wi-Fi and if you can notice as we're going through these attack vectors the opportunity of these types of attacks and maybe the impact is going is less and less because you know we started off in the internet

that's very wide open to everything fishing now you're on the internet but you've got to rely on somebody on the other side to have some sort of action take place now we're on Wi-Fi so this exact example attack vectored Wi-Fi requires you to at least be within physical proximity to some Wi-Fi right some sort of signals that you can get that doesn't necessarily mean you know you have to be right next to it or you have to be within the building there's great examples out on a lot of websites of really good Wi-Fi antennas that you can buy pretty pretty inexpensive to really boost your range from what a normal laptop or desktop normal cards

would have so for Wi-Fi you do have to somewhat be within the vicinity we'll talk about that in a bit a couple of things examples you'll see within manufacturing environments within general manufacturing is on average about like 10 years behind what most the research says of the normal security cycle so as an example that within this Wi-Fi you see WEP you see WPS being used in most organizations IT offices you know that's that's not common you're not connected to wet you're not still seeing that being used there's pretty well awareness about the vulnerabilities associated to that both in a manufacturing environment you know that cycles a little longer maybe that's still being used maybe the devices are

limited and that's why that protocol Sibley news that's what those older devices know how to talk to you and can talk to you so here you see it you know attacking attacking the web it's exceeding of course and then you know getting access to that Wi-Fi network this is an example of another you know passive reconnaissance get an idea of what type of network you're on what kind of devices are there you know what what can you see from where you're at where are you because you know when you're red teaming you don't always have a pretty manufacturer a pretty diagram out there showing you which network you're on what those IPS are what devices are on there

you know you kinda have to do all that on the fly you have to try to understand where you're at you know what do you have access to what devices are you seen and going back to the passive you know netstat Wireshark dumping information and looking at routes on the systems trying to determine what other systems are is this system interacting with or what are the systems around the network without actively blasting it with the scanner so a replay attack as you can see here we're on the control network I'm a lower network and we're gonna will show you where this is in the greater diagram right next here once you're on that control network a lot of those

devices have trusts kind of inherently built in maybe because they're older maybe because that's their model they're expected to be working on a trusted network replay attack is possible with these types of devices typically it's unauthenticated it's using you know the passing the protocols cross if you can maybe sniff something with Wireshark and replay it and have this same device repeat that same you know on/off or switch something make it off instead of on or replay something to make it do the same thing again that's an example here so with Wi-Fi you can see Wi-Fi on the Supervisory network you can see within the control network you can have devices connected to it so with this vector we

essentially possible to get you know supervisory network control network but maybe not at the office IT Network you know if you're red teaming the objective driven maybe the ops IT was never an objective anyway you're trying to get to the control network and with this attack vector you know you can get straight to the control network without making all the noise and detection capabilities that are on the office network that may not be on the control network yeah so why it's very important to look at Wi-Fi on the network on the manufacturing where networks is because as you can see the Wi-Fi is kind of off to the side here on the supervisor and control

network it's so we could actually in some cases potentially access Wi-Fi outside of the plan itself so basically if we have a plant wall or something around our plant or plant fence or whatever we could actually access the Wi-Fi outside of the plant so this is why it's very important to to look at Wi-Fi yeah I just want to say that yeah and we'll bring that up in mitigation measures also you know how to use the property antennas on your Wi-Fi if you are using them directional versus you know spreading out to everybody in all directions the last attack vector is the physical we have example there of a security fail so you know you put up a

gate you could absorb great but there's no offense you know supporting it people can just walk around kill those brushes physicals as we're going down again you know less opportunity here you have to physically be there within the manufacturing environment you have to attempt you have to interact with other humans unfortunately you know and a lot of the same social engineering techniques that you've seen otherwise would work here to you know fake ID with a bit of nice talking hey how's it going great weather whatever whatever you want to say you know get access look like you belong play nice or you here the other route you know look official look like you're somewhere higher up where they

shouldn't talk to you or something like that that works too oftentimes within these assessments there's there's boner abilities everywhere right but humans are typically the biggest weakness to solve that with the example of fishing you know they're gonna click something if they think they're gonna fit looks right or if you entice them with a something shiny they may click it you may get them that way humans are more unpredictable than machines so typically they're the biggest weakness in the link and again the main concern here is safety so in a manufacturing environment you have this big robots doing things you don't want to be caught on the wrong side of those things if you are physically assessing a

system you know you want to take a look at what you're doing what you're going to get access to what kind of position that'll put you in once you're in there you don't want to be in a bad position for your own safety and this is just an example of what you can get access to right so bypassing all the control now go straight to the control network you know bypass the office IT network and you have physical access to physical buttons and can either you know change processes or manipulate things that way turn things on turn things off or in a lot of times you know you can implant something so implant may be something as a c2 command

control to give network networked access to you walk out of there now you have network access you can go home and you know work from the internet so a lot of a potential for physical it just requires that you're actually there and you know there's greater chance you're seeing etc etc so within a physical you can get straight access into control network maybe once you get past the perimeter just go into the manufacturer inside you're assessing the controls on the perimeter as well you know are they looking at badges do they care do those badge scan do they care if they scan you know what kind of policies and procedures do they have in place for

somebody that's unknown did they follow those and that's the kind of thing you're looking at when you're red teaming on the physical side so maybe they have all these processes and policies in place but you find out when you do this maybe they're not following them maybe a nice wave and a hello good morning works right maybe if you look trustworthy enough that's all that matters so worst case scenario as you're doing this Red Team assessment you got to figure out what's the worst case scenario you got a kind of brainstorm in your mind what's the worst thing that can happen from you doing it and also attackers so is denial service is shutting down the made fast rating

system the worst case scenario that can affect the bottom line send commands to process you know manipulating the process in some way and that way not being detected is that a worst case scenario we're getting the sensitive information out of out of that facility you know is it proprietary and you want juice want the information so you take it and go use it somewhere else or sell it so there's endless opportunities a couple things here on the right of show is the attacker objectives so loss denial manipulation can you manipulate the processes without you know a defensive measure picking in or taking notice can you deny things happening on the network stop network traffic deny things or loss

loss of control the network um so some of the lessons learned we we learned from our experiences in Red Team manufacturing so manufactured networks are using very outdated software hardware protocols things like this this is a common issue and and I think all of IT security but it's very prevalent in manufacturing environments I mean you know we see XP running all very old protocols like WEP things like this and so it's it's important to to to be safe when you're red teaming manufacturing environments so don't bring anything down no denial service attacks things like this but it's just an important thing to mention that these networks are usually very old and some of the old attacker techniques still

work against these against these networks like like the why WPS Wi-Fi pixie attack that cracks WPS protocol and things like this whet attacks yeah just attacks like these old texts that we haven't used and like Johnny said we haven't used in 10 years still work against these networks and then last but not least use vulnerability scanners very scarcely so even and map I would advise not even to use that just passively listening on these networks is the best way to go about it so using wireshark or TCP things like responder definitely probably is a no-go but passive listening techniques are the best way to learn about the network and some of the common findings that we we

discovered whoops Johnny I drop your clicker in this little hole here you're not getting that back some of the common findings we have in manufacturing this actually we pulled this from the DHS common common vulnerabilities and manufacturing but this is also true to what we found from from our various research so the first one would be authentication and access control there's just a lack of authentication for some of these older manufacturing protocols like we could intercept communications and see clear text passwords clear text commands things like this we are able to replay attacks very easily with no authentication needed and of course access control a lot of default credentials were in place the for example on the VPN appliance

two-factor wasn't enabled just some things that we consider basic and IT security is not necessarily on the manufacturing network or not enabled the next would be network design this goes into network segmentation and segregation there were a lack of of segmentation between different networks for example the Supervisory and the control network should always have some kind of a segmentation or a firewall in between and we sometimes didn't see this another thing about network design is networks for not we're not very well documented so the IT staff really didn't know and the security staff sometimes didn't know what was on the network and what how the network was laid out and there was no formal documentation for

this policy was another big one sometimes there weren't any there should be some policies governing basically security governance there should be some policies governing different aspects of manufacturing security monitoring if there was monitoring going on on these networks sometimes nobody was watching the screen so that's a problem obviously if you're gonna have monitoring someone should be looking through these logs and finding us because we were able to do a lot of these attacks and in our example scenario especially with without anyone actually seeing what we were doing until after the fact configuration and maintenance this would be general asset management so basically know your assets this would involve doing a asset inventory of your manufacturing

facilities and ICS environments and really knowing what's on your network what software what hardware basically knowing your network and sign it out and this goes along with also patching and maintenance it's so patching is difficult in IT security but also in manufacturing facilities a lot of these this hardware and software is is very dependable it's built for liability and not for security so it's yeah it's difficult it's difficult to patch these things but the first step is is asset management and knowing what's in your environment and finally security awareness so this is also a problem everywhere but one thing that we've noticed is usually the the manufacturing floor like the people actually working on the manufacturing environments and in

the robots themself they may not have any idea of what security is so it's important to to to train these employees and just to so they understand that if someone strange is walking around the environment unplugging things and their network and things to kind of question them and say hey what's uh what's going on okay so some of the mitigating measures we have safety of security awareness training which I just mentioned train your staff strong remote access policies so this would be the line along lines of strong VPN policies configurations enabling two-factor expiring accounts that should be expired things like this proper network segmentation which I already covered this would be um yeah installing firewalls where they

need to be installed in ensuring each level of the network has a separate separation and segmentation policy and auditing of the Wi-Fi configurations so make sure you're not using WEP make sure WPS is off things like this and then asset management which already covered this kind of covers the entire network but just really know your environment know your assets your software your hardware your robots what needs to be updated and when and this is really the first step to to proper updates and securing the environment and then we have yes monitoring of course this would be which I already covered basically monitor the environment for anomalies monitor the environment for attackers and and so you can catch you know the

red team you can catch us on your manufacturing network sorry guys I don't have the clicker so regular patching which I recovered and finally make sure to do regular security assessments and red team assessments penetration testing audits these these these techniques really help you understand your risk in your environment so it will cover this in the next slide but it's very important to know your greatest risks because right in IT security everything's a risk everything is critical but really this is a great method especially red teaming is a great method of finding your critical your critical risks in the environment and what you need to fix first so one thing I want to say is it's very easy for us

to stand up here and and say how bad manufacturing facilities security is and how awesome we are what great red teamers we are and how we found your network but it's blue teaming is much harder I'll admit and so it's very important to to to know your risks and this is mainly the main thing that we want to get across here just know your most risky parts of your network so complete a risk assessment and this is if I would say the first step in securing a your environments and know your assets perform a security assessment or a Red Team assessment and find out what is the greatest risk to you and your environment and this can be done with

using the example red team methodology we have here kind of picking a based on your threat model pick a type of attack that you would want to perform on your manufacturing networks and really determine determine your greatest risk this way and finally create a plan to start mitigating these risks so pick your top three and create a plan to to start remediating these so an example would be so begin taking inventory of your assets asset management audit your current access controls and your current access control policies and finally implement a strong network segmentation so these are just three things that we think you know could be quick wins for you and your environments but it could

be anything just it's it's a really facility-specific yeah I just add to that you know like you said it's easy for us to say this was broken this was broken but on the flip side it's very hard to secure these types of environments and really get them up to the same par as maybe some of the other side of the IT world their life cycle is longer they have you know a real need they need to be up 24/7 most of the time they're doing a job that you know must be done can't be interrupted so that brings about challenges on its own so we just want to highlight here that you know it's easy for us to say this and

this and this is broken but we realize it's also hard for the defenders that are monitoring the network or the system administrators that are trying to make their network secure here on these manufacturing environments and we're trying to you know just point some quick wins like I said yes so these are some of the resources we used in our in our presentation here so these are all awesome this is why we included these and sure take pictures of it these are especially DHS industrial control systems manual here the DHS common cybersecurity practices is good my favorite is the industrial control some cybersecurity kill chain this is awesome but these are just really really good documents you can use

to to start with this with security assessments in your environments yep and this is just a few there's a lot that you know on both sides of this there's a lot to dive into red teaming and how do that efficiently there's a lot to dive into an industrial security side and how to do that efficiently so we thought would provide some to get you started and get you down the right path but there's a lot of resources out there that that are good and with that okay we'll open up to questions hopefully we'll have answers you mentioned in your assessments or in your assessment plan that you're not going to do a do s but

from standpoint of the internet is that not a realistic scenario yeah so it's a very realistic scenario but we also care about our manufacturing facilities and if we bring it down this money you know yeah that's the downside of being let's say a red teamer penetration tester you have to find the vulnerabilities but you also have to do it in an effective way and not have you know a negative impact on the organization you're testing on the flip side of that you know on the internet that is there's probably attackers doing that all the time but specifically for do s we're talking about control Network supervisory Network where that's you know has more of a greater impact and they're those

typically well sometimes they're available on the internet but most typically they're not

I think just last month you heard about Tesla General Motors they didn't attack you but they attacked when your third parties and got manufacturing specs you know the tolerances you put on your machines what do you see any change especially from General Motors and maybe do a little bit more to third-party vendors because obviously if you're doing a good job protecting but in your information is sitting on third party and gets owned and dumped then how does that influence with your red team actions maybe with third parties or vendors right so good question I don't have the most up-to-date information on that you know I will say like you mentioned it was the third party I'm on the General Motors

side I wouldn't have the most update information on that but I could probably get you a better answer later do you mean like mainly the techniques or how the actual that actual attack affected [Music]

obstinate until that from trying to inspect on door wondering if it obsolete actives that you maybe

right yeah I understand your point that that's difficult apart third parties you know may have some data that's related to us unfortunately I don't have more information on that but I can talk to you after this and try to get you a better answer how often how often do these teams have development or test systems that you might be able to run more intensive scans on without you know taking down production right and that could go back to scoping so it could be that the manufacturing organization would like to know you know instead of a red team assessment what are all the vulnerabilities associated to this box right so they set up that and you know

your scope is let's just take a look at you know this box and see what's gonna happen here so you can definitely scope out an assessment that way although it won't be as useful but like we said you know it's there's multiple layers to do here and that may be one step in a greater assessment later on in the future of taking a look at a manufacturing example manufacturing device like you mentioned and just using that scope and throwing everything at it seeing what's available and you know giving them that well that would be very specific so a scope of like here's one box you know take it down that type of thing or find everything evil wrong with

it that falls more into penetration testing so we have done that but that would be a penetration test the red team means more of ejective driven use anything you can to get to that objective so a question I have is is not all of us will have manufacturing systems but a lot of us probably have SCADA systems is there anything that you would change or do and when approaching this from a skater perspective right so that's a great question that's why in our beginning of the terms we tried a level set with ICS that's like our all-encompassing term of like SCADA pcs all the acronyms that go under it most of the references we we showed you here

would be applicable to SCADA also that's just a subset of ICS so we generalize and call it ICS but most of this would be applicable SCADA and a lot of the mitigation measures are also the same right so defense-in-depth are you doing user awareness do you have policies in place are you looking at your access logs do you have access logs so a lot of the defense-in-depth mitigation measures would be the same and a lot of the attacking of that network would be similar hey thanks for the presentation you guys mentioned a few times shying away from active scanning on the network because it can have adverse conditions on the equipment and you suggested using just passive

monitoring of the environment using wireshark or tcp dump or whatever else as a way to you know get the information that you want how do you deal with the amount of data that you collect and how do you derive interesting findings from that traffic are there any specific tools or techniques that you use to dig through you know gigabytes of peak apps and to find the juicy stuff that's otherwise given on a silver platter and on an SS man yeah so let's see I would say Wireshark has been the primary tool but Wireshark has a lot of awesome plugins for for ICS and for manufacturing I would say there's a plugin for for most protocols in

manufacturing and ICS so one of the common things to do is to use the white shark search tool and search for certain for certain strings certain certain protocols for example maybe I'm trying to do so if I was doing a like a reconnaissance you with with Wireshark I would search for ARP or something like that to try to find what what system is talking to another system through layer 2 things like this I would look for HTTP connections outbound things like this yeah I think what you're shooting it for here is so let's say you do use Wireshark and you're creating all these manual search terms and trying to dig through it they're also open source

software out there that'll take a peak app and do all that parsing for you and then you just go through and look at the interesting results so example of that would be netminer that will take a large peak app you feed it it'll say here the what we think are clear tax credentials here's what we think is you know this type of traffic so that's one way to go through that sort of data very quickly but up it's up to you eventually as the red teamer and as the Assessor to look at that data try to get the meaningful information out of it and then choose the actionable data to keep moving on hi I'm just kind of

curious from the remediation perspective a little bit in terms of wine how much traction have you gotten also has there been any pushback in terms of some of the recommendations on what needs to be fixed how is that prioritized and who is the ultimate customer I hope that helps okay so there was a lot of questions there let me try to remember them all so we have found that doing a red team assessment is much more impactful to say the customer whoever we're doing it for whatever organization because they'll see the real the real results right so a lot of times they get handed a security assessment and it's it's a scanner results right so you have this

vulnerability in this one and it's hard for them to interpret what that means what what is the real risk here what does that actually mean but when you do a Red Team assessment it's much easier to drive that impact home by saying here's all the plans to product X we took last week that you didn't find right or you didn't see us so that really helps really drive home the impact of the vulnerabilities and that usually gets you know their ear and they start to pay more attention because they see what's a tangible result out of these having these vulnerabilities there versus a scanner that may just say it's a medium risk to have this vulnerability

you know that that's much less tangible I think there was a second question there on the true by me typically it's more of a helpful atmosphere maybe at first there's some you know back and forth on what what took place they usually want to know what happened and as we mentioned in scoping maybe not all of the employees within the organization know what's happening so that's to be you know expected but once we get pushed past that and show the results usually it's how can we fix this you know what do you think's the best route to continue yes oh I think successful in fixing definitely yes a lot of these as Kyle mentioned a

low-hanging fruit as soon as you're aware of them it's much easier to drive the remediation to fix them it's maybe this configuration change that doesn't take much time but some of the other ones are more long-term you know how do we fix the environment how do we fix the network what can we do next generation or how do we fix that next year so there's always planning involved point of that I can't speak for any company right GM or my company but from my experience people are very very willing to work with us because a lot of these manufacturing plants don't necessarily have a security staff right and they they they're just learning how I T

security is so important from the news and things so they're very willing to to work with us from my experience so a kind of along the same lines you know you guys are talking about culture and working with people and on their systems have you guys handled the culture of this is my system and and my stuff and and even to take it a step further if you broke something how did you handle that

yeah this so this is I think we're out of time this is kind of a loaded question I really want to talk about this so if you could come see us afterwards that'd be awesome yeah two minutes can we tell a story about how he brought something I don't know my bosses in the audience I'm not going to or my old boss I'm not gonna say who it is but can I your first question I mean yes I have broke something is it was intended it was to show an example similar what you mentioned you know here's a box what's the real vulnerability of what can we do so your first question was how did they push

back on that right so like I said earlier sometimes there is initial pushback because some of these red team assessments may not be knowledge to all employees so if if it affects let's say a particular sysadmin who really thinks they're doing a good job with their with their systems specifically they may be defensive on you know what vulnerability was found where and why that was like oh I'm about to fix that you shouldn't be there but once you kind of explain the situation and usually provide the information and know that you know we're not targeting a person we're looking at the environment as a whole you know it's not whether this system was vulnerable it's you know how did these how is the

network or how is the system as a whole interacting with each other and the ways we exploited it so that's one thing make sure that we don't target specific people or services or processes that we're trying to convey you know at the entire environment and that can help mitigate some of that you know finger-pointing and then again once they see the impact of these vulnerabilities sometimes they're they're more receptive on okay you know I want to fix this how do I fix this quick last remaining question we still have maybe a minute or two okay well thank you guys so much [Applause] thanks everyone [Applause]