Building An ICS Firing Range: Sharing Our Journey & Lessons Learned

um thanks everybody for for being here and thanks for having us um we're going to present in the next 30 minutes a project that has been we've been working on moritz and i for the last year and a half um and i was building a firing range for industrial control systems and we did that in our kitchen because of corona so there was no way to do it in the office and that's why that's that's why we're going to you can see we didn't bring so we actually created a model we didn't bring that but we're going to show some um some videos of it later and then for everybody to take away as well

the lessons that we learned so if you're planning to build something similar in your kitchen then you know what to do and not just to do hopefully um so roughly that's the the agenda for the next 30 minutes um we try to get everybody on the same page what we understand is firing ranges what we understand as ot or operational uh technology then mort will talk us through how the building of the ics firing is actually unfolded we're going to demonstrate it and then the lessons learned and then at the end the questions about us i'm niko i'm the red team lead for nvizo i've had my fair share in i.t security and i focus on rt security as well in

the past two years and most of you sure my name is summers i'm part of the software and security team and r d team at enviso and i'm an iot and ics enthusiast and i try and get a hold of as many devices as i can try and break them um a few words about that viso um it's it's a company that's roughly nine years old originally founded in in brussels a noun office in frankfurt in munich we have 150 colleagues at the moment so it's um it was just a very small subset obviously um but what's interesting is that inviso is dedicated to invest 10 of the revenue into r d and that is how we

can fund this project and the time that went into it um so that's um that's how good that that happens now firing ranges um what do we understand as fine range of course it's it's somewhat a controlled interactive environment um it's an abstraction of a real environment think of a toy train for example instead of having the real train of course to play with it should be as realistic as possible and that's what we try to do as well so you will see that we built a a model of a basket bridge so which that has leaves the liftings we try to move letters as closely to the real thing as possible it should be reusable so we want to use it

for trying out stuff um in our context it's i.t security so we want to try out attacks for example or defense mechanisms and of course if we run an attack we should be able to do it again and again and it should have that playground factor so it should be something where we can approach things from different angles and just see what happens if we try that way and so on why do we need a fire range so the first that comes to mind and that was our motivation as well at the beginning is for training purposes so we we've done a lot of id security ot security was fairly new to us back then

so what we wanted to do is to train our people as well and ourselves on how to approach ot next networks and how to do security assessments on those the the lab itself the way it turned out is very suitable for awareness training as well because it's very visual so if you run an attack for example then you can make some some lights stop blinking or the bridge leaves are not moving anymore something else is happening there's a buzzer going off so for awareness trainings for example um very useful for testing if you had neo2 components for example pll plc's that you want to try out that you've never played with before you just plug it into

the lab or the firing range configured and then make it work and then last but not least for further development of tools for example for forensics so we actually used that lab for forensic training as well um which you'll see then at the end as well very briefly now um ot and ics operational technology are computing systems that are used to manage industrial operations for example the manufacturing process and the subset of that uh industrial control system so the integration of hardware and software components to control processes for automation and humanitarian institute mentalization we do see two candidates here on the slides on the right hand side you see a programmer programmable logic control it's very difficult for me to say that

plcs which is essentially the core unit if you will of some automation process so it's it's the cpu that that runs was connected to two sensors and actuators that drives for example conveyor belts and so on and then on the other side that's an hmi human machine interface where the operators normally oversee the process they can start stop processes or react on errors and and so on so it's just the interaction with the machinery behind it now um i bet you're all familiar with it the what many might not be familiar with is that there's a difference between security requirements for operational technology and i.t whereas in iit normally you would probably argue that confidentiality is

has high priority so you don't want a customer database for example to leak to the internet that would be very severe of course if you shut down that database i.e um in regard to availability then that can have some impact but it's probably not on that same level whereas in ot it's very different confidentiality usually doesn't play a big factor but availability does so if you think about power plants for example if they just switch off or steal a steel mill and that is corrupted somehow the process then you need to restart that might take days so there's a financial impact obviously but also the the production behind just the delay it takes to get every

everything back up and starting so availability is key now attacks against ics this is the sun's ics cyberkill chain it usually comes in what as far as we know it comes in two stages the first stage usually goes against the it network so the way we know it there's a resource development phase at the beginning then delivery where for example phishing emails are sent to somebody working in the office they click the attachment um the the payload installs um on the workstation ensures persistence and establish c2 communication and then the the adversary can act through the c2 channel and yeah their objectives whatever maybe now if the attack goes against ot then this is also true so we are that that stage

where we have a foothold on the the enterprise um we then recon where is ot located we identifying possible jump servers or dmz that we need to pass through in order to get there once we're there we locate the targets we develop capabilities testing them so that might be payloads that we want to download on plc's for example then we download them install and execute the attack which could be the disruption of some manufacturing process for example the objectives of those attackers are commonly either the the permanent loss of view of control for the operator so that they can't see the process they can't interact with it anymore it could be temporary only for vm control again but

also safety which is a very important factor and the same of course for manipulation and the safety aspect is also very different to what we have in i.t there's rarely at least i don't know of any attacks against i.t infrastructure where you would say safety was a concern safety of people specifically and we had a case in 2017 um which hadn't happened in saudi arabia against a power plant where they found a malware that targeted safety equipment it didn't succeed but the assumption is that if it would have compromised those safety systems um it would have then disabled some of the procedures that would have triggered in cause of emergencies for example if gas is

leaking um into the outside then those safety systems would prevent this now if you modify or manipulate that then you couldn't or you wouldn't prevent it so this would not only be then of course um an issue for the um the provider here but also for everybody that lives around the power plant if hazardous gases are leaked to the outside now moses is going to talk us through our ics-5 range please right thanks so how did we actually go about building this ics firing range well let me maybe first start with uh what we had in mind what were our driving motivation uh behind building this so as niko pointed out we have a difference in security

requirements in ot comparing to i.t right and we then have to adapt those differences obviously into our approaches to performing security assessments right so most importantly what we have to do to pay attention to is of course the safety aspect we can never know really what is the safety impact of of hacking a ot device if we don't have accurate depictions of the documentation of what actually those devices do so what we want to do and what we then succeeded to do is to build a playground where we can perform internal training to then perform research and development on this so we developed the concept of a water treatment plant which was comprised of three

water filtration units and then as actuators pumping stations which pumped water from one filtration unit to the next one and to back this up and to represent a more or less realistic environment that drives this we had also scada and an enterprise network behind this fully virtualized so when we worked on this we were then contacted by clients who said well we really like what you're doing we would like to have this for us as well because we are actually operating critical infrastructure and as part of this they operated bridges escort bridges and they wanted to have this such a playground for their own digital forensics and incident response teams to find out okay what happened

post-compromise and this should be also a mobile solution so they can then do those trainings at different sites so we adapted our concept to to do basketball bridges and this was our very first concept so here you can see it's basically an aluminum frame on wheels in the very top there you have a scale model of basical bridges then in the lower half there's a front and a back side in the front side you see rt equipment such as plc's power distribution power supplies and the back there the the area that you can't really look into because there are black panels there should be the virtualization server so we were familiar with building virtual environments

we were somewhat familiar with assembling stuff but not very much with 3d printing and they're just designing this this bridge for example so that's what we started with you can see in the center there's our um our c80 model of the bridge um you saw there were two bridges right so the other side is just a flip version and we 3d printed those so what i would like to point out here in particular are the most prominent parts so here you can see the bridge leaves which are lower and grays there are switches that are our sensors which detect whether our bridge is now in a fully opened or fully closed state and of course we also have traffic lights

and a road barrier that goes up and down and when all of this works it's synced so that there's a process where the bridge opens and then the traffic barriers are down and the lights indicate that no traffic should happen here and it should all hold and then they go back down again the road barriers go up again and then traffic should commands there as we said as part of our title we built this in our kitchen this is my kitchen and that's my oven actually much to the dismay of my wife we did this in my kitchen just to keep those prototyping intervals very very short so typically i would design a model in

the evening kick off the print this would take hours and hours and hours and sometime in the afternoon it would be done on the next day and i could take it out of the printer and try and assemble it and see if it works or if i have to adapt anything about it so at that point we got a model but it couldn't really do anything on its own so we needed to do more to it we first assembled it then we got a hold of those aluminum profile beams and built the frame and the back plate for the electrocabinet inside we put all the ot equipment in there wired everything together installed the bridges and wired them

those are roughly 50 individual connections that we then put down to the to the ot equipment we put panels to it because they look nice and they of course hinder you from actually reaching into it while it's operating you shouldn't do that uh some sweet backlighting we put in there so you could actually see something because it was getting dark and we did a lot of testing an awful lot of testing many hours were put into debugging all of this and we at some point even had to recite using an oscilloscope as you can see here to debug the motor control signals because they were not really in sync and they were off so we had the physical build but this

only gets you so far what we were really interested in was actually powering it with a fully virtual environment in the back right so let's get to the network infrastructure that was then used for example for kill chains this should again be somewhat realistic and should be extendable so here's our network infrastructure those are four networks the networks in the dotted rectangle they're mostly virtual while the ot network down there is mostly physical let's go through all of those they were implemented in an e6i server and they all use opn sensors for segmentation and segregation so just for routing and for the firewalls now let's let's see so in the enterprise we had an enterprise network we had a

domain controller and a domain joint office workstation which we heavily abused for initial foothold in our kill chains so there would be for example the case where we had a phishing mail and this attachment of the phishing mail got opened and then it was compromised there was also a web ui that would allow office workers to then have a look at what the process is currently like in the at the bridge uh then in the skydive network we had an operator workstation which would allow you to more closely monitor what's actually going on in the process taking place like what are the individual pieces of the bridge doing there was an historian which actually grabbed production data and stored it in

a time series database and a jump post to which we will get back later the dmz honestly states uh very empty for most of the time and still is we plan to put a windows update server maybe there but we didn't and then in the ot network we have of course all our rot equipment so there are four plc's one rather new model so there was an s7 1500 they were all siemens plc's that was the area supervision that one would orchestrate the other ones then there are two more or less budget models s7 1200s they were the lifting substations and they drove the leaves and the barriers and then lastly we got a legency model

the s7 300 which is if i recall correctly close to 30 years old and that drives actually the lights and of course we have the manual override hmi so where you can actually tap and change stuff around physically we also built in a cctv system using raspberry pi's and pin pen tilt cameras and here's one last uh virtualized station so that's the engineering workstation that will be used to actually program the controllers and that would also be more or less the last step in our kill chain where which we could compromise to then um if it effectively compromise the plcs because this one could program those right so uh knowing about this let me show you

what it really looks like when we all assembled it let's see if the video works ah nice okay so uh here you see the model on top and it will just start its cycle of lowering the the barriers and here we can see in the hmi that what you see on screen is actually sync to what's going on in the ridge model so now for example they are raising again taking a look inside we can see a whole bunch of wiring stepper motors and actually also active cooling we had to do this a whole bunch over the whole firing range because it got really really hot then there on the back you can see a

desktop case so that's our e6i server there's a big switch and an ap for wireless and easier debugging and that's pretty much it and that's what it does all the time and now at that point we can go there and try and compromise it then we have some backup slides in case the video wouldn't work you can skip those now for the lessons learned what did we learn we learned a whole bunch first of all regarding the ics lab setup we learned that especially with the hardware compatibilities and dependencies that you've got assembling such a fire range can get somewhat complicated when you're not a main expert we are not ics engineers we are not mechanical

engineers so those were things that were rather new to us you have to take several things into account right so power supply what kind of power source do you need what kind of protocols do those uh devices support what kind of protocols are relevant to your cases to your scenarios maybe so this can get this can be quite a bit uh then the the software licenses are quite pricey we'll get into this later and a practical thing that we figured out was that the stepper motors they tended to overheat and that was actually very bad for 3d printing again we didn't really do this prior to this project so this was all new to us

we are not mechanical engineers so especially the part with the bridge leaves opening and closing this was quite challenging for us to develop to design and then to implement and to make sure that it doesn't break because it broke a whole lot and then factor in the fact that printing is very time consuming as i said i would kick off a print um at night and it would maybe be done in the afternoon next day we had prints we had individual parts that took up to three days to print so 72 hours and if anything goes wrong you're done it's bad and also if you're not an expert in 3d printing you will run into a whole lot

of problems ranging from from a to z so we had for example problems with the prints not sticking to the print bed so the print would just go and the filament would go all over the place but it wouldn't be there where you actually needed it and then to have models that you can actually adapt to your needs and to to modify further you would do this in cad again we didn't know cads prior to this so this was also required a whole bunch to learn but also there were practical challenges so this is this is the firing range it was 80 centimeters by 120 centimeters in size uh only the frame was in meter in

in in height and if you factor in the wheels and the models it's one it's more like one and a half meters so every time we wanted to move it anywhere we needed to get a very very big van which was mostly empty but we needed it to put it there upright so we needed the vertical space and then also we had problems with the models right so here you would see that the road barrier would collide with the lights up there wasn't very nice the stepper motors were heating up there was the wiring which was all over the place and there was very little clearance for the moving parts this was iteration one so we it one more

time and there iteration two we addressed all of those problems here you can see those lights are set back so they don't collide anymore there are actually fans that we built into it so there's active cooling and you can just let it run for hours now it won't every heat which is nice if you want to do maintenance or if you have to do it there's a central back plane where all the wiring is just going which is nice and of course the clearance we have way more clearance here and then no more parts scraping on other parts so that's great what's the bottom line though uh we started this in january 2021 we put more

than a thousand hours of manual work into it more than 900 hours additional to the first 1000 hours which is net 3d printing time and a whole lot of this time was overnight and over the course of the whole project we burned through more than eight kilograms of filament we spent roughly eighteen hundred eighteen thousand us dollars on the whole project most of this for hardware but as you can see three and a half thousand euros are use dollars just for licenses which i hadn't expected in the beginning but here we go and we actually destroyed some stuff over the course of the project so there were two stepper motors that we actually burned through one plc that died

presumably from 40 wiring and one motor driver that also died due to 40 wiring and at times we lost parts of our sanity to be honest at that point i would like to hand over to nico again about what's next what's next thanks for it okay um so you if you paid attention i'm sure everybody did the video deferred the video that we showed deferred from the 3d model for example um so if it wasn't clear we we actually created two iterations the first one where the problems the rules that that most mentioned and then to make it better we created a second model so that's actually two models what we should be doing is creating a third

iteration um to address some of those points that are still not not that great so for example mobility is an issue you need still to rent a van to transport it and it was supposed to be very mobile it really isn't at this moment um so it could be for example that the disassembling of the entire um firing range is somewhat easier um maybe um yeah that parts could be taken out of it um also the modularization is something that we we had in mind for example so we we talk about basket bridges now but maybe we want to have a small town or toy train or something running on top without changing the hardware underneath because it's most

likely going to be very similar so we just um would replace the the top on the model i.e the bridges with something else and then download new software on the plcs and then it should be done so this is something that that could be done in a third iteration whether or not we're doing this we have to see um it's not clear at this moment what we certainly will be doing is developing scenarios so this the intention of this firing range was to use it for training and we've used it for forensics training in ot for example so we had a two today workshop on this which were quite great um but we currently don't have scenarios

for example for penetration testing or red teaming in ot so this is something we want to to work on as well um similarly uh monitoring and detection ot i think is a topic that's very interesting and when many people are interested in looking that as well even in our company so for for this developing a scenario would be very worthwhile um now if if you're interested in ot or want to know more about this project um we're here to answer questions of course we're going to be outside you know after this talk you can reach us via these email addresses um if you want to know what nviso does for ics there's a page for that as well and otherwise we

have two twitter handles and a blog that you can look up on the block there is a series um about this fire range as well so parts of what we presented a bit more detail as well in some in some areas and just check it out and see for yourself good any questions thank you thank you thank you so do we have questions i'm sure for the audience the front microphone if you want to take any questions so okay i see chen please come up it's hella cool so thank you very much for this um i was wondering what other materials did you consider when putting this model together like did you did you think about legos or

something like this or did you go directly to printing i think we lego was in talks there was some talks about using a bit more wood as well to to get some parts done i'm not sure what we did not decide on lego in the end to be honest not sure i think it just looks cooler if it's 3d printed maybe it was occupied by children or something yeah as well yeah i think the the probably the possibilities are it's just a lot more possibilities to create stuff if we have a 3d printer then we we're fully in control of how it should look like and especially the smaller parts than they are needed and i'm a long time no lego player but

i remember there were no real bridges moving bridges so you only had trains and stuff okay um do you have any um similar projects like um other companies doing similar test ranges firing ranges that you exchanged ideas with honestly we no we didn't we know that um not for bridges but but of course siemens for example they they have developed um also test environments for for traffic controls for example um but but it's a different price range as well and i'm not sure if they they would disclose too much about their projects towards us um but no actually don't know maybe over you find somebody from siemens over a coffee yeah um so do we have any other questions

right one coming up uh so uh most of the ics or the ot environment will have some physical control uh over the process not only that some malware can shut down the process while building your fire range did you also consider physical controls like you detect some attack or something and then there is somebody who is putting out switch or like why triton did not succeed and also scenarios like that so that's the hmi if you meant that so that's the hmi for the control so for example uh if you consider the titan thing it tried to shut down the safety system but of course it did not succeed there were also physical controls on site and all

to avoid this were you considering this to build in the firing range because that would be more realistic in an ot or ics environment right yeah so we can't put everything in that model that's i think um probably that's difficult um so we we um restrict ourselves to the couple of plcs that we have in the hmi um to and and with the stars around that so additional hardware equipment we did not plan for at least in in these situations so far yeah yeah this is why because it would not be practical to imagine that okay only through an enterprise network you get into the ot network because there would always be some physical control

over the processes as well sure so you think that the the attacker like they so they skip the it part and they go straight to ot and and uh carry out the attack from this is that no i meant like there would be some uh physical control that cannot be reached by attackers yeah avoid such scenarios and maybe in these kind of projects we can also consider this it could be it could be implemented in this show yeah i mean it's also extendable if this is a project for the future i think it's a good idea to include thank you thanks and are you going to put this on the internet for us to test as well

i guess you need data for forensic analysis so you just just put it on the internet and wait for us to test it maybe we think about it okay okay thank you very much