Industrial Robots And Machines Under Attack - Marco Balduzzi

hello hey wow a lot of people in the room thanks for being here it's a super pleasure my side to be here and attending the first edition of bide so I'm also happy that you have bides here in Prague uh so this talk is about industrial machines uh I work for TR mic research and over the last 5 years we look into this aspect because it's new uh has not been uh uh researched so far and there is a wide surface of uh vulnerabilities and and problems I'm also uh president and co-founder of noad is a akish conference similar to this one that we run in Italy in uh in October it's in English so you're all welcome to

participate it's a uh almost free entrance so it's very easy and relaxed uh so if you want to come over just contact me and uh I will put between VIP line and we can Network even outside of the of prag cool uh so why we're talking of industrial machines is because uh we have seen a a huge Revolution over the last uh decades so you can see machines have been introduced already many years ago but they were just pieces of that's machine right like a piece of metal while lately uh there was a push on so-called smart machine someone can also say industrial iot so similar as you have in your home building where you

have like a smart TV or a smart refrigerator anything should be smart here is a kind of similar situation so actually at European level uh there is a push for uh selling more and more of this machine so uh if you have a company and uh you're going to buy a machine for producing anything if you buy a small machine uh which is compliant with soal industry 4.0 uh usually you get a special um discount or tax reduction so uh that's why most of the people go with uh those machine that are uh connected and uh this is cool right so there are a lot of functionalities because they can easily get integrated into uh into the soal shop floor so

where you have it and OT that converge at the same time time so those machine that are connected either via uh uh Wireless or U radio propriety radio protocols as well as traditional TCP normally uh tend to be connected to um uh to the cloud for example for management they connected to the uh to the database to know what is produced so it's pretty useful right so if you have a huge Factory you want to know like uh what get produced or you want to interact with a machine remotely to change the configuration and change the pieces that are produced so of course you know there are some benefits of this integration of operational environment

so what is in uh in Orange um and the traditional it uh infrastructure but you know with functionalities usually the problems comes especially uh when the code that runs on those machine uh might not be the last uh the most recent code but there might be Legacy code uh so this is the setup in which we uh uh we did the research and uh here I'm going to uh give you two short example of a previous work we have done on the topic and if you are interested you can find more on uh reference and then we will dig deeper into so-call CNC machines so machines used for manufacturing plants everywhere so let me start with protocol

gateways uh so these are devices used to connect OT network with it networks uh so are um gateways like you know for example the router Router you have at home to connect with internet no the fiber or the ADSL router but this one are used to convert OT protocols like mod bus Prof profy BS where a lot of like protocol specific to networks uh Legacy protocol often serial on serial bus to uh tcpap and uh these are essential part of your uh deployment uh because you can basically bring a legacy machine uh that uh uh was used to uh was connecting Olivia serial into your uh TCP AP Network so we look into this devices in

the past and uh we found a few interesting things I want to summarize here quickly before uh moving forward with the rest of the presentation so uh normally gway is installed as you can see here in the middle of the screen uh this device here and what it does it was saying is basically translate all of this which is the what's called field Network so the OT Network into the traffic that come from the tcpap network so you can have a database you can have an engineering work station HMI which is the graphical user interface used to uh control what the production is doing and uh this this gentleman here does the conversion uh often you also have soal

IC firewalls uh so this one are very critical component because they make sure that the communication from the green line to the red line makes sense so you can for example specifying rules that says Okay I want to pull dat out of the motor to know what's going on but you know if I'm not admin I don't want basically to mess up so to be able to write something of a motor for for example change the speed of a motor because this one might create a problem right we might basically burn the machine completely or you might stop production so the SCS firewall is the element that specify Implement basically the policy for the operational

environment so problems here we found here it's just one of many uh in this work that uh if you specify an incorrect length in the packet uh the packet doesn't really get translated correctly get basically copy over so you can see here the bytes are always the same they get copy over but what change here is that the mapping here you see function code four it become function Code Zero have so what means is that in attacker that generate a read request from the historian down to the OT environment it actually um translate that request into a write request so a read input register request for reading values out of these devices get translated into a right

multiple coil so this one would allow attacker to bypass firewall and to uh write as I was saying on the uh Network a bunch of malicious things so this is one of the many problem uh and um if you are interested to this topic just give it of a paper it's like 70 Pages paper there was also a presentation we made at blackhe hat and uh there are some interesting findings here uh but Network like bosan are not only connected via tcpip they can also be connected over uh propriety radio protocols that's also an interesting things here uh there are more and more um uh machines that uh are operated remotely via so-called controllers so controllers

can be like this so you can have very simple controllers that basically allow you to Mo move the machine right left up down or controller that use joysticks um so in now in followup research we look into okay what's bad with those things right so uh what happened we bought some of them we went for the most prevalent one that are sold in different uh countries this is one of the most used one so as you can see here there is a um transmitter which is this one and the receiver unit so the receiver unit get installed on the machine and the transmitter is used by the operator to uh to control the machine so here is a different type of

research so uh on the previous one we mainly work with fer to uh find vulnerabilities into the protocol gateways here instead we work at Hardware level so what we did we uh exract the FW of the device we look into uh how the modulation of the radio protocol is done so I'm not sure how many of you uh have experience with software defined radios but this research go exactly in that direction so the goal here was to reverse engineer uh the radio protocol of this devices to know if were there were vulnerability so in this specific case here for example yeah we decoupled the the chipset on the board uh is a Texas instrument CI you

can read there 1 one20 so from there you can try to guess which kind of modulation schemas are used based on that then you can do with the coding go from analog to digilog digital sorry and then you can try to uh uh reverse engineer the payload and here are uh basically what we did so uh at the end of all of this me and my colleague here uh we went for some uh real world testing so we develop a tool uh that um implement uh malicious uh radio transmission system so here you have for my laptop with a bunch of uh software defined radios which are this device that you connect on the internet Dev on the on

the USB port and allow you to generate radio packets and we went out different facilities to test wherever you could possibly hijack certain machineries and I have a small video here I want to show you so uh let me see if it works yes so here we went into a construction site so this is a colleague of mine in the car with a uh with a laptop this is me holding so this is the legitimate transmitter here that I have here that you use basically to control the crane as you can see here in the video I'm not touching the controller but what you will see here is that my colleague from the car is basically able to to move a

crane while I'm basically you see so these are the sort of things it could do right as you can see the impact of this is super severe it can be this this is another construction site here as well that's my laptop with a soft defined radio uh attach here to the to the laptop the legitimate transmitter here nobody's touching it so I run the attack and you will see basically that uh the crane is basically moving up about doing nothing here is another

one here we you see here we generate the signal this is crafted by the

attacker this is pretty huge is a HDE

crane yet again so in this research we test uh not sure if right statistics here but if I recall correctly was something like seven vendors and they were all vulnerable to the same problem so you can see here there are General problems with all these Technologies and um and this bring me to uh the core of my presentation which is on CNC machines so after having looked at how this machine communicate both on the um on the wir on the network and with radio protocols uh then we went deeper into understanding fundamental problems of manufacturing machines so these machine are largely used in the industrial world and uh they Encompass wide diversity of terms so there are

Drillers for drilling holes that letter machine Mills Grinders 3D printers I'm not sure how many of you had a 3D printer at home but you know 3D 3D printer are CNC machines as well the small one and um these machine are very complex like this one is a machine which is installed in the uh poly Technic of Milan uh it's a local University in Milan and it cost something like € 250,000 to give an idea and this one has a Precision of 0.001 mm so 10 at minus 5 so what happen is that when the machine moves it can move up to 0.1 mm so to give you basically the idea of a Precision on how

this machine operates and is used for crafting very small like metal gears uh for example that you use then in the automotive industry of course there are tons of this machines and um uh as you can understand we serve critical asset right for all the Productions uh globally the other interesting things here is that you have a controller which is this one here and the controller is often the same across multiple machines in fact here uh so the clicker has stopped working again wow in fact here we have a very complex supply chain so take for example this machine I showed you before uh so who's going to develop the machine it may uh develop his own

controller which often is not the case or it might go ask to a NC manufacturer so basically uh the guy that develop the controller to provide a controller in fact there are a lot of manufacturer of CNC machines overc machine for doing anything basically but there are not many vendors of controllers so that interesting things it's like to say okay cool like there are many Linux distribution out there right that no s s Ubuntu but the core operating system which is Linux is just one right so the kernel is always the same so here it's kind of similar so um the supply chain consist of the simul the controller itself which is basically the kernel here then there is the

machine manufacturer which is actually develop a machine based on the uh controller the seller the integrator and the end user and all of this is very complex as you can see here there might be bunch of problems all around the supply chain so for This research uh we decide to focus at on the side at the controller why because if there are problems on the controller itself all the supply chain get impacted by those vulnerabilities like to say if you have a vulnerability in the kernel of an operating system like say let's say Linux all the different flavor and variants of of the Linux distribution we have a problem is the same things we have seen like last week

with this new library that got uh a nice um back door implant in the library basically all the software using the libr are affected something similar so what we did here uh we consider four representative vendors of four different countries uh to make basically the study um uh gen General right um to be able to apply the study uh worldwide uh we consider a controller from us controller from Japan one from Germany and another one from Japan so these vendors are uh probably the largest vendor on the market as you can see here our vendor that have been established if you take haenam more than 100 years ago so you know it's not a new

one on the market they are very large uh companies like having more than a billion revenue and thousand of employees and uh so what we did we work uh with both simulators and um real world machine for the experiments of course we start with the simulator first because you don't want to break machine and then you go down to to the machine so the simulator just for curiosity come to Flavors so uh there are uh software simulator uh that runs as a virtual machine so you can just run it on your computer and test the virtual machine the have simulator instead that are physical so they are basically like the device that you connect to the machine

but the periperal are um virtualized all right so uh how our research approach looks like so we use an approach which is the same for all vendor to make sure that we have a fair comparison of the potential problems that uh we could identify so we did an initial scraping with General vulnerability assessment scanner like nesus and there are many just to get rid of very common vulnerabilities that we are not much interested about um so the focus is not on okay cool uh it runs Linux and Linux is okay we know it it's more like which kind of technology is domain specific so specific to the CNC machine and might have problem that's why we then move into investigating

domain specific technology so technology that are specific to this domain as you can see here there are a lot of technologies that I was not aware of when we started the research but these are technology that are uh kind of common in like for example opcua is widely used Mt connect is widely used but all in this specific domain and no one before look at this so if you're interested to work in the in not security there are a lot of like technology very specific it's similar to the automotive domain uh in car security for example is the same thing so like there are a lot of specific domain and this is what really the juicy part is uh

then we did threat analysis to understand okay what could go wrong here right and then we develop a PC code that we tested on simulator and as well as the real world machine and then we work with cesa which is a cyber security agency in the United States that help us to get in contact with a vendor which is not easy right so we are not if you find the bug like in let's say whatever like uh Google or Microsoft you know they have a b Bounty program they have like security officer they have like people that know about security so you have a d you have a person you're talking to that understand what you are what your

problem here is here it's not really the same thing so uh and I'm I'm going to discuss that later so it was very helpful you know to have someone that allow us to talk to the vendor about the the problem so what are the general problem we have found uh Legacy software so similar to as a protocol getways I show you before or even the U the radio protocols I show you before so there are a lot of in all this OT space often you deal with Legacy software so uh that's one problem uh second problem you have much of mitigations like uh if you have a buag on software running on Windows Windows has a lot of mitigation uh

functionality embedded in the kernel like like a random stack address randomization or whatever are many uh uh technology put in place to prevent easy exploitation here is not really the case so usually a bug means a problem all right like an exploit and an impact uh a lot of of leakages so these smart devices as I was telling you before they are open for connectivity so they are open for sharing information they're friendly with anyone right but in your mean to be friendly with anyone you often provide way more information of uh what you uh should uh provide we'll see example problem with authentication bypass and the problem with uh uh Access Control poorly implemented

so let me give you an example here so this is a Mt connect Mt connect is a nice standard so all the vendor support Mt connect is a standard used for making the machine communicating each other so if you have a machine of a certain vendor and a machine another vendor if you want these two machine to communicate to be able to operate in a production line you can use Mt connect is a standardized language based on XML uh so here uh as you can see here XML I'm pulling some data out of the machine so like uh what the machine is doing blah blah blah blah blah blah so if you can spot this it means that you're already

expert in the domain so this is a the line of code that is currently executed on on the machine so this is in gcode gcode is the programming language used here and it's very similar to programming language used uh for uh robots so what you can do here you can write a small script that pull this variable out you can basically dump all the code that run on the on the machine so you can see here it's super severe because if I am a if I a company and I have a machine that produce a piece and an attacker can Dum the code basically I can steal all the intellectual property of a company because I know what is

produced how is produced I can copy the piece I can make the piece better so you can see it's a huge damage and you know we are not talking of the most complicated uh vulnerability exploitation chain here it's basically like a python crer that con connect to the demon and dump basically that information from empty connect okay so this one was reported and fixed very rapidly in this case uh thanks to another big issue is the lack of resource access control for example on us which is one of the vendor we look at um they provide a um a properity protocol which is uh this one you can see on screen where you can query for example for the serial number

of the machine remotely we can query the machine to know what is produced on the machine you can query that's very sensitive one that's why I read you can query for the configuration of the machine you can say okay what's the value of um variable 10,000 so variable 10,000 is like uh in near in a certain area of memory you can query for the value of that specific address in memory and you can also write uh certain area in memory you can see okay cool yeah you can write anywhere in memory what could go wrong well a lot of things actually by manual they say okay you can use this e command only to write certain area so

from one to whatever actually it was not the case so when we look at this actually you can write anywhere in in in memory with a lot of problems uh I had a video here I'm not going to show you because otherwise this presentation is going to take like 3 hours but uh um you can find extended version online or you can ask me I can uh give you more uh more detail whoop all right what happened here okay we are here no we're not here okay let me go

down oh we are we have done already a lot of SL it's a very interesting area of research here because uh it's pretty trivial to find a lot of problems so that's the impact here so we are our four vendors you can see here these are all the different attacks we came up to that map to the different uh uh class of attacks so you can uh uh the first one is compromise like airc remote code execution of the machine the second one is damage uh like we um damage for example the piece you are producing damaging the machine uh damage include uh damaging with for the operator right so a safety risk for the operator which is operating

a machine denile of service attacks of different form hijacking like taking control of the production and T of information like uh leak of um of information as you can see here the inut is huge so pretty much all the vendor have bunch of problems I'm going to give you some example of a tax we came up with um so let me maybe start with this just a sec what time I'm supposed to finish 11 right okay well 10 minutes all right so let's start with this one here you have on the on the uh right side here the attacker console uh this one in in Black here you have a camera on on the machine

so this is the CNC machine which is uh under attack so this is a real world machine we use for the experiment and this one is the console so this is what the operator view of the machine as you can see here uh the operator can can see uh the name of program that they get executed uh this is the uh source code of a of a program so we are information of a program like the spindle so the speed in which the tool is is moving here have the position on the axis uh so this is a uh driller so it has a free x y that that's position in the space and this one is the interesting part so here

tells you basically uh how the machine is compensating so let me explain a bit of this so basically when the machine makes uh holes so the point of the driller uh get consumed so when you start drilling things are okay after you do like 1,000 holes the tool gets shorter and shorter because uh you are basically consuming the tool so what the machine does it automatically compensate that to make sure that you always drill at the same depth okay so this is how it works this one is automatically done by the machine so here uh the attack so what we did we develop a code that would instruct the machine to create a piece that is like this so like

a hole of 5.05 mm dep so this is what our code should do so what we do here we uh remotely basically tamper with the memory of the machine and we introduce an error I'm not sure you can see here because it's bit small we introduce a compensation error of 0.25 mm so a quarter of of millimeter so what the machine does basically introduce a micro defect in how the piece is um uh developed you can see here uh when we perform the attack the machine runs with the same code so we didn't actually change the code right but the machine introduced this compositional error of 0.25 mm of course for those machine I show before that can operate at 0.001 mm

you can also introduce 0.1 mm of error so that's nothing it's not even visible I can easily bypass QA and um in the in the company so what what the result of this is that uh if you do this sort of attack on on a company company gets out with a lot of pieces that uh may have def so think about the company that produce brakes for for cars so if you introduce a defect like this and you go to the market then you can have a you can have a major issue right because you need to recall the pieces back and this is a huge problem for the company because you're going to lose a lot of

money uh for recalling all the defected pieces uh in term of images can be a problem because you know if you do a such attack and uh the the reputation of a company probably would go down right so this show how easy is to remotely hijack a a machine to introduce an error uh we also did uh a similar experiment here uh I'm going to show you so for this uh I'm going to show you how an attacker can actually damage the machine itself so what we did we use a 3D printer we print uh the uh the this is the The Tool uh of the driller in plastic you can see here that's a plastic point we put into the

into the machine here and then we did the attack again but in this case instead of having a compensation factor of only 0.25 mm we went for 10 cm so that point the machine does not even understand that 10 cm is huge so what it does it basically when you start drilling it compensate it go down 10 cm into the metal p and you basically crash completely the plastic point so this is of course you say it's an experiment but uh usually when you're drilling for example aluminium uh you need a point which is harder than the aluminium itself so in that case you go for titanium so points that cost easily like uh 15K you know like 50,000 Tios one

point just because they're made with tanium so you can see this sort of attack can damage the machine easily right so this is another example this one goes back to what I was showing before uh with empty connect so here again we have a attacker machine here on the on the right side this is a Pock that's the uh the machine that got attacked as you can see here this is another vendor is a different user interface but basically it's the same stuff so here you can see what the machine is doing here you can see the coordinates um here you can see the code so this is what the operator sees on the machine so in this attack uh the

attacker as I was saying you before remotely pull out the information from Mt connect so you can see all these different points because it's basically uh pull information out as soon as you find a new instruction uh you have a v here so you can see at the end you can basically remotely recover the source code of a program and this is even more severe because when you D the code the code here is interpreted I mean it's not compiled uh and this is the same for um CNC machine as well as indust robots so it's not a compiled code it's like a scripting language as you can see here so if you are an attacker you don't even

need to reverse engineering basically the the program because it's super visible here and this brings you to the next attack which is uh this one I'm going to show you here so as you can see here in the code here you have a value called a variable called K which is vc1 that means K take input from a general uh variable uh called one and I was as I was showing before since you remotely you can write to certain memory address in memory you can alter that value so if you have a machine that is drilling two holes because of this value holds variable two if you're an attacker you can remotely connect to the machine change

that value in memory at that point the machine will start drilling five holes so this is the experiment we did here and shows how a criminal can can hijack the production and basically take control of the machine without changing the source code of the machine can influence the machine in doing something different from what you're supposed to do so this also super severe because in a production line you're producing pieces and attacker can basically hijack with production and change what get produced you can even go farther so this is another example from uh this is another machine so here the machine is executing code a code called z uh 1,00 with the source code here so what the

attacker does it can download the program change the program for example add this comment here reupload the comand and the Machine at the next execution automatically pull the new code and start running that one so that's also super severe it's a writing of a program executed on the machine this another example you can reset the soal advanced management tool so this is something similar to the compensation uh problem I show you before so here is a basically system in which allow you to do um Regular processing so if you have like a program a line that you produce pieces under pieces constantly you need to change tools when they are wasted out so you can compensate you compensate then the

tool it's wasted out you throw it away the machine pick up a new one continue to work when it's wasted out pick a new one so it's called here you can see the list of tools that are picked by the machine to be able to continue the production so what happen here there is a counter that count how many holes the machine has done so in this case on this experiment you see we have like the first tool is exhausted because he has three already 14 holes so we are now working with a second tool which is tool number 10 so the attack what the attacker does it can reset tool number 10 so means that when

machine is producing uh and start drilling with tool number 10 the ATT do you know what uh res at the counter for Tool number 10 so what the machine does continue to use tool number 10 over and over and over so at that point you are basically either you are um creating a problem say you are uh damaging the piece under production or damaging the machine completely because you start drilling with a with a point which is exhausted and you continue drer and you might crash basically the the machine against the piece you are producing with a with an impact on the uh on the cost of production this is cool so uh this is

another machine it's us um it's an us vendor actually for those of that are passionate of Formula 1 you know I love Ferrari because I'm From Italy as as his own like a Formula 1 car actually they start from Formula 1 and then they went into CNC machines this is what I learn so this is an n machine you can see here there is this nice button called feed hold feed hold so what is this it's it's a bit like emergency stop of the machine so when the machine is doing something bad you basically push this button it's basically like uh if you're driving the car right and you have your break so if something happen you just break things

it's like emerg stop so here again by tampering certain area of memory you can disable this this button so it's super severe because it's like if you're driving a car and someone remotely connect to your car uh and disable the brake so while you're driving you have the first curve you break brake doesn't work and you crash so here it's the same and the Machine even tell you the feed hold is suppressed so when you do the attack and the and the victim operator press the button machine said you know what I'm surpressed so I'm not going to stop I continue the production in this example instead is how you this operator which is producing

something on a real war machine here is how you can uh start monitoring the machine remotely so you can dump information like uh the name of a machine the serial number of the machine uh the type of controller which is controlling the machine which program is at the moment and execution here is not running here it's running here the the line that is currently executed on the machine of a program here have a number of pieces you can see the counter here that go from 148 149 so this shows how an attacker can dump anything like he can know like uh which machines are installed on a certain Factory uh which uh programs are executed on the

machines uh how many pieces are produced by U by machine and you know this is like a game over this is another example where you can trigger fake uh um so fake alarm so called software alarms um so machines support uh alarm which is physical like the know there is a uh lack of oil on the machine so there is a problem as well as software alarm similar to interrupt in operating system and this is an example where an attacker can trigger this uh software alarm remotely at that point the machine just pop up alarm and it stop and at that point the operator should go on the machine to uh the buug What's happen and restart the

machine from uh operating and be super severe because you can easily cause AEL service attack finally you can also run some of those machine so this is an example where we compromise a machine we met exploit and we manage basically to um to screen lock you know the use user interface of a operator asking for a rans so that's also um an example of the N service attack so with all of this I want to say that industrial facilities are cool uh because they are deployed more and more with uh smart um devices that often have remote based functionalities which could be like wire traal tcpap functionalities or as we've seen in the begin of a talk

you know based on radio protocols uh the problem with all of those functionality applied to a word which I think is not ready for uh uh for dealing with all these challenges in fact it wider uh the surface the attack surface and it open the uh possibility for a bunch of remote vectors uh so as as uh researcher in security you are responsible of conducting this sort of research to show that there are problems and to identify those problems before criminals can abuse them uh that's our responsibility and um for that reason we here we spent month um going into a responsible disclosure process with a vendor it took us I think something like four months to have

uh I'm not saying all of the is problem fixed but a large majority and this responsible disclosure it turn into a training so we often add to educate them on on the problem of having certain functionalities but this is at the end you know why we do research so to help the vendor into moving to a safer uh environment and I want to encourage all of you to look into this era of research because uh it's just the point of the iceberg this one so uh there are more and more functionalities we did investigate from lack of time and we live for future work that will for sure offer plenty of interesting um uh research um possibilities so with I

conclude uh this are my contact so feel free to connect with me over Twitter uh let's keep it under this name or uh online and I will be happy to have few questions if let me see if time permit uh maybe I don't know just one question if someone has one let me see yes hi uh thank you for the presentation it's a very good research I you've been doing this for a while now right uh talking with vendors and doing this disclosure and training did they mention anything like to any of these attacks being seen on the wild like maybe hey now you're telling me we have these malfunctions are is this being actively

exploited do you know the T of information yes so there are like cases of malware that arrive on a computer that can uh that is connected to the OT Network and and this basically Mal damp information from the machine so it's uh both cases been observed a lot yes as well as ranser it's very easy to run some and block both machines yeah cool I guess we are out of time I will be here all the day and for sure tonight I heard there is a great social event so more than happy to have a beer with you and continue the discussion thank you