Industrial Scale Hardware Hacking - Anthony Clark

all right back in Action so next up we have Anthony Clark Anthony has been a cyber security researcher for over 20 years he was a researcher at Al National Lab and is currently a GU scientist welcome [Applause] Anthony it uh so a little bit more about my background I founded the red team atos was one of the original sea team members I built the tech security program for the nuclear weapons division of the lab and then I founded a number of right now I'm running company called red cor lab uh one of my previous companies I did Consulting for a hedge fun called bridgew Associates and they have a concept called building a machine which is a way

of Designing a process around what he doing in their case it was around stock trading and so what my talk us about today is we built a machine process around reverse engineering hardware and software and the reason for this is uh with the Internet of Things avionics RS uh Industrial Systems the internet network communications are prevalent across so many devices that uh there's a large need for security assessments at the hardware level and we're assessing so many devices that we needed a rapid process to be able to do so I'm to cover and our specialty is taking uh technologies that we've never looked at before learning how they're supposed to work reverse engineering them and

leveraging them in some way for attacks effects vulnerabilities um and then the types of devices that we're looking at are scientific instruments uh avionics machines inlight entertainment systems physical security systems like electronic blocks that might have a we've worked on General Motors Hyundai Automotive Systems reverse engineering their passive entry passive star for example so this process applies to across the board on all types of hardware systems so this graphic gives you an overview of what this process is uh that we've developed and I'll give you a high level and then we dig into details on each one of these areas so when a new device comes into our our lab the first thing we do is catalog the device collect all

of the information model manufacturer Dimensions uh high resolution photographs inside and out all of that type of information then the next thing that we do is an open source review which is where we look across uh FCC databases uh vulnerab databases vendor websites anywhere that we can get information about the device that we accessing once we've covered that area of information we do a RF radio frequency assessment of the device and I'll show you some slides later on what that looks like but essentially we take the device put it in an isolation chamber and record any admissions that it makes or any Communications using Wi-Fi Bluetooth Zig other protocols other things that we can do to it

perspective as well uh then we do a full Network survey of the device what how does it communicate on the network does it have Services uh web apis and along those lines our next step is a hardware analysis we're looking at debugging ports um you know chips memory where it's storing its firmware inputs and outputs to the device those sorts of things once we have all that done our next step is software analysis and this involves downloading software from the vendor uh dumping software from the device itself and then running that through a reverse engineering process where we're looking at potential vulnerabilities ways that the software is use bu capabilities things like that and then

we build tools so for example in a pen testing environment you might be using tools like metas oress or any of these kind of off the shelf tools that use to scan networks and software from our perspective we're usually building custom in house device that we're looking at on C available in those the next step is to build a knowledge base all of this information becomes useful when new versions of the device comes out other types of devices that use similar Hardware components or similar software so we build a large that we can refer to as new devices and then finally my favorite phrase is you didn't document you didn't do so we document all of this and

provided for our customer the process all right so in order to enact this process you have to have a lab to be able to do all these different steps and the way that we've approached this is a bench concept where we have a bench for each one of these things so for example we have r test bench where um we collect any signals leaving the device we have tools for all the different protocols that could be seen in the device we have a virtualization infrastructure because often we have to um emulate the devices software or you know many of the devices we look we look at are based on arm and so having an emulation

capability for arm where we can do destructive tests without actually affecting the device itself very useful and then a hardw test bench so for lab equipment what are the types of things that we use uh these are some of the basites in this screen we have logic analyzers we have cillos Scopes we have multimers um and alongside that a whole host of different types of gear so uh being able to isolate the device on a standalone Network or air gap network is important many devices these days communicate back to the vendor and if you're doing um destructive tests or offensive Tes into the device you don't want to impact a vendor or another customer on accident

so doing all the sort of testing in an isolated environment is key we also have a host of small programmable devices that we can use to uh affect the hardware that we're testing these might be Aros or Rasberry PES some cases we have uh USB devices that we plug in in order to ulate keyboard effects one good example of this is um with AV onx and flight entertainment equipment if you've ever been on a plane that has a little movie screen on the back of the seat a lot of times those are running an Android operating system and there are uh secret sequencies that you can do to get into the operating system of the panel and so

having a USB device that can emulate a keyboard is helpful in a ATT uh also in addition to those sorts of devices we have a an Imaging bench where maybe we want to take a very close look at the devices Hardware components chips CPUs uh Network traces and so you're you're going to need something along the lines of a uh electronics microscope uh we also have high resolution cameras for taking closeup pictures and I'm going to walk you guys through a case study of how all this works so with such a complicated process you also need a way to track what you're doing track your steps coordinate between different team members know we have a hardware Tech we have a a

software person have a engineer so you want to build out a a board or a process for tracking everything that you're doing when you're doing this Hardware testing in our case we use it's called canand board it's usually used software development we use it for tracking the steps that I described in the diagram so this is an example of what Target cataloging looks like uh in this case we're targeting a metronics uh medical device and so what we do is we do a tear down of the device documenting the steps you know what tools you use what uh size bit do you need to remove the the proprietary screws or whatever whats outputs the device have so in this

example you can see we've got um you know a phone line input we've got power and we have connections to other types of medical devices we catalog all this information in our knowledge base including any data sheets that apply to the components on the board um if there's difficulties in removing the case sometimes this Hardware will have security features such as a a coding over the chips to prevent you from holding them or identifying what they are so that's the catalog step then we dig in really deep to uh specific key components on the board for example um Eon man flash anywhere where the firmware might be stored on the device as well as uh pcpu at

pgaas and then catalog all the pins and configurations for these different components because these are uh critical in the attack phase of this process where we're trying to find ways to compromise all just briefly this is what the open source review might look like um there's a fcci database for any hardware devices which use radio frequency Communications and a lot of times you can find a great info there as far as you know frequencies interference noise um you know they do a great job of testing Effy devices and so you can lever some of that

information so for the RF survey part of the testing we have an isolation Fa Day cage uh normally we don't have a laptop in the page would made it easy to demonstrate the photograph and so you would put your Hardware device in here there's ports for antennas um ports and then you're going to measure different things you want to see what the device does when you power it on does it make any radio communications you also want to see what the device does under normal operation so user interacting with it connecting to it over the network does it generate any animations there so we use a spectrum analyzer U our fa page goes up to 6 GHz um we typically don't see too

many devices going above that but it is possible um um emanations indicate potential attack surfaces or areas where we can snoop on traffic for our Our Fair Day Pages we have pass through ports for Network USB and power that way you don't have to um have penetrations in your our children so there's higher sophisticated tools for doing this type of work I'm going to talk about the types of tools that you could buy as aumer hobbyist uh so for our lab we have things like the hack rf1 uh running in spection analyzer mode um and that's what you see here collecting animations from the device we have tools for RFID tools for Wi-Fi tools for

Bluetooth software defined radios all of that sort of thing one additional thing very useful in this kind of testing is what we call near build analysis this is where we place a probe very close to components on the device board uh so in this case this is an example we have a Raspberry Pi and we have a near fill probe you know just a few millimeters from CPU at a specific angle that gets attached to a uh a high gain low noise amplifier so you can pick up any radio animations coming from the device and then that's connected to an oscilloscope in Spectrum analysis mod so this lets us collect very fine emanations coming from the

device and what's interesting about this is that you can actually tell the difference or fingerprint different processes running on the device so for example uh we put a crypto minor piece of malware on the Raspberry Pi and we could distinguish that from say Apache running on the Raspberry Pi so doing that level monitoring you can actually finger print processes running just by want C

you now for the network uh survey portion of the testing this essentially involves running Network scans uh doing full packet captures very similar to the RF approach device goes on an isolated Network every single thing the device does from a network perspective is captured and cataloged and analyzed um we have a bunch of different ways that we do Network tapping this here is uh hack five plunder bug it's a very small almost like USB key that you can attach to the network and put of the device and capture all traffic now why we do this is things like uh a lot of these devices especially Internet of Things uh industrial devices they're Securities about at the 1990s level uh often they

don't use encryption for the nwork and you can capture things like clear Tex passwords which this is an example of that for the web server that's running on that device there uh next something people are probably more familiar with just regular vulnerability scanning in this case we're running a burp scan against the devices web server looking for kind of loow hanging impr vulnerabilities as the previous speaker said often times you don't have you just need donutes you don't have to go to really extrem in order to to compromise a lot of this Hardware uh in this case the uh the session token is is available this device doesn't use https the session token is in the URL that's communicated

by the user and capture that off the network and then recap it once we have an understanding of how to communicate with the device from Network perspective uh the next step is to locate potential effects so what this particular device does is it's a network controled power strip so like in our lab when we have devices under test we don't want to be going in and pluging unplug you turn things off and on it's helpful to automate that process for example when you're doing the RF testing you want to be able to remotely turn the device on and off to capture the rfm so this device is Network controllable so we're looking for effects against that device in order to

do something is unexpected in this case from capturing the web application traffic what we identified was is the user just sends a command via the URL to turn the power port on and off and there's no real authentication or protection on this Comm and so our next step is to build a tool that we can run in this case python script which sends the same command to the device over the network and turns the power off on the so in a you know I've seen these in we did some work for a satellite company and they were using these in their test lab and we could turn off their satellite sensors for mely just using

this effect all right now for the hardware analysis phase this essentially involves uh disassembling the device looking at the circuit board and trying to identify anything that we can use to attach the device and interact with its firmware so these might be JTAG debug ports Ur ports SBI ports uh and there's a web database called JTAG test that provides pinouts for a lot of these different Dev ports you can go look those up um in our case like if you look at the middle picture there one of the first things that we do is take a multimeter and measure every single pin that we're able to access on the board uh in order to identify which

pins are likely to have data whiches are ground things those on and this is what our pin out analysis table might look like that so after we've touched every pin on the board we're writing down every value that we find and trying to identify what could we hook up to where might we be able to have an effect one of my texts made this so this is using the Sally logic and what you do is we use a Raspberry Pi for this example as well you connect the logic analyzer to all of the pins on the board and then you power up the board and the logic analyzer will actually tell you which pins have data on them

which on ground so it's an automated way of doing that just describe the the other thing that you can do with the Sally logic analyzer is decode data that is being transmitted so in this example we are transmitting hello world capture in a real world environment the board might be communicating to not Hardware device or providing information so this is one useful way Capt all right so once we've done our Hardware analysis next step is software analysis because most of modern devices today have web interfaces or Windows software that you can use to communicate with it and so we're going to be looking at executable files B AR it's a Linux based operating system which a lot of

devices are now forw reverse engineering and trying to extract data from the uh the memory on the device so our process ends up generating I know this is probably not legible but I'll just talk you through it it ends up generating a uh a database of information about every file that's used in the software for for that particular Hardware device so in this case you're going to get Che some file sizes if the binders are compiled with security protection it's like memory randomization um any useful strings libraries that the software depends on and all that sort of information from a vulnerability analysis standpoint this is very valuable information so once we've cataloged all of the components of the software then

we do assembly level reverse engineering so in this case uh this is for the metronic medical device that I pointed about earlier this is what the flow graph of the software looks like so every function of the software is it's connecting to pieces of code this is what it looks like it's helpful to see that from the cap perspective and then you can also find interesting things like when we dis disassembled this particular software we got an error pop up in our disassembler that gave us a hint as to where the developer of the software was storing their code and their debugging symbols for that code so that's this line here this is something that you might be able

to go search for on the internet and if you get this it'll provide very valuable reverse engineering

information so from a reverse engineering C point we want to identify uh functions that the software is easy to control the device that we might be able to take control of such as let's say you're connected using the the manufacturer software and you want to watch where the software is writing files to your computer by pulling data from the device that say it's a cpad machine it's going to be pulling data from the pressure on the hose and writing that to a a database file or a text file that might be useful from a standpoint to know where that's getting ready to so when we our we look for all that sort of thing same thing for register keys if

the device writs to registry keys we want to know about that we also want to know what the file permissions are the registry permissions are uh service permissions these are all things that from our ATT perspective we may be able to to prob about 10 years ago Adobe had a situation where their software would install uh service on Windows and that have vulnerable permissions you could then leverage that service to get admin on on the computer just by way of them having Adobe inst of these so another thing we end up doing is disassembling Java many of these devices are using Java software for control and when you reverse engineer that you can find um where encryption

keys are created types of Hardware the software is design communicate in this case the ftbi drivers for USB uh we've been able to inject code in these sorts of things and cause effects on the hardware so that was the static analysis part of of our process now is the dynamic analysis part this involves running the software itself and capturing everything that the software does on on the computer uh file system changes registry changes artifacts and memory uh processes that get created network connections ports that get open and so we use tools like CIS internals for capturing that information audit D or osquery on link and OSX and then debuggers this particular screen is a I think x64

debug and in a debugger we can see the software loading libraries we can see the uh functions that get called by the software and set break points on these things so that we can watch while the software is doing its operations and pinpoint where vulnerability might be so in this case for that metronic software by setting a break point uh on particular function I think read file in this case we can see that it's reading in a URL for it to communicate with so that might be something interesting from a a testing perspective what's at that URL can we emulate what that's doing and about Hardware make it do things it's not supposed to

do okay so many pieces of Hardware now have mobile control medical devices do this uh home iot devices and so we have a process for reverse engineering overlaps as well and I'll just talk you through that briefly uh first is static an analysis of the mobile software so forro APK files usually you can find these APK files on the site like APK pure uh download that run it through a tool called mobsf which will give you a report like this that tells you vulnerable permissions that the APK is using if it has any clear text files or sensitive data all kinds of useful things we also use the same disassemblers and debuggers like we would on uh

other types of software uh another thing that we do is injecting a certificate into the mobile device so that we can snoop on the SSL protective Communications there's a tool called APK man in the middle that allows you to um remove certain configurations that would prevent that from happening and then you can just watch the traffic going back and for to the Target we did an analysis of Tik Tok to application because it was in the press a lot for you know being potentially a malicious piece of software and it's a very large APK so what we had to do was spin up a huge a AWS mode with a lot of memory and processor power so that we

could break up the APK remove his certificate protection inject the certificate and then monitor all TI to tr and uh the result of that process was Tick Tock from uh software and network perspective is no better or worse than any other social media software meaning it steals everything just like Facebook does or

Instagram all right so we've taken care of the RF side of things we've taken care of the network side of things and uh the software side of things now at this point we have some ideas of vulnerabilities that we can leverage that we can P so we build our own tools to make use of that and we provide these tools for our customers so if you go to the red C app GitHub these are all free you can download them we have tools for doing that file scan that I showed with all of the you know dependencies and check SS and file information we have tools for exil trating data over HTP from devices um

useful uh a Unix Recon tool so many of these devices are running an arm processor and some Linux Vara and if you can get this tool on there it will tell you everything about users passwords groups uh file system permissions running processes it just does that in an automated way and send it to you so check those out if you get a chance these are a couple of examples of our Tools in action uh I know it's not legible but this particular tool what it will do is it'll if you install a piece of software from say Amazon you have a an Alexa this will record every file that gets installed on your computer where it

goes what file permissions it gets um what type of file it is check sums all of that and this is a similar example for portable executable files like Windows based files once you have your tools built you've collected all your data you've identified V abilities you want to build a knowledge base and this is an example of uh web application knowledge base that we have that we're working on where we store all the photographs reverse engineering call graphs file information like what I showed you a minute ago from my scripts so that way you have a central location you can go search every time I have this component what devices does it belong to what vulnerabilities

does it have is it related to a different device that we can be looking at and then finally documentation if you didn't document it you didn't do it generally we end up with very large reports 100 Page Plus reports with a lot of dependences containing all this information uh and this is what we provide at the end of the day to to the customers uh and usually one the documentation should have roughly three audiences executive management technical staff and then other reever Engineers or Security Professionals deep technical staff executive summar is one page high level details maybe five pages deep techical details and I'm out of time for the case study but if anybody's interested I'd be

happy to show you off on at that point I'm open it up questions

hello I got uh two questions one is um have you ever done any testing on space system equipment or parts or uh Hardware or software and my second question is are you have you ever been shocked or surprised or either about finding individual Hardware Trojans or back doors or the the the amount that you do find sure thank you absolutely um we have tested space related telematic system so the software that's talking to sensors on Space devices we've also tested some sensors that have gone on satellites for example U nothing as far as like propulsion control is more scientific sensors um and then as far as the prize yeah definitely the we tested a badge reader system that's used in very

sensitive places and the software for the badge reader had a backrow password and it was made by a foreign company um and the password was the name of the software that was that was very shocking we're often shocked with how vulnerable these devices are but only a few times if we found like an intentional malicious backb troen in the device actually is pretty rare unless you're getting bra market devices from China thank you any other

questions there Jonathan Security operation space um thank you interesting talk for sure um questions one is there an average timeline to the type of contracts you get and then second seems like it's really interesting that um your companies able to to come in uh and reverse and these problems it seems like perhaps it's a bit after the fact um can you speak to anything regarding um is there is their Intelligence being put into you know these clients of yours that they might want to to have this information before they go and purchase Hardware is there is there much uh work being put in that area sure absolutely um so it is a bit act the fact often we're

working with Hardware manufacturers themselves and so in a few instances such as with Avon's equipment we've been integrated into the the production build of the devices so it's part of the iterative you know as they Dev alling things we're testing getting feedback we can verify it um if it's a customer that's implementing a device I usually try to dissuade them from hiring us I say are you sure that you want us to test this because you've already bought it and we're going to break it and then you're have to do something about it um that's the most common scenario that we see is someone bought something and someone said oh maybe we should have this tested it's kind of

you know it's good to know that you're rable but at that point they've already purchased it it's hard to get leverage on the vendors to things often from that perspective uh we did a test on some electronic keys that were protecting sensitive uh materials and the keys were super vulnerable but they expect millions of dollars replacing all of their locks and keys and it was a little bit at that point um as far as the timeline I would say depending on the the device 3 months is about normal um for small iot devices we can be those or two it often depends on how deep the customer wants to go they want to know what a funded highly capable adversary

is going to do then months I I saw somebody over here is there another question all right well thank you very much oh