Blowing up Gas Stations for fun and profit

[Applause] hello everyone 11th edition of besides uh Lisbon it's a it's a great pleasure to be here with you today it works so welcome to blowing up gas stations for Fun and Profit uh it's it's a joke of course uh don't go on blowing stuff that you don't own if you own a gas station it's it's quite dangerous to blow them up so please don't do it either um so let's let's begin now what do we have on the agenda for today we have uh adgs what they are and how they are used and why am I looking uh we are going to talk a bit about the device prevalence in terms of geographic uh

distribution and uh we are going to talk about the Legacy protocol issues this is something that has been studied before uh at least part of this Pres presentation uh we are going to talk about other devices newer devices the issues that they have uh we're going to talk about two handfuls of new vulnerabilities and how the disclosure process went about those new vulnerabilities and finally uh my quest for physical damage is it possible to actually uh achieve physical damage by exploiting this this type of vulnerabilities you guys are late so so anyway what are atg systems I usually ask the audience uh have you heard about atg systems before show hands nobody likes to show hands so I

won't do it but usually there's not a lot of folks that heard about this before uh for me it was the same like two years ago or three years ago I didn't knew about atgs and I I was curious before because I I found out about this protocol and it spk my my curiosity but essentially an atg is an ICS system that measures and controls big fuel tanks so this includes uh measuring the temperature the humidity uh inside the the fuel tank but it also uh interacts with the with the outside via not only sensors but aters so it can trigger alarms trigger ventilation uh trigger turn on and off the fuel dispensers uh emergency shut

down vales and things like that other peripherals so they are very important for inventory management so if you think about a gas station uh the fuel cannot run out so essentially there's a whole logistic process attached to those atg machines that kind of predict when the fuel is about to run out so it triggers the the request for a refueling agent to to come and add fuel to the tanks uh it also ensures compliance with environmental regulations in in the US for example uh if you have a big enough fuel tank you have to have an atg because they have a process that Auto automatically detects if there's a leak and if there's a leak and the leak uh is

is big enough there there will be fines so uh it's it's very important for for environmental uh compliance now they are not only used at gas stations gas stations is the the use case that that comes immediately to mind but wherever there's a gas tank big enough um those atg systems are used and we are talking about hospitals we're talking about airports military facilities wherever there is a emergency backup generator that's big enough for example an hospital has to have a big backup generator right because it has to power the entire building and the machines there so the atg systems also uh exist at at those locations as we verified in the while now why are we researching atgs

now if if you know bits side um we we scan the internet at scale uh and and it's it's common for us to find uh IC Systems that are exposed that obviously shouldn't uh they they have become very prevalent in our society and to that effect we have been building uh an industrial control system physical laboratory so in in our lab we we we are able to test these devices in a more controlled fashion now this is more very important for us uh because it allows us to properly and safely identify these protocols and these devices as you may know it's they they are quite different from a regular web server for example you can scan a web server and

it's not you usually doesn't crash but an IC System if if you go wild it's pretty easy to to crash the device to reboot it we don't want to do that we are very careful when we are designing our probes uh to have the minimum impact on on the systems that we scan and IC we are we are extra extra careful so as I said the the the protocol had a decent prevalence this was an image from a study that I did in back in October and and yeah I was curious about atg systems automatic tank Gorges they control fuel of course there's something there that it's interesting and speaks to a security researcher right now it's not the only reason they

are also uh this this this threat is not theoretical there are there are evidence nowadays that these systems are current being Target in the Russian Ukraine War uh this is an image of voltage of Team One fist it's a a group that usually publishes uh screenshots of IC Systems that they attack uh on Twitter and this is just one of the models about uh that's a Russian atg that that they attack I don't have any evidence uh about the attack or the vulnerability if there was a vulnerability exploited they they might as well just be connecting and shutting things down um without exploiting any vulnerability itself now throughout the presentation I'm going to speak about old devices and

new devices and what I mean by that is the old devices I'm going to refer as devices that only speak the atg protocol or the Gil baru protocol TLS protocol has many names it's a protocol that runs on for 10,1 uh sometime it's usually identified as atg protocol but it's not uh the name is not consensual and on the on the bottom uh left you have the atg protocol devices that that speak this the atg protocol uh I'm not sure if it's very visible it's supposed to be a world map with blinking lights so there's the different versions We are talking about around 6 6,500 uh Legacy uh atg devices that are online uh today

and the devices that don't speak the atg protocol usually they have some sort of web interface so these are the the newer ones that that I will refer to and the vulnerabilities that I found is about the the newer ones now let's talk a bit about the Legacy protocol so you might have heard the this story before if if you follow IC vulnerabilities back in 2015 HD Moore and Jack shadowitz they they published uh a set of Articles where they flag this as a security U as a security issue uh the protocol the atg protocol is opened by default which means that there's no security controls implemented in the protocol and anyone could connect and and do a series of of things the

protocol is essentially a Serial protocol that was ported uh so it's able to speak tcpip so so you send in essence you send just serial commands to an IP address and you get the reply so yes what could possibly go wrong there um so potential exploitation include resizing tank information that may cause overfill shut down dispensing shut down Leak Detection uh shut down the the network itself so you have the N of service conditions um loss of compliance data so all those potential exploitation impact was actually recognized by the vendors on their own web page where they uh urge the urge the the the users to secure their their devices and take them off the internet which is

curious because it's also one of my recommendations now you might expect that things got better from 2015 to current date they haven't so back in uh in 2015 they worn about 5 5800 atg devices found online and exposed uh last month alone there were over 6,500 uh of those devices exposed excluding gas pods and excluding the devices that may have a security code because one of the one of the features that this protocol has is that it is possible although a bit cumbersome it is possible to add uh an optional six-digit code that can only be six digit and usually it involves you turning on a dip switch on the actual machine itself so and then you you can choose this this

code and and protect the device with this sort of password now it's it's no wonder that usually those devices don't have a password which is just a cumbersome uh process to actually set up a password right now there are many ways to go about fingerprinting this these devices uh even if you don't think about the the password but what what you're seeing there is I I wanted to understand okay how things change from 2015 to current date and am I seeing the whole sorry I I wanted to understand there might be a way that all devices now are password protected so we don't have the the same visibility as before so I started to to

to fingerprint the devices um actually looking at the dcpip stack replies to try to increase my success rate in scanning the devices and one of the reasons is when you try to scan a device that that tries to speak serial uh sometimes the the the the replies get broken in half sometimes they don't reply in a in a fast enough time there are many quirks that Mak scanning a more difficult test than just scanning a regular web server for example and one one interesting feature that some some brands have like the TLs uh 350 is that they use a very particular uh TCP window size so if you're just sending a a syn packet and you analyze

the reply and you have this uh strange window size of 2047 you can almost be sure that you're talking to one of these devices it's just one trick that helps you steer your scanning and then you can conduct a more um slow and more uh you know careful scanning on that device excuse me so uh using those those methods and others uh I was able to get the version counts version counts per Port they they are not just listening on this port there are other on the standard atg port for example if you go to showan you you can search Port 101 but there are other ports where this protocol is actually listening on Port 8 801 is one of them

there are others about password protecting so one of my uh questions where I cond conducting all of this was am I not seeing all these devices because they are password protected and could could could I do something about having just some sort of indicator about devices that could be PR protected so one one of the things that that I did was just okay let's just use uh the 1 two 3 5 six security code which by the way uh comprises between 17 and 10% of all six digit pins in all the word lists that were studied uh so I I took this number I just ran uh the the same scan the same IP address space just to have

an idea of of how many devices would reply 57 instances of devices reply if you do some some dirty Mets uh I I could infer that probably around uh 600 devices out there are configured using using a a security code if uh that table is right right so are could there be like two or three times more devices with the security code sure could could be are there 10 times more devices it's it's very unlikely so the fact that we don't have more visibility on those devices because they are all password protected uh I don't think that's that's the case at least not not not at a very big scale so yeah 1 2 3

five six now now we're going to talk about other so this was all about the older devices the the old atg speaking protocol speaking devices now talking about the the newer ones so if you think about it you know most older devices have this this old protocol implemented and and the industry moved on so there are more and more of those uh new systems being being sold some of these new systems actually have the the option to turn on this protocol because it's so widely used it was the the most successful protocol for this industry so it's it's still supported by by newer machines uh but most newer devices have some sort of web interface right so you

have more protocols you have more interfaces you have a wider attack surface so you know you have more vulnerabilities at least in theory but also in practice now back in March uh there was 10 new vulnerabilities found on six this different atg systems from five different vendors now this was a a crazy week of going through all these devices and tried to fingure figure out if there were uh actual vulnerabilities and what was concerning here at least for me is not the the amount of vulnerabilities or the amount of systems that were vulnerable it's the type of vulnerabilities that were found and and the fact that this was all found in in one week is also uh important

because it's part of a a criteria of the unforgivable vulnerabilities not I'm not sure if you heard about the unforgivable vulnerabilities but uh if if you haven't please look it up it's it it's a set of criterias that Define vulnerabilities that really shouldn't exist at least in 2024 um some of these vulnerabilities that you're seeing here have been fixed in the past but they weren't properly fixed so it's they were easy to to bypass uh other vulnerabilities like for example you know hardcoded credentials I'm sure they were meant to be a features for the the vendor to uh you know fix things in in in the machines remotely um but yeah now about the how these new devices are

distribut again I'm sorry for the very dark map uh but you can roughly see the US on the on the left side and and Europe on on the right side and over there it's uh it's Taiwan uh Thailand and Malaysia um there are some models at least the models that that I'm talking about there are some models that that were more sold on the US for example the protos uh it only happens uh or at least uh I only saw them on the US there are some geography bias um they're heavily focused on us Brazil and Thailand and but some models seem to be exclusive exclusively sold on Europe even if they are from from uh vendors that have

businesses and in both continents now before I go into this I'm going to talk about the the vulnerability types that that I found but I'm not going to to go deep in into any details uh I'm sorry maybe you want some details but the if you look into the vulnerabilities they are so simple that it's it's they are trivial to find so um yeah so vulnerability types command injection uh if you if you dig into iot devices and you try to find command injection you you I mean if you've done this before you know where you have to look to try to inject command so yes um they were pretty easy to find hardcoded credentials um you

know undeletable super administrator and and things like that that that were found um passwords also the passwords that were just too Elite to to be guessed were found which was quite nice authentication bypass um I found a lot of authentication logic client side that makes no sense at all um SQL injection aided by full error log so you just try to do something in SQL and you have the entire output of of your query with the error so you can precisely change it in a couple of tries to to do what what you want uh boring cross F sweeping not going to go into that PR escalation like if you log in as a guest then you have

full administrative power so all of this yeah oh pet rsal not all it's it wasn't not not even pet rsal it was just direct uh file access it just indicate the file that you want to read on the operating system so it's not traval at all now these were all what we could call unforgivable vulnerabilities so the the the criteria for unforgivable vulnerabilities around five five different points first there has to be some precedence so many other vendors or developers made the mistake in the past you know uh there are reports about The vulnerability there are those vulnerabilities are including vulnerability databases and so forth check uh documentations there there is documentation about all this types of

vulnerabilities you know books articles blah blah blah check they are obvious I I think we all agree most of the things that I if not all are pretty obvious so check the attacks are simple uh you don't need to you know encode your payload or whatever they are uh the most common types of attacks for all these vulnerability types check and the found in five uh the issues could be found in in five minutes uh of manual testing or code code review I don't think this is a hard you know limit like five minutes I think 10 vulnerabilities in in one week also classify it as the founding five thing but I I leave that to to your criteria

so now it was a a lengthy process about how we did the disclosure of all these vulnerabilities we wanted to coordinate the disclosure of of them all because they affected the same type of devices and they were quite similar um I can see many faces here that went under underwent through the vulnerability disclosure processes many times so you know there are challenges trying to disclose one vulnerability with one vendor so sometimes there's a a language barrier sometimes you cannot get to the engineer that understands your your your your issue so they can fix them sometimes the vendor doesn't reply or you cannot find the responsible vendor at all that's for one if you try to

coordinate 10 at the same time there's there's it's a huge challenge right so what we did was we partnered up with DHS cesa they they really played a critical role in coordinating all this process uh they they did the the the vendor reach out they they connected with the teams inside the the different vendors so we can explain the vulnerability so so they can fix it um there was some challenges that's that is true uh if you see the the advisor from Caesar some some of the vendors are still uh we are still trying to get a reply some of the vendors were very responsive um we get it all but 6 months after we we finally disclose all the

vulnerabilities they become public on the 21th of September you might have seen this uh on the news somewhere uh and mostly because of of the impact now before I go any further I would like to change to share with you a live demo this is all your fault uh but now I need I need a couple minutes to to make sure if this works because I'm supposed to be showing you a webshell no a a shell give me a second I have some time we more let's

see there some

okay this is the sound of our IC lab in Boston and I'm going to show you the images from the from our lab in Boston if I can put the VLC window on that window the hard part is done yes ex thank

you yes so this is an atg this is uh real time from our lab in Boston and what I'm going to do is I'm going to exploit some vulnerabilities and you will see the result if it all goes well the sound will be quite annoying as

intend let's see first part it wakes up the display if it works oh it does and then it exploits a different set of vulnerabilties showing the complete power of an attacker over the device it first sounds the alarms then it starts blinking the lights because you know it can then it changes the actual output of of the of the window uh so the the operator cannot really trust what he's seeing on screen uh so it it doesn't know which button to to press this can be used in many different ways for example ransomware or just you know if you decide you want to troll some some operator and and do something funny so all of this that you are seeing here

is just using living off the land tools so all of the tools that the attacker can use are already inside the the the this device in particular and because this this is uh besides what you're seeing here is none of those vulnerabilities this is a a new zero dat now I'll try to move on with the

presentation and it worked cool so this this is something that that an an attacker can do with all the vulnerabilities that that that I I mentioned before but there's more I was on a quest for achieving physical damage I I wanted to do a live demo here and and bring an atg that I have back home but the the probe that I wanted to connect that I actually have it's it's much bigger than me it doesn't fit my car so it will be quite awkward to you know walking around with with a pro and an atg and uh so so I didn't bring it so and end the video um so why why was I doing this I think in

our mindset as acers we we also we always try to have the maximum damage or the maximum impact on a system that we try to expl exploit we explore like the extreme case scenarios right so this is something that when I first started researching the atgs I knew it controlled fuel tanks and that was the ultimate goal uh in my research in my head how can I how can I make something blow up hence the title right um and so I I don't think I need to explain the why it's because we can because we like to do it because it's it's our job to explore this the scenarios right and there were many ideas that I have I

don't actually own a gas station so many ideas I couldn't test right um but I could test some living off the land uh techniques uh I I could explore was there something more in the Legacy protocol that wasn't explored before that that I could so so I went about these questions and tried to solve them so how how can you approach this right so den naing of service is Trivial when you get admin privileges on on an at right you just you know reconfigure the network everything stops working uh doesn't make a gas station explode so it's not interesting so what might so you can think of at least in my frame my my my frame of thought was the following

so you might attack either the sensors or the actuators the sensors you know uh they measure fuel they measure water they measure the temperature uh you can actually interfere with the readings but that's almost the same effect as resizing the the tank information so you can also do that and it's easier than to to spoof the readings uh what this does and and this is actually something that concerns the the folks that work on this industry uh resizing the fuel tank information like telling that the fuel tank can hold like 20 tons of diesel instead of 10 reduces the the reaction time that the person that goes to refill those tanks have to stop the process so they don't

have have time to know that the tank is full and there can be spillage if you spill diesel uh you have environmental hazards but if you spill gasoline or or plain fuel it's much more more dangerous so you can have a potential fire there um you can also disable leak dete um you know there there's other problems that I texting that I can explore I know the atg experts are are concerned I am concerned Why shed GPD thinks the there's a a fuel tube going into the radiator of the truck there but you know who knows now much more interesting is attacking the actuators the actuators actually interface with the with the physical world and all this all these

atg machines usually have an internal or external relay board where you can connect your peripher peripherals right so they they trigger alarms they they trigger HVAC systems they trigger pumps uh and it became obvious to me that abusing the relays would be a very interesting attack Vector now depending on the relays used it's you can actually go online and find the me mechanical and electrical characteristics and they usually have this this this chart in some way or form which is the the durability how many times can they be switched on and off before starting to fail and you do some quick map map to understand if if you switch them on and off fast enough can you achieve those uh

those times and in fact in fact you you can and I figure out that um well if you if you work with with this type of relays or you have an electrical engineering background maybe it's obvious for you for me it wasn't uh that you can actually turn a relay into a light bulb which is pretty cool um now of course not all relays will fail uh in this particular uh way um but this is actually relay uh from inside one one of the one of the atgs that I have I I tested them off the atg because atgs are expensive I didn't want to you know break the atg so but the the conditions

under which I stress stress the relay were were real and they were actually using again uh living off the land techniques it's this while true in this particular model the the board that controls the relays controll by uh i2c uh there are other models that's control with the the relays are controlled under uh under a different um interface but in essence what you need to understand is if there's a relay and there's a way to interact with a relay you can probably abuse the relay in in in this way they might not you know Catch Fire like like this I'll play it again because um they might not catch fire but they will eventually stop working and they

will stop working pretty fast if you switch them on on and off really really fast and that leads to physical damage now how about the the old protocol can can you do the same thing in the old protocol without you know uh gaining admin admin uh access it turns out that you can so if you read the 600 something pages of Serial commands that that those devices support one code is very interesting it's called set relay orientation so the the old protocol has no way to actually turn relays on and off but it has a way to configure the relays and you can tell the the AGG that hey your relays are should be normally

open but if you then tell it again hey your relay should be normally closed is e will will understand the current state that it is and it switch to the right state so by just reconfiguring the relay you can actually achieve the same result and the same output which is turning the relays on and off really fast although if it's uh via the network of course it won't be as fast as you seen in the video it will be much slower it will probably won't catch fire but doesn't need to catch fire it just it just needs to break and more importantly uh um it just needs the even if the relay doesn't break what's connected to it

might break faster so if there's a pump connected to the relay I'm pretty sure it will break faster than the relay itself now where am I yes so in a nutshell is physical damage possible yes it is it's possible directly uh the atgs with mechanical relays uh can have those components that the actual relays damaged by an attacker and the peripherals that are connected to those relays can also be damaged by by turning them on um I enough really fast even if you are using optical relays there there are boards that might have I never found one but there might have uh there there might be boards with Optical relays that they don't suffer from from this um from this stress

testing but the peripherals that connected to to those relays will will suffer and this is direct damage so but there's also indirect damage right so if you modify the the tank geometry or if you achieve a spilling a spillage uh if there's a fire there's indirect damage so you you can actually have a you know gas station blowing up uh but also if you just change the tank information and switch diesel uh to gasoline for example when someone's going to refuel their car they might get the wrong fuel so you instead of damaging the the atg your damaging the cars that are refueled so there's a lot of damage going on now conclusions ads are usually part of

critical infrastructure um there are more uh adgs exposed now than 10 years ago and the new ones have a even bigger attack surface so it's um it's good for security researchers um there seems to be some some sort of luck of investment in security of those systems judging by the of vulnerabilities found physical damage has proven to be possible and not only on the newer ones but also on the older ones and we know that atg systems are actively being targeted by techer something needs to change and I I keep walking on stage and talking about uh IC or Hardware or iot and stating that we need to change we need to take this uh

we need to take security very seriously uh like we like to say but um we don't see a lot of this in practice and it's scary so what can you do to drive change if you're a security professional you can identify any atg and get them off the internet the vendors recommend that you get off the get them off the internet so how much clearer clearer does that message needs to be the vendor doesn't want their own device that speaks internet to be on the internet if you're a manufacturer you know should use secure by Design principles understand the supply chain cyber security principes uh you can try invest building programs that accurately detect misconfigured or

exposed systems we we are working with with some partners that they do just that so the vendor proactively tries to understand how their customers are using their devices and try to tries to reach out and and actually uh suggest fixes thech Mis configurations and so forth so I think that's part of the future of solving this uh like c i problems and work for policy makers um should understand the risks of exposed industrial control systems and I think one important part is quantify the impact so when usually when there's money involved things start to move and if if folks understand the potential impact in terms of money that will they need to spend if something bad goes

wrong uh something bad happens uh maybe there's there there's some change change in the Horizon and I said if you work with icot and you you want to reach out feel free if you look at AG security before uh reach out please or if you work on the fuel industry especially if you want a gas station I I would really like to talk to you I I promise I won't blow your own gas station up but we we can try some something else and that's it and I'm open to questions if you have them thank you

[Applause] don't be shy oh there's one question two

questions hi uh just a quick question when you were um one of the demos that you had uh with the um op codes that you were sending did you try to fuzz those to find undocumented functionality that might be there etc etc uh so the the the number of functions that are implemented vary from adg uh to atg we don't all the the the the atgs that I have uh in in my lab I I did try to to First sum uh they usually break so but I I didn't find any undocumented stuff it's pretty well documented actually uh they they are I don't think they they need to hide stuff because you can do pretty much

everything off

thanks well great stuff again you the responsible disclosure process is not easy but I'm wondering about the patching because for this type of devices it shouldn't be easy uh what about your experience about the patching here yes so now uh to be fair to to to the vendor some of these devices are are very very old um they were designed uh and and the protocol itself was designed uh before uh the internet be as it is I would I would assume so you're having like devices that speak the serial protocol to connected to an IP address that's how old they are now when you think about how how you go about patching when you have a series of new devices in the

market and maybe the folks that designed and and coded um the interfaces for the old ones maybe they are not alive anymore so so it it makes things harder so what what they can do is to provide Rec recommendations to to old customers uh they can try to understand how how how their systems are used even if they are unable to produce a patch and I think what they started to recommend to everyone is is the same thing get those devices off the public internet because although they are still vulnerable and and you can get you know access to a local uh Network in many different ways uh at least the the long they they won't be the low hanging fruit

right but the the patching has been hard there there are some uh some of the devices uh if you go look at the advisory that there won't be any patch for for them so it's hard it's a hard [Music] challenge great presentation uh from the world map I have the impression that Portugal was not so much affected with these vulnerabilities if so if you if so do you dig on this do do you try to discover which atg is Portugal gas station F yeah that's that's true uh there weren't a lot of uh AGS exposed in Portugal uh at least the models that that we are talking about here um I'm not sure if we are doing things right or

we are not using this these particular models so I I don't have a better answer but I can tell you that I was speaking with some folks from the the oil industry uh in Portuguese oil industry recently and uh they are interested in in digging a bit more about it and try to understand the reasons behind this yes any more questions nope thank you ped thank you