An IOT War Story- Jason Spencer

good good morning everybody um hello and welcome to my talk my name is Jason Spencer um my talk is called an iot War Story uh just quickly a little bit about me um you can find me on Twitter at uh breaker of ss I'm a security Analyst at Orange cyber defense where I've been for the last three years um what I enjoy doing in my personal time is surfing golfing and uh hacking on the weekends uh um so this talk will go through kind of the emotions of an assessment from scope fails to the learning I required to do an assessment how clients put things together um I'll then focus on two different hacks I did

and then um driving a struggle bus up Sony pass trying to find a solution um the original title for this talk and maybe what I should have kept at was am QPS AWS API iot assessment or acronym soup and how to deal with it um I was on an assessment about a year ago and I read through the scope of work and I said something something's off it was an API assessment but it kept referencing Some Cloud component and I was like maybe like an API in the I don't know so I go to my friend Simon who worked on the scope and I said hey dude what's what's up with this something something just seems

off um so he says no no it's all good it the cloud comp just ignore that this is a Swagger assessment very basic you know go ask for the Swagger files in the kickoff meeting and you'll be fine so you know feeling like this is a pretty standard assessment I decide super confident I'm going to walk into this meeting with my notes everything I need um you know I'm confident I can tell this client what I need um so big smiles in a meeting with a lot of people and I say hey guys can you give me the Swagger files and genuinely this client started laughing at me um so I'm starting to feel a little less confident in this

meeting but I'm still I still know what I'm doing so I say so then he just starts spouting acronyms at me just throwing words all over the room like we're all just supposed to know what he's talking about AWS a amqp iot and I was like what um and then he had the audacity to stop look at me and say what do you know about amqp and let me tell you honestly I had never heard this term in my life um I kind of felt like a deer in the headlight the first thought I could possibly have is well [ __ ] you Simon You' you've got me in trouble um didn't really know what to do

just sweating beans um thinking you know like well well [ __ ] um fortunately one of the first things I learned when I joined sense post was the best hackers are the ones who know how to Google w so whilst he was throwing acronyms around the room I thought to myself the best thing I can do right now is to Google the thing I know the least about you know right now I didn't even know if amqp was an acronym um so fortunately I had done that I Googled and one of the first things that um came up on the Google search history uh Google search was differences between amqp is and mqtt and I figured you know

if there's differences there's got to be similarities so I said to the client well I don't know much about it but I know mqtt and I know they're basically the same thing and it must have been true because he bought it um so kind of did feel like a bit of a superhero moment I jumped over the car um you know feeling pretty proud of myself I could go on and do this assessment you know sweat off the brow moment first thing I had to do was tell Simon that he was wrong he didn't know what he was doing he shouldn't be my my boss and uh he sucks but then after that I needed to figure out what to do so now

we're going to talk a bit more about the boring stuff the learning that you needed to do in order to perform this assessment and then some hacks that we did um before I jump into the learnings we I needed to have a look and understand you know what is this what is this architecture what are we testing um before we get into the nitty-gritty of what is amqp so this is the client's architecture so a device or an iot device is speaking to a manager that manages that iot device with mqtt that manager speaks to a message CU with amqp a consumer reads off that message queue with amqp and pushes data onto a global

data Lake and then from oh I'm behind on my numbers and then from there a customer portal uses an API query to query that data L for data relevant to them so as I said in my original but this is an amqp AWS API iot assessment so um the a AWS is this rabit mq endpoint lives uh that's the cloud component the rabbit mq endpoint is an a AWS API Gateway so we're testing not from the iot device to the manager we're we're not testing from the consumer to the data leg or from customer portal to the data link it's simply from the manager to the Rabid mq um API Gateway um so I had to say that I assumed the

breach of that manager so that I had credentials in order to interact with that um that service but just for interest sake let's have a look at this section here all right so one of my very smart colleagues George and later Rogan and William repeated it had to test this attack Vector here so unfortunately in a rather trivial start to the assessment um for the client um George was able to uh boot the machine in recovery mode add his own root user and as the Brits might say Bob's your uncle he had everything um so George did a bit of dumpster diving and one of the first things he found was configuration files lots of

fun things in there but most importantly for me the Rabbid mq credentials um an important fact about this manager is it is delivered to the various customers so it lives on site at the at different customers and isn't managed or controlled by the specific client with testing which means it opens them up to local physical attacks which georg's attack is then plausible which makes my attack plausible because I was able to get the credentials um so we're going to assume we're going to go from the manager from here on out so we can just erase this and just assume we have control of the manager and we're trying to push data to the customer portal um so now as I said before we're

going to discuss some basic stuff um learnings just to understand what iot is you know for those who may not know it's just things connected to things it's quite simple it's sensors or Internet connected sensors that um gather data about itself or its environment and push that data to a hub or a Gateway that Hub or Gateway will either analyze the data there or it will push it further to get analyz somewhere else in our case of course we're pushing data to a Gateway which is then pushing data onto the cloud in order to get analyzed and then we'll push that data back to the iot device so one example of using iot in our lives is cars that are using U

Google Maps or ways in order to generate the fastest route so they have the ability to look at other cars see where there is traffic and push you down a direction that will make it faster for you another example is wearing a SWAT watch now in my instance my SmartWatch is gathering data about its environment me my heart rate my V2 levels my sleep patterns it's pushing that data via Bluetooth to my Gateway my cell phone um which is then being analyzed and pushing that data back to my watch so we use iot in our lives um to live uh to have a smarter easier life you know we want everything to be kind of maintained or be in a state of

homeostasis so for for businesses we want iot to help our customer experience to save us money to save us time um in order for us to further uh sorry in order for us to make the lives of our customers easier because everyone else is doing it if your job if your um business isn't making the life of the customer easier they'll go somewhere that's faster for them to get out of the shop on the flip side however it adds risk so increasing our attack surface every time we add something to the internet right our digital footprint expands so a Smart Lock which might add security because you can check remotely if I my door is

locked you know if I can log in on my phone and have a little bit of extra usability because I unlock my door before I get home or before I get to the door however should that be hacked of course that opens up further risk for us right if that can be opened by anyone so most people who have iot in their lives probably have a little Ros reie um hosting some sort of Home assistant um and they use it to just manage their home environment their life right you know are my alarms said are my windows closed what's my airon doing recently I was talking to my boss about this and he was showing me his home assistant

implementation and said how and said how when he's on his way home he logs on the platform check checks the uh temperature of his rooms and retro and um corrects that so that when he gets home he can um be in a comfortable environment now what if I could modify the temperature he is seeing um you know if if I could modify the temperature he seeing he might do the adverse of his intention he might make the the room hotter than it already was um now this wouldn't do anything serious other than rarely annoy my boss um but what if instead of was his room it was a fridge or a set of fridges in a in a

warehouse and we could ruin the produce well that's that's the Crux of our attack you know I'm not attacking the iot device itself I'm not attacking um the the data lake or the customer portal I'm trying to modify the data that's seen at that customer portal in order for the data that's being sent back to um to negatively affect the iot devices on on the way back now if we were at scoped in order to test the iot device itself sure I could try and send um illegitimate commands in order to make the temperature hotter itself on the device however I can if I can push data to the customer portal and get the the a

legitimate command from the customer portal to affect the room the device the iot device will have no way of verifying that it's incorrect and will just act accordingly so it might seem more legitimate to come back that way so naturally the first thing that we might ask when you hear am QPS AWS API iot assessment is what in the hell is amqp so an advanced message queuing protocol system is quite simple it's actually just a it's like a simple Q it's mqtt or anything else it's just a data comes in and it get puts on a que and it comes out the other end um amqp is an application layer protocol but it's also um it defines find the network

layer and the high LEL architecture of the broker so when I when you see rabbit mq that's just this specific implementation there is other Brokers that can utilize amqp but amqp defines how that broker must work now you can also you can also Implement certain of your own rules but there are defaults that need to be implemented in order for mqp to function um so to start this let's try and understand what exactly a q is all right so cues are extremely simple in their design generally they're comprised of three parts a producer the message queue itself and a consumer message cues work um in an asynchronous communication method um meaning that a producer does not need to wait for a

consumer to absorb um or deal with that data that's being pres um sent in order for the producer to keep working sorry a nice example of that is when I send an email to Leon de is way too busy and too cool to talk to me so he never replies um so I can keep doing my job I don't need to wait for him otherwise I would never get work done um cu's work synonymously in real life as they do in message cues and that's in a first in first out manner which is first in the queue first out of the queue um the clients are the producers and the consumers whereas the server is the middleware so that's the

broker that we were speaking about um when I originally gave this talk to Leon uh he's unfortunately for him had to sit through it many times um I showed him how amqp works and even him who is much smarter than me said dude that was so complicated I have no idea what you're talking about um so we're going to be using the analogy of me going to buy an Apple at wws in order to understand how this works now wws I'm just using because their cues work synonymously with Rabbid mq um whereas if I went to pick and pay we separate into different tills this is just one large queue um so if I was to go

shopping at w for an apple um I join the queue with my Apple I stand behind this lady and when I get to the front I can buy I in this case am the producer the apple is my message and um the man at the checkout counter is the consumer in this case after he's passed on the message he happens to pass that apple back to me but what the consumer does to deal with that message is not my business I just need to give it to him sorry this is me getting to the front so if I oh sorry so again we're at wellworth there is not just one CU there's many cues so now I'm going to go

through the concept of how consumers work so multiple consumers can come to work every day so w wor there's many consumers I am in dark here the fourth person with my Apple I will the first lady will go to the first person the second man goes to the second um till the lady there the third person goes to the third till and so on uh sorry the F the first till again if two consumers came to work that day um so an important facet of how rabbit mq and will works work is once I go to till one I don't go and show my food to till number two that is only um visible to the first till

person so if I am chosen by the first till person that is who will see my data and that is the same as with rabbit mq if I am the first person in the queue I'll go to the first person who subscribed to that queue that's the first consumer now the next question we might ask is who can consume now at wws same as at um the same as in rabbit mq we have the ability to Define exactly who can do it so obviously somebody from Pick and Pay can't go and stand behind a counter in will wordss and say Hey I want to serve you so you need to be authenticated to serve somebody you need

to have credentials in order to subscribe um but the next thing is okay cool but can any wws employee go from anywhere in the world well wws can say anybody with a badge can go and stand behind account and help pable wws also has the ability to say hey only people from this very specific Branch can go so in rabid mq terms if I have a very specific IP address then I'm able to consume off this network but what happens but wws also can say okay cool but I don't just want anybody in this Branch to go I want people in this Branch who have a who have been hired as a counter worker so again in rabbit mq terms that

would be somebody who can subscribe to a very specific virtual host which I'll speak about in a bit more detail now um so somebody from a very specific Branch so an IP address on a specific queue in a specific virtual house so rabid mq has the ability to say only certain people can consume or they can say anybody who's got credentials so again we're at ww and now we're in the food section um but there's actually two cues I lied to you earlier there's one cue that everyone can go to and there's a second queue where only people who have 10 items or less can go now the second cue is what would be referred to as a rooting algorithm right

so do I have 10 items or less or not I.E am I just here with my apple and if I am then I'll join the root the Q so in rabbit mq terms that would be an exchange of type direct or the default case for how it works right so the default way is if I match the rooting algorithm I'm allowed to join that cue um an exchange is what we use to bind um messages onto qes so it it Roots it's a rooting algorithm to buy messages onto Q's so do I match it or not the default way built by the rabbit M Q's framework is a direct Q now this rooting algorithm we either have 10 or

we don't have 10 items so if I do have 10 I join the queue if I don't I'm not allowed to now naturally if I were to have more than 10 items they're going to say get away from here you've got two bags full of items and this line is only for only for people with less but what if you go shopping well words on a Saturday now let's be honest it's it's a it's a [ __ ] show on a at lunchtime on a Saturday everyone's there buying their lunch their weekly groceries and you know it's a mess so there's very few people who are actually there buying 10 items or less so if sorry um so if this

was the case a worldw employee might say to you hey come with me let's join the shorter queue even though you have more than 10 items now in rabbit mq terms if I was on an exchange of type fan art that would mean that it doesn't actually matter if I if I have the right routing algorithm I can just send my data there all I need to know is where the exchange lives I.E I've entered wallwords that means I can go on any Q now it will just push me onto a que because you are in the shop format now speaking of different cues I'm now going to go through the concept of virtual hosts all right so if

we have a look at this diagram in wws there's more than one different section right so we've got a home section here donated by um the APO theary section we have a clothing section which is donated by the um the sporting good is section and we have a food section which I'm not really sure what that is um those are the only images I could find uh so we have three different virtual hosts right so I can go and Shop in wws anybody who walks into wws is immediately authenticated right so let's just assume that you have credentials to speak to Rabbit and Q um endpoint so the next thing is if I was to go shopping in the food section well

now I would be expected to pay for my goods in the fooding section because that is where I am now authorized to purchase Goods right so I have been authenticated by walking in the shop but I'm I'm only authorized to pay for my goods in the section that I currently am um but well let's again has the ability to have a LAX ruling of this you know sometimes I've been at wws I've filled up a trolley I've looked at the queue and I've said well the Homing section is dead empty I'm going to go there and be mean um so if I took my clothes to the clothing section and pay for my goods there I have been

authorized to to shop in a different virtual host what this means is in Rabbid mq terms I can say only people who shop in food section have to pay in food section or I can say look as long as you're authenticated go and do whatever the hell you want I don't really care so a virtual host is kind of exactly what you would imagine it to be it's a namespace with its own exchanges with its own cues with its own bindings and everything else um but how we authenticate that is secondary to authenticating to the Rabbid mq node itself so we first authenticate by walking into wws and then we authorize ourselves by going to the cube we're

allowed to go to so essentially this right this is a message broker all right so we've got a producer that can send into a virtual host um sorry um into an uh a producer that can send a message into an API a broker API endpoint the broker has one or more virtual hosts and each in each virtual host you have one or more exchanges which have um a binding to one or more cu's and that Q can be consumed by a subscribed consumer or one or more subscribed consumers so before we go on to the hacks and stuff let's have a look at the client architecture again let's try and understand exactly what is going

on so as you can see here we have each manager belongs to a SE a separate company so a manager belongs to a company it has one or more iot devices those iot devices send their device information with a device ID to a message Q broker um the consumer consumes off of that sends that data into a global data Lake and then each customer portal will uh GA send an API query to the global data Lake and ask specifically for the data on um within their company so some important things to recognize here is obviously they each have sequential device IDs this is not just within the organization this is Cross organization so device id1 belongs to the first

company if I was a second company and loaded my iot device next I'm device ID to um when we hit the rabbit mq end point we have an exchange of type fan out now what you'll remember about that is it doesn't matter if I know the rooting algorithm which is obviously going to get important so this is an example that I set up um with a amqp fan out type on the exchange with a rooting algorithm of us. that's us. um wild card so it can be anything I just needed to know the US dot or in this case because it's fan out I didn't need to know anything as you'll notice down here the rooting algorithm I

actually centered in with is DN I don't know if anyone can see that it's quite small um but I sent in DN and I was still able to push that traffic onto the queue uh yeah that is obviously then sent with mqtt onto a data L and then from there the API query Works onto the DAT L cool so why are we here all right so we've had to enjo a pretty stressful kickoff meeting um next we had to take a step back learn figure out you know what exactly is going on everyone's a little bit confused um but I'm sure you're all thinking give me what I came here for this is cyber security and I want to see some hacking

so next I'm going to go through two different attacks um that we did an evil producer and an evil consumer and then I'll go through how we struggled to find a solution so the evil producer now we're all sending into the same queue that's all going to the same data Lake and everyone's just quering for it so I'm sure all of you who were sitting there were thinking Jason this is obvious you're just going to send in your uh to the other device ID from your um manager from your company that you own you're going to use those credentials send into the manager and then this client data's Integrity is going to be ruined because

they're now pulling all of my data now obviously I can produce tons and tons and tons of data so I will overshadow any of the data that's being sent by the actual device and in this case uh when I did this uh like I said Leon's unfortunately for him had to sit through this many times on about the fourth or fifth time he looked at me and he said Jason how how did you do that like how did you send traffic in I said what do you mean like I I I wrote like a tiny little python script and Leon was quite angry with me he said how have you waited this long to tell me that you

actually wrote something for this now I I did wait that long but that was just because as you can see it's not it's nothing impressive but Leon said well you've got to put it on get for people to look at so it's on git now um and because it's on git I had to delete all of my other crappy repositories so nobody could see that um didn't need anybody looking at like Kudo program it was dog show up there um anyways yeah so there a couple things that you needed in order to interact with this endpoint so obviously I needed to create a connection with to a channel like I need to create a channel connection so I needed to know

where the host lived I could get that from the configuration file and then it was understanding what virtual hosts there were you know what other data I needed in order to connect to it now this was aqps so it was slightly trickier although all I needed to do was add context and say I didn't really care about any certificate you have so I didn't verify anything just sent in the context uh the data there the information that I needed was the virtual host that was the tricky one to to find but because the client was very nice they gave super verbos credentials and I was able to actually interact with the management node so I

could I could use a rabid mq CTL tool that they've created in order to act interact with the Rabid mq node and from there I could pull what cues were in use what exchanges were in use so I was able to get like the general information that I needed from from um interacting with the rabbitmqctl tool I then used a p Library which is a python Library used um in order to create and manage amqp so you can publish data you can consume data and you can actually start a server um there so I created a connection and then it's a very simple to um uh command where you just need to publish the data knowing a rooting key

again didn't need to know that so I just wrote anything there to the fan art um exchange and then it was pushed on and then of course um the important thing being the body um where I'm sending that this um refrigerator running at 69 degrees so hopefully they'll hopefully they'll fix

that I didn't realize that

so um okay so the next attack that uh we looked at was is well as I spoke about with Wills you know who how restrictive are you being when you say you can consume all right now in my case um not was the answer so I could create my own evil consumer um which would R Traffic off of the the queue now as as you'll remember the first consumer is going to read the first message because they've created the queue we can assume that they have a subscribed consumer to the queue so I'm the second consumer so the first message comes in from a company and I can read this second message off now that's obviously important for a

couple different facets the first being customer 2 never actually receives their data um so they've like lost their AV availability and the second being that I have read sensitive information um yeah so one important thing here is obviously they've now created a single consumer but I can create hundreds of consumers so if I have 100 consumers they will read the first message and I will read mess message 2 through 101 and then they will read message 102 um because I have the ability to create as many as I want I can pretty much make what they legitimately see invalid or you know useless because it'll only receive that every hundredth message 101st message so again the how um again

very simple uh you just create a connection string and then I needed to create a loop um in order to continuously consume so it didn't just kill off the after the first um read and then it's just a basic um uh thing to consume so I needed to identify in this case where the que was not necessarily the exchange so I used rabbitmq to view where connections were coming from so I could read where their consumer lived and What channel it was connecting to so I could attach to that same channel and read data off of it um yeah so those are the attacks and now I want to have a look at potential Solutions and what uh what struggles we

went through to why that wouldn't work so the first thing we thought was well you don't you don't have a a virtual host well let's let's separate each client each company into their own host um the data will go to different places um you know I can't send to your virtual host and then you know it'll go up uh one thing I forgot to mention uh I'll tell so here's a a little story when a while back I took my friend out to a fancy restaurant and um they they only served one like you weren't allowed to pick your meal you have to eat what they were eating and it was like was Squid Ink

with like some puffed popcorn I don't really understand it and I can only imagine I can only imagine that Two Chefs came in that day and one was like I am making Squid Ink goo and the other Chef was like I'm making popcorn so they were like well we're serving squid ink and popcorn um in when you have many cooks you know everyone's got their own opinion and everybody says this is the way I'm doing something so the person in this case who created the squid ink part said this is not going to work uh a virtual host will not work because I'm using mqtt and mqtt has no virtual Host this is something we

learned in the Google search differences between mqtt and rev Q so they said there's no way that we can work with this because you virtual host won't work so I I tried to assure them all rabbit mq uses at least the root virtual host so you are you just don't want to separate it more and they said no no no we know what we're talking about and I've only done this for a week at this point so I'm like all right well you know what you talk about so um but technically this wouldn't have worked anyway and the reason for that is it actually doesn't matter if I send if I can't send into their virtual host

because we're all going into the same data link so if I send my traffic with the incorrect device ID to my virtual Host this gets consumed by the global consumer and push to the global data leg um then customer one is still going to read off all of my information so it doesn't actually matter um so the second thing we thought of was well let's not let's not say that the attack is impossible but let's like reduce the likelihood significantly so we'll we'll move it away from device IDs and we'll put really hard to guess GS now the person who was in charge of the popcorn said to me well we've already agreed to device IDs and I am

not going to touch guits that's that's too much extra work sign up SLA no that's not going to it's not going to not going to fly with us so I said all right cool um but technically this wouldn't have worked anyway um and the reason for that is because we have the eil consumer so I have created a consumer I can read off the cube um which means I can steal the sensitive information regarding that guid and then of course once I have see once I have um the gued or I have that senstive information I can then send traffic into the queue and the customer portal is going to read my message so it didn't

actually matter so so what do we actually do like how do we how do we solve this so personally my first thought um and I kind of still feel like is the problem is we have this global data link right we're just dumping everything into the same place and was they're using the customer portal to like query it um but we don't the customer portal has no way of verifying what data is valid and what data is invalid in the customer Port their sole job is hey isus give give me everything for my device ID um but you know that's that's not a solution we can't just say that's the problem so what do we do we need to we

need to actually try and help our client right so we can't just say we'll put virtual hosts in because they can't do that we can't just say we'll do Goods because you know they can't do that so we need to come up with a solution where they can actually actively do something to fix it so the final thing we thought of was all right well let's set the data that goes into the manager we'll sign that data at the manager level um the consumer can decode that message and then read what's inside so if manager has sent something with device ID one and they are sorry device ID 2 manager 2 um then it will go into the data L if

however they send something for device ID three they need to log that drop it and report and say hey why are you sending traffic in for the wrong device ID um so only valid messages should actually ever get to the data link the next thing we obviously recommended uh well not obviously is you know you shouldn't let everyone consume so they said no well we need to consume in order to send um traffic back from the customer portal to the it we need to be able to talk backwards so the best way we could say do that is you can only consume off of a very specific virtual host so in this case you should have a

virtual host at least one going forward and one going backwards and you only let people consume off of the one going backwards so that this attack path is impossible um yeah so in conclusion you know at the beginning of this talk I said well honestly [ __ ] Simon and that is the way I felt in the beginning but had Simon read the scope properly he said he never would have actually put me on this assessment so I wouldn't have had the fun the learning opportunities that I did so instead of um freaking out running around like a headless chicken and swearing assignment a lot you know take a step back you will figure it out

and it's a lot more enjoyable than doing a normal web app or mobile um yeah so we have iot in our lives um to make ourselves easier you know make our environments um more more maintainable smarter um but when you put things incorrect together incorrectly it can be a bit of a mess and lastly you know third parties can distribute the workload um but when you have too many uh Cooks in the kitchen you're bound to end up with some weird squid andk goo and popcorn um yeah uh lastly I just want to say some thanks to some of my colleagues who helped me both on the assessment um Simon for messing up the scope Leon for

helping me with this talk um orange and then besides and you guys for sitting throughout the thing um I I really hope you did enjoy some of it or all of it um and if you have any questions you know shoot them here or message me on Twitter I am the breaker of science

for

[Music]

okay

yeah

yeah um sorry I don't know if youone heard that uh correig me wrong the essentially if we are you know seeing data that's out of sync should we not delete that at the customer portal level at the at the end point or you on the way to the customer portal if we see 69 and a fridge should only be running at 25 um then we should throw out that data that is invalid in yeah you look at data Trends if something's way off you know deal with it that way I think that's a really valid approach um in this case obviously I'm trying uh um to to flood it with invalid data so I'm hopeful that if you were to

implement something like that as an attacker I would hope that I am flooding it with so much data that it becomes the trend maybe um but I do think that that's a very valid approach and something that you should anal something that you should analyze um is that data at various points like at the consumer level if we are seeing something invalid that's why you know we look at that signed data is it correct is it not is it coming from the right place like you say the iot device itself could be malfunctioning or sending from that level um a colleague of mine suggested it wouldn't have worked in this solution too many cooks um but a colleague of mine suggested you

know signing the data at the iot level so that we aren't ever getting to a point where the data could be manipulated and centered um yeah I think I think that's a very valid way of approaching it I just think that there's security and depth means you need to also account for the fact that if somebody floods it with data that needs to happen or you know at the consumer level can I consume and steal your data and then you're not seeing valid data

anyway

yeah thank you for

the

yeah

say than