BSidesCharm - 2017 - Jose Fernandez - Frony Fronius - Exploring Zigbee signals from Solar City

all right i'm ready to go so how's everybody doing today yeah all right so i'm really excited to be here this is the first time that i'm presenting at a besides charm conference here in what's now my hometown um i was supposed to uh present many uh and during the first one but i was unable to due to some uh last minute engagements but i'm very excited to be here today and i'm very thankful for the organizers and who gave me an opportunity to come here and talk to all of you today so thank you um are we good adrian awesome all right so my name is jose fernandez and i came here to rock

so the name of my uh talk is fronifronius um exploring zigbee signals from solarcity um i am on twitter so if you guys want guys and gals want to follow me then please go ahead um so the agenda for today uh i'm going to describe uh the research goals for for this project i'm going to illustrate some past works that have been done in the field of of zigbee um i'm going to highlight some of the signal analysis that i did and the methodology that i used to do it the multiple avenues of approach that you could take when you're doing security research into the grand scheme of things um what i call case one so a lot of

those avenues have approached i focused on one of them particularly and that's why i'm able to talk to all of you today and for the research that um i'm looking to do and i'm you know part of this talk is a way for to to help me find other people who are interested in doing research because you know it's no fun being john rambo and you know just doing everything yourself right it's always you always get better results when you integrate involve more people and you know the opportunities are there but sometimes you know we just don't know where to start so about me i am an infosec researcher for the purposes of this talk i consider

myself a mad researcher a mad scientist i focus mostly on attack and defense mostly and i'm a veteran i'm a phd student and this presentation you know legal swimmer right it's it's my it's my own it's my own views my own research it doesn't involve any of my employers past current right so took care of that so all this started a year ago um i had access to solar city equipment and i was like ah this is pretty interesting i mean how does this all work like how does it communicate i saw that these things had antennas that there were devices that i had to plug in you know inside a network and i was like

huh how does how does all this work so i spent countless hours uh during that year just learning about sdrs learning about zigbee um reversing among other things but i had to wait for the right time and before doing any of the actual like hands-on research and my hypothesis was someone like me who had very little experience exposure knowledge of zigbee could go from zero to now in front of all of you due to the lack due to the vast amount of work that has been done on zigbee research over the years i'd be able to provide meaningful contributions into this field for the purposes of defense so i pitched an abstract it's like hey

you know i have a pretty good feeling that you know if this idea gets picked up you know i'll be able to run it and be able to pretend in a short amount of time so now i have an actual reason to do all this work right so i thought about how to attack this research and what what i was going to do to understand all these technologies and all the things that make it work and that was uh you know long hours long days right because i'm sure all of us already have nine to five jobs and you know maybe more than one right so i was able to devote a lot of my free

time in the pursuit of this my hypothesis was if a large pool the past works exists in your field how feasible is it to replicate the findings on systems that are unreported right so in a lot of these other researchers a lot of the other research that you'll see it may focus on on certain things but you know in this case i wanted to see how solar city actually employed their communications there was no existing public research on either solar city or fronius which is the manufacturer of the power inverter that this company is starting to deploy that tight zigbee analysis until today so i'm adding to that to that knowledge base right now by just

being here and presenting now zigbee research done to date period secretary spicer that's not true i don't even believe that he said that but nowadays maybe he did so these are you know a lot of past works done by other individuals you know um let's take a moment to give them a round of applause for all the work that they've done there's so many projects that you know it's one thing for you to do research and then not tell anybody about it but when you share it with everybody um you grow that body of knowledge but at the same time you know you put yourself out there right so you're gonna be subject to critique

but um a lot of these individuals and many many others right i couldn't just put all of them in a slide but there's definitely a lot of material out there that if you're interested in this technology you can just pretty much search for it and run with it now here comes the perfect timing in all this so the digital millennium copyright act of all pieces of you know legislature that are out there now allows infosec researchers to conduct research on consumer devices the research has to actually start in 2017 so that may mean fiscal year for the government but you know i literally started around january of 2017 doing the actual hands-on aspects of my

research so i spent about a year thinking about how is i was going to approach this problem and how to best approach this i contacted the vendor to start a dispo responsible disclosure channel and i received approval to present today so thank you to them we came to an agreement and to limit some of the things that i've already uh shared with them in terms of findings while they you know internally figure out how they're going to permanently fix some of these issues so in in my case solar city or frontiers they didn't have public like contacts it's like they didn't have a bug bounty they didn't have any of that so at first it was a little bit uh tough

to figure out you know how to actually do responsible disclosure because sometimes you know if you're doing research on a particular vendor the information isn't out there and you don't just want to call the you know 800 number and so hey you know i got vulnerabilities since close is like you know let's talk about it because you know it opens you up right so i used um some you know i watched the news so i knew that tesla had pretty much acquired solarcity and tesla does have a bug bounty program so i was able to use that poc i contacted them it's like hey i know you guys aren't solar city but since you guys merged

perhaps you know who i could contact and they actually brokered the communication so my initial observations was a solar city starting to roll out the fronius equipment to solar customers when it comes to the um companies like this where you know it's like smart metering and you know generate your own power there's there's multiple vendors out there but in this case solar city started rolling out some of the fronties equipment and you know the device um i would assume would have to collect that metering information and report it back somehow so if this thing if there's this power inverter which i'm going to show later that is like physically outside of your residence or place of work

it communicates with an internet gateway that is inside your residence in place will work right so there is some form of communication from something on the exterior to something inside

this particular vendor uses equipment from fronius and digi who's who's heard of digi yeah they have a huge ecosystem of products and uh in this case even though i did the responsible disclosure to solarcity you know they use their products so if you try to approach all three or more at the same time you know your mileage may vary in terms of acceptance so the communications flow between the devices this is what the power inverter looks like in this case uh that is a fronius primo model as you can see there on the bottom that little black thing there that's there's an antenna there this thing communicates to the internet gateway which is you know a small

small little device you know also has an antenna right it looks it looks kind of innocuous right it looks kind of safe that talks to your land which talks to something right because it's not really clear at first just by looking at these things how is this actually communicating over the net so the solar gateway needs three lights to to function properly right ones are going to be your power the other one is going to be the network link on your on your lan the other one is going to be the zigbee link so three lights [Music]

i'm sorry john luke there's actually three lights so my initial thoughts on the equipment were if they if they gather that consumption device of the energy that you produce you know what what else are they collecting so the company has already you know disclosed that you know they can actually fingerprint the devices inside your house based on how they draw power it's pretty interesting so they can actually tell what model of like large appliance you're using like a washer dryer refrigerator they can actually fingerprint those things based on the consumption of that's pretty cool right and then you know for me yeah yeah i guess i'm okay with that right now because you know let's say

you had a really old dryer and they're able to fingerprint it and all of a sudden you start getting marketing mailers for like discounts for a new energy efficient one right so this is a marketer's dream but at the same time what do you what do all of you consider could be the repercussions of once this technology actually becomes very precise let's say that there's certain devices out there that they don't want people to own and you power them and they're able to fingerprint that where does the the the sharing of that information stop right so there are some concerns there but you know for now my initial thoughts looking at these things there's like you know it

looks kind of flimsy so it's like are these things really secure right i keep hearing about iot and the iot for me means the internet of threats and i keep hearing about how all these other like vendors and corporations they keep getting it wrong so i was a little bit suspicious of course i would want to remind everybody here that you know there's always when you're doing infosec research there's three things that you need to consider uh you know this digital millennium copyright kind of like thing that that came out that's okay but you never want to be the first person to have to test that in a court of law so you always have to

consider the difference between what you can what you can do what you could do and what you shouldn't do at all times if you see a lot of the other presentations that you know happen and let's say like conferences like defcon stuff like that sometimes they'll do the research and and you know they're pretty much violating a lot of the title 18 laws especially the computer fraud and abuse act right and sometimes they get lucky and you know they don't get sued they don't get right they don't get threats sometimes the talks get pulled right because they don't go through those proper channels so always consider things that you could do within the scope of

yourself and your own reality right because there's always many more things that you can do and usually some of those can get you in trouble at the same time you shouldn't be afraid to explore and just open things up and figure out how they work because that is part of this community right if we don't take screwdriver and a flathead to things and figure out what's actually there um we're doing a disservice to our communities so are these things secure uh my initial response to that answer is yes i am going to continue doing this presentation as i continue the the process of responsible disclosure and hopefully that'll always remain that way you also have to consider don't give

away free pen tests to companies when you're not invited or you know just just don't do all of this work when nobody asked you to and nobody wanted you to do the work either right so don't just do all this work and then just give it away to them for free right there's something else to consider never value your time at zero there are multiple avenues of approach to kind of understanding how the vendor actually ties all these devices and communications together so there's the communication between the power inverter and the internet gateway the communication of other power inverters and internet gateways and by that i mean you know the ones that you may or may not own

the communication between the internet gateway and the company [Music] uh the communication of a web app that they have in order for you to communicate with the company and then you can see you know how much power you're producing and things of that nature they have a mobile app so that's a different that's a whole different avenue of approach how does the mobile app communicate and tie into all this and you know for me the most interesting one was the communication of the gateway and its internal components right so this presentation is only going to focus on the very first avenue of approach which is the communication of the power inverter and the internet gateway because at this

time it would just be illegal to disclose some of the things that i've discovered and i'm still you know working to put all that together also so you need to give the the vendor a time to fix or just doing a disservice and you're opening yourself up for lawsuits and other nasties so recently digicorp actually put out the internet of threats security balance between economic cost and benefit so perhaps the people in the back cannot see this very clearly but so it's pretty much the cost to scale ratio between you know here's some at the bottom there's some communication attacks yeah i shouldn't point things um so in there you have some some man-in-the-middle attacks that you can

do um there's some software attacks that involve apparently malware and social engineering are up in the scale i would argue that social engineering [Applause] you know costs almost nothing right so um that's just my perspective though but over on the right side we have invasive hardware attacks and non-invasive hardware attacks so in the non-invasive channel aspect of this they have side channel analysis uh jtag right so they know what people can do to their devices that they're aware but they put a cost kind of like risk balance to all this because the the things at the very top were the um invasive hardware attacks they figured that those things could be you know it could take a lot of time and

resources to to actually produce so you know they know that those things can happen but you know they see the likelihood of trying to make preventative measures to defeat those things it's kind of like you know right now it's just when you do cost benefit analysis it's not just it's not worth it at this time it's a business decision so you might be asking yourself why why are we only talking about the first thing and why are we talking about the the first thing why you know why how did you get permission to do this it's because there's a physical security problem with these devices right these things are usually outside of somebody's residence or building

they have to be accessible by you know not only the the power company that it's tied into but you know if there's like a fire or some other type of emergency you know those uh first responders need to potentially come in and have to you know turn this equipment off so there's a physical security prop problem in the way that it gets integrated and deployed to customers anyone can access this panel in fact you can pretty much operate this thing uh just using the dashboard and you can get some some pretty cool you can do some pretty cool things with it and you can also do some very damaging things with it so i i know

i notified the vendor of the particular problem and you know i i kind of expressed that you know this this is worth talking about because this is already out there and it's been out there for years and nobody's really talked about this openly so you might be asking why not the second one why didn't you try to figure out how the other people's power inverters communicate right so i did a poor man's war drive with the zigbee module and i had some ah you can barely see it but um i had i had some e-waste in terms of a laptop that i was throwing away but i took out the the wireless antenna because you know when you throw this you

know different electrical components and you know old hardware out you can still scrap some things and then you might use them in the future so i was able to do that and then i was able to amplify the the signal strength of my poor little xp device you know i tried but it was very difficult because i found that very few owners around my area actually had the same configuration that i had from my experience i was told that hey you're one of the first people to start getting this new equipment with this type of particular configuration uh sometimes they do the configuration where it uses wireless ethernet right those things are also very interesting and they suffer from

this fa the same physical security uh program uh problem so uh solar city has a web app where you can actually find other solar consumers near your area that's pretty cool right so um i i use that web app and i started driving around it's like okay i know where that street is and then you know i just used visual analysis it's like you know just look at the roofs you'll find the solar panels right it's not you know you don't have to get too technical with this and uh i i did some some non-invasive scientific analysis by that i mean if a device popped up on my screen um i wouldn't have tried to interact

with it further it would be the equivalent of using something like a wi-fi stumbler or something like that ah you can't read that so i i kind of stopped doing this too because uh for me it just wasn't as interesting as other things um but if you do want to help with this initiative um fernie feronius at protonmail.com is going to be the email that we're going to be using so that way if you are interested in assisting with this research or just getting involved that's going to be that or you know talk to me after the talk um ideally i'd like to find somebody else who has the similar solar equipment so that way we can

compare notes right and local would be better because i'm pretty sure somebody else besides me has this and they may or may not live close by so when you look at their web app so although all those pins on google map those are other solar customers that they have when you click on the details of those pins the information that it gives you is how much power they've produced that's pretty cool right there's no they actually took some time to to make sure that you can't just like grab this and you know find gps points and stuff like that so they did take their time to kind of reduce the exposure of that customer information but it's still pretty

interesting right you can pretty much find people who are solar friendly by using this so the easiest way to start like most things that you know you you know without taking a screwdriver to it is look at the labels in this case i had two devices so i looked at the label for what was uh outside and you can't really see it too clearly there but there's an fcc id number there right so then when you do a search for that it's in 2.4 50 megahertz right nothing talks like that right yeah that is an empty empty frequency no it's it's not it's unlicensed so a lot of devices you know are going to operate in those channels

for me i was just like okay so there's there's a lot of things going on there let's find out what's going on here so i started speaking to some of my peers it's like hey what's the best way to approach this right that was told just put a usrp on it okay so that looks like this right no it's not that easy it's bad advice so then i was told just throw a usrp at it see what that looks like nah i said doing signal analysis so i purchased this uh very small um kind of usb dongle from texas instruments it's a cm 2531 i think it cost a little bit less than fifty dollars

and it's specifically built to to help analysts and researchers find zigbee communications i used stationary analysis to identify the communication between the devices and i determined that you know there were going to be other iot things talking also so i came up with the hypothesis that you know if i remain at a stationary position and just look at the different channels over over long periods i'll be able to determine which is my device just by the amount of data is being transmitted i did this for 14 days and by that i mean in the morning i would wake up i would select the channel i run it and then when i'd come home in the evening

i would stop it and i would look at the look at the information that was collected so in my case it was channel 20 the one where my uh the zigbee equipment was communicating on i did the same thing with killer beat it uh it took a minute to id the the the devices and the channels that they were operating on another three minutes to pcap on that channel and it maybe took an hour to flash those razer usb sticks they have right because just because you get it you used to have to flash it in order to kind of support packet injection and the other cool features so looking at this i'm not going to show

you guys the p caps because it's it's too too intense to be honest so it was beaconing every three seconds i was like three seconds you know yeah odd number but it's like why why three is this really my stuff so i i compared the the mac addresses on both labels i match both of them it's like yeah it's my stuff and the packets differ from from texas instruments called packet sniffer it has an awesome gui and it's great eye candy but you know i need picab this thing outputs to a dot psd and it's not photoshop so i was like packet sniffer data it's like come on guys that also means something else so they they actually have a wireshark

converter so then you can take those psd files and convert them into wires into pickup then you can you know do more things with that there's also these other like third-party drivers where you can output like a fifo file and then open that through wireshark but i found this to be the easiest considering i had collected tons and tons and tons of decapitate it's a good way so hmm so i decided to take a peek at the things that are outside right what's in the device because you know there's a physical security problem uh program a problem with it so let's open it up and what information can i pull from it and i kept thinking you know i kept

thinking you know there's encryption involved and stuff like that you know do i really want to like offset and do all this stuff let me let me figure out what's there first so this is a picture of the center of the of the pcb board for the power inverter and you can that cap is huge and there are many caps just like that one so all of those things want to kill you so even if you power off these things you still need to discharge it and you need to be extremely careful i mean i've seen this thing uh work with 19 amps at any given time yeah you know it takes almost less than one to kill you so

be careful right don't just go in there and you know try to try to mess things up or you know just be very careful when you do it so this is what the daughter board looks like so i was able so this thing has a daughter board and yeah see i already see head shaking right so um so so can somebody say what's wrong with this picture any takers so uh one of the things that i found pretty curious you know it's not only the xp stuff but it's that capacitor oh my god too much coffee so you see that that cap there and then it's going through the antenna wire i was like oh maybe this is a trap

maybe you know this is some form of like physical you know maybe i need to be careful with this like indiana jones but no um so it turns out that the that the way it communicates from the outside inside it's using xp pro right and that chip isn't soldered in so you can literally just pull it and put it on to the device there on the right which is an xb explorer let's do some analysis on that i loaded it into program called xctu who has not heard of xctu let me tell you just doing this zigbee stuff and then learning about xctu i was kind of this exists like whoever like all the people that developed that

um they spent a lot of time incorporating features that only like engineers would care for right people like us also so there are so many cool features and i was very surprised with what i found so when you load xctu you tell it hey i want to use this zigbee device so the device that i pulled is the one on the right oh sorry the one left it's it's working as the router right now the one on the right is the coordinator i'm not going to get into zigbee specific you know aspects and communication all this but those things are important so it's able to pick out that other device that's inside a building right

cool thing is you can double click on the on your device on the left and you can actually reprogram it on the fly you can read the config and reprogram it very cool it has many many other features which i will not cover today but needless to say barrier to entry is very low right so so there you know i kind of mentioned this thing beacons out every three seconds there's my three seconds the the scan duration so every three seconds was scanning so i was able to answer my own question so i can read this all the information is there great what else i was looking for that well-known zigbee alliance key right because i figured hey this is

maybe maybe right maybe this through the faults you would find your answer here this is part of the things that i've agreed with the vendor not to directly talk about today but it would be that easy for anyone to open this pull the config and read um encryption specific information that way it's all there it is this was the part that blew my mind you can actually remote into the coordinator with with the thing that you just pulled from the outside so yeah see i can already see the looks on your face is like what yeah and this is what cyber should look like in my opinion because if i just videoed how easy it

was it wouldn't be as cool you're in the coordinator just like that i was really surprised i was like i was not expecting that to work but it's built into the the capabilities of the the spec actually it's supposed to be messy programmable right um from my experience in the past i thought these were only used in like dev kits i didn't actually expect to find this in something you know out in production but it was and then you can start um seeing the the actual way that device is configured to communicate um not only the channels with the pan ideas and other other cool stuff right so you can pull even more relevant encryption aspects

from this remotely so you took something from the outside you went in and you're able to read that config and you can also configure it too so you can change it to whatever you want i thought that was pretty neat so if you want to do kind of like more ziggy analysis i would always recommend that you know you get something where you can attach an antenna to it just so you know you get better results you get you can see more things now the cool thing is you can actually use xctu to just look for other devices right so you can program this you can start mapping out different things and you can probably

remote drill into those things too yeah no i thought that was cool no yeah would you agree all right so i like getting things that you know you're able to connect antennas just for this very purpose because you look at things like um the razer usb stick there's no way to connect unattended to it unless you know you get very creative and sometimes you don't have to you can just use commodity hardware it's already out there to do stuff so in this case i started doing you know poor man's signal replay and that that video there on the left was me replaying traffic just very very crudely but i'll be honest it didn't work because i was a victim of

geigo a lot of it had to do with the way i was replaying traffic and what i was actually replaying so i wanted to keep the three green lights at all time and you know within half an hour of just doing these weird replays without you know manipulating dates or you know accounting for the different aspects in this it dropped right but there's always the potential that if you spend time doing this you might get results so but propose that you know to secure the the the problem this vendor needs to stop using this this pen id that seems to be used you know through through other solar companies they need to start either changing the

pen id or you know just mixing it up right to not make it as easy to be able to identify these things um if you want you can actually use xctu and change your encryption on that coordinator first on the thing that's inside right because when i looked at this device it had 31 things that i would have to unsolder and i didn't want to do that because you know i can barely on a good day solder most things so 31 things and then yeah i didn't want to do that so you can actually set encryption however you want remotely then you can do it on that power inverter that thing that you extracted from the

exterior then you set it that way you know you've you've done that it has limitations because anybody can still pull these things and they can pull you config so maybe you know the vendors can get creative and they can do things to either do program this this small module or load it from another data source i mean there's different things that they could do but you know is it feasible i can't really answer that you know definitely i would say at least solder the damn thing in make it hard right so some of the future work i looked at a lot of the passive communication as to how this thing talks out to the mothership how does it report you know

the metering and all that stuff i was able to kind of find similarities as to the end points where it was actually talking out to so just take some time and consider the things that are on the right side um you know it's there was no clear text default passwords none of that stuff they addressed all that right so still interesting you know communicates over the net i spent most of my time doing research on the actual device itself so looking at things that i could pull from memory that's what these are you know kind of like proofs of work that i was able to pull that i felt were relevant to show these are the kind of things that

you can extract from from memory and there's always you know better ways to do things also right so i use very rudimentary process to be able to do this but you know that i can figure out you know what version of at least python that they're using with this you can see some some of the companies and softwares that they incorporate into this um you can actually see like c code and stuff like that it's pretty cool so there's definitely a lot of work to be done that's why i'm i'm here today i'm looking for other people who kind of want to assist with this help take it further right call for allies you know let's work to solve cool

problems together and you know let's be ethical about it too right because nobody wants to go to jail um i'm not really looking to get into like any bug binding programs because they'll usually tie you into an nda and then they'll limit what you can and can't say that's frowny fronius protonmail.com any questions yes

earlier i kind of asked if somebody has this i'd like to be able to talk to them so that way we can share and compare notes um you know just looking at the roofs is one thing but then when you're looking at the side of people's buildings and they're out in the street and they're like hey what's this character doing yeah it's like ah it's okay i'm an infosec yeah that doesn't trigger any alarms so and that's an interesting point because when you see all the man in the middle things usually they want you to install things on the client device first but this is a small embedded thing you can't really trick it into it's like hey no use this

from now on so you have to get a little bit more creative if you want to do those things it's definitely still an avenue of approach though any other questions comments well that's it

thank you so much for being here and um hope you enjoy the rest of the conference

yeah right now