Vulnerabilities and Defenses for Distributed Energy and EV Charging - Jay Johnson

I'm just kidding.

Oh yeah. >> All right, let's get going here. So Jay Johnson is the chief technology officer at DE Security Corp. J is coordinating the construction of cyber security protection, detection and response technologies for EV chargers, renewable energy installations and energy storage systems. Previously, Jay was a distinguished member of technical staff at Sania National Laboratories where he led research projects totaling 25 million in the areas of power system control, optimization, and security. Jay has authored more than 150 academic papers, 10 patents, and was interviewed for Wired, Forbes, NPR, and several other media outlets on his EV charging cyber security research. Excellent. Yeah, take it away. Thank you for being here. >> Yeah, thanks everybody. You you made it

back from the lobby and from the coffee break, so I appreciate that. Uh there's there's been a lot of talk about kind of the IT areas of cyber security at this conference so far. How many people work in OT operational technology, the things that run gen sets and power systems and water and wastewater and those types of things. So little there. Okay. Some here. Yeah. So this talk is all about that. It's all about power systems. It's all about the things that actually run our society, the critical infrastructure. So I'll be talking about vulnerabilities and defenses for distributed energy. Distributed energy or distributed energy resources is the deer in our title of our company. Uh

that that's your like battery systems and it's everything from you know Tesla's power walls in your garage all the way up to these mega packs and bigger systems that run our power grid now. the amount of battery systems going into the power system right now and especially colloccated with data centers and the massive boom there is tremendous. So there's there's a ton of development in that space. So as as was mentioned previously my background is working for the US government at Sania National Laboratories. Uh I did a lot of development of the communication protocols used for these deer devices. So Sunspec Mbus TMD3 I2030 protocols that you probably don't run across in in the IT space, right? These

are the things that actually tell you telemetry from this equipment and move control signals back down to the end points. Uh, I had the honor of of leading red team assessments on military micro grids on a bunch of EV chargers, level three chargers. Those are big DC ones that sit like this out by Walmart. um and and I did a lot of power system modeling uh as well. So this talk is going to be split into a few different chunks. The first little bit is going to be kind of a threat briefing to talk about some of the attacks and vulnerabilities on solar infrastructure and electric vehicle charging. You're all going to get to be the first people

in the world to hear about a new CVE that I'm finally releasing after a couple of years of sitting on today. Um and then I'll talk about OCP or open charge point protocol attacks and that's the communication infra uh communication protocol that's used for a lot of the EV charging uh networks and then lastly talk uh briefly about some of the cyber defenses. So how many people here have heard about Volt Typhoon? Maybe you've heard about Salt Typhoon. So Chinese threat actor working in the telco space and sucking up a lot of private information. Well, Volt Typhoon is another A out of China that is really embedding themselves persistently in US critical infrastructure right now. And

there's a fantastic podcast by M Nicole Horroth that I strongly recommend you take a listen to and it goes into the details of how they're getting into these systems and they're using living off the land uh techniques and methods to basically sit uh on these systems. Um and they will like just check their credentials every 90 days or so to make sure they still have access and then log out. And it's extremely difficult to pick that up because there's no indicators of compromise. There's nothing there, right? And so um what do we do about that is the big question. So few different ways, a few different attack vectors to penetrate deer systems. Uh I'll briefly touch on these.

Um kind of over on the on the left side here, you can see the head-end systems. So this is your cloud infrastructure and your management systems. So the vast majority of PV systems and EV charging networks are cloud deployed. Um they've got your normal, you know, SAS providers out there that provide these services. And so if you um lose your access to that uh infrastructure, you have insider threats, then you can control these systems in bulk. Uh there's other um APIs and apps, mobile apps that have been the source of a lot of like um vehicle issues over the last decade or so I would say similar things here where there's uh vulnerabilities being reported nearly every six months or so

in this space lots of them uh take a look at some of the for scout research um that's recently come out and then um unfortunately a lot of these protocols are unencrypted and so man-in-the-middle attacks and things of that nature that replay attacks can happen um once you're on that OT network and then uh at the device level, which is what I'll be talking about here, um, is there's there's a lot of interfaces. So, if you tear apart one of these big DC chargers, there's USB ports, there's serial interfaces, there's internal networking, right? Because there's multiple boards. And so, if you can tap into the control signals there, you can actually manipulate the operations, you

can turn off the cooling loop for the the charging cable, you can shut down certain power supplies that exist in the back end. And so there's a lot of vulnerabilities associated with that. Uh a couple of months ago we published a white paper um called solar energy cyber cyber attacks and vulnerabilities. And in that we talked about this history of different vulnerabilities in the space and um as you can see it has been increasing. We'll see where 2025 ends up at the end of the year. U but there there's there's been a lot more publicity in this it doesn't mean the systems are getting worse, right? It it means more people are paying attention and talking about it and and looking

into this. So that's that's uh great. And so I want to I want to briefly speak about these different attacks that have occurred uh over the last uh five years or so. Uh the first one is in March 5th, 2019, there was a DOS attack on a uh organization called power and they manage wind and solar assets in uh California, Utah, and Wyoming. And there was an outage because there's unpatched Cisco router there that was being DDoSed. Um, and this is actually the first cyber attack on the US power system uh in in the history of the US power system uh where they lost visibility. They eventually patched that system and and got it back up online. Uh

the Palo Alto networks unit 42 has been reporting that uh some of these um gateway devices that take they basically translate one protocol to another protocol down to the endpoint device. Um they've been used uh through different code injection techniques to be compromised into the Mariah botnet. Um and then in 2023 there was some installer level credentials that were used to modify some grid support functionality inside these devices in Romania. 2024 800 uh again remote monitoring and kind of gateway devices that pick up on the power system data from these uh PV systems. These are actually mostly in Japan uh were compromised and used to fraudulently route uh bank transfers. So it's it's believed that there were uh

nation straight nation state actors that were um basically routing fraudulent bank transfers through this botnet in order to hide their tracks. 2024, a pro-Russian activist group called Just Evil targeted Lithuanian solar monitoring solutions, and they um they were able to um it's believed just get regular credentials either through brute force or some other social engineering campaign to get into these systems where they can control these things. What's interesting about this is it was actually tied to some government facilities, schools and medical facilities. And then most recently the FBI and NSA have reported that China linked actors have been compromising solar uh devices. So pretty recently a Greek researcher um this this gentleman Styus uh discovered that um there are six

different solar platforms. These are cloud platforms that uh he was able through uh various different means gain super admin access to. And what that means is he could see the full visibility of all of the PB systems attached to these backend systems. And so you can see here it's only almost uh 220,000 devices that he had access to. And with that access he could perform remote firmware updates. Uh had a back door into these networks and he could um potentially just brick the devices entirely which is a huge risk to the power system, right? Because in a lot of cases these devices are a significant amount of generation to the point that the disruption there will cause the grid

operators some real challenges in uh in you know maintaining that balance between generation and load. And then more more recently, Reuters came out with a report saying that there are some u undisclosed communication devices inside these commercial and industrial and and utility scale uh inverters uh that would allow um Chinese access to the equipment. Um over here you can see there was an interesting case where Dier um which um had a had a financial agreement with Solark um to basically maintain the backends. Uh they decided to brick you know no longer allow the devices to operate in the U US all the inverters that have been installed in the US without uh explicit permission to have connections

to their back end. And so what's frightening about this is we have some numbers from Europe in particular that controlling just three to four gigawatts of energy there could lead to disruptions with like somewhere around 10 gawatts of control uh leading to a blackout and the um uh the reporting out of solar power Europe most recently has indicated that there's somewhere on the order of 200 gawatt of solar capacity there most of it coming from China and several manufacturers have more than that 10 gawatt limit right so at that point you're at the tripping point uh so similar to what we saw in Spain and Portugal for the Iberian blackout uh that can happen here so let's switch

over to the EV uh charging cyber security in this case there are several different uh interfaces to these devices. You've got your credentiing system. So So think of like public chargers. Walked up, you have an RFID tag or some other, you know, credit card swipe or maybe your your phone has an app on it. You you uh connect to that. You say, "I am who I am, and then you're able to couple into your vehicle and start the charging session." There's also uh backend communications that we've talked about before up here, and then maintenance terminals as well. And so these these are the four interfaces that uh can be potentially compromised and have been in

the past. So let's talk about each of these. Um the charging cables themselves, there's actually communications that run between the vehicle and the charger. So the vehicle is communicating, hey, I'm at this state of charge. I have this allowable amperage that I can take from the charging infrastructure. Um, and if you hijack that communications or tap into it, there's some additional functionality there that uh, most people haven't looked at. So, you can actually SSH from a uh, charging cable into the EVSSE, the electric vehicle supply equipment. You can actually most recent very recently here there was a a Tesla charger that was compromised because there was a firmware update capability over that charger over that charging

cable where you could um essentially if you plugged in from a Tesla that was internet connected, it would push updated firmware to the device. And so there was a pone to own competition where they discovered this uh communication pathway and they they were able to compromise that in a way that they could they could route the device. Um and then out of the University of Oxford there's been a ton of work that's that's really fantastic about um compromising the charging sessions uh using a side channel attack where you basically broadcast RF signals to that uh and couple into that charging cable so that the communication link is broken. And without the communication link, it's no longer able to

communicate. And so what you see is these stop charging messages when that occurs. And so there's their setup in the in the boot since they're in England. I don't know if this audio will come through, but I'll show Well, all right. So user interfaces. There's there's a few different things here. You can clone the RFID tags. There's a lot of talk right now about fraud in this area because these RFID tags can be manipulated or brute forced in certain ways. Here's an example of a of a charger from Electrify America where they accidentally left a Team Viewer session on and they were able to log in with their smartphone and interact with the equipment. So, it's a

Windows machine, which isn't too common, but Windows machine and they're able to navigate to the using their phone here just accidentally let this session on. So inside these devices there's tons of interfaces, right? There's all this communication and capabilities there. Um I'll I'll point out one example here. So some research that came from Pentest partners. Uh what they did is they found inside this EO box and I've actually found other devices that have this where there's just a Raspberry Pi sitting there, right? Raspberry Pi is great except that it doesn't include any secure boot. There's no encrypted drive. It's just an SD card that holds everything. So, you just pl take that SD card, plug it in your laptop, add your

credentials, plug it back in, log in, SSH in. You can basically modify the entire operating system. Um, so that that's a that's an issue. Um if if you want to uh you know uh maintain any source control over your software or you know control over the communications from these systems and here's a supply chain attack where um in Russia um at the time of the beginning of the Ukraine Russia conflict what happened is the Ukrainians figured out they had supplied some equipment to Russia for this EV charger and they said well we have access to this I don't know if it was legitimate or a true back door or what, but they were able to push firmware or

an update to these chargers to basically prevent them from charging. So, a traditional supply chain attack. And so, what it, you know, it basically prevented the the chargers that ran on the M1 or M1 corridor between Moscow and St. Petersburg from operating and displaying, you know, Putin is a big head and glory to Ukraine. So, here here's work that I recently uh well not so recently uh did but started about a couple years ago got halfway through dropped it and picked it up for this session. So congratulations you get to hear the latest and greatest in this space. So we've got a new CVE that I'm I'm announcing here. Um it's for this guy right here this juice box. It's an

ENLX way um device, level two charger and essentially the vulnerability associated with it is almost trivial. Everybody over there at the the capture the flag game could do the same thing, right? So you you connect this to your Wi-Fi network and from that there is access to operating system commands that allow you to extract PII, change certain settings and manipulate the file system. And so let's take a look at what what this actually is. Uh there's this nice uh Wi-Fi module that is provided by Silicon Labs inside this device. And with that comes an API where you can run all these commands. So many interesting commands. Look at all this stuff. You can do ACP uploads, factory resets, ping

to potential DOS or DDoS attacks. Uh you can formate format the flash. You can you can do a lot of different things. And for that reason, if we look over here, you can upload and download files. You can read and modify the Wi-Fi password. You can reboot the device. Luckily, there's only about 20 of these things out there on the internet. So, that's good. But this is a severe vulnerability with this equipment. So, this is what it looks like. Basically, you just netcat into this thing on port 2000. You can get all the parameters doing a get all. Here's a throwback to my Albuquerque people. Here's your Wi-Fi password. I'm the one who knocks. And uh

the all Goodman Wi-Fi is right there in plain text for anybody to read. So if you can find them on the internet, you can get the Wi-Fi password for these. And so there's a lot of uh C.WE associated with this as you can see here. um probably should bet some of these with CISA or others, but pretty fascinating. One of the things that is on this device, if you look at the operating system, is a a PEM file. It's a x509 certificate. And that x509 certificate is for a DNS record called evconnect.com. And evconnect.com is is, you know, this is kind of like a third party backend management system. And so that I believe they never actually fully implemented

this. This company is actually um gone gone bankrupt and or they got rid of the assets which I'll talk about here in a minute. But I think they were on the verge of of using this as their OCP open charge point protocol backend system. So there's actually kind of this other market for that um where they can they can um hook up to the to these other networks and then communicate that information to them and then they handle the management of it whereas the physical devices are manufactured by a different company. And so what what does this give you? If you in this case, you know, credentials were expired. I don't think they fully implemented it. But if

you could do this for another charger, what would you be able to do? Well, there's a lot of things. So if if you're at the headend system and you compromise something like uh styus did or if you're down here at the EVSC level electric vehicle supply equipment that level there are different things that you can do and you can try to compromise the other direction right so from up here you have the ability to push remote stop transactions you can stop charging sessions you could change the tariff rates you can make it really expensive or make it all free uh you can um do firmware updates, things like that. From this side of the system going that

direction, you can do your code injection attacks. You can try to compromise the headend system to to compromise the entire fleet. Uh you can manipulate the measurement data. So, uh I didn't really charge that much. Don't bill me, you know, or uh manipulate um some of the transaction events so that it doesn't look like you actually need to be build or you did anything. you can modify serial numbers or vendors or other things. So, this is this one's kind of interesting because you can manipulate all of the the equipment there, right, to basically ruin the database. So, you're just like pushing garbage into their database and so now they don't know what to trust. You could

iterate over the serial numbers and models. So, cause a bunch of issues associated with that. Uh, all right. So, we've got a timeline here of what happened. basically back in 2023 reported this to Vince and then um notified the company of of the issue. Didn't get a response. As I said, they've they've kind of closed doors. Uh but uh they they suggested I go public. Um and I eventually did today. Uh so this this is the uh timeline of that. Again, these are stranded assets that I believe most recently this Voli group has actually tried to help out the customers of these equipment uh by providing the back end. If you want to protect your equipment, you can actually

set a password for the SIS for the kind of remote terminal interface which is not there by default. Uh or you can disable it with these commands. And there's a lot of other work that uh somebody else can pick up on with this if they're interested to reverse the app or the kernel or try to get that information modified. And so I realize I'm closing in on my time here. So I'll I'll just briefly talk a little bit about some of the defenses and then hopefully get a few questions. So the the thing I was talking about with Vault Typhoon is that they kind of skip over the the cyber kill chain, right? So normally you're

you're doing all this recon and weaponizing stuff and then doing this fancy stuff where you're um delivering these payloads that get executed and then indicators of compromise show up along along this chain. But what they're doing is they've got valid credentials. They log in like a normal user, probably using a normal user's account, and then just go straight to deploy, right? Just to manipulate the OT system, however they choose. And so our our company deer security is focused on the cyber physical layer down here, these actions on objectives to actually look at the physical impact of what's going on and detect it at that level. So we provide a network intrusion detection system that does the normal

stuff network analysis using Z snort or those types of things. But we also extract the information in those protocols convert it to physical meaning voltage frequency power and then combine these two things to say hey does this stuff make sense? Is it valid? Are there anomalies here that would indicate that there's something going on at the power system level that's that's actually preventing this from working correctly? And so as an example of that, here is a autonomous grid support function called volt bar where at low grid voltage you're injecting reactive power and at high grid voltage you're absorbing reactive power. And so normally you follow this curve because that's grid supporting grid helping. But if you flip

a sign in that command, you actually get grid damaging. You're injecting reactive power at high grid voltage, which pushes pushes the power system into an unstable state. Interestingly enough, in the Iberian blackout, there was a solar plant that had been misprogrammed in this way where when there was a voltage pertabbation, it did the exact opposite of what it was supposed to, which actually exacerbated the problem and caused more issues for the grid operators there. And so this this type of thing can happen accidental or not. Um and uh our adversaries are are paying attention to that and causing um well pre-planning for these types of things. Could talk a lot about that. All right. Briefly, uh the response

characteristics for different things is very important. So hey, you picked up there's an issue. What do you do about it? Right? So in a lot of these local things what you can do is you can reset those settings figure out which IP did it who did it how they did it and then isolate that uh pathway and so you want to roll back to a known good state you want to roll back to where it can actually support the grid operations. Um similarly you can roll back firmware if the if you see a firmware update and all of a sudden the device starts uh behaving oddly and then at the kind of headend system here uh you can revert to

um known good settings you can also do this analysis you're going to you're going to do all your normal stuff blue team stuff where you're going to start incident response you're going to look at who did it how they did it and so forth but uh it's it's very important to as quickly as possible bring the OT system back to a known good state and uh prevent further damage to the operational characteristics of your critical infrastructure. All right, so in conclusion, threat actors are prepositioning in our in our energy networks and other critical infrastructure. We've got several vulnerabilities with these uh PV systems and EV chargers that I talked about today. And then we need some ML driven

AI driven CA capabilities to respond to that and uh basically restore operations as quickly as possible. So I think we have a few minutes for questions. Yeah. >> Does anybody have any questions? This is such an important topic like you said they're planning, they're preparing for C2, right? So for these chargers, do they not do any like firmware validation or like well why why are like the most simple basic uh things that we would do with any other system ignored, right? >> Yeah. Well, there's a lot of answers to that question. One, regulation. There is very little regulation in the cyber security space, especially in the US. Europe's a little bit better, but here it's the wild wild west. Uh the second

is they get away with it, right? There there's um there's such a market push, so much pressure on these companies and it's so difficult to compete that they take as many shortcuts as they can. And there's actually other research uh last point on this. There's other research in this area that uh was conducted by a researcher called Sebastian. If you if you go on GitHub, you could see him under vegan MOSFET is his name, where he's he's um essentially modified the firmware on this equipment for several different devices to have it perform operations that it should not, including uh tripping the breakers in his own home by switching the IGBT switch uh switching signals to that that

power stage. And so very lowle controls. Yeah. Sorry. >> Okay. I have a question. So you were talking about rolling back to image and things like that. So I'm kind of curious about protection around like the golden image for firmware and things like that. If I was a bad guy and I wanted to have persistence in your infrastructure, I try to have my code be persistent in your >> right >> if it were me. That's where I would go because that's kind of like your crown jewels, right? You can always knock me down by rolling back, but can you knock me down? in your image already. I mean, you're just killing your dude. >> If if you're in the build, you're in the

build, right? Um, yes, that is a that's a that's a fair point. And so, how how do we maintain the gold image is a very big challenge. >> I would say for the parameters, it's a little bit easier. So, there are are a set of parameters for these energy devices that are required per the US interconnection standard. They come from IT E547. And so that standard says you must be within these ranges of adjustability in order to operate this equipment. And then each jurisdiction, each utility will will set those parameters slightly differently, but they're they're pretty well constrained, right? And so you can you can hit the parameters pretty well. The firmware images, I agree, it's

challenging. How do you know you haven't been compromised in the supply chain? Yes. Um, so is Alex Adams here or are you There you are. Okay. Uh, we're gonna take one more question and then we're get you swapped over. All right. One more. On a local level, can you use Wireshark in order to see if there's a vulnerability in that charger that you know that model that you showed on your slides? >> Can you use Wireshark to see if there's a vulnerability with your charger? Uh in this particular case, I don't think people are regularly communicating on port 2000. It's just a maintenance port or actually there's there's kind of a web interface, a local

web interface that I believe is using that to populate the web page. And so that's why it's open and exists there. And so I would say that in this particular case, you're not going to see much external traffic in Wireshark to that device. So I I would say no, you won't be able to pick that up. I will say a lot of other deer equipment though, if you monitor the traffic there, you're going to see some very strange things and a lot of communications to like Chinese servers and other uh servers that you would not expect. So it would be interesting to tap into some of those communications and see where they're headed. Such a great talk. Please give Jay

Johnson a round of applause. Thank you all for your questions.