End of Life Equipment Should Not Mean End of Life (Your Life)

Welcome to Besides Las Vegas's I am the Cavalry Track. A few notes before we begin. First, we would like to thank our sponsors, especially our diamond sponsors, Adobe and Iikido, and our gold sponsors, Drop Zone AI and Profit. It is their support along with our other sponsors, donors, and volunteers that make this event possible. These talks are being streamed live and as a courtesy to our speakers and the audience, we ask that you check to make sure that your cell phones are set to silence. If you have a question, there will be an audience microphone placed in the aisle thusly. Please use the microphone to ask any questions so that the stream and the recording can hear you. There's a live

stream. And as a final reminder, the Bides Las Vegas photo policy prohibits taking pictures without the explicit permission of everyone in the frame. These talks are being recorded and will be available on YouTube in the future. Our presenters have kindly granted permission to take pictures of them, so you may take pictures of them, but not anyone else. >> Only Thank you very much. >> If my eyes are closed, no sharing, please. >> All right. Um well this is the final very highpowered panel of the day. We are super excited to welcome Ms. Stacy Higinbotham and Mr. Silus Cutler and Mr. Paul Roberts to help us understand how to deal with this problem of end of life

equipment so as to not end our own lives. over to you. >> I [snorts] feel like our title may be a bit of an overstatement. [laughter] I hate to start on that, but I'll start with my name is Stacy Higin Bothotham. I am a policy fellow at Consumer Reports. Can everyone hear me in the back? Yes. Okay. And I'm going to let each of my colleagues introduce themselves. >> Hi, I'm Silus Cutler. I'm a principal security researcher at Census. Uh I also work as an adjunct for the Institute of Security and Technology. >> And I am Paul Roberts. Uh my day job is a director of content at Reversing Labs, which is a um malware analysis and

software supply chain security firm. Uh and I'm founder of a nonprofit called Secure Resilient Future Foundation. Um which is focused on promoting smart cyber policies for the future. >> It's a very broad goal. I like it. Yes. Okay. So, why are we here? >> To scare the pants off of you. Are you ready? >> Um, just to get this started, we're going to talk about this issue from the consumer, enterprise, and OT side. Starting with the consumer. But first, what are we talking about? >> We're talking about the risk of endoflife devices on networks. And is this even a problem that we should worry about? Yes. Um, on the consumer side, Consumer Reports has done some work on

this, but you don't trust us. The FTC actually in November did a study and they noticed that of the hundreds of connected smart IoT devices, only 11% actually clearly told consumers what the end of life product or policy was. So, when those things would go offline, which is crazy. We'll talk about it in a minute. So additionally last year uh at census we found 400 uh exposed human machine interface devices HMI devices which were mentioned briefly on some of the talks before today uh found 400 of them in the US uh related to water treatment plants that you could visit by with a web browser and see the tank levels for uh water holdings in various communities

across the US. And you know big problem that we have right now is a population of between 900 and 100 mil 900 million 90 million and 100 million uh SOHO routers uh deployed in homes and small businesses throughout the country. A significant portion of which contain exploitable vulnerabilities andor have been declared end of support and end of life meaning they no longer receive security updates and patches from the manufacturer. These are being actively targeted by state actors including China as well as cyber criminal groups. Um and there are uh immense repercussions from that. We're going to talk a little bit about that. >> Yeah. So starting right off uh and continuing with the the things to worry

about. So a lot has been talked about about the sort of threats that are being seen across across various sectors right now. And one of the ones that most people are familiar with uh is Vault Typhoon. Um, so, uh, I come from a background investigating ransomware actors and targeted intrusion actors and I've struggled actually for quite a bit with Volta typhoon specifically. Um, I keep trying to break it into a model similar to what I remember from like ransomware groups where it's something that like I try and wrap my head around them as a group because I want to find an office or some unit that is the ones that are day-to-day conducting these intrusions. Um, but as I've as I've

looked more into Volt Typhoon, thank you. Uh, and as I've continued to track on things, uh, it's important to think of Volt Typhoon not necessarily like a traditional threat actor in the sense that um, you know, that might be one unit. It's not going to be something like that. Something like Volt Typhoon and the Salt Typhoon actors that people are talking about these days as well. Uh these go to something actually that we started talking about in the security community back in 2018 2019 called supra threat actors in which it's multiple different units all working together towards what essentially affects amounts to a national effort and we've talked about this with China as part of the

Taiwan Strait conflict um specifically the technique as I've always been told uh what they're trying to accomplish is what's called a fire sale this was part of stuckset well supposed to be or uh nitro zus I think was the parent operation Uh and the idea is that in the event of a conflict, you want the ability to disrupt uh transportation, financial utilities, power, all the critical parts of a country's utility sectors. Volt Typhoon represents fundamentally the targeting of the energy sector and building this long-term capability capable of conducting a disruptive action within a timed window as part of a larger effort. Um so the challenging thing is is we trying to to put it in

con in like a shape that we understand often. Uh folks may be familiar with things like the KB botnet which Lumen has done some fantastic reporting on. And it's important to remember like the KB botnet itself likely is only one part and it may even be a contractor supporting this broader volt typhoon effort. Um but groups like this are are highly sophisticated. They know how to conduct these attacks. Uh, the term living off the land to this day bothers me a lot. Um, so we'll dive into a little bit more as we go. >> All right, so we've just scared you. Why does EOL matter in a connected world? And we probably instinctively understand this because many of us are probably

dealing with connected devices in the cyber security community. But connected devices establish this ongoing relationship between a consumer and a manufacturer. You no longer can sell something, throw it out into the world, and pretend that you can leave it alone. You have to continuously update these products and continuously communicate their status. And for a consumer, and by consumer, I mean anyone who buys these products, but specifically consumers, the lifetime of a connected product, it no longer ties directly to its its physical life, right? So, I think of this as a a light bulb. A lot of us probably have smart light bulbs. Historically, when your light bulb burnt out, I know this is before. Pretend we still have

incandescents. The little tungsten thing, it burnt out literally and your light bulb would turn off and you knew it failed and you would replace it. My smart light bulbs don't do that. [laughter] When they are end of life, I have no idea, right? But then they become part of this larger attack service. So when you think of the EOL issue, this is about reducing the overall attack service and capacity for people to use like to target or hurt the US. All right. Nobody likes maintenance and the industry isn't set up to support it. We have heard this like a million times today. The issue is we're all excited about innovation. We're excited about doing new things,

but we're not excited about going back and making sure the existing things we have stay operational and are still getting security updates. I mean, it's basically like changing the air filter in your house, right? How many of us are excited about that? Literally no one. Okay, so I'm sorry. You are on the maintenance team and bless your heart. So, what happens when we ignore the end of life issue? At Consumer Reports, we call all of these connected devices in the home, it could be in the enterprise, we call them zombie devices. And I like to tell people that today zombie devices are running a muck. They are everywhere. And we just don't know it. And the other

issue is these are supportive devices with unsupportive owners. And what I mean by that is in the enterprise, especially in the consumer home, if you're not doing automatic patching or updates, the owners of these devices, they have to do it. And a lot of them aren't paying attention to it. And underlying this whole issue is the fact that we need herd immunity against malicious actors. So again, reducing the attack surface, getting everybody if if everybody pays attention to their end of life, then we reduce it for everybody and we're all ultimately safer. I know this is a hard cell, but this is the cell we have to do. All right. So in this talk, thank you

for coming. We are going to explain the problem of zombie devices on the consumer side. We're going to show how EOL affects us in IT and OT environments. We're going to expose this obscurity by security disinformation that is promoted by vendors. Paul is going to do a great job on that. And we are going to provide practical ideas and policy suggestions that you can advocate for or vote for if they ever come up. And yeah, are we ready? >> Yes. >> Consumer time. How many of you guys ever do an audit of your smart devices in your home? And Wow. >> Love you. And y'all are security people, right? So every I do it in October.

>> Not representative of the population. >> Yeah. This is absolutely not. No. No. >> But every October because it's cyber security awareness month. I actually go through all of the connected devices in my home and I check to see if they're still supported. I used to run 80 something devices in my home. So this was a long process. Since I've been doing this in the last two to three years, I think it's year three was this year. I only have like 40 devices now in my home. But I do call and a lot of times y'all it is calling because most vendors still don't have this information readily accessible. I have to call or email and about a third of

the companies never get back to me. So this is what consumers are dealing with and it's no wonder there's all these consumer zombie devices running around. We did a survey in December where we actually asked consumers about this whole issue like how well do they understand that any connected device in their home is a potential zombie. The answer is not well. And then what is their impression? So for 40% of consumers who have purchased a connected device, the only time they figured out that it stopped getting any sort of feature or security updates is it just stopped working for them. That is insane, right? I mean, if you're just relying on something and suddenly it just stops working. But what's worse

is only 39% got any sort of notice from their manufacturer. And this is just people who realized it's it's it's a zombie, right? There's other people who have no idea and they're not doing the security audits. So, this slide has way too much information for you, but I'm happy to share it. We have it online. But basically, what this is trying to show is how consumers don't actually understand that their connected devices will one day die and they will die because their software stops getting supported. because I don't know how else I could explain to some the 43% of people who think their Amazon Echo or their Google smart speakers will remain working after software updates. They're

like, "Yeah, it's totally going to work for me." And I know some of you guys are like, "They can hook it up via Bluetooth and then they'll still be able to control it and use it offline." Don't. Most consumers aren't doing that. I mean, I want them to, but no. Okay. And this is just a general sense of how long consumers expect certain devices to last. >> And if you can see here, a lot of people have some kind of crazy expectations. I don't know how many people really think a smart light bulb is going to last 10 years. Actually, I do because they're right there. Um, I will tell you though, just in the let's call out some good

actors, Philips Huegh light bulbs, I've owned those for 10 years and they are still getting software support updates and I can go to the Philips Huegh website and they will act if I if you Google your Philips Huegh light bulbs plus end of life or security updates, they have a nice little chart that shows you exactly what they're supporting and how long. That's the sort of change we want to see in the world. So on the consumer side, what do we want? We want manufacturers to set up what we call minimum supported time frames on products. That is a clunker of a statement. Think of it as an expiration date, but we can't call it an expiration

date because I want to give manufacturers the chance to extend it. So we're calling it the minimum supported time frame. And basically when you buy a product, I want you to be able to look on the label and say, "These light bulbs will get security updates for three years. Awesome. These light bulbs will get five years, even better." And I also want the manufacturers to be like, "Hey, I'm going to extend that and I'm going to let you know via email or on the website and it's going to be easy to find on a product page." We also want manufacturers to proactively notify comp consumers. So that 40% who didn't know their stuff was off like going to stop

working, we want the manufacturers to be like, "Hey, in six months it's going to stop working." We also want to help consumers understand what to do when this happens. It is very hard to tell a consumer that that $180 Amazon Echo that they bought back in 2015 no longer is getting security updates. But you know what? If you have one of those Pringles can Amazon Echo devices, it's end of life. It is not getting security updates. You should probably unplug it. Just PSA for y'all. But it's hard for consumers to do that and I don't blame them. And then we think longevity and support should become a market differentiator and that disclosure upfront is going to drive

that. We've actually seen this in the smartphone industry or yeah, smartphones. The iPhone used to have like two years of support and then Google was like, "Hey, we'll give you three." And then Apple was like, "Okay, fine. We'll give you three." And now we're up to seven. We're seeing it in the smart TV space. LG and Samsung TVs, they're both offering seven years for your smart TV. By the way, that's only seven years for your TV. I don't know if you think your TV should last that long, but you know, you at least know. Okay, so those are some of the things on the consumer side. Now, let's go to Paul and talk about enterprise.

>> Sure. First of all, let me start by saying that in 1986, my mother redid our kitchen and installed a Subzero 550 refrigerator. And 40 years later, that refrigerator is still operating. Um, so all those people who said, I don't know, maybe like five years for my refrigerator, 10 years, try 40 years. Um but that's obviously not most refrigerators but um but our but it's a way of sort of thinking like the way we're thinking about these devices and their lifespan has been curtailed by by vendors and it's good to kind of pop out of that sometimes think back to when things lasted a lot longer. So my name is Paul Roberts. I'm the president of

this nonprofit called Secure Resilient Future Foundation. Again, we're really focused on promoting uh policies that promote uh secure, transparent, and resilient future. Um addressing cyber security risks. Um so, yeah, as as Stacy said, I'm going to talk about this sort of um endof life problem as it relates to what we would consider it, uh whether that's enterprise, just kind of the the technology that businesses and nonprofit organizations, public and private sector organizations rely on, a lot of our economy relies on. And I know at conferences like this and black hat and defcon even up here we have consumer OT it kind of siloed right the way we talk about these problems but the but the

truth is of course as we know they're all intertwined with each other um and we've had incidents in recent history that have made that really clear and I one of course as as you would indicate up here would be the Colonial Pipeline hack back from 2021 that was compromise of Colonial Pipelines IT infrastructure I think through a compromised VPN account if I recall um uh by a ransomware group and to you know uh kind of contain that breach Colonial shut down their entire pipeline system that transfers uh natural gas and up from uh Florida up into the Northeast um or or actually gas just not even natural gas just um and huge disruptions uh lines at gas

stations and so on. Um was it a compromise of their OT environment? As far as we understand, no it was not. But the result was the same which was a shutdown of this critical infrastructure that provides you know fuel to a big chunk of the US population in the northeast. Um so and and that's not the only one. More recently we've seen um attacks on uh St. Paul, Minnesota. Um, and I believe Adelene, Texas. Um, where we again ransomware groups compromising public sector infrastructure as we all know, local governments, uh, you know, state governments, uh, underfunded, understaffed and not secure. Um, and once again, those organizations shut down critical services used by residents in order to contain and respond to the

ransomware risk. Um so when we talk about IToT consumer technology yes they are very different um different vendors different architectures different support ecosystems written in different languages communicating in different protocols those are all legitimate distinctions between these different technology types and consumer IT consumer electronics IT and OT are often deployed within the same environments and are interlin. Um, vulnerabilities in one uh can lead to compromises of the other, whether that's a a direct compromise uh or an indirect compromise like with Col Colonial Pipeline. And of course, as we know, the malicious actors, the volt typhoons really don't care what their uh mode of access is to your environment, right? They are resultsoriented. they're focused on the

objective they want to obtain. If they're coming in via um a smart television set in the um you know kitchen uh where your employees gather, that's okay. Uh if they're coming in via you know a router switch, firewall uh public facing application server, that's fine, too. Doesn't really matter. What matters is the is the objective. And we all have examples of this. There was the I think casino that got hacked via the um the fish tank. Yep. That was a found one. We remember Target that got hacked via the the HVAC system and a contract that goes back whatever 15 years now. So we all understand this, but it's important to keep in mind that these

distinctions um are kind of um you know don't really have hold much water. Um and an example of that of course is SOHO routers, right? So, Soho, small office home office routers. Um, there are, as I said, around a 100 million of those these deployed in the United States. That's based on statistics of about 130 million fixed broadband subscriptions in the US and around 80% of those households own a broadband router. So, that gets us to around 100 104 million uh SOHO routers out there in homes, small businesses, you name it. Um and these are actively targeted by malicious state actors uh vault typhoon uh cyber criminal groups, ransomware groups um and others uh Chinese uh Iranian, North

Korean, Russian, you name it. Um and why they build botn nets out of these devices. We we heard we knew that years ago with the Marai botnet which targeted IoT devices and those botn nets are used for a variety of purposes. could be distributing fishing emails and other content that are used to gain access to sensitive accounts that they then use to move laterally. Could be malware distribution. Could be for command and control and and moving data that you've stolen off of an environment. Um password spraying uh you know hard um uh brute force attacks to crack accounts. One of the advantages that these have and why actors like Volt Typhoon like SOHO routers deployed in residential uh

settings is because it gives them a really good way if there are thousands of them or tens of thousands of them of spreading out malicious activity across a large population of devices that get makes it much harder for the target to spot patterns and also avoids you know the types of threat detection it's going to say oh that's traffic that's coming from Russia or from, you know, Eastern Europe or China. And so we're just going to block it. Um, hey, that's traffic that's coming from, you know, suburban Boston. Um, this probably legitimate. Um, so they they serve a whole bunch of purposes. So, Soho riders are a great example of kind of this cyber risk problem around smart and

connected devices. So, I'm just going to ask you all um raise your hand if you've got a, you know, broadband SOHO router uh at your home. Yeah. Okay. I think that's an under representation, but okay. Um and raise your hand if you actually purchase that yourself um deployed and actively manage it. Okay. Raise your hand if it was supplied by your broadband provider and they Yeah. I'm gonna I'll raise my hand too, right? Um Okay. And now raise your hand today if sitting here today you can state with confidence that that broadband router has not been declared end of support or end of life. Pretty good self- selecting population. Okay. [laughter] Not reflective of the

population in general. But and and I'm going to say I'm not 100% sure. I you know I logged into my router maybe a couple months ago but I didn't go and say oh you know what version of firmware is it doing? Is this still supported? Is this device still supported? Do I need to upgrade? Unclear. Um, one other question. So, how many of you have actually hardened your SOHO your your SOHO device? So, strong password shut off FTP and TNET and that's okay. Okay. You guys aren't representative of the population. I'm going to stop polling you. Um, [laughter] the reality is of course these devices are kind of set and forget. um for for many, you know, of that 104 million,

what percentage are set and forget? People have no idea. You know, they haven't been updated in months, years, decades. Um that's a big proportion of what's out there. Um and uh as Silas said, there's been a lot of really good research done on um botn nets uh by like Lumen and Black Lotus Labs um cyber criminal and state sponsored that rely heavily on compromises of these devices. So a substantial population of these devices whether it's millions tens of million are end of life end of support meaning they no longer receive software updates and patches um and therefore are vulnerable. So if there is they're running a vulnerable version of a known that's been identified in firmware

attackers that can identify that router can compromise it using that that known vulnerability. Um so end of life Soho routers are a big target and um this is a just a quote from from Lumen kind of explaining why you know um they're associated with home and small business users. They don't have the the ash Stacy pointed out consumers generally don't have the time knowledge or expertise to actively maintain them in the way that many of you are. Um and because they're SOHO routers they actually can they're they're fairly flexible. They can handle a lot of traffic. They can they're very effective as as infrastructure for for conducting malicious activities. Case in point as Silos mentioned is the KV

botnet. Uh this is a botnet that dates back I think to 2022 was kind of first identified uh called out by Lumen and Black Lotus in 2023. It's not huge. It's not like it's a huge botnet. I mean it's hard to know exactly how big but I thousands tens of thousands of devices maybe. Um but highly effective. Um and it's been used in attacks by Chinese state actors Volt Typhoon uh targeting US critical infrastructure uh as well as those in in you know US allied countries um and it's being used to basically target um communication systems um a you know with I think the idea is in in an incident like invasion of of Taiwan um

being able to shut down the ability of the US to communicate and coordinate with its allies in Asia and Taiwan and so on seems to be the pattern. Um, and the devices by and large that make up this botnet are end of life Soho routers by Cisco, Dratech, uh, Netgear, uh, more recently Axis IP cameras as well. Um, so these are all being targeted and this is ongoing. So I mean, this was written about two years ago, but this this botnet is still out there, still operating. Um it technically got shut down by the DOJ um back in last year. Um really what that meant was they you know you had a court warrant. Um they

according to them removed the malicious software the the the KV botnet software and cut off access to the command and control environment. Although Silus says that might not actually be what they did. >> There's weird legislation around where they're allowed to uninstall from what I understand. Right. >> They they had a notable impact. >> Yes. >> Yeah. >> However, those devices are still out there. They're still end of life and unsupported, >> right? >> And somebody could also just walk back in. >> That's right. >> And the thing is is these environments as well are are target rich and actor heavy rich as well. Like >> Yeah. >> Miri is a routine thing where someone

will break in, break back in, somebody else will break in the next week. That's right. >> Tip off the existing mirror infection with their own. Right. They're like abandoned homes basically. It's just like who's squatting in it this week, who's squatting in it next week. One of the things that KV botnet operators did when they when they compromised these devices was remove other malware. So like a lot of these devices already had malware on them, but but the botnet operators were like, "No, no, we need to be the only malware on this thing." Um but but for the DOJ to be like, "We shut this down." It's like doing an endzone dance on like the 20 yard line, you It's

just like, yeah, you took care of this version of these devices being misused, but you know, >> mission accomplished, right? Um, you know, the other thing I'd like to point out just before before I hand the mic to to Silus, um, is that this issue is not of end of life devices and endof life software is not specific to the consumer space. It's not specific to SOHO routers. Um and an example of that we've seen more in the traditional IT space is with the Ivante VPN appliances. Uh these are really widely used virtual and and physical VPN uh appliances for obviously virtual private network remote connection into into sensitive networks. Um by their very nature they are public

facing right because they're giving other people access into your environment. Um and these have been the target of a string of attacks going back three years uh targeting serial remotely exploitable zeroday vulnerabilities in the Avanti firmware. Um Volt Typhoon is generally you know these attacks are generally attributed to them. There have been numerous numerous reports written. There were I think there were like there was one month where like five separate OD days exploited in a in a single month. Um and the targets were Department of Energy, EPA, Transportation Security Administration, the UK's National Cyber Security Center, New Zealand, you know, Five Eyes targets. Um so these are highly targeted state sponsored attacks. Um what's really interesting is that Eclipsium and

after this sort of string of like attack after attack after attack, this firm Eclipium that does embedded device security decided to like really take a look at what's going on with this Avanti firmware uh that the stuff that's running these devices. Um and Avanti actually is one of the few vendors out there that actually encrypts their firmware images. So it's it's not straight not as easy to sort of reverse engineer them, figure out what's in that firmware. They do that ostensibly to secure them. Um, but what Eclipsium found really makes you wonder if that was the actual purpose of encrypting those binaries because what they found was end of life code. The base operating system for the Ivante devices was CentOS

version 6.4. That was originally released in 2013. It was declared end of life in 2020. Um, and this analysis was happening in 2024. So these devices out there deployed in three-letter agencies, sensitive government agencies are running a endof life unsupported, you know, open source operating system. They're also running an endof life version of the Linux kernel 2.6.32 that had been declared end of life in 2016, so 8 years before the analysis happened. Um, and they were running a version of OpenSSL that dated to 2017. Uh so again these were software that had been updated patched um secured uh but those you know like with open SSL or with you know Linux kernel there were more recent more secure versions of it

those had not been adopted by the vendor also and this is something I see through my work at reversing lamps and the work we do at supply chain security very vulnerable code. So, Eclipsium found numerous outdated libraries, open source and third party. Um, 396 CVEes or software vulnerabilities that had a rating of high um and uh 111 of those around a third of those had known exploits uh and two of them had uh remote exploits. Um so this kind of speaks to the larger question. So end of life software, how long vendors are going to support software is one aspect of a larger problem which is around software integrity, software supply chain security and so on. Without clear

guidelines around support and maintenance um as well as transparency about what's in software and the quality of that software, >> huh? >> SBOS. >> SBOS. That's right. Software bills of material. um consumers businesses critical infrastructure are at a huge disadvantage. Um so my my my take on this is the emperor has no clothes. Emperor OE emperor. Um [laughter] um and this kind of comes out of the work I've done with with the right to repair movement. So I testified in front of Congress back in 2023. I've gone to a lot of hearings and advocated on uh for for a right to repair. Um and yes, is there is Yes. Yeah. Yeah. Yeah. Um the the

message the sort of I I would call it disinformation from the software industry including cyber you know including vendors who sell cyber security is that you know and this is what they would say in hearings like you know the security of our products and the security of our customer data is our top priority. There is nothing we take more seriously than that, you know, and you know, any regulations you pass are really going to, you know, hinder our ability to innovate and protect our customers. Um and then there's this sort of there is security and obscurity um uh line of argument you know let us build a walled garden that will be a beautiful you know pure uncorrupted

place where we will protect our customers you know data and and environments from compromise. Wall gardens are beautiful and safe. Um what that means practically is monopolies on service maintenance and upkeep. um latitude to kill products whenever you deem them no longer profitable. Um and really limiting the ability of customers to actively maintain, repair and um uh sustain the products and technology that they bought. Um so there is this kind of idea that third parties you know independent repair operators or the customers themselves can't be trusted to patch maintain update their own stuff. You have to have our authorized people come in and do it. Um and that end of life is a purely um you know vendorbased

decision. It's our decision when to end life end end of life or end of support products. It's based on our own calculations. there are no you know that you're going to bear the burden of that but it's our decision uncorrupted uninfluenced by anything else um so this is you know this is a major phenomena in the industry software industry um and there are a lot of cyber security consequences to it what are the solutions so Stacy had her ideas um I think there are a lot of uh one of the things that we've proposed is um model legislation around Okay, we're not there. Okay, >> some of the things that we can do um is

mandate vendor transparency. So Josh mentioned software bills and materials. Absolutely. What's the ingredients list for your software? Are you using a uh 10-year-old end of life operating system? If so, that might influence my decision to buy your product, renew your subscription, what have you. Um and uh what is the support life cycle for this product? Like Stacy said, if vendor if if consumers know how long you're planning to, you know, provide security updates and so on, that might influence your decision and might create positive market forces to do more support rather than less. Um and um one idea might be mandating patching uh for crit high-end critical vulnerabilities um even for end-of- life products. And when you

think about it, that would really for customers address probably 90% of the issues with end of life declarations, right? It's like as long as you're getting security updates, but you're not getting any more feature updates, that's probably okay for most vendors, right? And would have a huge impact on just the security of the larger population. Um, and then some kind of graceful handoff around end of life event of support. It can no longer be like we've just declared end of life. We're turning our back and walking away and the burden falls on you to either replace the device or just take the chance of getting hacked. Um some kind of graceful handoff where you say we're declaring it

end of life but we're going to open source the software or we're going to put the software into escrow and let third parties come in and maintain and patch it and so on. Okay. And with that so those are all ideas to discuss and we can talk about them more. I'm gonna pass baton to silos talk about OT security. >> Thank you. So I will fully admit I'm a fairly new person when it comes to entering the OT world. I've started doing a lot more of OT work as part of my role at Census and trying to understand what we see on the internet. Um and one of the things that has historically been told to me by every

every person I've ever talked to in the IC world or in OT world uh is that OT systems function differently. They are designed differently. they have different priorities which in fairness there are some very critical ones but there are a lot of overlaps between our worlds. Um one of the kind of things I've heard from both of you actually has been uh how critically important it is for folks to be paying attention to be aware of their devices. Um and the thing I heard when talking to a lot of folks going into this talk was um yeah end of life is commonly found in OT networks but it's justifiably that way. um it's infrastructure that they have a plan

around that they know about and so uh I'm actually here to say that there's a lot we can actually learn from the OT space in how to bring bring better practices and better planning around around end of life uh in terms across both enterprise and consumer as well um as I said there's a very different focus for OT environments the focus is often on availability you're expected to maintain these systems not just for uh you know the 20 40 years I think you said it for >> 40 years or 39 years. >> That's incredible. Um >> that's unusual. >> Yes. >> I wish I had gotten four years out [laughter] of my last refrigerator. Oh

my gosh. Um but the it's like the those devices are designed for a short lifespan. The sooner they replace it, the better. Whereas if you're um think about like a traffic light system through a town, that's an OTA environment. Those traffic lights and the interconnection of all of those um that is not designed for four years. If every town had to come and replace their traffic lights every four years, it would be a disaster. Um, so end of life is very common to find. Uh, but it's justifiably so. Next slide. So, uh, there's a couple things we can take away from the OT world that apply directly directly to enterprise and consumer. Uh, so let's talk about what some of the

healthy practices look like. Uh, everyone here has also talked about the advantages of multi uh or of open standards where if your smart light bulb communicates through the Philips app, but there's an alternative app that you can use. >> Ziggb, >> yes, love ZigGB stuff. Um, and Meshtastic I think is built on ZigGB as well. So there's other implementations where these open standards allow for a lot of really cool innovation as well. Um, >> we won't talk about Ziggbby profiles, but yes. >> Yeah, that's fair. Um so one of the common things as well that I found is uh when folks in the OT side of the world are looking for uh looking to maintain

infrastructure longterm is they have plans around how to deal with uh obsolescence of devices over time. So if they know that they're going to be putting the system in they're expected to run it for the next 40 years and that's when there will be a refresh. Along the way, they're going to have to replace parts, and they're going to need a longer term relationship with their vendors to ensure that they not just know when this device is going to be end of life, but also when support is going to end, when part availability is going to end. Um, and something as well that I really wish was like there was a better way to measure this, but it would be

sort of the um like end of attention, like when is when do folks stop looking at it? And interestingly from the stat from earlier when I talked about the 400 HMI devices, one of the my favorite things so far with engaging more with OT uh users and vendors uh they are much much more responsive. So of those 400 that we were able to identify, I think we had like a I'm going to estimate low because I can't remember the exact number right now, but it was either like 91 or 93% remediation. It was incredible. When I've done malware cleanups in the past, um, if we have a thousand victims and we clean up 20, that's a great number. Um,

>> because going individually to each vendor is incredibly difficult and we wouldn't have been able to do it uh, for this cleanup without the support of uh, EPA and others as well. Um, additionally in the OT environments, there's other things that help I guess uh, maintain the uh, attention of operators longer. So for uh some different utilities there are various regulations that they're going to have to adhere to um such as the uh NERK CIP standards NERC so North American Electric Reliability Corporation. Um additionally there's uh IEC 62402 which documents how an end of life uh obsolance plan should be laid out and some of the key things that are going to be in that. Um,

yeah. Yes. H my slides don't advance at the same time. That always trips me up. So, uh, I wanted to kind of close on one one other thing with OT. And unfortunately, this slide has been a fun bit for me. So, as I've like followed a lot of the OoT threats over the years beyond just the work I'm doing now. Um, I did research on one of the precursor variants to Stuckset. I was heavily involved in a lot of the research around Sandworm when they were going after controllers in Ukraine. Um, a lot of the times the threats that we see daytoday to a lot of OT environments aren't necessarily uh the sort of CVE like

remote exploits that you typically would think of. Unfortunately, that statement was turned on its head recently with a vulnerability that came out in uh how train coupling protocols work and >> such a cool vulnerability. Uh, but often vulnerabilities that I'm seeing regularly for OT environments typically fall into two categories. either it's vulnerable by design which which would be such a um like HMIs that you connect to the internet you have full administrative control there's no passwords that's poor design standards um alternatively things like misconfigurations maybe a device wasn't meant to be connected to the internet but somebody decided it would be great to access it while on vacation uh but more often uh tends to be the vulnerable

through supporting tools uh so groups like the Iranian one mentioned in the previous talk cyber avengers they have been prolific in targeting uh open remote desktop open VNC and um even if you go on showdan they have a site where you can view like live captures from SC uh VNC and RDP screenshots that they see and you'll see things occasionally most of them will be like log into your Windows 2010 server uh but there'll still be quite a few that uh will have a SCA interface up on it uh and those are key examples of vulnerable through implementation or supporting tools uh and It's very rare that we find adversary tools that are truly designed

to operate against OT systems or things that would speak OT protocols like Modbus and things like that. These are rare. Um, so when we see them, usually they're incredibly fun to look at as well. So, all right. So, we promised you solutions. Um, we [laughter] these are these are not it's security. We're going to get like part of the way there, y'all. And before I start with solutions, I want to caution folks because on the consumer side, I hear a lot of this. They're like, just shut off the devices when it's end of life. Kill it. You cannot do that in consumer homes. You cannot do it for so many reasons. One, there's a lot of really

old people who spend a lot of money on this device. And maybe they rely on it for something important like, I don't know, detecting fires in their homes. And many of these devices, they still will do that even if they're vulnerable or being used by malicious actors. So you can't shut those off. It sucks. That would be the best like easy. Well, it wouldn't be the best. It would be the easiest. It would be terrible from an e-waste, from a consumer rights. We're very against it. So just that is not an option if you think it might be. Don't do it. >> Bad for the planet as well. >> Yes. >> Everyone hates that. >> Yeah.

>> Except that it's easy. So might we we do see people suggest it. Yeah. >> Um, what we have suggested and that Paul foreshadowed for you is Paul's organization, Consumer Reports, CDT and Per, we actually are putting forth model legislation at the state level. It could go federal, but that's going to not really happen. Um, advocating for a lot of the policies we talked about. So, disclosure of how long something will be supported, proactive notification. We even have a line in there about your ISP. ISP equipment, those routers and set top boxes. Set top boxes are incredibly powerful. They're not managed well and they are end of life in a lot of ways and or not a lot of ways,

they're just end of life and they are used in a lot of ways in attacks. So having your it requires your ISPs to maintain this. And I'm not talking about Comcast, they're actually pretty good about it. I'm talking about like the wisp in Iowa who's like oh and the what? Okay. Um there's another thing on the consumer front the US cyber trust mark which you guys may have heard about. It might be coming. I hope it's coming. It's probably not going to be coming when we anticipate it. I'm part of the organization that's setting those standards uh or the committee that's setting those standards. It's really great. that will have um a minimum supported support minimum supported time

frame on the label behind a QR code. So theoretically in the future if you buy a connected device that has the US cyber trust mark you can scan a code and you can be like oh my god it's supported for five years. It is still up in the air whether or not the FCC will say hey if you support it for zero years that's sufficient to get the mark. If that's the case, pay attention to me because I will totally rail against it and you could all submit comments. Please do. Okay. And then we could provide incentives for organization to build non-cloud dependent devices to combat e-waste. I'm just going to Okay, I'm sorry. I'm I'm going to pitch

Consumer Reports just a little bit more. Uh [clears throat] we are actually going to put out research about longevity by design. So it's like security by design. It's going to t tackle a lot of these things for connected consumer devices and it covers cloud legal models. It's very exciting. Anyway, look for that. And those are our policy solutions. And here's some practical solutions. Oh, I have one more. Hey, Oasis, which does enterprise software. >> Not the group, >> not not not Liam, >> not the Gallaghers. >> Uh, [laughter] no. Um, I was trying to make a firewall joke. I was like, it's not going to work. Don't do it. Um, they have proposed standardizing language around

end of life. And this is really important because some people call it end of support, but what the heck does that mean? Like, I'm only going to support it for six months. I don't know. Um, so we can't have that level of confusion at the consumer level especially because they're very unsophisticated. But even at the enterprise level, it is confusing when you're like the end of life, end of support, end of service, end of maybe I'll provide security updates if I feel like it. It's terrible. All right. And then, oh yeah, this is my one. So, uh, one of the things that like regularly folks have talked about is, uh, actually it was talked about in the last talk as

well, uh, where cyber incidents are treated as like weather phenomenon where it's an outrage that can't be avoided. Um so one of the things for organizations which can uh devise a path for offline operation if there is the likelihood and knowing that ransomware attacks do have the ability to disrupt businesses practice it have a plan in place for if you were to have a situation which your internet connectivity went offline could you maintain operations still um and this goes into things like I love the crash card idea for for hospitals because that's such a cool idea and especially if it could be done in a situation when communications are broken But care still needs to be delivered.

>> Uh okay. Yes. End of life plan. Routinely audit. Practice your fire drills. And finally, avoid hardware on the US sanctions list or the covered list as they call it. And maybe if something is talked about being on the covered list, but hasn't quite made it because it's quite the process, maybe reconsider sticking that in like a really important place. >> Yeah. And even in some cases, we may not know what's coming on the sanctions list, but like when you hear politicians talk about TPLink routers all the time, like maybe consider that when when making purchasing decisions. >> I will say that I've Yes. Okay. [laughter] >> And also start looking at things like can this device that I'm buying run

software other than the vendor software, right? It's a really especially with something like a Soho router. Really important question. Um, I can tell you for an example, and this is based on a little bit of work I did with Reversing Labs, that um, you know, if you even if you look at OpenVPN and the the free version of OpenVPN versus the premium version of OpenVPN and analyze the security aspects of those two versions of basically the same software, but one's premium, one's free. Um, you'd be really interested to find which one's actually more secure. >> [laughter] >> All right. So, hit up the mics. It's time for questions. We're going to zoom zoom zoom through it.

>> Okay. Uh, quick hits. Um, the end of life that could lead to end of life. Sometimes you have an OT IC piece of equipment from like a Schneider Electric. I'm saying that because they're one of the ones putting an effort in on their newer models. Um, but the owner and operator continues to use a long dead unsupported end of life much less secure by design or default. So there's a, you know, could you try to describe the shared responsibility between putting out defensible, maintainable, higher quality things and availing yourselves of them? And is there some way in the context of this vault typhoon ticking clock to maybe have a program to drain the swamp or

cash for clunkers or something like that? And then the second point, if you can weave it in, we started making a list of bad edge devices or dangerous red edge devices like TPLink or those with frequent known exploited vulnerabilities. We had a harder time finding a list of if there's a naughty list, what's the nice list? So, is anyone like Consumer Reports or or Census like looking at the least bad options? Because if we could tell people stop using something, we should probably hint to them for the current available pool, not the future available pool of what to use instead. Thanks. >> It's like the UL model, >> right? >> So, Consumer Reports does have Okay,

this isn't exactly end of life. Um, but we're we're building up a cache of research. Um, what I have looked at recently is the top 75 smart home brands. I have actually looked and researched whether or not they have a dedicated security contact actually most of them do and the few who didn't the year before we when we started it have actually added one so that's one and whether or not they have a VDP a vulnerability disclosure program so not end of life but still an indication that they are secure and are developing mature cyber security policies I don't know what else you asked >> oh that's back to that herd mentality I don't know how to communicate that to

people so I'm pass on that. >> I mean the notification work we do is small and we're not going to reach everyone. So um part of it is recommending better products as well. As he said >> there is there actually is interest in Washington. It's a like the FCC manages a program for getting out Huawei gear out of small wisps and ISPs. We've actually talked about it in the health care space because there's lots of hospitals running ancient stuff that should not be running. It is an easy thing to get people excited about because they're like, "Yeah, yeah, this is a really important." It is very hard to get that funded. I don't know how we do it. I'm

that's why I think the resiliency and the practice kind of elements come into play here because and >> I mean one of the one of the things we're missing here is a government that a federal government that actually takes and I mean you know we have in the last 40 years as all of this infrastructure has grown up has been a period that in the history of this country is going to be remembered as one of intense deregulation and high degree of skepticism. about the role of government in everything. um the the example you mentioned about you know managing risks I mean you can easily imagine you know looking across the country at the critical infrastructure the providers

and saying you know let's let's figure out what are the key providers and and assets and resources and really direct a lot of resources towards securing those and understanding how attacks may play out. Is that going to extend to every suburban home in Soho router? Absolutely not. But if the focus is on coordinating that response and preventing huge disruptions, that's very doable. But we're living in a system where, you know, the lobbyists descend on DC, the EF emphasis to get that done. It's okay, you know, and nothing and nothing happens. and and we all kind of bear the bear the results of that. Hospitals going down, uh St. Paul, Minnesota getting hacked, um you know,

you you name it. So, I mean, I it is I I do think we need to re refframe our thinking about what a solution looks like. Yeah. >> Okay. 30 second question. 30 second. >> Yeah. Okay. So uh you talked about life cycle uh u time time frames from manufacturers that will actually come with uh EU cyber zillsect where the vendor must specify for how long they will support and that will likely spill into the US and so it will go in the direction of the US cyber trust. Is that correct? >> Yes. It actually takes 80% of uh 306 uh 303645 um the European it's great actually if you are a consumer buying connected

products any product certified with the CE mark for the EU after August 1st oh oh as of today anything coming out as of after whenever >> that's going to be really good it will have incredible cyber not incredible really decent cyber security next question >> yes >> hi so to make make a pitch for the manufacturers without any clothes. I'm at Seaman's Energy. We're highly economically incented for customers to buy new control systems. And we have a several thousand person sales force whose only job is to sell new control systems. And yet, when you look at the fraction of customers that buy a new control system for a billion-dollar power plant that decide not to buy the

updates for that control system, >> right? >> It's well over half. >> That's right. Right. >> So, you know, I see the you smart people up there in the policy space saying, "Oh, the manufacturers need to do stuff." >> Yeah. >> What do you want? What more do you want us to do? >> Yeah. No, I I agree. I mean and the and the policies aren't should will not necessarily all be focused towards the manufacturers or the vendors. They will also right the point is to set up guard rails and and so right so maybe they have the right to not do that but that creates more liability for them if some adverse incident happens and that

changes their calculations about whether to invest in that update. Right? So I will say uh I had two slides just on seammens as defining several different things because end of life procedures first published thing I could find like they're easy to find easily available >> and and profitable but today >> yeah I think we've pretty much gotten there but enforcing economic incentives on people. So, you're talking about requiring them to There we go. A little higher. Uh requiring them to patch for a certain amount of time. Where do we get the teeth? You're saying the government, you know, we've got deregulation, right? Are these going to do any good if there's no way to go and hold them accountable?

>> I'm inspired by the I'm a member of the I'm on the board of the repair coalition and I've spent the last five or six years in deeply involved in the right to repair movement. So that had a bunch of very thinly funded nonprofits going up against trillion dollar corporations like Apple, Microsoft, Google, Samsung, and we won. We have about six states that have passed comprehensive electronics right to repair. Some of them ban parts pairing uh and so on. And so that's my model. And so yeah, I'd say DC is a tough environment right now, but the states um are actually and in fact Texas just passed a pretty decent electronics right to repair bill. Um th

this is a bipartisan issue has bipartisan support. Um really speaks to the needs of small businesses and consumers. Um and so I I feel a lot more optimistic about that and and I I do feel like these are issues that that state legislators might be willing to take up >> and all of them depend on the attorneys general. >> So the next stage is actually enforcement >> in enforcement. Yes. Right. And that's where once these get passed, help them get passed and then start tracking who the heck isn't doing it. And then CR will do the job of like organizing and taking things to the AG's office. >> Yeah. >> And now we're stopping. [laughter] Now I

love you. I'm sorry. >> All right. Join me in giving a hand for our fabulous presenters. [applause]