Spies, Saboteurs & Scoundrels: How Russia, China & Nefarious Actors Are Hacking IoT

Augusta are you ready to rock all right you wanted the best they got stuck in Florida so you got me so uh so hopefully you're in the right place by saboteurs and Scoundrels again my name is Brian contest a little background I've been in cyber security way too long about 25 years I've been building companies most of my career I started with disa and then later Bell Labs I lived in Brazil for a couple years but then just started building startups rip Tech Arc site imperva solar networks verden a couple IPOs couple Acquisitions I wrote a couple books my last book was with the director of the NSA and then I recently did a documentary for HBO and cyber warfare

with General Michael Hayden so I've been doing this for a little while so I've learned a lot of ways of not to do things certainly lots of great Lessons Learned but this presentation is really about X iot or what we sometimes call extended iot now X iot is really three General categories the first one I think most of you are familiar with it's Enterprise iot or Enterprise Internet of Things printers KVM switches lights out management HVAC systems security cameras UPS racks things of this nature the other portion are OT and there's been a couple talks in OT today but scada devices industrial control systems plcs that genre things that are digital equipment that controls physics so

temperature flow mixtures volume things of this nature the third one are network devices these are specific to switches and routers network attached storage load balancers wireless access points and so on so collectively these groups have a few things in common the first one is their purpose-built hardware and firmware so they might be just as powerful as a laptop or in some cases even more powerful but they have a very finite use case the other portion is they're Network connected there's some OT devices that are actually not Network connected and there's some iots that actually aren't as well but in this conversation we're talking about things that are network connected it's usually TCP but it could

be modbus dnp3 other Pro other protocols Etc and the last thing is and probably most important for today's conversation is you can't put any kind of endpoint Security on these devices so while they're running Linux or Android or BSD or VX works or other relatively common operating systems or real-time operating systems and they have all the same power and storage and capabilities and ports and protocols as those systems you can't put any Security on them there's no anti-malware there's no host based IDs or IPS most of them don't have runtime protection they're just there and they're just vulnerable so that's kind of a perfect storm for attackers hence this whole presentation so I think most of you are

probably familiar with Showdown if you're not it's just like doing a Google search to show me what kind of devices are internet accessible within certain categories so I did a very scientific study and I typed in words like camera UPS voice over IP just to see what might come up in these but look at cameras there's almost five million cameras and there's probably a percentage of these that are honey pots but even if it's five percent which is probably less than that but look at UPS's uninterrupted power supplies what would be the use case of having your UPS internet accessible probably not something that was done on purpose a quick question sure the audience here what do you think

is the most common UPS vendor out there I'm sure you've heard the name before but APC so I did some more scientific research I said I wonder how hard it would be to find out what the default username and password is to a APC UPS does anybody want a hazard to guess what the default username and password is not even that hard so using this hacker tool called Google I said default password APC UPS and it's APC APC in lowercase now just to give you some feedback so my company phosphorus we focus on X iot stuff we've been doing this for years and years we have a bet and that's we if we ever find an APC UPS system that does not

have the default password APC with the default username APC will buy everybody in the office a steak dinner we've been eating a lot of chicken nobody ever changes these so you've got about 13 000 plus devices that are internet accessible which are probably pretty important and they probably have important things plugged into them and there's about a 99.99 chance the username and passwords APC APC so use that however you like uh so here are some stories about these devices has anybody heard of a Russian X iot hacking 2 called fronten okay this is kind of a unique tool so the Russian FSB had a Consulting Group build an X iot hacking tool for them specifically to do

the following find xiot devices like we just did with Showdown compromise those devices control those devices use those devices then to attack it Assets in those environments and then exfiltrate sensitive data It's actually an extremely extremely powerful tool unfortunately for the Russian FSB it got stolen by the digital Revolution hacking group that said this is way too powerful for one nation to have so if you go to your favorite Torrance and other locations where you like to download malware and hacking tools and if you can read Russian this is a really really powerful tool that you can leverage today and by the way it's not just Russia that builds tools like this everyone's building tools like this now

because hacking X iot is the the new new and we'll talk about why that is China takes a little bit of a different approach to this China says well that seems like a lot of work to hack something after the fact why don't we just ship it with the malware already installed we'll just skip the middleman it's a huge Time Saver it's very efficient It's very effective so the U.S House of Representatives actually passed a bill Banning certain vendors things like ZTE Huawei hickvision which is one of my favorite that's a camera up there it actually ships with the malware already installed um it's it's pretty interesting how blatant is as well so essentially when

the camera is recording audio and video there's a light that's green and if you say hey camera stop recording audio and video it says okay I'm going to turn that green light to red but guess what it's still recording audio and video and it's still streaming that information back to a location that eventually makes its way back to China where they're doing temporal analysis volumetric analysis pattern Discovery anomaly detection to correlate massive amounts of data so you can think of some of these cameras are probably in very sensitive areas Financial Services critical infrastructure boardrooms things like this so that's the hick Vision it's not to slam this particular company I'm sure they have some products

that are very good that aren't Laden with malware but they happen to be so high profile that they are banned by the US in government use and with government contractors yet we see them all the time and fun fact the most common camera on Amazon the last few years with the highest number of reviews came pre-loaded with malware as well so this is a very common occurrence for security cameras now this Hikvision camera something else this happened very recently so August 23rd this year it was announced that there was over 80 000 exploitable Hikvision cameras and that number actually I think is way lower probably by an order of magnitude but a patch came out for this exploit

back in September 2021 it doesn't surprise me that it wasn't applied though because nobody ever patches these devices when these devices get deployed they're pretty much the same way as day one as they are you know year 20 when they get ripped out of the office building so no one's fixing these so it's kind of a sad State of Affair for hickvision that they ship with malware and if that's not good enough for you here's a bunch of exploits you can use against it as well it's not very fair but I guess life's not fair it's not fair that carbs make you fat but I'm Italian I love pasta and there you go so who's heard of Mirai the Mariah

botnet quite a few of you this is like the poster child for xiot attacks this is way back in 2016 which in the xiot world is ancient history but this is pretty interesting so the attackers started off by searching for cameras just like we did with Showdown further they refine the search to say hey is tone net running so Port 23. okay I found a camera it's got telnet open third thing does it have the default password well most of them do and if it doesn't let's try one of about eight or ten different password combinations and sure enough that gave this botnet enough power to take down Sony Reddit Netflix Twitter PayPal GitHub major telecoms

there's just a few security cameras had more aggregate processing power and bandwidth than Google and Amazon combined so it just goes to show you how powerful some of these botnets are but the plot thickens in the xiot world there's a lot of shared libraries and there's a lot of white labeling which means I have a security camera and it's got this build but hey this guy over here is building a printer and he doesn't want to use all his original code so he's going to grab the shared Library this bit of white labeling this and that so now this printer has the exact same vulnerabilities it's running telnet it's running default passwords as the camera

and so is this AV Equipment and so is this Voiceover IP phone so now you're seeing that this spreads to multiple X iot devices and this is indicative of this space all the time because a lot of these companies aren't tech companies that build this equipment some of them are some of them aren't HP they build printers they're a technical technical company there's agricultural companies there's companies that focus on Healthcare and other things this isn't their daytime job so they bring in contractors Consultants that don't have any type of security development life cycle they've never heard of a sdlc they just want to make the Blinky lights blink make everything work and then they move on to the next job

so we go into organizations today and again this is back in 2016. that their devices are still vulnerable to Mirai in fact we still see organizations today that their devices have already been infected by Mirai and they're still being controlled but as long as they're printing or the Voiceover IP phones working or the security camera is doing its thing no one's paying a lot of attention now this was kind of the first generation of X iot attacks which were opportunistic let me just add you to my botna and then with that botnet I'll do black hat search engine optimization phishing attacks malware distribution DDOS so on and so forth but this has evolved the next generation is lateral movement

and this is the big one so when I think of the the whole world of xit attacks it's really three areas it's that opportunistic botnet stuff it's using xiot to attack it assets which we're going to talk about here and it's that physical world stuff we talked about like spying unlocking doors shutting down the power things of this nature those are the three big categories but this one's called quiet exit and this was released by mandian earlier this year so the way this worked is the attackers got in through traditional fishing techniques on it assets like a laptop through social media through messaging through email I'm going to go ahead and get some malware on your

laptop but I know you've got network security and application security and endpoint security encryption and all these things you're spending all this money on so I don't want to stay there I want to quickly look for xiot devices that you're not paying attention to and in this particular case they were looking for network attached storage security cameras load balancers wireless access points voice over IP phones things like this again on the network side it's all BSD and on the iot side that they were looking at it was all pretty much Linux and a little bit of Android so they weren't like crazy unknown operating systems so they got access to these devices and once they

were there they installed uh drop bear which is just a reverse SSH tunnel so once that was recompiled to be loaded on these network devices or these iot devices they could control them remotely so I got in through it I pivoted to xiot and now I can control that xiot device once there they were making AP API calls to local Exchange Server and Office 365 in the cloud they started extracting emails email attachments all the messages going back and forth especially for executives security teams M A groups so on and so forth the reason they moved here was because they could evade detection and they could avoid any type of sort of security tools that might have been in

place had they looked at an I.T perspective so I'm going to maintain that persistence I'm going to evade a detection I'm going to continue to make these calls and exfiltrate data and almost every case they were there for over 18 months over 18 months of making API calls and grabbing every single organizational email and they did it from some X iot devices and the fun part is why compromise One X iot device when you could compromise thousands it gives you a much better way to maintain that persistence in your network so here's a recent takedown this was a Russian botnet called our socks and what was really interesting about this in fact the only really interesting thing

about it it was such a successful botnet actually it was so uh well constructed that they were renting it out for about thirty dollars a day so people could use it for DDOS attacks for a hundred dollars a day you could rent it out and you had full 24x7 technical support as well so they were running running this like a real company but what I think is interesting about this one is the primary target where industrial Control Systems which were mostly real-time operating systems now industrial Control Systems sometimes are Linux sometimes there's windows but usually VX works and other things of this nature but they didn't do it so they could blow something up or you know

change voltage on something or screw up some mixture in a batch manufacturing plant they just did it to add it to a bot and there were some network devices there were some iot devices but it was mostly industrial control systems but it was very very powerful that takedown as you know most takedowns the way they work think of it like DNS you know you've got all these subsections reporting finally up to a Master if you can take that out you can kind of take the botnet down at least for a few weeks until they rebuilt someplace else Siemens had an attack that came out pretty recently with a horrible name the S7 plus crash attack uh this kind of

tells you how easy some of these attacks are on the industrial control system side so essentially think of a device like this that controls temperature and in in the OT world we call these set points so the temperature can go as slow as 70 degrees Fahrenheit and as high as 80 degrees Fahrenheit if it goes lower or higher I'm going to send an alarm to escape device it's going to alert me something bad has happened okay so it's there so let's say I log in and I change it and I say hey I want to increase the temperature now to 120 degrees which should be way past that point But as soon as I do that I start sending

packets and I can use something as simple as netcat to send a packet on TCP to Port 102 by simply sending packets on Port 102 to the Siemens device it does a Dos attack it can't communicate it says oh something bad's happening the temperature is going way over the point it should be going on my set point but I can't tell anybody because somebody said me a packet on Port 102 and I'm still trying to figure out what that thing means because I wasn't designed to understand that so it's important about this particular attack was Siemens came out and says look guys we have some great security fixes for you the first thing is patch your system

so we think you should update your firmware the second one is enable password control a lot of these devices don't even come with password control enabled that's a checkbox third thing once you do that set a password and then the fourth thing is turn off Services you don't need like clear text clear text protocols Bluetooth low low energy things of this nature these are pretty basic things right these aren't things that in the IT world that we think of as very groundbreaking in fact I like to say this that xiot security today is like I.T security was in in 1995 and I made this slide before Coolio we all know he recently died so rest in peace Coolio but in 1995 that's

about when I got started working in I.T Security in my very first project this was at disa I had to go around and find all these U.S robotics modems that were plugged in and nobody was tracking because that was the great evil of the world at that point hackers were going to get in through dial-up modems and take over the world and launch nuclear weapons and all that stuff so I downloaded this tool that ran on dos called Tone Loc written by Minor Threat and mucho Moss all it was was a war dialer and the PBX guys hated me on all these bases because it would ring every single number so Queen sequentially within the entire base looking for the

response of a dial-up modem and then once I found that modem I had to figure out to have an owner if it doesn't I put it in a bucket I had many many buckets filled with these U.S robotics modems the next thing I got into in that time frame was there was this tool called Satan I don't know if any of you ever used it but it was one of the first vulnerability scanners it was before iss's freeware version I think it stood for Security administrator tool for analyzing networks I had it loaded on my son pizza box I think it was Solaris 2.51 is what I was running and I was using this for vulnerability scanning

and somebody said hey we want to install this new thing called the firewall what's a firewall nobody had really talked about it too much at that point it was all ACLS so the device that I was doing my vulnerability scanning on we actually had set up as a firewall because I was the only person that had a workstation that had two Nicks so we could have internet facing and internal facing so it just goes to show you how basic things were back then so password management discovery of your devices firmware and Patch management these things were in the early early days and it's the same way for X iot today for those of you who are just starting to

get into cyber security take a serious look at this being the new new because this will probably carry your career for at least the next two decades this is what everybody is talking about this is what all the buzz is now on the dark net this is where a lot of nation state Investments are going right now cyber criminals are going after this in a big way if you're new to this space or you have more time left in the space consider this as an area of focus so we have a lot of interesting research stats that we've discovered over the last five years looking into X iot we've looked at millions and millions of

devices across all geographies uh private sector public sector so on and so forth so here's some interesting stats what we've discovered is there's roughly three to five x iot devices per employee for every company so a company of 10 000 people has roughly between 30 to 50 000 X iot devices that's a lot it's a lot more than you'd think and in fact it's so much more than you'd think that on average when we go into a company and I say hey guys guess how many devices you think you have if they tell me 50 000 I know in the back of my head okay they got about a hundred thousand because they're always off by

40 to 60 percent because oh I forgot about all the door locks oh I forgot we had that many security cameras Oh I forgot about KVM switches and lights out management we forget about all these things because no one's ever managed them because there was never a way to do it before so they're just sitting there vulnerable open exposed now there's a bit of a bell curve to this law firms have a little bit less than three to five uh retail has quite a bit more in industrial manufacturing both batch and discrete have a lot lot more so but on average again three to five devices per employee um so audience participation what percentage of X iot devices do you think

operate with default passwords 95 see you're also pessimistic so 50 roughly now again there's a curve to that too audio video equipment is closer to about 99. if you have like we talked about before in APC UPS system good luck right now when we talk about this also what you find out is when those passwords were changed maybe they were changed once at the time of implementation because it required a password change rotation frequency every 90 days complexity probably a four digit PIN and length again probably a four digit PIN so not really good password policy because how are you going to do it you're going to send somebody around with a paper clip to reset every single

video camera in a casino probably not right so there was no way to do it at scale so what percentage of X iot devices operate with end of life firmware okay 70. okay 26 so 26 end of life however on average six years old so imagine trying to operate that smartphone that's sitting in front of all of you right now with an operating system or any app on there that was six years old it probably just wouldn't work right but these are the devices that are controlling critical things and a lot of these devices help manage other devices UPS racks KVM lights out management so on and so forth so if this is a big

problem and with old firmware comes vulnerabilities now this one when we started researching it we had to go through the data a few more times because this is really sad 68 percent had CVSs scores so CBE ratings you know 1 to 10 10 being the worst of 8 9 and 10. almost 70 percent had eight nine or ten which means a hacker with very little to no skill in fact we would even call him a hacker we'll just say a person with a laptop that can access the device that for some reason it doesn't have the default password oh my gosh how will I get in well at this level it attack you can get

administrative access with little to no effort and little to no skill right so that's a that's a huge power if I told you this was on your it assets today you'd get up and go fix this right away this is Earth shattering but again these are Linux devices these are Android devices these are BSD devices they're being compromised they're being taken over the only difference is we're not monitoring them we're not protecting them like we used to or like we should be so who are the biggest offenders this is fun so we went through and there's actually a very large list we have that are the biggest offenders I can't spend the whole presentation talking about it so I

limited it just to a handful but um we'll start off with this one KVM so keyboard video Mouse so KVM switches suck and this is why you can have a KVM switch and I think most of you are familiar but if you're not think of a rack with a KVM switch one Mouse One keyboard one monitor plugged into a dozen two dozen three dozen servers okay it's that simple the problem is a lot of these guys are running Ubuntu version 10 which is about a decade old I guess you find something that works and you just stick with it the issue with this is that's filled with vulnerabilities but you don't need to really worry about that because it's

always a default password nobody ever changes the passwords on their KVM switches so if you can get access to the KVM switch you have direct access to the servers it's connected to without worrying about all the other pesky details so this is a really really powerful capability so if you can get access to the KVM switch you can access everything else next one help me out here lights out management controls suck and this is why they suck so lightsab Management's actually even more powerful than kbm they're usually installed into the back of your most critical devices you might have heard terms like ipmi ILO idrac these are just little Linux servers that's all they are

little Linux servers that are connected to the network that allow you to do things like turn the power on and off you can change network settings you can open up a shell you can spawn a virtual terminal you can upload software so you have very rich capabilities people always forget that these things are there and this is one of the areas that we say guess how many devices that you have that are connected that we can discover oh we forgot all about our lights out management devices is that really a problem it's a huge problem right so if you're not tracking these devices it's really opening you up okay server cabinets and racks suck and this is why just like you so

these things have uh Cable Management they have tamper management they have ups and power supply capability which means that if I'm going to reboot this device because I do a firmware update it means everything connected to that is going to be have to be power cycled as well which means nobody ever does it these things once they're installed they're their default configuration which leaves them vulnerable in everything that's within those devices vulnerable as well so this is one that people really often look over so I like to call these devices that manage other devices the KVM switches the lights out the server cabinets and racks okay physical access controllers suck so this now you guys are getting it so this

is the thing we we went into a financial services organization and said hey let's let's find out what kind of xiot devices you have oh we see you have a lot of digital door locks in fact he had 6400 of them within about I want to say 10 seconds but it was probably closer to five but we'll go ahead and say 10. we're able to show them how we could lock and unlock any doors in the building at will with no hacking required including things like the front door which is probably a pretty big deal and they said wow that's really a problem that we have all these digital door locks that no one's actually paying

attention to I believe these guys are all running Linux as well there's a very popular company called nordex security that ships with uh level 9.8 out of 10 and level 10 out of 10 vulnerabilities in their default systems and it's not to say this is a bad company it's just again to explain to you there is no security development life cycle for many of these companies they're not looking at this from a cyber perspective at all they only look at this as a network device that's there but it's doing a physical thing so we're not really doing any cyber security checking on these devices so big big vulnerabilities when it comes to door locks and this could be

CAC right this could be badge readers this could be digital keypads anything of that nature okay printers suck printers suck primarily because they're super promiscuous oh Brian what do you mean they're super promiscuous well I'll tell you they want to connect and talk to you on every single port and protocol possible because they want to be easy to use they're wired they're wired lists they've got talent open SSH HTTP https and everything else in between they want to be easy to use the problem with these guys is most of them ship with about an 80 gig hard drive the Enterprise ones by today's standard it's not really big not really small but it's pretty large

they're pretty easy to get access to passwords are usually again default and if they're not default passwords there's a lot of vulnerabilities that you can compromise once you get onto these printers here's the fun part you can start looking for other it devices on that Network to start attacking an exfiltrate data so we are working with one customer it was a major Hotel chain they had about 60 000 printers across their hotel chain they had about 2 000 of them had been compromised what was happening is they were on these printers these attackers they had their malware on there it was going out it was enumerating shares it was grabbing sensitive data off networks it was

sniffing traffic it was pulling that down onto the server and then it was exfiltrating over icmp has anybody here ever done icmp data X fill so when you do it over icmp it's low and slow so it's like I have a bag of rice and I have an empty box and I take one grain at a time and I stick it in the Box it's very monotonous but it also flies under the radar and icmp Echo request 3 is always allowed out because someone on the network Administration team opened it up because they wanted to Ping something and do a test and they forgot to turn it off nobody talked to security so it's up and open for all

times forever that happens all the time attackers know that so you see if I can get onto your device I can steal sensitive data I compress it I exfiltrate it over icmp and nobody will detect this thing so it's a great great way to exfiltrate data and these devices are always vulnerable and the biggest part of it is is there's a lot of them organizations have tons and tons and tons of printers okay voice over IP phones and video conference systems suck now here's something fun so this voice over IP phone not the one pictured here because not calling anybody out but there was a voice over IP phone that shipped with Port 22 SSH enabled completely

undocumented so much so that the vendor didn't even realize they had Port 22 enabled and running on their phone we talk about shared libraries white labeling all the way down to not even knowing what ports you're running on the device that you're selling to people we see a lot of these video conference systems as well we talked about spying before a lot of these will be in executive boardrooms they're always on even when you think they're off and they're always streaming that data back these are very very powerful exploits most of these guys are Android that operate on these systems by the way so out of all the systems we've talked about which one do you think absolutely

sucks the most what x iot device do you think is the number one worst offender okay yeah cameras you're right security cameras suck the most and this is why because when you go into organization you tell them how vulnerable their security cameras are it's kind of like the end of Spider-Man where they're all pointing at each other who's supposed to be securing this well it's facilities right no it's not us it's it's I.T no no it's a network team uh-uh it's the security no it's a third party vendor nobody wants to take responsibility why would you we were working with a casino and these numbers are staggering when you think about when you think about it

makes sense they had over 50 000 security cameras 50 000. just incredible right um all their cameras were compromised and they're all being they're all crypto jacked so they're all being used for mining crypto well they're very powerful most of these cameras were really high-end cameras much more powerful than our laptops right and they're all running Linux I think maybe busy box and some other things but mostly Linux and on these cameras they were doing crypto mining guess how they detected that they were hacked ah power bill that's correct so casinos I'm guessing have pretty high power bills to begin with their power bill increased by a thousand percent so that's an anomaly um I would love to say Splunk and some

correlation rule or some might notice that the person that managed the power bill said hey what happened right and there was actually another case at a different Casino where they had a x iot thermometer in a fish tank just a thermometer and a fish tank that was accessed by some attackers they used that to jump into the database that they keep for all their high rollers all their whales so it had personal information it had financial information had travel details it had a lot of very sensitive information extremely high net worth individuals all from a saltwater fish tank thermometer right so on the security cameras again the interesting thing is they can be used for those

physical attacks to spy on you they can be used to add to a botnet they can be used for crypto jacking and things of this nature and because of their the fact that there's so many of these devices and they're so vulnerable and you saw in that early Showdown search how many of those are internet accessible they're a massive Target now again you don't need to be internet accessible most of these attacks are getting in through traditional means through I.T than attacking xiot to maintain persistence and avoid detection that's what you have to think about these are just other I.T tools that are being used to attack the rest of your it environment so beyond we've talked a lot about

Enterprise a little bit of industrial but this maps to the military as well there's a lot of Internet of Battlefield things on soldiers on aircraft on terrestrial vehicles this stuff is everywhere there's Health Care specific devices as well smart ships smart buildings smart cities all kind of fall into the same category which is interesting if you think about it I think the first the first unofficial X iot device was a Coke machine a Coke soda machine it was put on arpanet like 1988 right and they tracked how many cans of coke were still in there pretty basic just a fun little research project the first official iot device was a announced at interop and I

think it was 1995. and it was a TCP toaster right same thing so in 1995 we had a tcpip ex-iot toaster today in 2022 we have complete smart cities Dublin Ireland is the world's largest interconnected smart city in the world that is being used kind of as the beacon for all other cities that want to be smart Cities traffic emergency services water and sanitation how people are moving Health Care everything is interconnected and this is the way the model is moving after this I'm actually flying to London for a week and then Dubai for a week and about 50 of the time I'm there it's about how they're moving to a Smart City design so everybody's embracing this and think

about that with all the stats and figures and crazy stuff that we just talked about before stuff starts to get pretty real really fast right because we didn't have to really deal with this in 1995 with it but we're certainly dealing with this now with x iot so I don't like to give these talks and just talk about all the kind of bad and just leave it like that there's there's ways to address these things and I don't want to pitch any products but I do want to talk about a new category of solutions and at the end I'll share some various vendors out there that you guys can check out a lot of them have free

versions that you can kick the tires on these are called Enterprise X iot security platforms something completely new it's not a Sim it's not a firewall it's not a vulnerability scanner it's a completely new genre of products and some of these companies you may have heard of some of you may have not but I'd like to give give folks some Solutions on the other side of these so the first big issue with xiot is Discovery as we talked about before when we go into the organization we say how many devices do you think you have people are usually off by 40 to 60 percent they simply don't know what they have and again just like it was back in 1995

people didn't really have a good idea what devices they had in their Network and there's some issues if you look at some of the traditional ways of doing Discovery think about a vulnerability scanner a tenable a qualis a rapid seven they're all predicated a 1990s vulnerability scanner technology which works by setting malform packets to devices and seeing how those devices respond do any of you here work in industrial control system OT kind of scada worlds so a few of you you're never going to let somebody go into your environment start running a vulnerability scanner because a lot of your devices were built in a vacuum and your tcpip stack will say look I don't know what that is so

I'm just going to roll over and die it will impact availability in the world of OT it's all in CIA confidentiality Integrity availability it's all about availability a little bit of Integrity but almost all availability so you cannot use those traditional means and for even Enterprise iot devices some printers and things like that it can knock those over as well so you don't want to use that well if I can't scan what's my other option well maybe I can sniff okay well sniffing's not bad it's hard to scale throughout an entire environment because you need a lot of TAPS and span ports and things like that it can be very very ineffective from that approach the other

problem is most of these devices do use encrypted communication so while you can do Discovery you might not be able to glean the metadata necessary in order to make an empirical decision of what this device actually is I might be able to say something like I'm 80 sure you're a printer or I'm 50 sure you're a voice over IP phone but heck you could be an MRI machine I don't know so that doesn't really work as well so then you've got sort of asset Discovery tools that work by looking at Mac addresses so the oui the organizational unique identifier or the first six octets of it anyhow and they basically say oh you're jet direct

so you must be a printer right not really jet direct could actually be tagged to things that aren't printers as well so there's a whole slew of reasons why old school approaches to Discovery simply doesn't work for xiot it can either be dangerous or it can be inconclusive but either way it's not a very good combination so the way these Enterprise iot security platforms and again I'll share a couple of these uh players at the end of the presentation the way they work is they actually do interrogation and what's that mean so think of C-3PO from Star Wars he could speak like a million languages one of them was like water evaporators and things like that you

have to be able to speak the language of the devices that they were designed to speak so they've all built these abstraction layers to say oh you're a printer I'm going to talk the way you expect to be spoken to oh you're a PLC device you're a network attached storage device this device expects some type of specialized Windows client this device is SSH this device is telling that this is this you kind of get the picture there's thousands and thousands and thousands and thousands of ways to do this it's nuts but in doing that you're able to actually interrogate these devices and find out what are you what firmware what version what model number what are your

passwords what are your ports and protocols what's the status of your certificates things of this nature right so you can actually interrogate these devices and identify what it is empirically if you can do that you can move on to the next stage which is much more important but at least now you know empirically what you have in your organization and you can reinterrogate your environment as often as you need to to kind of build up this list is it going to be 100 probably not but it's certainly better than nothing and it's probably close to about 90 95 percent in most cases so it's pretty darn good and again the big thing about interrogation is it's safe and it's

scalable right I've seen this being used in military I've seen this used in manufacturing Healthcare financial services so on so know what you've got kind of Step One now that you know what you've got you can go to step two which is remediation which is really the big thing because it's one thing to say hey look I found a hundred thousand devices and by the way they've all got default passwords and they're all vulnerable and they're all running clear text protocols so have a good weekend right it's nice to say okay let's go to step two so one of the things you want to do in step two is actually fix the firmware and Harden so this has to do with

upgrading these devices so a lot of these devices as we talked about before they're really old they're end of life firmware in some cases the companies have got out of business in some cases they're you know six years plus it's not uncommon to find these devices that have um firmware that's over a decade old especially in the operating or on the OT side where you're looking at things like I have a device that's a PLC that's running a turbine and I depreciate it over decades like I do this turbine so it used to be if I can't fix this how would I fix X iot devices in the past I'd VLAN them I'd spend a lot of money

investing in my switching architecture to put all these devices behind vlans which is very expensive for even a mid-sized company we could be talking about millions and millions of dollars in fact I was working with a company in Atlanta yesterday where I think they were quoted like five million dollars for just vlaning technology which who's going to write the check for that nobody it's ridiculous and I'm not against vlans I don't think vlans are bad I just think it's a very expensive way to try to fix something more importantly it doesn't fix it the analogy I like to use is let's say I'm working on something I cut my left hand open and I should

really go to the doctor but instead of that I'm going to stick a plastic sandwich bag on my hand and I'm going to wrap it up with duct tape so now I'm not getting blood on my keyboard or my right hand but I still have a bloody hand in a bag which is probably a bad idea right well that's what a VLAN is it says oh I'm protecting everything else but just keep this bloody hand over here so you still have vulnerable devices so if you can upgrade the firmware if you can patch it I know such a novel idea that takes you a long ways well the problem was historically Discovery wasn't accurate enough to tell you

empirically that I know exactly what this device is so because of that now I can upgrade all your printers or all your voice over IP phones or all your KVM switch is so on and say forth but now you can now you can actually push those and this is really cool actually you might find out that you've got let's take a security camera it has a log for J vulnerability I think most of you have heard a log4j I think it was it was the big news about six months ago well these X iot vendors don't move at lightning speed when it comes to creating new firmware so they go look we know we have

a log for J vulnerability and we're gonna have a patch for this in a year so we're going to get it done really quick for you but you say look I'm on version five you're not gonna have version six for a year maybe I want to downgrade to version four that didn't have that until that happens to sort of be a compensating control so now you can actually even downgrade these devices and during that initial Discovery process for these Enterprise X iot Security Solutions they can actually tell you if you're on version five and you want to go to version seven you have to go five then six to seven as your upgrade path or you can go right from

five to seven that's a huge huge Time Saver to know that also you don't have to hunt down the firmware the biggest problem for firmware we've all done it you're at home you're like man I should probably upgrade my wireless access point or my printer so I'm going to Google this model number and I'm going to spend an hour trying to find this firmware and spend another hour trying to figure out how to update it and probably either you do it or you just give up and you decide never to do it again so we've all we all fall into that I have a printer next to me at home that I have no idea it's I'm

sure it's already hacked and everyone's spying on me but if you have the ability now to say I don't have to go find the firmware that firmware is all going to be kept in some Google Cloud cryptologically checksum that's how most of these organizations work and then I'm going to just say upgrade on my HP printers from version 10 to version 11. all my cameras from this to that all devices in Augusta today all devices in Atlanta tomorrow that sort of thing that allows you to really scale and then you're not falling into the case where you're saying hey I'm 50 sure this is a printer so let's go ahead and do the upgrade of the firmware and then

oh zoinks it was an MRI machine and people get really mad when you turn a two million dollar MRI machine into a 500 bubble Jet right so it's good to have that empirical knowledge furthermore I want to be able to harden right remember what Siemens said enable passwords add a password patch it turn off stuff you don't need well this comes up to Turning stuff you don't need shut off telnet tftp FTP HTTP just run https and SSH don't run wireless just run wired don't run Bluetooth at low energy etc etc so you can start hardening these devices the next step is credentials so now I found my device I've upgraded my device I've hardened my device across my 50 000

X iot devices now I want to manage credentials most of these devices work by integrating with tools like Pam privilege access management tools cyber R cache Corp psychotic there's commercial versions free versions whatever but they store the actual credentials problem they don't speak to X iot devices nor do they want to invest the time and effort to make their devices speak to all these X iot devices so what they've done is they said let's let the Enterprise X iot security platform speak to those X iot devices and we'll go ahead and manage the credentials on the back and through apis so as you're going through that Discovery process it's automatically enrolling them in the Pam solution then

every 30 60 90 days it's doing the password rotations on those devices again using the xiot solution as the middleware now the cool thing about this is when you're in this situation you actually have the knowledge to say well this device can only take a four digit PIN well this device can actually take 25 characters but no special characters well this printer can take everything except a backslash or a hash sign or this can't do the letter B we've seen the weirdest password configuration policies that you can ever imagine for these devices but it takes all that out and then certs almost every single wireless access point we run into has a cert that's like

TLS version 1.1 or 1.2 it's self-signed it's expired this is very common because there's so many of them it's hard to manage these things and keep track just like you manage the credentials you can manage the certs so now I found my device I've upgraded the firmware I've hardened it I've managed my credentials and now I'm managing my certs that's pretty great actually that's taken you now on par with what you've got on it so that's kind of Step One Discovery step two remediation my favorite part is this so now you've spent all this time fixing everything so you spent the last three weeks finding your X iot devices upgrading the firmware adding good passwords Etc and now you want to make

sure all that works stays good work in perpetuity and you're looking for that environmental drift so everything that's working stays working and it's a very simple process I've got 50 000 devices those devices are going to be reinterrogated every single day and oh this device was in version seven now it's in version three somebody must have walked by with a paper clip and poked a little black dot on the back of the device or this device did have a great password but now it's set back to default or I wasn't running telnet but now I am running telnet so out of 50 000 devices today I can manage by exception because here's the five devices I need to look

at right now and it actually allows me to scale and when something changes they integrate with things like slack Splunk you know titanium de misto all the traditional tools that you use for ticketing service now Etc to do that alerting and then you can report on this data as well give me a report that shows me all the devices in my environment right now that are end of life show me all the devices that I don't have managed by my Pam solution show me all devices that have level 9 and 10 cves just having that type of invisibility gives you so many steps beyond what the bad guys are hoping for they're hoping you'll be apathetic they're hoping

you'll be passive that's why Russia and other countries and cyber criminals are investing so heavy fronten wasn't developed because they thought there was some hedge use case where it might become handy frontin was developed because they knew the whole world was being caught without knowing that this was happening and I guarantee it's happening all day long always everywhere this is the new new like I said earlier if you're thinking about getting into cyber if you're new to this game this is area you really want to start focusing on so sort of sum up things by the way here's a bunch of companies a phosphorus wear one you can certainly talk to us but order zingbox nozomi

armis Etc there's a few others as well but these are all organizations that play in this xiot world so some of them just do Discovery some of them do Discovery and Remediation some of them Focus mostly on vlaning whatever I don't even want to say one solution's better than the other but they're definitely things that you should start checking out and for a lot of you these might be very new names to you right they're not as familiar as Splunk and tenable and things like that so get to know these get to know these types of companies because organizations simply don't know what they have because they don't know what they have they don't know what to fix and if they knew

how to fix them they can't fix them at scale and if they were able to fix them at scale they're not able to monitor them to make sure they stay fixed so that's kind of the net net of these so if anybody wants to get in touch with me here's my here's all my details but I think we have a couple minutes do we have a couple minutes for Q a so I'll go ahead and take a couple questions and we have some gifts to give away too so any questions out there uh yes sir

yeah so there's actually a lot of great sort of iot security conferences X iot is a bit of a loaded term because it's this umbrella thing but check out some of the iot shows there's some of these offered through black hat and Defcon I'm doing one in London called The iot Science Foundation one next week so there's a lot of these things coming up there's a lot of reading on iot hacking so these are all things that I think you could take a look at to sort of build up your skill set yep great question yes sir more popular

yeah there probably are some that are best so what are the things we do for fun in our Leisure Hours is we go to swap meets and we go on uh eBay and places like that and we buy a whole bunch of this gear and I live in San Francisco and I don't know why but our swap meets are filled with like healthcare X iot devices especially like IVs mobile IV systems and when we pull those in and we look at what's on those devices oftentimes you'll find network maps of organizations passwords all sorts of sensitive details that came out of like a healthcare provider things like that so some of that's being reused and

resold through Ali also some of those devices that you're getting again you get what you pay for and some of these devices do come with malware pre-shipped so let me ask a couple questions to give away the gifts so first we have this book go hack yourself who wants to tell me what xiot stands for I only have one book so someone raised their hand okay right there yes you got it and that will be for you when you're ready and let's see what was the name and raise your hand what was the name of the attack that mandian discovered that gotten through the fishing attack on it and then pivoted to ex-iot I think you were the first one go ahead

quiet exit and you get this uh wireless adapter deal awesome all right everybody thanks so much for your time I really appreciate it