Talk 1 - Cameras, CACs & Clocks: Enterprise IoT Security Sucks - A Story of Two Million Interrogated

hey everybody i'm brian cantos i'm the chief security officer with phosphorus cyber security today's presentation is cameras cachs and clocks enterprise iot security sucks uh this is a story of two million interrogated iot devices this slide i just said yikes i didn't really know what else to call it so i went over to showdown and i think most of you are familiar with that the showdown search engine to see which devices are um internet accessible if you will uh and i just typed in a couple terms that you would think of for iot devices i've got camera voice over ip uh printer and ups so not very scientific just typed them in but what we find here is almost 5 million cameras

you know over 250 000 voice over ip phones uh you know 83 000 plus printers and almost 14 000 ups systems which oh my gosh and the thing about ups systems i won't tell you but you can google this the password is almost never changed and it's a very very simple password on these it's actually the same credential for username and password on these devices i don't know why they would be exposed on the internet but iot devices don't need to be exposed to the internet for these attacks to happen but certainly the fact that many of them are increased the likelihood also we're seeing a lot of nation states now paying attention to iot and

investing quite a bit and this is globally so russia has this tool called frontend which was developed for them by some contractors specifically for the russian fsb and this was an iot hacking tool designed to find compromise and compromise iot devices install command and control software and do whatever they wanted from that point or also use that as a jumping off point to get in deeper into organizations interestingly a hacking group called digital revolution got their hands on frontend and released the code so you can get it from torrents and all the all the places that you like to get your hacking tools from another interesting case is uh there have been iot devices that have been

banned actually the us house of representative has passed bills that prohibit federal agencies and contractors from actually using certain iot devices so i mean they're that well known to have that level of vulnerability that they're they're just not allowed our research what we've actually discovered and this these numbers actually blew me away before i really started getting involved with these organizations there's about three to five iot devices per employee so this is the the craziest looking uh possible chart to show that but if you just look at ten thousand employees for example ten thousand employees you probably have somewhere between thirty thousand and fifty thousand iot devices that's a lot of devices and it's way more devices than

you could possibly manually manage in terms of taking care of the firmware and taking care of the passwords and turning off unneeded services it's just completely untenable you need automated tools you need a centralized process to handle that but it just kind of shows you the breadth and depth of this problem now if you look at law firms for example they'll have a little bit less you look at retailers they'll actually have quite a bit more so it kind of depends on industry but on average take a look at your organization there's probably about three to five iot devices per employee right and we'll get into some of the iot devices that we see quite a bit of in

the worst offenders and you'll start going okay yeah if i start including these things i can completely see how that goes and just like with the stock exchange we were talking about those cesium clocks when we go into organizations to do a proof of value and we say hey how many devices do you think you have iot devices and they'll say i don't know i have x almost a hundred percent of the time their numbers are off by about forty to sixty percent so again they just didn't know about half of their devices in their environment so we talked about printers a little bit before printers are really a special case because everyone's got them and everyone's got a lot of

them we work with some folks in the hospitality industry for example where they have tens and tens of thousands of printers just printers different models and brands and you know versions and things like that but they have lots of printers now the thing about printers is most sort of enterprise level printers have about a 20 gig hard drive it's not huge it's not small but it's a pretty good size hard drive and what had happened is some attackers had gotten access to these these printers which by the way most of these are uncommon operating systems like linux or android and things of this nature in fact across all iot a flavor of linux or a flavor of android is by far the most

common thing that you'll find and by the way they have a lot of the same capabilities services um in the case of printers however they're far more promiscuous in running so many different protocols they've got every protocol imaginable because they want to be easy to use they want people to be able to connect to them and use them it's the point of the printer so from that perspective it makes great sense but attackers know that and because they know that they take advantage of the fact that those printers are so promiscuous and there are so many ways to connect with them so in this particular case some attackers had gained access to the printer they

had uploaded some of their tools this was only to a few dozen printers at the time but it did expand greatly to hundreds later on and they used those to go out searching for other critical devices on the network i t devices primarily and from them they were extracting data this was all about um intellectual property theft they were extracting the data and then they were storing it on those 20 gig hard drives and i'll get to what happens next well a lot of this activity because it was all being controlled through various remote connectivity uh controls and and c2 and things of that nature it created logs and those logs were showing up in splunk and they were

anomalous they said hey we've got a lot of printers that seem to be pretty chatty and we're not really sure why and then what they were seeing was data exfiltration and these printers they were being set up to exfil data over icmp why because everybody allows icmpo and even if it's shut off network ops turns it back on to do some testing they forget to turn it off just a really easy way to exfiltrate data you have to make the packets pretty small takes a little bit longer but it usually goes under the radar so you're using these printers to grab it sensitive data compressing it up exfiltrating out over icmp which eventually caused some alarms

to to go up and what they found was when they started looking at these printers and say god it's not just one or two it's we have several dozen now more than 100 now they're infected it was just the firmware they're running really old firmware some of these devices hadn't been updated in uh up to 10 years most of them hadn't been upgraded for about four or five years at least so they were just sitting there vulnerable they were doing their job they're working fine as printers and usually most people don't pay attention to a printer until it stops well printing so the fact that they are running all this old firmware just made them a big target

with a big juicy hard drive and an easy way to exfiltrate that data out now talking about firmware about 26 of the devices we counter have end of life firmware meaning that there's just just nothing we can do with that firmware it's completely end of life if you're running it you have to get on a newer version of it of the remaining 74 the average age was six years old um think about your smartphone it probably wouldn't work if you hadn't updated things in six years and i'm almost certain it wouldn't work um but now you're talking about your enterprise devices again all of your key systems especially the systems that manage other systems and that's a really a scary step

because if we think about it again there's not a lot of difference between an it device and an iot device in terms of capability network access and the type of negative ramifications it can cause upon your environment and again with iot you can even cause physical harm so if i went and told you 26 percent of your it devices were our end of life firmware you'd jump off this video right now and you go fix those that's how serious this is so when we start digging into the actual statistics around the vulnerabilities what we found and this is this is probably one of the the most frightening statistics so far the cbss scores so scores of of 1 to 10

10 being the most grievous 50 of the devices have a score of an eight in additional 18 have scores of nine or ten these are high to critical level vulnerabilities these are the type of vulnerabilities where you don't need local access you can do remote exploitation they're very they're very severe the great majority of the vulnerabilities discovered on these devices were eights nines and tens that's a scary thought again this is one of those things where if this these were your it devices you'd be like oh my god what can i do but because iot devices historically have been so hard to first of all just discover then manage then manage the passwords address the vulnerabilities in the

patching and the firmware managed services etc nothing's been done let's dive into the biggest iot offenders and these are the devices where we see sort of the most grievous issues across the board now we see a lot of other devices that i'm not listing here but uh you know we just had to sort of condense it into the the ones that keep on ringing out as being problematic again the the pictures that you see here are not the actual brands i'm not not going to call anybody out here that's not the focus here but kvm switches again most of you are probably familiar with kvm switches but kvm switch you stick it in a rack

you connect it to multiple computers and it lets you control um keyboard and mouse and video and sometimes you can cycle power as well we come across a lot of these devices that are running like linux ubuntu version 10. well that's from 2010 right i think the latest version now is 21 point something it's a little bit past 21.1 um so this is it's running old stuff and because of that it doesn't just have a few but they have a hundreds hundreds of vulnerabilities now think about this i've spent a lot of money on my endpoint security and application security identity this and network security that and now i've got this kvm switch that can back door into

all these devices running a version of linux that's uh more than a decade old that if i'm an attacker and i can compromise that i can do a lot of malicious things so those devices that are connected i can probably power them off i can change configurations i can you know modify stuff steal stuff whatever i wanted to do so that's a huge back door and that's one of the reasons that a lot of nation states and we talked about russia earlier are building these tools because why bang your head through the firewall and the ips and the encryption that and application security and endpoint security when i can just go around that back door again like 1995

with that u.s robotics modem plugged into the back of that windows nt 351 server in the back of the data center that everybody forgot about right so kvm switch is one of the biggest offenders that we see out there another big one lights out management controls um so if you look at the back of this of this server we see a couple ethernet ports on the right and something that looks like an ethernet port where that arrow is well that's not an ethernet port that's a lights out management port right that's that's what you're going to use for management these devices well the thing thing about these is there's a few different flavors there's hp has ilo

dell has idrac super micro has ipmi uh so there's a few different flavors those are the main ones they're just little linux servers in fact there's malware specifically designed to target those little linux servers for lights out management systems and again like the kvm switches but a little bit more powerful actually these lights out management controls bypass all that other security that you put on top of that server so you could have 10 million dollars of endpoint security on top of this thing but if that interface isn't secure if it's running a default password for example or a weak password or it has a vulnerability in it that can be exploited because that version of

linux or vx work that's running on that has not been updated in the last 10 years that's a problem because now as an attacker the things i can do for that is i can shut the system down i can change the network settings i can run a shell some of these guys i can pop open a virtual terminal as well i can upload software or malware i can do all the bad stuff that an attacker would want to do and again most of your critical servers are going to have these devices and some people say oh well you know i don't even know that thing was there and that's the problem again because there's no good inventory for discovery

for a lot of these a lot of these products server cabinets and racks this is another one that people tend to go oh those do have a lot of iot devices they sure do ups systems which are notorious for having the vendor's default password which takes about five seconds to find if you google it cooling systems cable management tamper resistance um sensors and all sorts of other things depending on how fancy that rack happens to be and they're really cool and they're really capable they're almost always running the oldest firmware known to man and the reason for that i think the primary reason is if i want to upload all the firmware on my rack including my

ups system everything else i need to reboot that system in most cases which means all those devices that are relying on that switching gear routing gear computer systems etc they're going to have to be rebooted as well so there has to be a change window and when people schedule change windows they don't always think about iot devices unfortunately which causes this problem so these devices are frequently frequently vulnerable and a lot of these are also tied to kvm switches that we talked about earlier exposing you to a lot of problems physical access controllers here's another case where you probably have oodles and noodles of these throughout your environment whether it's biometrics or you know a pin or a scan card or a cac

system whatever the solution might be again these are these are linux devices that are that are running on the network and in one particular case we're working with a customer and they had uh the system deployed it was uh it was all default passwords but if those passwords were not default and it's crazy to find the default passwords on door locks it's crazy to think but you always find it they had three critical cpes so had that default password not been there it would have been easy enough to get through one of the critical seats one of the three you could pick the one that you like most to get into that system and we're sitting with the cso and

their team and we're actually able to show them with a click of a button we could have locked or unlocked all 6400 doors door lock systems that they had throughout this financial services company and again the iot tax iot devices can have that impact on the physical world printers we talked a little bit about printers before they're certainly one of the most commonly attacked devices especially from nation states but cyber criminals like them too if any of you were at black hat back in 2019 in vegas there was some research that was released where they found critical level vulnerabilities again these are level 10 vulnerabilities on over 10 000 different printer types and brands and there's

just so many different types of printers out there but that's that's pretty incredible over over 10 000 devices had critical level vulnerabilities and again they're highly promiscuous they're running a lot of services they've got wired and wired list connections they probably have bluetooth they might be running other other protocols as well you can manage it via https or ssh some of you can tell that into they're just very very open and that's taking advantage and again because a lot of them have that big storage drive it's great to use them to attack it devices and exfiltrate your data out the way we talked about it before but again it's one of the biggest targets for state sponsored hackers i would say

it's probably in the top three next one voiceover ip phones and video conferencing systems like printers organizations have a ton of voice over ip systems um and even when people weren't at at the office there's still so many active iot systems that we actually saw an uptick in the number of attacks on iot devices but now i guess they're calling it the great return here in silicon valley we see you know the likes of apple and facebook and google people are are all going back to work again traffic's increasing because of that which is usually a pretty good sign probably an iot device could detect that but there's a lot of these voice over ip

phones and they're usually running a flavor of android what we find with a lot of these devices is they've got undocumented ssh administration capabilities running on them with default credentials again and a lot of this goes back to what we were talking about before with the manufacturers of some of these devices aren't necessarily software development houses they're more they manufacture things sometimes like farm equipment and things of that nature so they have a small crew they're not putting a lot of time and effort into into testing certainly not looking at security if they are it's very little and because of that they're saying well i'm just going to white label that i'm going to use that library from that

group and they might not even know they have that undocumented um ssh with default credentials right and they it just got rolled out that way because they hadn't done that level of testing so that's a that's a pretty scary thing um with this one organization i guess you can call this a beacon of hope but maybe it's kind of sad they had 31 000 phones only 700 only 700 had a critical level cve as we know it really only takes one so these again are uh an example of something that there's just so many they're so voluminous 31 000 phone systems if you wanted to upgrade the firmware or rotate the passwords every 90 days that you used for managing those

or whatever steps it would be impossible to try to do these things manually and then the last group i wanted to cover were secure our security cameras these suck the most there's so many of them and there's so many problems again some of these cameras shipped directly from their manufacturers with the malware already installed so you skip the middleman you just go straight to straight to being infected and having your device controlled these cameras have been known to actually turn on when they're supposed to be off they're known to record audio when they're supposed to be on you and they also take these streams and they can pipe them back to various countries so that's a problem

right there again a lot of these devices have been banned but cameras are devices that if they're compromised they can certainly be used for spying we mentioned some of the attacks that are very common in it and iot and how they're the same from ransomware to data theft ddos command and control uh malware distribution things of that nature one case we were working with a customer i think they had something like 9000 or nine thousand five hundred a lot they had a lot of video cameras and they had been crypto checked uh which most of you probably know that means i wanna take this device and i wanna use it to mine crypto so they had all these cameras

that were doing crypto mining um and the way they detected it i'd love to say it was some cool sim tool or alerted this it was their power bill their power bill was really really high and they didn't know why if you've ever looked at organizations that do crypto mining it's uh it's pretty energy intensive so that's how they found it um these guys are usually running linux usually it's some form of like busybox which is a common a common piece that's on there and the thing that's interesting about video cameras is a lot of the older ones are designed so i have 10 000 cameras but they all have to have the exact same

password that's talking to the management console like something like you see here in the middle um which which is an architectural flaw i mean you could set it up and there's some newer ones where you have the streaming password is the same but the management password is different so it has two different passwords uh because of that issue so when you're talking about password management on these things uh you have to keep in mind that hey if you're if you are gonna change the password and use something complicated now i have to do it for all 10 000 devices and it has to be the same thing it has to be a group password again from

a secure perspective not a great design but sometimes you have to work within the limitations of those devices as well so again virtually every organization has hundreds if not thousands or even tens of thousands of these types of devices there's often finger pointing when it comes to cameras well i thought facilities took care of that no i thought i t did no i think it was supposed to be security and it's kind of like spider-man right they're all pointing at each other it just it it's just very commonly um targeted and they tend to be very easy to take advantage of and on the consumer side again we're not focusing too much on the consumer side here but one of the most

popular cameras on amazon last year most popular camera highest rated you know approved this and that shipped with malware already installed on it to summarize um you know iot devices they're virtually always vulnerable i wish i wish there was another another way to say that it just sucks hence the title of the presentation um historically it was really hard to just find these devices just say i don't even know what i have i know i got a bunch of cameras and some door locks i think and lights out management i hadn't even thought of that but just finding them that's that's a problem then the next step of course is once they're found remediating the risks which is the

bigger part of the equation right i just don't want to find it i want to fix that and then i want to keep it fixed right so that's really important now there's solutions to help with this right enterprise iot security platforms somewhat of a newer concept in this world and i put a couple versions down here of course this is something that phosphorus does but we're not the only people that play in iot security you can take a look at some of these other vendors everybody has a little bit of a different approach to this some a little bit more legacy some a little bit more modern but now these tools can help you discovering the device and what

vulnerabilities are on my device updating my firmware managing those credentials and those certificates and hardening those devices and then what's really critical is integrating with all my other it security tools for logging for soar capability for ticketing systems and then being able to pull all that together with a reporting capability that makes this easy for me to manage at scale if you're able to do this not only are you able to sort of discover your iot environment and lock it down but you're able to do it at scale with these automated tools and make these devices secure you're just not hiding them off in a vlan just not closing your eyes and hoping nothing bad

is going to happen and you're greatly reducing the risk on your organization with something that's very easy to do in most cases running discovery from these enterprise iot security platforms is like running a vulnerability scan right the firmware is just kept up there you don't have to go hunt for the firmware yourself the integration api integration with the pam tools is already there it just works and it just fits so something you probably want to check out if you're trying to address your iot security so again my name is brian cantos i'm the chief security officer with phosphorus cyber security and thanks so much for your time