Cameras, CACs & Clocks: Enterprise IoT Security Sucks

all right uh so brian cantos with us today um uh first i want to let you know what's going on across the street brian's uh brian's competition is a is a talk called i hate you so you're in the right place you don't want to be hated brian loves you no it's across the street it's uh talking about getting along with developer teams so but here we've got brian cantos presenting cameras cacs and clocks enterprise iot security sucks a story of 2 million interrogated devices [Music] hello knoxville all right never choose your walk-on music after you've been drinking all night [Music] so we're past our hdmi problems let's go ahead and jump into this presentation

so a little bit about my background so i've been cyber security for about 25 years uh two ipos eight acquisitions uh or my wife likes to say about five pounds per company little does she know it's actually closer to 10 jokes on her but started my career uh in cyber warfare with disa down in fort huachuca arizona from there i said i want to get about as far away as i can so i moved to brazil and then after that i just started working with tech companies building startups from then on till today currently the cso with phosphorus cyber security set on a couple boards wrote a couple books my last book i wrote was

with the director of the nsa uh that's focused on physical and logical security convergence which is this whole idea even though we wrote that a little while ago about iot and kind of merging that with cyber security and figuring out how they can enrich each other so what is iot if you ask each person in this room they'll probably have a different answer uh i'll throw out a couple ideas and i'd like to hear what you guys say like a printer a mri machine a scada device for example uh even a satellite how about you guys let's go ahead and just yell out some iot devices coffee makers absolutely refrigerators microwaves routers switches cars so pretty much everything right

so the way to kind of boil it down and again there's a zillion different definitions but a purpose-built device connected to the network with some type of embedded firmware and generally speaking that's android it's linux busybox vxworks ubuntu things of that nature on the router side usually things like bsd etc but there's a lot of things that fall into the iot ot network device group i'm going to be turning my head a little bit to look at these because my laptop's way over there because we have a short cable not trying to be rude uh we've started looking at iot devices in about 2017 and it's a continuing project but now we've looked at uh multiple million iot

devices mostly across fortune 500 global 2000 and government agencies so you didn't really focus on home iot devices roomba vacuums lg vac lg washing machines ring doorbells alexa that's a different category the consumer stuff it's cloud managed cloud upgradeable we're focusing mostly on enterprise level stuff the stuff that you'd see in the business this was a global exercise continues to be a global exercise but i'd say the majority of the customers were based uh in the united states so a little bit of history to kind of tie into this because i really think it helps frame this whole concept of what we're doing a quick raise of hands here who thinks we probably live in one of

the most technologically innovative points in human history right not a trick question i think probably most of us would agree um although i've heard that if the greeks hadn't been sacked by rome in what 323 bc that by 1492 when columbus cruised over to the bahamas that the greeks would have had a man-spaced expedition to mars if their technology continued to increase at the same rate but my dad told me that and he's from greece so consider the source um but as you see as we evolve from cave paintings and alphabet written language paper etc every once in a while we have a pretty major leap a quantum leap if you will one such event was in 1804 joseph marie

jacquard invented the automatic loom looked like this some would say this is the world's first computer now it's made for making textiles specifically it was making blankets and as you look at it it's a predecessor to computer punch cards or more so computer punch tape which most of us weren't around for but this design allowed these textile manufacturers to go from making about a dozen blankets a day to making hundreds of blankets a day if you look at it it's got input output storage memory it's got all the things that we would consider a computer it was about two decades before babbage and ada lovelace and all that so it's a pretty awesome invention he just marketed it right

so you've got this machine and the the employees are like holy cow we're going to lose our jobs to this thing right so they're a little bit nervous about it so what did they do they broke into the factory at night and they sabotaged it now some historians will say that's where the term saboteur came from but i always thought it was ironic that the world's first computer suffered an insider threat so now we jump into sort of the modern age post arpanet we see things like tcpip the first computer um the internet becomes commercialized if you will all the way up to what we're looking at today with iot devices where we have satellites receiving data we

literally have zetabytes of data being transferred back and forth and everything now is being connected healthcare devices uh discrete and batch manufacturing devices military equipment so on and so forth so pretty incredible now if we look at how the threat window has changed looking at sort of from commercialization of a specific innovation to when it was actually attacked it's a really interesting dichotomy as well so take 3000 bc the mediterranean sea that's when they first said hey you've got stuff over here i've got stuff to trade over here why don't we stick out on a boat and trade the first notion of trade 1300 bc were the first recorded pirates so it took about 1700 years before

somebody said why don't we just rob that boat that's a pretty big threat window then we have air 1859 john weiss said hey i'm going to take this hot air balloon and i'm going to commercialize it i'm going to use it to move mail and the project crashed and burned it literally crashed and it literally burned but we didn't have the first hijacking until 1931 not 1700 years but a pretty gosh darn big threat window 1962 telstar just a couple years after sputnik owned by at t first commercial satellite used for television broadcasting we haven't seen too much as it relates to satellite warfare there are some isolated incidents with nation states but usually the juice isn't worth the

squeeze because those types of satellites have a dedicated base station so you have to attack the base station to get to the satellite and usually they're in closed off environments so it has happened it is pretty rare but in 2015 we saw truly data theft which was a command and control device that was being used to operate off of satellites so we have seen it yet pretty rare 1988 first commercialization of the internet nsfnet said hey we got a bunch of people using email mci said so do we let's charge people and they can email each other what a novel concept pretty much a threat window of zero so if anybody here remembers the morris

worm which used our commands and finger it was supposed to spread about a dozen devices per day and it spread at thousands of devices per minute and pretty much any internet accessible device that was accessible was compromised before this later on a paper was written at berkeley called warhol worms that said in the future the internet could be taken down in 15 minutes or less which was based on andy warhol's statement in the future everyone's going to be famous for 15 minutes that brings us to iot in the iot toaster which was the first official iot device but iot became kind of commercially popular about 2010 in about 2016 is when we saw the first

iot specific malware and botnets and that was tied to mirai so again we're seeing compressed zones throughout this entire ecosystem and i'm going to go into mariah in a little bit as we start talking about the most hacked devices and share some examples with you so first i just wanted to throw a couple stats out there to kind of frame this 67 percent of the companies we talked to again fortune 500 global 2000's government agencies 67 percent have had a breach or attack related to iot so these breaches are things like data theft so they'll attack the iot device move laterally to your it devices and exfiltrate sensitive data emails database information file server information stuff on people's laptops an

attack could be taking over devices and use it for crypto jacking we see a lot of that we actually had a customer that detected that they got compromised on their iot devices they had 10 000 security cameras i'd love to say it was splunk or some incident responder that caught it their power bill their power bill went up by 7 000 so which is an anomaly um so if any of you have done any type of crypto mining uh legally or illegally you know it sucks a lot of power but 67 percent we don't have great stats on the people that don't know they've been attacked so there's that security teams able to identify the

majority of their iot devices twenty percent honestly most companies don't even know what they've got and then enterprises increasing iot security spend this one's positive at least people get it they're like you know what we have a lot of iot devices they're not secure they're connected to our it environment and they're posing a massive risk we better start putting some money into it so that's a good thing who heroes use showdown so here's a fun little take home project type in something like voiceover ip security camera printer or ups and see what's connected who would ever connect a ups system to the naked internet well you can't really read it there but almost fourteen thousand

why would a ups be connected to the internet let alone fourteen 000. and here's the fun fact one of the most popular ups is apc we have a running joke in the company that says if we ever find an apc system that doesn't have the default credential apc apc lower case you can google it i'm not sharing anything we'll buy the other guy a steak dinner we've eaten nothing but chicken since we started this almost every single apc system that's probably plugged into really critical devices has the default password and 14 000 of them are internet accessible if you could read the part here that said uh security camera it's about 5 million something else that's interesting who's

heard of the russian frontend iot hacking tool so this is really neat this was a tool designed by contractors for the russian fsb the russian fsb wanted a tool that would allow them to target iot devices compromise those devices move laterally to it siphon information from there to the iot device then out to whatever location they want to send it to very powerful tool a hacking group called digital revolution found out about it stole it stuck it on a bunch of turrets and made it available to the world so now if you'd like to get this tool and play with it have at it it was paid for by the russian fsb another interesting thing about iot some

people know that some don't there's some that come shipped from the factory with malware pre-installed so many in fact that the u.s house of representative passed a bill that said these china-based firms huawei zte hickvision and a couple others are completely banned in government agencies and government contractors they have malware they have phone home capabilities they have a number of problems that they're saying they're not even allowed here another fun fact two years ago the most popular consumer based camera on amazon came pre-loaded with malware you completely skipped the middle man the malware is already there it saves you time it's more efficient so stories from the trenches let's get into the good stuff so mirai botnet i mentioned this a

little bit earlier this happened in 2016. this is always the poster child for hacking iot so i'm going to show you this one i'm going to show you something that just happened a few days ago this tells you how non-complicated these attacks are so what these hackers started doing was scanning the internet for a specific type of video camera this video camera had telnet running on port 23 which we usually see in port 2323 which we sometimes see so he said i'm looking for devices with a tool like showdown like we looked at earlier would tell that open okay so far pretty basic right next thing they did is they said let's go ahead and try the default password

and the default password worked on more than half of those devices they said let's try some very common passwords and that worked for about another 20 percent they had created such a massive botnet that they were able to take out paypal reddit sony netflix these are companies with pretty juicy networks using a botnet made up of mostly cameras i say mostly because here's the kicker in the iot world they're not usually tech companies that are building these iot devices they're traditional manufacturing companies so what do they do they use shared libraries they use white listing and when you do that you get shared vulnerabilities and shared default passwords and they have teams of maybe two or three people that aren't

incentivized to do security there's no security development life cycle it's a piece of farming equipment and you ask steve to make the internet accessible and he wants to get it done as quickly as he can so he can move on to the next thing that's kind of what we see juxtaposed to like alexa or something where they have 10 000 engineers focused on it that has a higher level of security so because of that white labeling because of those shared libraries it just wasn't this camera that the hackers found that same code was on printers voice over ip phones desk set boxes because they had those same shared libraries and white labeled systems in there so it didn't become just cameras

became cameras plus a whole lot more what's even worse is because the half-life of firmware updating is so low on iot devices you'll find these iot devices living with really old like decade old firmware that some devices are not only still vulnerable to mirai they still have mirai running on them and nobody knows the camera's still flickering the uh this other device is still detecting so what do i need to pay attention for so that's 2016 still relevant today this just came out a couple days ago this was released by mandian it's called quiet exit and this is really pretty interesting so the attackers were targeting network devices layer 2 layer 3 network devices things like san arrays load

balancers wireless access points some switches some routers again vsd systems and they're also attacking iot systems like voice over ip video cameras ntp clock systems and some other things like gunshot detection devices that can actually detect if a shot was fired they have them on a lot of college campuses unfortunately it can triangulate say there's an active shooter so they're started targeting these devices and those are mostly linux and android so they said we're going to compile the software which is just dropbear ssh client server pretty well known stuff we're going to write it so it runs on these bsd devices and it runs on these iot devices why'd they do that because no one's watching because no

one's digging into this stuff no one's looking at the logs and who's going to expect the switch or the wireless access point or the video camera could do anything malicious it looks so safe so they could hide out there essentially these are a hide once they had access and command and control activity they did it through a reverse ssh tunnel they were able then to make api calls to local microsoft exchange and cloud-based microsoft 365. using that they targeted the emails of executive security staff corporate development m a teams etc and then they pump that information out through those compromised network devices and iot devices back to the attackers and in many cases this was going on for

more than a year and a half i guarantee you there's organizations in the thousands that have these devices already compromised running this type of malware on their network devices on their iot devices that's siphoning off their information today it's a very very quiet very stealthy way to laterally move for the organization because nobody suspects this it's like a land war in china you don't expect it so question for the audience what percentage of iot devices operate with default passwords any idea okay 87. did somebody say half half 50 and if i told you 50 of your it devices were running default passwords you'd probably be pretty concerned and leave the conference right now and go fix them

but we haven't thought about it that way previously so fifty percent a hundred percent of the passwords on these iot devices network devices in ot are out of policy they're not rotated every 90 days they're not complex at all and they're usually as short as they possibly can be now there are some iot devices that say i can only be a four digit pin and it has to be numeric or i can be 10 characters long but no special characters or i could be 20 but not the letter b we've seen all sorts of stupid and crazy ways that this stuff is approached but 50 percent and of the 50 that aren't default they were changed once at the time of

installation things like major financial institution dash 2013 right that's the type of thing that you see all the time so when you say hack it's not necessarily hack it's login that's what we're doing so here was an interesting case this was at a stock exchange and as most of you know when you're trading stock milliseconds matter if you have a two or three millisecond advantage over me that could be millions or tens or hundreds of millions of dollars so they have these really really expensive ntp systems there's cesium clocks atomic clocks now these clocks are not cheap they're incredibly expensive more most more probably than a lot of people's data centers but what they were able to do

with these ntp things are keep stock trade trading safe so we went in there and they said how many of these guys do you have and they said we think we have about you know about six we found out they had 15 of them they didn't even know how many they had because they had a poor inventory back to that other stat only 20 of organizations really know what they have even when they're super super duper expensive cesium clocks right within their network so keeping an inventory is a huge issue now another interesting thing about this clock is there were no vulnerabilities there was no cbes not because it's uber safe and secure because they've only made like 100 of

these there's a very finite audience for such an accurate clock willing to spend that type of money and once you get past stock exchanges it's pretty much nobody maybe military this i like to call the most uh horrific chart to simply say most companies have about three to five iot devices per employee so if you have a company with 10 000 employees you probably have between 30 and 50 000 and you're saying brian that sounds like a lot and you're right it is most customers also yes about 40 60 off hey guys how many iot devices do you think you have in your company i don't know twenty thousand and then we take a look oh you've got forty thousand or

fifty thousand consistently forty to sixty percent off based on their assumptions juxtaposed to evidence-based information that says look at all you've got and we're going to go through a couple of those examples and dive in the most horrific offenders that we see out there but three to five is quite a bit so printers printers are interesting this this is one of the most commonly attacked iot devices by nation states period why because they're super promiscuous they've got every port every service open i'm wired i'm wired list i've got bluetooth bluetooth low energy i've got all these different ways of accessing me because i want to be promiscuous i want you to connect to me i want you to be able to print

they also have about 20 gig hard drives on them not big not small by today's standards but enough so if you can get malware on there you can do some interesting things so we're working with a customer that had over 250 of these printers compromised they found out actually through splunk after about three months they said oh we're getting a lot of traffic from these devices and they look like printers why is that so the attackers had compromised the printers it looks like they actually got in through a phishing to somebody else's email then look through printers so they could again attack and hide like we saw in quiet exit so they went from the laptop to the

printer that was their base of operation from the printers they loaded some of their tools and they went out and started looking for information on databases active directory enumerating the networks etc they compressed the information on the drives on the printers and they exfiltrate it over icmp has anybody here ever done data exfiltration over icmp a couple people it sucks you have to break it into the smallest little bites it's like saying here's a bag of rice i want you to take each grain and slice it about 50 times and then slowly put it in this bowl that's what it's like but if you automate it it's okay and it goes under the radar because

everybody says oh it's just a ping it's icmp echo request three it couldn't be bad and it operations and network ops they always leave that open for testing and validation just to make sure everything's up and running so they're exfiltrating data and the biggest problem was the firmware that was running on these printers in some cases were older than some of the employees at this company so which brings me to this question what percentage of iot devices operate with not old firmware but end of life firmware any ideas

end of life any ideas 70. 60. pretty pretty close 26 26 of the firmware so old is no longer supported and oddly enough a lot of iot vendors too are kind of they come on the scene after a few years they go out of business they're not even around anymore so you can't even get the firmware but what's worse than the 26 percent albeit really bad is that the remaining 74 the average age is six years again we talked about the mirai botnet and the reason it was so effective because the half life on firmware patching sucks nobody does it because it's manual who's gonna manually go through three to five iot devices per employee thirty to fifty

thousand devices for ten thousand employee company and manually change the firmware where you gotta stick a paper clip in there and stand on one foot and hop around right it just doesn't work in the old days it didn't work we're going to talk about some tools that will help this was an interesting one because this one actually impacted a microsoft azure customer so this was all about vulnerabilities these firmwares aren't just old and if they did change the default password so you can't simply log on there's so many cves especially level eights nines and tens our favorite cvss scores of tens that they can compromise them compound that by the fact that they always seem

to love running cleartext protocols not just telnet and ftp but like tftp from like the 90s right and all sorts of old stuff goes back to the white labeling and it goes back to the shared libraries when you've got developers that can't write hello world without a library so you end up in this situation where you have highly vulnerable systems they were able to compromise enough devices where their aggregate processing and bandwidth was rivaling that of amazon and google with just a bunch of iot devices and they took down customers with a ddos attack in microsoft azure um when it comes to cbs scores i mentioned the numbers fifty percent of devices have a level

eight and another eighteen have nines and tens for those of you who aren't familiar with the scoring system is one to ten ten is the worst ten means with hardly any effort at all with remote access i can get administrative access to that device with very minimal skill if you know how to like type an ip address that's kind of all you really need to know so what i like to equate this to iot security today was like i.t security in about 1995. did anybody here work in security in the 90s the mid 90s nobody so two people so we're the only two old people here so that was the days of windows nt 3.51

that's when hackers came out you're going to see that later today it was an interesting time because we didn't know where all our it devices were we had really crappy credential management patching was sporadic at best and i'll tell you about some of my early projects i worked on back then some of you might recognize this us robotics modem that was the bee's knees back in the day if you had that modem you were the cool kids but i was on this project with disa called the umap unauthorized moment abatement project which was a really fancy way of saying hey brian you're the new guy why don't you run around and find all the modems out on in the

environment and see if they're being used and if they're not stick stick them in a box all right that sounds like a great job i'll get on that so i downloaded this tool that some of you might have used which is a war dialer called tone loc it was written by minor threat and mucho mas and it ran in dos and basically instead of ip addresses i would put in a bank of telephone numbers and say war dialed these and it came back and said oh that's a phone that's a fax machine back when there was fax machines or that's a modem and the pbx guys loved it because it actually drove them nuts um and then i would go around to people

and say physically face to face hey steve do you need this modem no i don't well i put it in a box hey bob do you need this modem yeah the vendor hat needs that for autobahn access and that was how i spent some of the 90s then i found this tool called satan security administrator tool for analyzing networks written by dan farmer it's how i learned how to write pearl which for about a decade was a cool skill now not so much so i ran that on my sun spark one plus pizza box that i had running solaris i think slayers 2.5 or something at the time so i used that sun box running

satan to do vulnerability scanning and help identify all the vulnerabilities that i could find in the network and that was a lot of fun i said that's way better than looking for modems then they say we need to install a firewall say what's a firewall kind of figured out what it is so it was on three and a half inch floppies we had checkpoint firewall which we loaded on this device i never uninstalled satan because why would you who knew so we had our firewall also running our vulnerability scanner which seemed very prudent because we were saving money because it was another 3 000 bucks to buy a server so that kind of frames what the 90s was

like for security and it's exactly where iot security is today it really really sucks so this is the fun part these are the biggest offenders these are the devices that time in every single time we go into an organization they are just utterly messed up so kvm switches keyboard video mouse most of you know you've got a switch in your rack instead of having five monitors five keyboards five mice you've got one of these you can access all the other devices most of these that we come across are running ubuntu linux version 10 from 2010. we're on version roughly 22 right now so you've got all this endpoint security and encryption and application security and multi-factor authentication and

network security and all these things that identity management but the kvm switch has 10 year old or more software running out just filled with vulnerabilities and easy to compromise so you can access all the systems connected to them why because a lot of people don't even know those devices are little linux servers that you can connect to next one just as bad i would say even worse lights out management so this this little guy well these are your ethernet connections while this looks like an ethernet connection it's not it's your lights out management port same idea except the fact that this is actually built into the server so this here is a little linux server running on your

big server whatever that happens to be so by connecting to that you don't just have the ability to log in change network settings you can launch a virtual shell a virtual terminal you can upload software or malware and today there's specific malware designed to target this because once you get in again it doesn't matter your application security your data security encryption whatever i have direct access to these you might have heard terms like hp has ilo and dell has idrac and super micro has ipmi these are all different names for the same thing they're all linux servers and they're almost always vulnerable and never upgraded so a great way to get access racks and cabinets no one thinks about

this stuff they've got ups systems in there and we already talked about how secure those are apc apc they've got cable management they've got cooling systems they got tamper protection a lot of iot devices in there and the reason they're not upgraded is because all the servers that sit in them they have to be rebooted when this thing has its firmware upgraded and it has to be rebooted so when you schedule downtime to update that database that web server that critical device you don't think about doing the same thing to the rack again like kvm and like lights out gives you direct access physical systems this is a really interesting one because in the world of

iot it's not just attack on the iot device and the lateral movement to cyber but it can have impact on the physical world i can take a security camera and turn it around and spy on you with both audio and video i can unlock your doors i can shut down your power when it comes to batch and discrete manufacturing industrial devices ot when it comes to military grade devices internet of battlefield things can be far far more destructive but we were working with a financial services customer and we showed them remotely how we were able to gain access not hacking just gain access to these devices and open and close at random 6400 doors including the front door the door to the

data centers and everything in between nor tech security who makes a lot of these physical access devices were well known for shipping out devices pre-loaded with vulnerabilities with cvss scores of 9.8 and 10 out of 10 which again is remote full administrative access over these devices huge huge hole who cares about the cyber security stuff if you can just get in through this stuff and then go steal the server we talked about printers i'm not going to belabor that too much except to say in 2019 at black hat there was a discussion where they said here's 10 000 different printers versions and types that came with critical level 10 out of 10 pre-installed again they're super

promiscuous they're multi-vector access bluetooth bluetooth low energy all these ways of getting access and state sponsored attackers love it most of the stuff we see out of russia most of stuff out of china and other nation states are going after this one they're insecure two everyone's got them and three nobody suspects them this is an interesting one as well voice over ip phones and video conferencing systems there's a voice over i p phone that's extremely popular and ships with ssh access undocumented in the manual with default credentials and a lot of the vendors that packaged up because of white labeling and shared libraries didn't even know it's running ssh could you imagine shipping out a piece

of equipment and you don't even know you have port 22 open or that nobody even bothered to do maybe a port scan of the device and see what's active crazy right we went to one location they had 31 000 phones and i don't know if this is good news or bad news but only 700 of them had critical cbes only takes one but they had 700. okay so audience participation i've just mentioned a bunch of devices there's one device type i didn't mention in that group which is the biggest defender biggest iot offender who said cameras yep absolutely right security cameras the biggest security cameras suck the most so here's the thing about security cameras

a lot of organizations we go into just don't have hundreds or even thousands but sometimes tens of thousands especially if you go into like retailers financial services military cameras upon cameras upon cameras but that's not the big problem remember the end of the last spider-man where all the spider-men were pointing at each other who's managing it well i t no security no physical security has that no some vendors doing it nobody knows who's managing it and again some of them ship with malware already installed some of them are illegal to run because they're so vulnerable um and virtually every enterprise has these and they're just being attacked all the time crypto jacking is another big one

you guys might have heard this story about the las vegas casino they had a iot enabled thermometer and a fish tank i don't know if anybody heard this one the thermometer got hacked and from that thermometer they moved laterally they got access to the customer database and more specifically the database of the whales that spend millions and millions and millions of dollars there their contact information banking information what types of things they like in their rooms and i'm sure there's very personal things in there as well all from a thermometer and a fish tank right so it can be something as innocuous as that where they probably didn't have tens and tens of thousands even though they had that for cameras

but it just takes one device so we talked mostly about enterprise iot security there's a lot of other fields internet of battlefield things we work a lot with the military on these devices some of these are on soldiers some of these are on vehicles some of these are on forward camps there's a lot of places these are deployed industrial internet of things that's when we get into things like ot scada plc equipment digital devices that control physics essentially right flow temperature volume things of that nature internet of healthcare things uh we were working with a healthcare agency healthcare provider that had mri devices and x-ray machines that were compromised not for ransomware attacks we always think

healthcare ransomware they're actually taking the information from the scans in hopes of potentially using that as blackmail or some other device you get your credit card information your banking information stolen that sucks your healthcare information though you can't really change that so hey maybe we're going to get somebody famous a ceo a hollywood movie star something like that so they're actually put malware on these devices smart ships smart buildings smart cities all different versions of the same thing a lot of these are ultra connected baltimore had an attack that ended up costing them tens of millions of dollars in their iot infrastructure police and emergency services were completely shut down all based on iot not traditional cyber and then network

gear we talked about quiet exit and how these attacks now are embedding and installing because they've been compiled for bsd on your san array on your load balancer on your wireless access point so we're seeing a lot of network gear because again it's a purpose-built device that's running dedicated firmware connected to the network that you probably can't install an agent on so it falls into that iot category so there's some ways to combat this and i'm going to share a couple tools with you guys at the very end i'll actually give you the brand so if you want to take a look at some of those you absolutely can but let's see what hollywood can teach us

about addressing iot attacks so issues with discovery so the first step is knowing what you've got again 20 of organizations kind of have a feel of what they have the rest don't and most of those people that are 20 don't really know so what's an old-school way of finding your iot devices using vulnerability scanners things like uh tenable qualis rapid seven and they're great at what they do but i guarantee you if you're gonna try to discover iot devices like that it's gonna really suck because it's gonna knock them over it's gonna mis-identify them if you're lucky it might say oh it's a jet direct interface so it must be a printer but maybe it's not a

printer maybe it's a jet direct interface that's on a mri machine right so it doesn't give you the data you really need it's just assumption based security so that doesn't work so some people use nbads and bads again great solutions on the cyber side but when you're talking about these types of devices most of the communication is encrypted anyhow and you have to have spam ports pretty much everywhere in your network to try to capture this stuff again you're talking three to five devices per employee ten thousand people thirty to fifty thousand devices you're probably not going to have nbad and span ports looking at each one of those pieces or if you are you're spending a

ton of money millions and millions of dollars in switch technology so maybe this isn't that big of a problem and the other one is traditional asset management devices what we find with those however is they look at the first uh six octets of the mac address so the oui the organizationally unique identifier it says oh well based on this your hp or based on this your adele device or based on this your wireless access point that doesn't really give you the detail we're talking about with discovery because if you don't get this part right you can't go to the more important section which is remediation so one of the best ways to approach this i'm going to share some tools with you

at the end this is a new category so it's not a firewall it's not a sim it's not a scanner it's called enterprise iot security platforms right this is a completely new category and i'll share a couple different brands and people that do this at the end but the way they work is they can integrate with all that old school legacy stuff as well if you want to pull it but in addition to that they can inspect that's like checking out a couple ports doing some banner grabs doing light stuff that's not going to knock anything over and more importantly they do this thing called interrogation where they'll actually log in if they can authenticate

or in an unauthenticated way and pull all the data what's your serial number what firmware what's your password level give me all the context so i know exactly what you are and what you do because if i don't have that i'm not a hundred percent accurate i don't want to turn my mri device that's two million dollars into a 500 hp inkjet right if it comes to upgrading firmware or something like that later on so that's really important and the c3po thing is they remember in star wars where i said hey i speak a million languages iot devices speak so many languages which is why these types of tools these enterprise security platforms are so

cool because they can speak all those disparate languages for those devices and tell you with 100 empirical evidence it's this thing which is very important so i found my devices that's great now let's go to the next level which is upgrading the firmware and hardening it there's no way to do this manually you can't update all your firmware manually you'd have to have a dedicated team of people running around upgrading firmware all day which would be the worst job ever right so you're not going to do that some people say well i'll just stick it all behind vlans so that used to be the way people approached this years ago we can't fix it so we'll put it on a dedicated vlan

well the problem with that the analogy i like to use is let's say you cut your left hand and it's bleeding all over the place you need to go to the hospital but you don't go to the hospital you stick a sandwich bag on it and you wrap it up with duct tape well now you got a bloody hand in a bag and it's not getting blood on your right hand but you still have a hand that's probably going to die and fall off on your left hand probably not good medical practice all it does on a vlan is you're saying i'm taking that thing with the default password the end of life firmware all

the vulnerabilities i'm going to stick it over here and hope it doesn't do anything bad and to do that at scale it's almost impossible so that's the best we had years ago doesn't really work today so another way to address this is with these tools these security platforms so you can actually get away from assumption based and say hey i want to actually know exactly what this is down to the firmware level so now that i can actually upgrade the firmware and this is where it gets really cool if a device is upgradable over the network you can upgrade it with these solutions and you can't just upgrade it but you can also downgrade the firmware

so let's talk about upgrade for example first so for example if i'm on version 5 and i want to get to version 8 you need to know that this you have to follow the path of 5 6 7 8 or this device you can jump right from 5 to 8. that's why discovery is so important so you can empirically say based on evidence this is the firmware this is the device again you don't want to turn the mri machine into a printer but you also might want to downgrade you're on version 5 and it's got a log for j and everyone right now is getting the internet for log for j and the vendor says i know it but i'm

not going to have a release out for three months because i only have two developers working on this thing so i can back call that to an older version if i need to without having to go stick a paper clip in something or some other esoteric way of doing this you can use these devices to upgrade and downgrade also hardening something we've been doing on it forever cut off my clear text protocols no telnet no ftp everything is over https and ssh okay also no bluetooth low energy no bluetooth and only wired no wireless ethernet and you can push these out devices you can push those out across the range it makes this scalable now

when you say i have 10 000 cameras i need to upgrade you can actually upgrade ten thousand cameras with a click instead of walking out with paper clips and trying to do it which means it never gets done it makes this problem tenable you can actually manage it now instead of just you know hoping and praying what i had one customer tell me he goes you know when it comes to iot we did enough to survive but we didn't do enough to matter we're absolutely not doing our diligence as a security team by ignoring this we don't matter next thing is credentials inserts right just like firmware you're not going to manually go change the passwords or

check the certificates on all these devices there's a lot of great tools out there like uh pam tools there's cyber art cachy cort vault psychotic all these solutions they're great for storing passwords but they don't speak all the iot languages but they can speak to an enterprise iot security platform which in turn can speak to the iot devices the ot devices the network gear and now you're not only managing firmware but you're also managing passwords and you're managing certificates as well and you can enforce it you can say you know what the password policy is every 90 days i'm going to rotate has to be long complex passwords or 20 characters or more with special characters except

for all these other devices because during the interrogation process you know this guy only takes four digit pins so maybe we rotate him every three days or this guy can't take a password over 10 characters with special characters any little isms that you run across and there's some really esoteric stuff out there that's weird so you have to have that flexibility so you can actually leverage your pam tools now your identity and access management solutions privileged actions management whatever you have to manage those credentials and furthermore manage the certs so now i found my devices i fixed the firmware i fixed the passwords and i want to make sure everything that is fixed stays fixed in

perpetuity so simply monitoring re-interrogating these devices on a daily basis to say hey this was version seven now it's version four it had a really awesome password managed for the pam now it's back to a default password so instead of trying to hope my 50 000 devices are secure i get an alert through splunk or service now dems titanium whatever telling me hey take a look at these devices these are the five you gotta look at right now because something happened somebody did a paperclip attack for example where they reboot the machine a lot of these devices have a little black button you can hit with the paper clip sets it back to factory default even the super expensive ntp

clocks and then the last one reporting it's great that you've got all these capabilities that you can find it you can fix it you can keep it fixed for monitoring but if you can't report and integrate with all your other cool it tools it's just some iot management solution right so you want to have the ability to integrate with all your reporting and there's some really good tools that do that today so in summary organizations don't know what they've got they simply don't have a good handle of all the things if they guess they go well we got some printers we're not that big we don't have that much but when you start talking about kvm lights out

management rack systems etc there's a ton voice over ip phones so they don't know what to fix if i don't know what i have i don't know what to fix and if i didn't know how to fix it i can't fix it at scale because i can't have a dedicated team of people upgrading firmwares and passwords and then if they are fixed they're not monitoring to keep them fixed because they don't have any tool to reinterrogate these devices without knocking them over and killing them to make sure that they're still up and running so that means your iot devices your it your ot devices and your network devices are insecure we talked about ransomware data thefts

buying destruction all these things that they can cause and finally there are a few solutions to do this some of these guys have like 30 day free trials you can play with armist nizumi zingbox uh we do these things as well everybody has a different slice and how they approach it but this is the first time there's ever been a way to really manage enterprise iot devices ot devices and network devices at scale when it comes to discovery firmware and passwords and it's such a huge hole and we talked about frontend the russian attacking tool at the beginning that's just the one we know about from one country that got released to the public there's stuff being pumped through

russia and china and other countries and stuff that's being developed here in the us by cyber criminals to engage in ransomware campaigns etc that make this a really really powerful capability if you're spending all your money on it security you're not watching these back doors they have a direct and immediate impact on it and all your data so with that here's my contact information my twitter my email and i'll take a couple minutes for questions i know we we kind of went over right because we started a little bit late but uh if there's any questions i'll be happy to feel them yes sir

yeah so an iot device from our perspective is a purpose-built device with embedded firmware that's connected to the network that you can't install an agent on now there's lots of other ways to define it but when you look at it like that a cisco router scada device all those kind of fall into that group yep sorry apologies we will have to cut it a little bit short on the questions just to keep things on time make sure we don't get too too far off but big round of hand thank you brian thanks everybody you