BSides 2019 Day 2 Track 2

This is the most important one. Seriously, I made this mistake before. I'm not going to tell the story because of the five-minute picture thing I just got shown.

So if you can create a wrap installer for the purpose you want to install and match your media player, and these installer packages, which try to get you to agree to install further malware, well, sorry, further apps onto your machine that you probably don't want. So one of the things is, at the time, they had a really big distribution deal with Babylon Toolbar. of you probably might remember that one. But along with Babylon toolbar, they would install another product by Crudware, which is a browser guard, which makes it so Babylon toolbar would take over your homepage and default search provider in your browser. And then browser guard tries to prevent any other apps from modifying those search and homepage settings after the fact. And

it even makes it really hard for the user to control their own browser homepage and search provider. change it away from Babylon's making money off of it. So what's interesting was at a later date, this browser guard would actually push software onto your computer. So at a later date, it would

strategy in mid-August to deploy to every single one of those machines the Tor Cefnit component, which overwhelmed the Tor network.

So now the next question is really whether this Crudware company was involved with distributing Cefnit or not deliberately. When we look at the app, the Trojanized installer for FileScout, if you look at the main FileScout executable that it installs, compared it to the CefNet malware, you would actually see that they're built almost at exactly the same time. They're built 15 minutes apart from each other. So the main FileScout application executable, not the installer, the app that it installs was built at pretty much the exact same time as CefNet, which is highly suspicious. So that's quite weird. Maybe a coincidence. But there were a lot of other similarities between the files being built and products being built by Crudware Company as well. these Zefnic components. So

they shared code, believe it or not. They shared, yeah. They used the same string encryption code to try to evade malware and antivirus vendors, because grayware vendors do that as well, by the way. They shared similar build paths which were exposed to the debug symbol paths. So for example, here are a few of them here. So on the left there, we've got a couple of the CRUDWARE build paths. That lower one, you'll see that they're both being built using Jenkins build systems. And the upper one there, they're both referencing code constant as a code name that they have. And codeconstant.com was actually one of the earliest Cefnit malware CNC domains we had seen, believe it or not. And that's directly in the build path of crudware apps, you can

see. And the kicker was that the builder subdomain of crudware when you investigated DNS history records did resolve to an IP address owned by this Crudware grayware company. So it was looking pretty scary. Evidence seemed to implicate that Crudware was directly building malware, maybe directly involved with building malware. But the problem was that they're a fairly big company, grayware company. They're about a $100 million valuation from estimates I could find. We decided to take the benefit of the doubt that they're probably compromised or have rogue employees who are misusing their systems to do this, basically. So we reached out to them and they reportedly, we gave them all the technical details, they reportedly fixed everything, cleaned it up, and it sort of did. Steffnit did

stop distributing for a while, but then about five or six months later we saw it begin delivering Steffnit again. Then they flew out for an emergency in-person meeting with us because we were trying to get to the root of the problem to clean up this distribution of malware. So I'm going to call their CEO as well as their CTO of the company came out for an in-person meeting. I'm going to call them Aviv and Vlad for the story. Yeah. Yeah. So...

To be honest, they're actually both quite helpful. They're very forthcoming in all the history of the Graver company, the architecture of the company. I wanted access to their build server because I had indications that that CefNet was being built on their build servers, our very build servers. And I asked for disk images of that server too. I didn't end up eventually getting the disk images, but they did give me RDP access in our in-person meetings from their laptop to have a look at the at that build server to do some ad hoc forensics. So I didn't have proper forensics tools because I'm using one of their laptops to access it. So the build server was actually building Cefnit. It was configured even to auto deploy it. Now

what's more interesting is there are source control logs, not only for the crudware apps of theirs, but there are actually source control logs for Cefnit, believe it or not, dating back quite a few years. Now, it was directly, it was these source control logs, they were directly implicating the CTO who was sitting next to me as I was doing this forensics. It was quite uncomfortable, I didn't exactly tell them that it clearly implicated him for obvious reasons, but I think he's probably seen over my shoulder what I was doing and I was gathering notes in my log book of all the evidence I was gathering. It looked like the CTO had previously been involved with CefNet, like I didn't see anything recent, but he was involved

about three years ago. He was the, so the company is from Israel, but they had an outsourcing team which is based in Ukraine.

The CTO was actually, I think, head of the team, head of the outsourcing team in Ukraine, and they took that team internal to the company. They hired them on as employees and let some of them go, I think, at the same time, is my understanding. And that team had basically been involved with building Sefnet prior to their contract work with the Scraver company. So the CTO was clearly extremely agitated in this case. He went into the hallway, had some very agitated conversations, I think with people back in Ukraine. I'm pretty sure I know who with because there was one clear particular person who was an ex-employee who had been the main developer of the Cefnit from the code checking logs. The next day he

received an email probably from I think I know who, basically saying that he's the developer of Cefnit. He knows it's been used for bad things. They're shutting everything down.

This wasn't, I don't believe that this was the CTO himself who sent this, it's someone else from Ukraine that I know of. In this case, actually, everything was shut down. They shut everything down with CNC, including all the Tor hidden relays. The entire infrastructure went down and we've never seen it since to date, actually, which is pretty good. Yeah. So I did, in this case, send a report to law enforcement, but unfortunately no one was held to account in this case. So in conclusion, Crudware as a company was doing a lot of sketchy things. I didn't go into all the details of it here, even outside of this delivering of Cefnit. So it was some of their outsourced and acquired coders, which were malware developers, and it was

particularly a couple of the rogue and ex-employees who were the ones who were using the crudware infrastructure to repeatedly deliver malware in different ways.

I met a malware developer who was a bit scary. He's, but that, yeah. I would like to thank my coworker, Hamish, who did some of the really interesting linking to the attacks of the delivery methods of CephMet in this case as well.

The Tor connections, this is a more up-to-date plot of the Tor users. see it never fully recovered in terms of the daily connecting users to Tor since it's botnet attack. So it sets up Tor to run automatically as a service on these infected machines. So a lot of AV vendors when they cleaned up Cefnit never went and removed and cleaned up these rogue Tor services being connected because it's a complicated cleanup process. So yeah.

In case anyone's interested in cryptocurrency, Litecoin, back when they were mining it, Litecoin was worth around $2.68. It's now $61, about 30 fold increase. But I'm sure they probably wouldn't have held onto it. Okay, yes.

they were not trying to DDoS tour, they were just trying to keep, just trying to move the whole botnet to I think, to a CNC method where they wouldn't be able to lose all of their five million installs so they can keep control of them. Because I think otherwise with that, yeah, otherwise with the, otherwise if we add, if the AV vendors all of a sudden added, figured it out, I wish we had figured it out earlier to remove that Adobe service that they added, they would have lost their whole five million user base. Or if we take out the CNC server used by that, that would have effectively killed it early on. Okay, so I've only got five minutes left.

The second story I'm gonna try to go through a lot quicker. I'm gonna jump over some parts of it. So this was an interesting click fraud, our family that was out there that I discovered using some exploratory signatures that I had written. So I had indications that there was this app on the computer affecting about 40,000 machines, was likely doing click fraud in the machines. It was trusted code sign by valid trusted code signing app. No security vendors detected it. So it was rather worth a second look. So I loaded an internet connected sandbox. It ran multiple hidden copies of Chrome in the background and it would automate visiting websites carrying out click fraud. Great, it's confirmed malware. So regular

process is you just add signatures, you want it to the cleanup, it removes it from all the machines, that's great. So I add a generic telemetry only signature against it, and then I added a generic detection on the code signing certificate that they're using because it was a malicious one. Signature lease goes out, I get 40,000 detections, everything looks great. However, when I look at the telemetry, I still see it's installed in 40,000 machines. something went wrong there. So I saw the detections, but meanwhile I still had it running an internet connected sandbox so I could actually take a look at what exactly happened as my signatures deployed. It received an update in the middle of the night as my signatures were being deployed to install into

a new folder on the computers. And they left the old install where it was to be removed by my old signature so it looked like it would work. However, they installed into a new folder with a brand new code signing certificate. I repeated that a couple times like, okay, maybe I just happened to get it at the wrong time. I repeated a couple times and the exact same process would happen. As I would release a new detection signature, they would install to a new folder on the computer. They would let me remove the old one and they're still in the box. So, yeah. So in this case, the Defender clients check for signature updates once per day. They must have set up some automation, which

is checking our latest signature definitions to see if their app was detected. And if it was, they would push out an update, beating our once a day update with our Defender clients at the time. So what's interesting is they were not evading my tracking signatures. So in this case, my tracking signature was the telemetry only, so it's not the removal signatures. So in this case, deployed the tracking signature. I waited until it reached 100% signature release to all customers, so all of our customers had it, and then I enabled it our Windows Defender Cloud, all at the exact same time around the world. So they wouldn't have any time to react to the blocking signature. So basically, deliver it to all customers and then enable it around the

world at exactly the same time in order to crush, remove it off all those machines. And of course, repeat it a few times for good measures because some people's computer may have been off at that specific time of day. So that was pretty fun and we've also got a super weapon.

If you frustrate an AV researcher so much or if you have a big enough impact, we've got this really cool tool called MSRT. So we are actually able to protect all Windows customers through the Windows update path through a one-time scan removal. So in this case, they also didn't know to check for checking our signature release process through MSRT, which is... one-time scan which reaches all Windows customers, not just Windows Defender users. They could be using Symantec or any other AV product and this will still reach them through the Windows update process. So in this case, it removed it from all the machines and yeah, that was mostly the end of it. Okay, yeah, thank you. That's all I had for today. Thank you.

We're hiring reverse engineers, PMs, security people, coders, everything. Yeah. How many iterations did you go through before you pulled that big gun out of the pocket? So the big gun has a really slow release process because like with Windows updates we have a very careful release process for that. So that probably would have come like more like four months afterwards to really finally clean it off of Windows in general. as far as how many times I released detection signatures before I realized they were gaming me, I'd probably say three or four times. But I think most of the time, when antivirus researchers write signatures like that, they won't be monitoring so closely the release process. They probably just would have released a

signature, 40,000 detections, good. Not really realizing that automations like this exist. This is the first time I saw an attacker with an automation system like this. I've never seen it before.

Yeah, thanks. If you have any more questions, I'll be out in the hallway and you can, I think we're about out of time. Yeah.

Thank you

very much. It's complicated because it's so long ago but they had backups of the records.

Carolyn,

Carolyn.

Thank

you. .

Good question. I think it's... Yeah.

.

Yeah, so we're gonna find, there's a bag somewhere that kind of brings the nap, because I'm not sure if George took it. If that's the easiest way, then yes please. Thank you. Of course, of course, no worries. No worries, this is just going to be ready.

Oh sorry.

It's a little early, they've closed the doors though. Should we just... We should just dive in? Okay. Well, I am very flattered that you all decided to come to my talk. Thank you for coming. And I'm Julius Museau, CTO and co-founder of Mergebase here. We've got actually 75% of our company is in this room right now. So raise your hands if you work for Mergebase. There's three of us here. And, I mean, you do the math, right? So number four, he decided, he wanted to stay in the office. He doesn't like conferences. Yeah, we're Vancouver-based. Well, we're actually Burnaby-based, for those that have heard of Burnaby. So I'll dive into this. This is to patch or not to patch. And my laptop decided to give me a free little

bonus gift on this here. So I'll just... there, hey? So,

yeah, I'm not talking about that, right? I'm not talking about, you know, your Cisco routers that need to be patched. I'm not talking about patching devices. I'm not, no, I'm talking about applications and I'm talking about when you kind of go under the hood, right? People don't make applications from scratch anymore. hand I guess if there are there any developers in the room or you know wannabe developers or you know on the sly under your desk and so the last let's say in the last five years did anything you develop did you do it all from scratch no dependencies no libraries no components nothing so just raise your hand if if you did it all from scratch

BC Aware conference, I guess that's like a month ago, all excited that this guy came to the booth, right? It's our startup, our first booth. We'd never had a booth before. And yeah, we're talking, right? And he's this postdoc at SFU studying ships and sensors and marine sensors. And yeah, he's like, oh, well, you should see if there's any vulnerabilities, any publicly published known vulnerabilities. He's like, well, I do it all from scratch. So how would that happen? I think you're the only human I've talked to in the last 10 years. It's just coding it all from scratch. So there is this one guy in SFU that this talk doesn't pertain to them.

So back to that, yeah. So we're not, yeah, we're not, oh, and I'd just like to thank Mars, of course, and Alex Dow, and the organizers of the B-Sides conference for accepting our talk and for helping us out with BC Aware too, that was great. And also of course I'd like to thank IRAP, you know if you're a Canadian startup they can be a huge help there.

It's a cute little tweet I saw. Why programmers like cooking? You peel the carrot, you chop the carrot, you put the carrot in the stew, you don't find out that your peeler is several versions behind and they drop support for carrots in 4.3.

So anyway, like I was saying, this, yeah, it's AppSec patching. So we're patching the components that are kind of under the hood there. Not talking about that. And, you know, my laptop, obviously, it wants to do that. It's kind of interesting. In this case, I think the answer, like, to patch or not to patch is yes, do it, right? Always say yes, right? Why? Because it's Microsoft. In this case... on Ubuntu 18.04, right, I'm on the LTS, the long-term support version of Ubuntu. So there, I mean, that's Ubuntu's way of kind of saying to me, Julius, here's the deal. Patches on the 18.04 LTS, they're kind of important. You should take them, and in exchange, we're gonna, you know, do some QA on that. We're gonna make sure that

hopefully these patches don't break your system, right? Whereas if you're on Debian unstable, as they call it, I believe, that's the name, right? Debian unstable, that distro, right? their contract is, maybe you shouldn't take our patches, right? But if you do, and it breaks your system, please email us, right? We'd be very grateful to hear about that. That's sort of the contract with Debian unstable, right? And then hopefully it'll become more stable for everyone else. So, yeah, so in this case, I mean, the answer is yes. Ubuntu LTS, yes. To patch or not to patch, yes, patch. unstable, now you're starting to... it becomes a little more questionable. But no, I'm talking about this. So this is our own tool, this is

Mergebase's tool. It's an app scanner, it just looks for vulnerable components. And normally I point at stuff on this slide. I guess I'll be pointing at the bottom. So what we've done here is we've scanned an application, Inside it, it has this bundled directory, this lib directory, and then we're noticing, oh, if you have that version of BADIC, you're probably vulnerable to that CVE. Or at least, if you go to the National Vulnerability Database, the US MITRE NVD, as it's called, there is a CVE filed against that version. The NVD gives you a version range thing, and it says this CVE affects that version. Now, is this system vulnerable?

know, right? Like, it could require some combination of factors. You know, you can, maybe there's an exploit you can reconstruct or download from Metasploit or something and try it yourself. But, you know, just because I download the exploit and I'm sitting there and I'm trying to exploit that, like, for me, I'm not 100% confident just because I fail to exploit my own system. I'm still, I feel nervous. I'd still feel nervous. I'd still, it's like, you prove to myself that I can exploit it or just upgrade. Which way would you go? One is a lot of effort. The other one, sometimes a lot of effort.

Yeah, so, okay, so if you had that report for your system, what would you do? And so that's what I was talking about. Would you just patch that library? Would you assess true vulnerability? Maybe look at your logs? the CVE and the NVD might have some information that you can sort of look for in your logs. The famous deploy a WAF, right? All your troubles are solved. For those, that web application firewall. Any other ideas? Because we've got to make the mic runner work for his volunteer vouchers.

Okay, and then two definitions for this talk. Pinning. This comes into these traditions, like Java developers, they kind of seem to do things one way, and then JavaScript developers, they seem to do things other ways. Python developers, I don't know, C Sharp, I don't know, but I'm very familiar with JavaScript and Java. So pinning is where you, when you set out, you say, my system requires these components and these versions, and you just nail it down. You're like, I require version one, two, three, very strict. Whereas, pinning is a well-known term in this area. I'm not the only one to use the word. Ranging, I am the only one to use this word. I'm sorry, I'm trying to get it out there. So there

really isn't a good word for this, but this is where you say, okay, I'm okay with one, two, three, or one, two, four, or one, two, five in my dependencies. Please, any of them, bring them down. It's all good.

And then, of course, right? If I depend on dependencies, well, guess what? My dependencies, they also, they can depend on dependencies. And either step of this journey, right, I could do pinning or ranging or a mix, and then my transitive, they can also do the same. Anyone ever done that thing in package.json where you sort of, your transitives are all ranging, but you don't want those, and so you say, no, the transitives have to use these versions, so you can pin the transitives with NPM. Right, yeah, so I depend on, let's say, Apache Struts, of course, and maybe Commons Lang and Commons Kodak. Meanwhile, these things, they're going to suck in a whole bunch of other components into my system, right? Apache Struts, it's not going to be

able to run unless it has all these other components sitting alongside it.

It's kind of neat because there's an example at the very top there of a transitive, but in an interesting way is in that it's being bundled into a single component. So that analytics client at the very top, you can see it has a Jackson data bind inside it and it has log4j inside it. That's not normally the way it's done, but sometimes that happens.

iceberg here, it's a feature. This is where the productivity gain is coming from. The productivity gain comes from the fact that I don't care, I don't know, just make my system work, right? The value prop, not knowing what you depend on. But then of course, sometimes once you see something, you can never unsee it, right? Like when you find out your children are making your clothes.

The talk is about to patch or not to patch and which is better. And it turns out there's sort of a natural experiment that happened in industry. So Java systems, they use Apache Maven to manage dependencies. You can use Ivy and you can use Gradle, but Maven came first. And so Ivy and Gradle, they're just kind of copies, right? They're sort of drop-in replacements. So they honor the same contracts. it was sort of this first generation library resolver, the application layer. Of course, I mean, RPM's been around forever. But it was the first one that the app layer the developers were using. Before that, you would just grab your jar files that you needed and you'd stuff them into your CVS. And sometimes you would forget to type

CVS add for that jar file and everyone would scream at you.

And so is Maven, are people actually using it, right? If anyone doesn't believe me, right? So, Esonatype actually, I mean, they are now becoming more and more a cybersecurity company, but they also happen to run the Maven central repository. So in 2010, they were saying 260,000 artifacts in the repo, 70 million downloads every week. So let's see how it's looking now. Seems to have grown a little. Yeah, so up to 3.4 million, so 13x, factor there and 1.6 billion downloads every week. So I wonder what their AWS bill looks like. Now the sad thing about Maven though is even though the XML file where you would specify the dependencies, even though it did support this notion of ranging

so you could say okay I'm gonna grab Hibernate, my system it needs 430 final, that's like the minimum that I need and but if you give me for four that's going to break me. So please just give me anything between those. So the square bracket means yes, that version is fine, like inclusive range. And then the rounded bracket means exclusive range. Right? Well, so that's Maven. And suppose you put that into your build script, right? You're saying Maven, please, that's what I want. And Maven's going to go off here to the repository. Hibernate core. Okay, so I'm going to have to scroll down. Sorry. we have it, 430 final and 440. So I just want everyone to look at these. Now the thing about

these ranging operators is you know, you're allowed, Maven's technically allowed to grab any version it wants that satisfies the range, but in practice we want the biggest one, right? Give me the most up-to-date one that still satisfies the range. So just in your brain see if you can think of which one you would pick if you were Maven. Okay? sorry that it sorted lexicographically, it did not sort numerically, so you'll just have to, so it's a kind of a trick there, look in the middle. So who kind of votes for one of those two would be the biggest one in that range? And personally I couldn't make up my mind between those two. Okay, let's

find out what Maven decided. So I asked Maven the dependency tree command, Maven, given that range that I asked for, which one are you gonna give me? Three, five, six. Right, and that's, there's just a bug in it. And that bug sat there for seven years.

And so, I mean, if you're a Maven developer, or a Java developer, right, you kind of have no choice, you can't use a version ranger. And then Maven 3 came out and they decided to take this seriously and fix this, but in doing so, they created this funny problem in the range resolver, right? So according to this, so I made ABC there, so B is bigger than A, so that's good. And then I ask it, is C bigger than B? That's good. And then I ask, is A bigger than C? Oh no, there's a cycle. So in the ordering, in the ordering relation there. So that's not a total order. So now you're back to sort of like it's gonna just spit out random weird

things now and again. And then so they fixed that in Maven 3 in 3.25, December 2014, right? And so you're looking at 11 years where you just can't use version ranging in Maven. Does it work now? So I plugged it in, asked it, and it said, okay, it likes that Atlassian one. 4.3.11 Final Atlassian. And yeah. So from 2004 to 2014, you couldn't use a version ranger. You need 3.2.5 or newer. Hey, guess what? You know, RHEL 8, of course, it's in beta. It's not out yet. It hasn't even landed, right? So some people might be on CentOS 7. They might be on RHEL 7. Of course, CentOS always follows RHEL, Red Hat Enterprise Linux. And if you're on RHEL 7 right now or CentOS

7, you're on Maven 305. So this problem, it's still out there, right? It's not even fixed yet, 2019. And so as a Java developer, there's no question, you're gonna pin, right? You're like, if you don't pin, you can't build your software.

So this is the practice of specifying all dependency versions explicitly. Gives you stability, gives you repeatable builds, prevents regressions from new dependencies just suddenly sucking in and breaking things. But of course there's this security implication. So if a library is vulnerable and you have pinned versions, right, someone needs to do something. The default action of doing nothing is to stay vulnerable. So here's the infamous Apache struts. If you would put this version range in, if the version range actually worked, it's nice. You'd escape the Apache struts carnage for free. You wouldn't have to do anything. The next time your CI system built the application, it would download the good version. I mean, of course, you need to be building your application for this to actually work. If you just

built your application and deployed it, and it just sits there on the production server for two years, okay, that's no good. That's not gonna help you. But as long as you had this in your build script and you ran a build, fix you, right, if you were on a good version of Maven. But that's not what Java developers do. This is what we do. I'm a Java developer. I've done this, right? I just put in the version explicitly. And so

the consequence is that someone on your team needs to make that patch and push that patch to get, right? Replace 2.3.15 with 2.3.32. Okay, so

So that's sort of like to patch or not to patch. So that was the not patching side of things. What if version ranges worked? What if there was no pinning? Nobody ever pinned ever. And what if there was a large population of developers living in such a world so that we could study them? Okay.

Maybe JavaScript? Anyone seen this one before? Node modules, that's where the dependencies go for JavaScript. So NPM, it's the JavaScript dependency manager. So if you were writing a JavaScript dependency manager, of course you would call it NPM. Stands for Node Package Manager. Yarn is a drop-in compatible competitor. Of course, it's not just for Node projects. Node is kind of like JavaScript on the server, right? It's awesome. I actually used to write build scripts in Node, because I'm just kind of that kind of person. But these days, if you're bringing Angular into your web app, into the client part, you know, or Voo or D3, it's NPM that's gonna be collecting all those dependencies, putting it all

together, and then the browser downloads the end result. So even though it's called node package manager, it's the package manager for all JavaScript and TypeScript as well these days. And the package JSON file is basically the build script.

we have the pom.xml file. I can't remember what POM stands for. And then they have this version ranging, it's happening, it's awesome. Interesting things that they do here. So that bug I showed you in Maven, where, what's it gonna grab, right? The three, five, like will it grab between the range? Interesting thing in Maven is if you kept your thing down to two dots and they were strictly numeric, then the actual, that old Maven dependency resolver actually works in that state, situation. So where the version, whatever you call them, the components of the version, if they were always numeric and there was no more than two dots, then the Java, the old Maven one actually does work in that case.

So I guess maybe Node had that same logic and maybe they realized about those same bugs. And so they just force you. They say, okay, you're only allowed two dots in your version numbers. And they have to be numeric. You try to say 1.2.3.5, 1.2.3 final, node just will, it just is like, no, I'm not dealing with that. You have letters in your version number, not allowed. And then same, the 1, 2, 3, 4, node's just like, no, this is not a valid JavaScript package. You have too many dots. It was kind of interesting, eh?

architecture point of view, allow everything and just write the code that handles it or really constrain things so that your job is easier as the developer. Okay, and then so you can do these, this is how you specify the ranges, right? You can use the tilde, the caret, you can use x, you can use asterisks, you can do a range like that. Of those five examples, which one can you tell immediately or which ones can you tell immediately when you look at them what what are possible valid version numbers, right? I'd say like 1.x, 1.2x, and these ones. I find the far left one and the far right one. You can tell right away what they're doing. So which ones does everyone use? The

middle ones. Yeah, they're very popular. The caret is the major, the tilde is the minor. There's some insane... if you have leading zeros, they behave very differently if there's leading zeros. You can go read that. I read it every day just to put myself to sleep.

And what do developers think of this? Well, it's gonna, you know, every time you run your build, it's gonna go out to the internet, grab the latest versions of those dependencies, run your unit tests, build your system, And you know, certain percentage of the time, if you're downloading let's say 300 dependencies and developers are responsible for those or randomly uploading new versions, you know, a fair amount of the time it's going to break you. Now of course it's easy to find angry tweets about anything. But I consider this more compelling evidence that in May of 2017, NPM themselves introduced this new package lock feature. So you can still have the ranges, but now you have to type NPM update. And NPM update says, okay, let's regenerate that

package lock file. And then of course there's that hilarious event stream attack. I know, you go to a cybersecurity conference, right, and it's just event stream, event stream, struts, struts, struts, struts, struts, event stream, event stream, right? It's like there's nothing else to talk about. It's just wild, right? So how, what happened here is that it's a very popular package, but it's dead, right? Like the developer hadn't done anything with it for like five years, and it's just sitting there. And then I guess it does something useful or important. And so it's slowly being accumulating, I don't know what you call it, like not stars, it's being accumulating popularity. People are just like, oh, that does

what I need, okay, I'll depend on that. And so on, right? this entire dependency graph, right? Like the node ecosystem has just thousands and thousands of dependencies and event stream had reached a point where it had, you know, I guess thousands, yeah, 1600 packages, sort of in many cases, transitively depending on it. And so I guess someone thought, hey, this is JavaScript. It's like running in millions of browsers around the world. What if I just ask the maintainer, let me have commit and publish. I'll send a few patches, I'll send a few commits, do a few PRs, just pull requests to look kind of useful, maybe add a unit test or do some translation on the documentation. And then the developer, he maintained like

hundreds, he had like over 100, I believe, NPM packages he was maintaining. So someone emails him and says, hey, I love this one, I'd love to take it over, can I have commit on it, can I have publish on it, right? And the developer, of course, he's gonna be like, oh man, yeah, that thing, that headache, yeah, bring it back to life, yeah, it needs some maintenance, here, go, have it, right? And so, the person asked for the keys, and the person said, oh, that's great, here, have the keys, right, because they're helpful. But of course, you have this carrot and this tilde thing happening. I don't even know if the timing here is correct,

right, because, I mean, this is what GitHub says. GitHub says September 9th, September 16th. But you could actually, commits, one says September 9th, one says September 16th, and you can push it in a single push. And I can't, I don't get to see the push logs. GitHub doesn't show you those on the public GitHub. You need to have GitHub Enterprise to see the push logs. So anyway, yeah, he pushes this evil 336, pushes a good 4.00, but of course tilde and carrot are everywhere in the internet, and they're going to grab the 336. And they didn't even, they didn't put it in EventStream themselves. All they did is ask EventStream to bring in another library. And then,

so I don't know if it was a week later. It could have been in the same push. But then they removed that. So that's it, you know? And then they rev the 400, the 336 turns to 400. And that's just the, if you go to NPMJS, it just tells you what's going on with EventStream. And this, because the 400 was pushed, you don't see that bad dependency there. So, you know, maybe the 336 was on this page for a week, but those downloads, they're not asking for 400.

So yeah, right? Like in that universe of 1600 packages, how many of them could even conceivably grab 400? None of them, because according to NPM,

four, when you rev that four, that's called a breaking change, right? You're saying, oh, I've broken reverse compatibility, right? So in Sanfair. And so tilde and caret and everything else, they just wouldn't touch that. If you would actually put 400 sort of preemptively because you were anticipating this attack because you're a very strange person, of course, NPM would just blow up, right? NPM would just say, no, there is no 400. You know, we can't build. So everyone out there and caret, they're just going to be grabbing that 336 like crazy, bringing it in to everyone's browsers. Yeah, this is an amusing tweet, right? In the subsequent carnage from the event stream bug. Quick tips, if you're going to use third-party libraries in a safe way, first, you should thoroughly

vet all your dependencies. Don't just install everything. Don't ever use nested dependencies. Pin to specific versions. Don't ever let them auto-update. And then check all your third party code into your repo so you can track changes. He's kind of saying maybe just build everything from scratch almost. Or at least read every line of code that goes into your system. So okay, grow your team, hire ten times as many people. Even for a merge base, that wouldn't even be enough. We'd need a hundred times more people. had lock files that's good that would minimize the damage because someone had to type npm update so that the lock file would grab the bad versions and bring them in. Of course I think lots of

people what they do is they run a CI job on their Jenkins, on their Travis, whatever that literally runs npm update. Did the build succeed? Okay it's good committed into master right and so it's no good for them. So yeah so the lock files you drink from the less often but you're still drinking from madness.

And again, lock files, they're basically pinning. It's nice because now everyone on your team is basically running the same stack, so that's good. That was always painful in the earlier days of NPM because I'd have that version of the transitive and they'd have that version and so who knows what we're seeing. So now you get stability, you get repeatable builds, your team is all in the same stack, it's preventing regressions, but now you're back to that old problem, which is if there's a vulnerable library, someone needs to do something. And the default action of doing nothing is to stay vulnerable. Now, to do is this now. It's a little bit better than that Java situation. In the Java situation, I had to actually

figure out the numbers, like 3.26.18. You know, you have to go to Maven and see what's available. You have to type it in. You can't have a... You don't want to have a typo. So this is easier, but it's still something, and someone still had to do it. And so, I mean, there you go. To patch or not to patch. Kind of inconclusive. Always patch is too hot, right? Even in the JavaScript world when they were always patching, they quickly backed off of that and they introduced lock files. Never patch, well then you have struts, so it's too cold. And then lock files, I say it's slightly warmer than never patch, because at least it makes that patching a little bit easier, but

it's still too cold. make this matter to me it's obvious and it's a ton of work and it's not it's not the funnest job in the world. Anyone ever purchased a Red Hat Enterprise Linux subscription? Or you could do CentOS right same thing and what happens right is that you have all these RPMs on your system and then because you purchase a subscription you're entitled to the patching in the errata right and so and then there's people somewhere that are deciding that these up versions of these RPMs they're good to go you you know our paying customers can have them so I think we need something like that in in in this application security land in this application dependency land yeah but at the application

layer instead of system layer so conclusion never patch you know it's not viable it's not an option always patch as we see it's an avenue for attack But the larger problem is, you know, there's effort required. You're gonna break your own systems if you always patch. And then there's a regression risk as well. I think bots, like there's these bots now, Clover or Dependabot, and they literally, they throw PRs at you if you're on GitHub, right? And so they'll be like, oh, stuff you depend on has newer versions. Here, have some PRs that just rev that. And... have, what's neat is if you're building all your PRs, right, your CI system, so you know, dependabot off GitHub, throws PRs at you, your CI system immediately builds them, so

you know if the build passed or failed, and then dependabot, because you have a public repo, sees that if those builds passed or failed, and then they can use that information to kind of improve their service, right, so they can say, okay, we have the cutting edge people that just build everything, and hey, the rest of you, just wait till those guys have built successfully before we let you guys have it. So Dependabot is kind of doing some interesting stuff there. And I think that actually really does take companies a lot farther along on this problem. And there's other solutions like that, like Dependabot. And yeah, call for startups, right? It would be a grind in cases right because what the painful thing happens is

you'll have something like you have all these customers on Apache starts 3.1 and 3.1 is affected but the Apache foundation themselves they're only publishing a patch for you know 3.5 let's say right so the 3.5.1 comes out it's got the patch but meanwhile you've got all your customers on 3.1 and on 3.2 so it's an interesting startup opportunity that work is very redundant across the world, right? Like if you trying to backport those patches to that 3.1 branch, to that 3.2 branch, and you do it in your company, I bet there's other companies doing that same backporting, right? And so it'd be kind of nice if one company or a handful of companies would start

doing that. Of course it's not the funnest job in the world, backporting patches and being sort of worried that you didn't do it right. As far as I know, nobody offers anything like this yet. Does anyone know of someone doing this? Don't be shy. Do you want to do them, Mike, for the... Oh, okay, I'll repeat the question, sorry.

Yeah, I mean, there's going to be something. Yeah. Well, I'm not a lawyer at all, so, yeah, I just have no idea. Oh, and I'll repeat the question. He's saying, is there potential liability? So if a company did want to go into this business of selling curated patches and then, you know, something bad happens, I'm sure they would have liability, so they're going to have to find some insurance. But I'm not a lawyer, so I don't really know. But, you know, Red Hat is able to do it somehow. Obviously, they're big and they're IBM now.

slide are your suppliers patching and there's some tools you can use to audit them so it's kind of fun running static scanners against your suppliers I mean not sure if that's always allowed strictly in your in your end user license agreement I'm not sure if it is reverse engineering or not because you're just getting version numbers like you're not really looking at the code I so no type can do that we have a tool, we have our fancy tool that you can pay for, or we have a free tool as well. Just email us for it if you're interested. And then, I mean, these are the three that actually can scan build systems or running

systems that are just sitting on disk, whereas a lot of the other stuff out there, it looks at Git, it looks at build scripts, right? And so you're not gonna have that with your vendors, usually. And if we had time, I could show off our scanner. And, I mean, you've already kind of seen it. I could do a live run. Yeah, and so that's the end. You can follow me on Twitter or hit our blog, post comments, or just email me if you like. And any questions for the room right here while we're here?

Okay, so the question is, I say that developers pin, and is it, why are developers pinning? Is that lack of awareness? Is it lack of training? And should they be pinning? Why is this happening? And what's sort of the breakdown for the reasons there? Would you say I got the question? My sense actually there is its tradition. So in NPM land, nobody is pinning. I mean, they're using the lock files now. in terms of actually writing down the dependencies or not. And so in Java, there was just this tradition established. And I think that tradition was there to begin with. So I think that tradition is part of why Maven didn't fix that bug for so long, because they're just dealing with Java developers, and Java developers just

would never even think of using the range operator. And then also because the range operator never worked, so we could never get going. So I almost think it was, you know, like the opposite of a virtuous cycle. I don't know what the word for that would be. And then it's this whole, this siloing effect that Java developers just, they talk to Java developers, they look at other Java code, so this whole tradition within that ecosystem. Whereas other ecosystems will have different practices.

The question there is, or the question comment is that perhaps the problem too was that the range operator in that Java case was unpredictable. Yeah, absolutely. I mean, that was the bug there. Not just the inconsistency, but what you will actually be getting in terms of features, or is it always referring backwards? Oh, the range. So... feature range, you can refer for it, you don't have to cap it, I didn't have to put the, like if I could, I just could do comma nothing with the round bracket and then it'll just keep going in that regard.

Any other questions?

And how are we for time? minutes yeah I could I mean if anybody want me to it'll be hard to do this with one hand but you know I did download JIRA I'm probably not allowed to scan it but I could

so here's our scanner and just you know just download JIRA and just pop it in ask our thing what it thinks

right? It's, well I mean that version of Jira, right? Version 802 Atlassian Jira. So I just downloaded from their website and I don't know if those are reachable JAR files in their running system and even if they're reachable I don't know if they're actually used. I don't know you know how that will play into the actual exploits that we're finding that are correlated with the CVEs here. But yeah, you know, if you have these versions maybe you should do something about it. Jira 803 or 805 or whatever the latest version of Jira is right now obviously this is something they work on themselves right so version 803 it won't have as many because they'll have

revved some of those. Yeah so sorry I'm on one hand here but I'll just do that so we can and it's kind of a small font

running it in the pager so I can sort of navigate around and take a look at them. Okay so it's done. And so I mean they have this EH cache right and then the EH cache, lots of Jackson right, Jackson is a Java serializer. Batic is what you use to make I believe it graphics files or something or charts or PDFs maybe and so perhaps some interesting vulnerabilities in there but again okay so it looks like it's mostly just Jackson and Batic here and then the Jackson is copied in many places throughout and the Batic also is copied in a few places so it's actually not that many really

and yeah so then the scanner just it I mean it tells you so it's looking inside zips inside wars inside jars it'll just recurse completely to any depth of nested zip files, jar files, war files.

Yeah, so I mean I'm doing the breakdown at the bottom there and I know that technically and I'm just using the base score and there isn't an extra high category, but you know, the high range, if you go to NVD's database, it goes from eight to 9.9, and so I'm just, or seven to 9.9, something like that. No, no, no, it goes from seven to 8.9. And so I'm just splitting that, I'm bucketing that, so the sevens are the high, and the eights are the extra high. Oh, yes.

Yeah, something, okay, so I'd say you get into sort of a development group sort of hygiene sort of thing, where teams that are really staying on the cutting edge tend to stay on, they tend to patch, right? And then other teams just, if it's working, it's working, we're gonna leave it alone. And then what happens is the team that is good at staying up to date with all their dependencies good at that and I think remains less vulnerable and then the team that doesn't, the gap just grows and grows and grows. And so I think you'll see both and it's a range but the thing that is just awful if you fall behind because yeah, suddenly you're looking at, you know, you might be looking at like

three developers working for three months on nothing but getting those libraries up

with all this regression risk. So it's like you're going to have to do some full regression testing for weeks, right? So it just starts to become a huge... It becomes so much work that the company becomes pretty averse to even doing it.

And so, I mean, stay tuned for our next talk, which is in this room, which is after the break, where we talk about maybe some different ideas that companies could... could try out for grappling with this issue. Because, yeah, we're not just doing a scanner. We actually have a much more exciting tool for helping out here. Okay.

Well, thank you everyone.

Hey. John. John.

I'll give you a quick info about five minutes and then you just will take over and provide more detail and discussion as well. Just wanted to start off with some background on what four problems that we see that

problem that we see. Wow, that's very loud. ...to start with, what's the core problem? This is one of the main issues.

So why is that the case? What are some of the root causes that this has become significant?

The first is that the other areas, for the security people are able to go to jobs in other areas. The network and the system where there is a very interesting talk yesterday, very impressive what all the developments have been in all these areas of security. And that, of course, has reduced the opportunities in other areas for criminals and recently focusing on applications and why. Well, very few enterprises today So let's look into more depth

into an application. It's not going to be a very good thing. It consists of a lot of different components. And so as we...

Okay. Is this better? Okay. What's the rest? Well, you've got 80 to 90 percent of the code that you're shipping or that you're using is reused libraries. Why? Well, to get more capabilities to market quicker. That's why everybody is doing that. That's a whole ecosystem of layered libraries. But of course, it's also a very large attack surface at 80 to 90 percent. It's a very attractive target for criminals because if can compromise one component. They can not only attack one organization, they can attack thousands if not millions of organizations with a simple and repeatable process. And that's even made easier for them because there's this whole exploit economy that is developed where they can actually purchase

or even download for free tools and or acquire services to get into business as a criminal. So it's very low effort. built a repeatable process to go after a lot of organizations and that makes it very attractive for criminals. The last reason is a lot of organizations are not optimally positioned to defend against this. To start with, they are reusing a lot of software. Why? Well, because they want to focus on something else, which is what their core value is. So they are not set up to really study in depth those pieces of software that they're using, they just want to use it. However, that's a complex ecosystem in itself with all kinds of versions and

variations and different distribution paths, patches. In essence, you need to really understand that well in order to protect it well. That's one of the reasons they lack the expertise on what they're using. The next is there's a mismatch in terms controls between the threat and the controls they have in a lot of organizations where the controls, because the applications is focused on the software development life cycle. Somewhere in the software development life cycle they have controls, scanning, they do other activities. So things will go into production and when everything is nice and stable, then they don't have to worry, right? Vulnerabilities are uncovered over time and it likely that when things are stable and in

production that vulnerabilities will be uncovered and so you need processes for that as well. Not everybody has that in place. So in summary, what's the reason that vulnerabilities and external components has become the number one source of data breaches? Well, criminals are focusing on applications. Within those applications the biggest attack surface is external components and it's a very attractive target for criminals. not all organizations are optimally defending against that. What kind of solution do we have in place? I'll quickly go over a few features and then Julius will get into in-depth demo. Our mission is to really secure the software supply chain. That's a multi-year effort. But the tool that we currently have in place

is a software as a service for IT staff to defend their applications against cyber attacks and their components. There's a few different elements to it. First is visibility. What do you have and what risks are associated with that deep within your application? Then provide some level of control. What are some of the actions you can take? Patching is obviously one for it, but also beyond patching, what other actions can you take to protect your application? Third, there's compliance to make sure that you stay within your risk appetite and policies as an organization. In terms of visibility, what we provide you with is deep insight into the application, into its libraries, versions, and variations, also which makes it unique across

all environments, including production. Also, it will take into account the actual usage of the application in order determine the risk that is associated with particular components. When you have components that are vulnerable, you can obviously, the best solution is to patch them. If you had been here earlier for Julius' presentation, you see that that might not always be possible. So it's the preferred solution, but it might not always be possible. In that case, there's a few other options that exist. So for instance, to block specific vulnerable methods or classes from being executed or to put them under close surveillance and collect much more telemetry on the actual usage to see if there's anything strange going on. From a compliance perspective, you automatically have

documented your risk assessment on an ongoing basis. You document automatically the actions you're taking as an organization, so that makes it much, much easier for the organization to assure themselves that they're staying within their risk policies. a quick overview of the application and its capabilities, then I'll hand it over to Julius to give you a demo of a simulated attack and show you what the tool looks like.

Okay, more one-handed mousing. It's my favorite way to mouse. Okay, it's up there. Okay. So we'll pretend that that we're a company and I don't know, we've got our staff out on the field and we've deployed this web app for our staff to, and it's on the internet and they can,

so our staff will, they'll log into this and they'll collect data out in the field and they'll upload it to us. So right, it's like sort of any typical corporate web app. And so, and we did build this with struts, right? So, you know, obviously we want to, attack ourselves and exploit ourselves and then provide a mitigation. So here is this web app. No.

And, um... Okay, we're gonna put some stuff here.

And submit that. And... I might have to put down the microphone for a little bit.

Okay, so we got that web app and now let's see kind of an attacker. Callie, and the console. Can everyone see that okay?

Oh, that's not mine too.

I'm trying to use the body down yesterday, but I guess the sacrifice is only just going to end. Okay, there you go. It's the man's on. So we visit the web scowl, it's on end.

and then the data, the data, the

user name, the tap, the tracking and now I can get into the database, that's the code over there, okay? So that's Metasport, the tracking system.

So what we've done is we've gone to that and we've applied to the merge-based inoculate. So what the merge-based inoculate is it goes in, and I'm gonna tell you the truth here, is rewrite all the jar files. Oh, sure. So, okay, so we go in and we rewrite all the jar files just to inject a little bit of instrumentation so that now when the jar file, vulnerable jar file is being used, we can do something about it. that instrumentation, it's gonna phone home to our, to our, uh, SAS dashboard here. So here's our SAS dashboard. And...

Oh, of course, I'm tethering on my phone. One second.

And there's our struts demo web app, right? And so those jar files what they're doing is they're calling into this every time they're used well not every time they're used we you know we do some stuff to optimize that uh and saying hey i've got some vulnerable jar files here i'm gonna just filter on the ones with vulnerabilities and so there they are so that struts web app that you guys saw under the hood it's using these jar files and that's sort of the struts when it's not, it's using like, then maybe that's not so great. So maybe let's stop that jar file from using. One sec.

Struts. You, I, okay, I'm gonna kill that jar file. I'm gonna tell that jar file you're not allowed to execute anymore. One sec.

Okay, now of course the struts jar file is um, is a big part of that web app, but nonetheless if you kill that jar file, so that jar file no longer can execute, right? But the rest of the web app is, remains operational. Uh, let's see if that has consequences for Metasploit, one sec.

What's it saying? Fail to bind. And that is the, but that is Kali. Am I still on, I maybe exited too much, sorry.

And I think it fails because the kill is in place. Is that right? Okay. So it's the same error message as before. Yeah. And so, yeah, I can't get in anymore. Of course, if I go to the web app itself, because struts is such a major part of that web app, that web app's not looking so great anymore either.

But so, I mean, I can unset that too.

and no action. Oh, there's the loading. Awesome. It takes about a minute. So it's just, I'm just waiting for this system to pull home to us to realize that that jar file is no longer killed and is allowed to run. Yeah. So now it's back. So, uh, And then we have some other data in here, right? We can see like how much all the jar files are getting called. For example, you can see log4j was called here 17,000 times in the last 24 hours.

I could show the, any questions?

Yes.

Yeah.

Absolutely. Like for example, here you see common bean utils might be a better candidate for killing because it has zero usages, right? Even though I'm clicking through the app and even though this jar files party app and is reachable, it's not being used. And so, um, you know, if I kill it, I'm sort of essentially, um, shrinking the attack surface, right? And it has, I mean, it has, this one has a vulnerability associated with it. If I drill down on that, right? So it has this, uh, CVE 2015 4852,

Oh no, no, we've got some Windows customers as well.

And uh, yeah, so it sort of takes that static report that you get from like Black Duck or Sona type, oh, here's all my vulnerabilities. And then what we do is sort of join it, bind it to production, right? And so not only are these all the vulnerable components that were sitting on the file system, know that they're actually being executed, that they're part of the code execution paths in your running system.

Yeah, you're saying like that jar file shouldn't be allowed to open files or something? Yeah, at this time we don't have that capability. I mean, of course if you had a jar file that was just dedicated to opening files, then sure, because we're really operating on jar file granularities. So this jar file does this, or this npm file does this, or this at DLL does this. So we're really kind of like setting up sort of these fences at that granularity. And then you'll know the name, the version, you're running that version. And then we provide these, you know, these mitigations. So you can monitor it, see if it's being used. We do allow you to monitor more deeply. So we would be able to surface the fact that

a file open routine is being called. And we're planning to extend those capabilities too, right? Because since we're in there in the runtime, since we have that instrumentation in there, there's all sorts of things we can do. For example, we're looking at developing a hot patch.

Yeah, at the time version one right now handles jar files and version two is going to be NPM. Yeah, and which is on the immediate roadmap. And then planning to further, you know, it's a grind, right? You just basically have to handle all the platforms eventually.

So we've been building this for seven months. And we've been working with some customers in POC, Proof of Concept phase. We have three customers right now for about a month. Yeah, we're a startup, right? This is a startup demo.

The thing about the run time, I mean, sure, I mean, as a precondition to offering this, we need to be able to even detect that the vulnerabilities are there, right? Oh, you're using a vulnerable component. That's sort of a foundational piece of tech you need to do this. So, of course, if the company wants to just use our stuff to generate static reports, go for it. And we've benchmarked ourselves against, you know, White Source and against SonaType and against Black Duck, and we are actually beating them just even on those static reports. But we think this is really important for the enterprise to actually, yeah, bind it to that production runtime data because it just makes the remediation so much quickly. As a developer, I mean, I

was, sorry, I'm talking too fast, but I was faced with this problem where I worked before. Now I work here. And I couldn't upgrade the jar file. Like, I literally couldn't because the dependency graph would get so complicated. But then if something like this would tell me, well, your jar file's not even being used, just delete it. But I couldn't do that before because I didn't know it wasn't being used.

And so suddenly a job that was gonna, you know, take me a hundred hours just to determine is the jar file used or not, you know, I can just, I'll know right away.

And similar for, you know, NPM once we roll that out, Python, Ruby, and so on.

Thanks for attending the startup demo.

Testing one two one two one two one two one two testing.

It's a matter of, it's a matter of game. Like there, which is a, I, I think with silence, utter silence, you know, it was really, it's because the game is on. Because we do the ambient noise is being pulled in. But like, it's not a thing we can really flex with, we don't have the time. Like I would need to basically, we need to jack the volume, like,

They're in the bathroom. Okay. Fire break. Yeah, they're in the bathroom.

here.

Well, good afternoon. Thanks for joining us. I guess you must all be the ones who queued up first because apparently there's people waiting to get in. So thank you for joining us. We're going to discuss a little bit of how we might apply some of the lessons that despite me being sick, it doesn't sound like I'm a good person to listen to about medical hygiene. But... We have a lot of challenges in info security and we know that the human challenge is one of the biggest problems all of us face. And actually, in my experience, in particular for people with a CS or an IT background, it's one of the problems that we struggle with

the most as well and that we're not trained in the social sciences often. We don't really have a good idea about how to approach getting people to do the things that we know are going to improve the security of our organizations, potentially protect our customers and our employees. what we do is quite technically complicated and most of those people don't understand why it's important and why it's necessary. So we're gonna share some of our experiences together with you today in that I have the benefit of, I work at Sophos and I travel a lot around the world speaking at events and consulting with our customers and other security people at events like this. And one

of the benefits of that is I actually get to learn what everyone is doing to try to get their staff with those policies and creative things companies all over the world have done that you might never have thought of that are really a good idea to bring the whole company along with you on this security journey. So I don't claim most of the ideas that I'm going to share with you today to be my own. Many of these things I've learned from others who found ways of getting things to work better than the traditional way we approach security education.

I'm a social scientist. I have a master's degree in human development, learning and culture. In my research, I work with a theory of motivation that's called self-determination theory. Most often I apply this to education, but it's also used widely in the workplace. It's used in counseling, it's used in sports. It's also used in medicine, both to train doctors and also to improve behaviors and thus improve patient outcomes. And those are some of the analogies that we're going to draw today in our talk. So I've been tasked with describing the problem and unfortunately, if you work, whether you're working internally in security or whether you're providing security solutions, we all know that human behavior is kind of the low hanging fruit these days. And if we look at

the big picture of every breach, whether it's the ones we don't hear about that are happening to every little organization around the community, or the ones we do hear about because they're in the newspaper and they're headline news. More often than not, they all start or involve heavily human elements now. And there's quite a few reasons for this. talk, but if you start looking at how effective we are at, for example, patching our web browsers and our PCs, we've kind of shut the door on drive-by-web attacks. You know, a couple years ago, 80% of malicious content beginnings of getting into an organization was exploiting Flash and Java and unpatched browsers and this kind of stuff. And it's literally the opposite of that now. It's well below 20% of attacks

have any involvement to do with an exploit against the browser. We've not, because we haven't closed the door, but we got better. We actually started doing better at protecting a lot of our systems and patching in these types of things. While we know we're not perfect, the human beings are now the easy, easy weak point for getting in. And whatever that low hanging fruit is, whether it's the fact that we're all quite bad at choosing passwords or whether we all have days like I feel today where maybe we're a little under the weather and not that great at decision making and make mistakes about clicking an email or opening an attachment. THAT'S ALL ULTIMATELY FALLING

ON HUMAN BEINGS. THERE'S VERY LITTLE TECHNOLOGICALLY OFTEN THAT COULD BE DONE TO PREVENT THOSE THINGS. WE'RE ENTIRELY RELIANT ON THE STAFF IN OUR ORGANIZATION TO NOT PRESENT THAT VULNERABILITY.

WE OFTEN PRESUME THAT THE PROBLEM IS THAT THE USERS ARE THE ISSUE, RIGHT? IF WE HAVE 800 STAFF MEMBERS AT OUR COMPANY, THAT'S 800 POTENTIAL actions waiting to happen, right? It's 800 attachments waiting to be opened. And, you know, part of that's our own attitude in blaming users when that's not wrong. I mean, the users are often inviting the bad things in, but we're also not necessarily looking at it from the perspective of how do we present these problems to them in a way that they can understand, right? It's not, you know, the advice that we give is often, you know, having the 72 character password and never clicking links in emails. I mean, who doesn't click links in emails? I mean, I've been doing this

for 25 years. I click links in emails. How is it sensible advice? I need to have better advice than that to give people, or they're going to either ignore it or I'm going to confuse them and we're going to continue to have more bad outcomes. And of course, we also expect perfection. We need 100% compliance and beings are never going to be 100% compliant. We have very unreasonable expectations. And we also have a tendency to measure failure. We started, Sophos, we started selling, this is not an advertisement, but we started selling a phishing assessment test like so many companies out there, like Phish.me and all kinds of other companies, right? And I was very disappointed when we launched the product initially because

the thing we measure and give you reports on is your failure rate. We tell you that 22% of the people in marketing clicked the phish. it's a very negative approach to the problem, right? We're always measuring failure and we're expecting 100%. And we're gonna keep fake phishing you until you never open an email and do your job ever again, right? And there's some truth to this. I actually had a guy report, I had a domain that I hadn't secured properly and a guy contacted me because somebody had hijacked it and was using it in a phishing attack. And I asked him, how in the world did you notice this and figure out that it was

me and call me to tell me about this? oh, I know that my company tests me for phishing all the time, so every time I get an email that has a link, I spend like a half an hour doing who is lookups and all this stuff to see if it's a trick for my tea or not. And I'm like, wow, can you imagine the productivity if all of our users started reacting that way to our demand for 100% compliance with our phishing tests? Maybe this isn't the right approach, right? Maybe this is not the way we get the outcome we desire. And the other attitude in many organizations is simply to focus on compliance. And I don't know if

any of you in the room were in my talk yesterday talking about payment card security, but those of you that were may very well understand the difference between what I talked about yesterday and what PCI compliance is. I mean, one is a checklist and one is understanding how it works so that you can protect it. And we're very much focused on these checklist kinds of things, and some of the business, you know, we do in 40-some countries around the world where we have an office, and in some of those places there's laws that we have to train our employees. So what do we do, right? We send them a PowerPoint that they have to read,

and then we send them a quiz to prove that they read the PowerPoint so we can... That's how I got GDPR compliant as well. I had a 22-minute video I got to watch, and then afterwards I got to answer whether something was privately protected information or not. We want to check the boxes and move on and I think a lot of in security in particular we go along with that I know a lot of people that go along with that because they feel well it's pointless I'll never go to teach people to not open the fishes so I may as well just do the compliance thing that management asked me to do because the whole thing is a waste of time and I'm not going to accomplish it

because I think I have to achieve 100% compliance and I can't get to 100% so I'll just do the thing I have to do which is check box and then once I've checked the box management will be off my back and then I'll worry about technical things that I could I can And of course we then often shame people or cause fear which ends up hiding our problems even further. You failed the phishing test, now we're going to report that to HR and tell your manager that you failed the phishing test and now you're going to go do the PowerPoint training that I just told you about because you failed the phishing test. that causes fear. This is going to look bad

in my review. This is going to be a bad thing. It makes people afraid to report things as well. So if you think about all those people in your organization, they're noticing things that you're never going to see in IT. People in finance have a different workflow than you do. People in accounting have a different workflow than you do. And if they're fighting, if they're afraid to tell you about things that they're seeing, that's making you more blind to the security things going on in your organization. And obviously this doesn't result in positive outcomes, which is why probably half the InfoSec talks at B-Sides that I've seen on this topic are that, you know, user

education is a waste of time because we can't get 100% compliance and all these other things. And users feel pretty demoralized about it. They feel like they can't do anything about it. They also often feel that it's not their job.

these IT security people came in, told us we had to do all this stuff. Well, I hope they realize that the KPIs for my bonus are that I get these three things done, which is my job. Like, I need to close support tickets if I'm a tech support person, and I've got to get the books closed on time if I'm in finance. And I don't have time to do your job for you. People don't feel like it's their responsibility, and when the outcomes are poor and when we're putting a lot of pressure on them, they just write it off. Not my problem. It's the IT security group's problem.

So, coming back to this idea of medicine and of encouraging good behaviors, many of us have also experienced this at the doctor's office. If we have a doctor who insists on perfection, if we have a doctor who focuses on our failures, if we have a doctor who tries to judge and shame us into better outcomes, we're likely to engage in a lot of less than desirable behaviors. anyone ever avoided going to the doctor because they didn't want to experience that? Have you ever just kind of hoped your problems went away? Have you ever hidden or understated your bad habits? I bet I would see a lot of, be honest, but this is a safe space.

Have you ever ignored your doctor's advice because you just didn't feel good about the interaction, right? So the big question, both in medicine and security, is how do you get people to do, to willingly do the things that they do not enjoy doing? We know that some behaviors are fun in their own right. There are things that were just, to do, you don't have to try and get somebody to do them. Hobbies are a great example. We call these intrinsically motivated behaviors. And then some behaviors are not fun, but we learn how to do them anyway and we do them most of the time. We call these extrinsically motivated. Some health examples are brushing your teeth, flossing,

a lot of forms of exercise that aren't particularly fun, eating more vegetables, taking your medications, quitting smoking. We used to think, well, intrinsic motivation is good, extrinsic motivation is not so great. We were only half right because there are some behaviors that we can never get to be intrinsically motivating. It will never be fun to brush your teeth. And what we usually do about these behaviors is we use rewards and punishments. Or we use rewarding and punishing emotions like pride. We haven't talked about that one, but fear, shame, and guilt are some that we've talked about. Rewards feel better, but rewards and punishments are really two sides of the same coin. motivation someone has to

engage in a behavior, rewards and punishments tend to extinguish that, actually. And they can also lead to us doing some really wonky things. For example, doing the bare minimum just to satisfy a requirement. We talked about compliance. Cheating or simply defying instructions. And, you know, that's not a great outcome. But fortunately, there is a better way.

cheating by becoming password compliant by adding 2019 to my password is not probably an outcome that we're looking for. Yeah, I used to iterate my passwords when I worked in a corporate environment. Increment by one. The truth is that we are actually primed to take on new behaviors, skills, and values, and norms from other people. This is most evident in young children if you spend a lot of time around them, especially when they're real little. They love to imitate you. They love to repeat everything you say with the exact intonation with which you say it. And although this is the most obvious in young children, it remains true across the lifespan. And the reality is that there are some forms of extrinsic motivation

are just about as good as intrinsic motivation. Most of us actually do brush our teeth a couple times a day, right? I hope, yeah? Alright.

So high quality motivation isn't so much about making people do stuff as it is about tapping into their own motivational resources. So what are those resources? First,

The better we're able to meet our psychological needs, the more likely we are to take on those behaviors, skills, values, and norms from our environment.

The first one, we don't usually talk about this one the first, but I wanna talk about this one first. We all need to belong. We need to feel connected to other people. We need to feel connected to other people. for by them and we need to care for others. A lot of social engineering actually relies on this as an attack vector. Someone I know and like, or at least someone who is part of my in-group is asking me for something and I should probably help them out. As a side note, in my preparation I came across a paper that did try to employ self-determination theory in computer security. Now, theoretically speaking, we talk about belongingness actually as relatedness. And these authors, who

I believe are part of a computer science department, they actually suggested that you can feel related to your data. That's not how this works.

value your data, but your data can't care for you back.

So relatedness is fundamentally social and cultural. Try as you might, you can't escape the human element. Oops, I'm doing things with speakers. Okay, thank you. That means that this all starts with knowing, liking, and trusting the people who are asking us to undertake these security measures. talk about that a little bit later. Second, we all need autonomy. That means that we need to feel like our behavior is coming from inside of us and from who we are and that we have freely chosen it. That doesn't mean that we can't act with and for other people. It just means that we need to be on board with what we're doing. We need to have buy-in ourselves. third need. The third need is

competence. Feeling like we can do things and knowing that we can grow our skills. This is where we tend to over-focus. We jump straight to the competence part of this. How do we make people better? Really the question, what's more important than that is why people would want to be better at this in the first place. In addition to needs, we can think about mobilizing people's interests, their preferences, their values, and their goals. Who are they? What is it about their jobs that they're protecting? What matters to them to protect? And why should they specifically care?

So, instead of assuming that the problem exists between chair and keyboard, that the user is bad and wrong, this means asking yourself, is the reason for the failure? How can we forge a partnership with our users? How can we inform them to make, sorry, how can we empower them to make informed choices? How can we help them to do better? And how are we getting in their way?

One of the simplest ways to support a sense of belonging is using intrinsic motivation. Make it more enjoyable to interact with you and your department. That's quite simple to do. You can avoid, completely avoid this shaming, blaming and judgment. People need to feel good about coming to you no matter how badly they've messed up. Communicate an attitude of how can I help you? You can also leverage existing relationships. Peers.

people in your department can be ambassadors to you. Peers can also be powerful models. Someone I like and someone just like me can do this, therefore so can I.

So as far as autonomy goes, autonomy is about giving people choices and getting them to feel like their behavior is their own. So

important ways that we do this is by anticipating and acknowledging resistance. Look, I know this is a pain in the ass and you may not want to do this, but it's really important that you do this thing that I'm asking you to do. We do this by explaining why in language that people can understand and in ways that matter to them. We do it by encouraging them to reflect upon their choices, it by inviting and accepting input. I'm squeezing this again. This stuff is a two-way street. If you ask people and if they trust you, they will tell you where they are getting hung up and they will tell you exactly what kind of support they

need. Earlier I talked about rewards and punishments. These are tricky, as I pointed out. You can use them. saying never use them. You can use them, but they should be small and symbolic. They're there primarily to communicate that something is valued and to acknowledge progress rather than to be used as an incentive, as a carrot, right?

And then the most obvious way to build competence is to build skills. We know that. is crucial though to building competence and to people feeling like what they do matters. If all you do with reports is you disappear them into the void, people don't know that what they're doing matters. You don't have to respond to everything. I'm not saying that every time a user reports something, you should send them a personalized thank you note. That would be ridiculous. But you do have to demonstrate that you are responsive people feel like they messed up. When we talk about health behaviors like quitting addictions,

this means framing slip ups as learning experiences. We're trying to focus on growth here. We can learn from this slip up and we can avoid it the next time it shows up. The next time the conditions are the same that caused us to slip up in the first place. We can, thank you.

We can increase awareness. This is another obvious one that we kind of usually do. There may be attack vectors that people have never thought of. And it pays to let them know that that's out there. And then one more way of addressing competence is to make tasks easier. We can identify and address the barriers and the speed bumps that prevent people from doing the right thing. provide tools and templates that also help people do the right thing, and he'll talk about that a little bit later. So

this is a little bit of an idea or a sequence that comes out of research about dental hygiene.

What they found was that this particular sequence, this way of approaching patients,

compliance very much. And the first thing you do is you ask. You don't tell, you ask. So you ask about their problems, you ask about their concerns, you ask what their practices are. What's the situation? And then you listen.

You acknowledge whatever feelings they're having about that, whether that's frustration or resistance, like, hey, you know what, I find flossing once a day hard. How can we make that less hard, right? You can make recommendations and provide rationales for those recommendations. And then you personalize the solutions. You know, you might not treat this as a one size fits all problem. For one person, you might recommend a water pick. For another, you might recommend like a different type of floss, right? In security, this might look like personalizing by departments or rule. on what kind of risks those departments will face. And then you demonstrate how to do it and you practice with them. So now we're

going to move on to some examples of how to put this into practice. I have this one instance that I wanna talk about and then I'll hand Mike back to Chester and he will talk about a number more that he has encountered.

As an alumna of UBC, I receive all kinds of email at my UBC address from various journals and things like that. Some of it gets flagged as potential spam. They preface it with a message that says I need to forward it and flag it as not spam if it's not legit. Cool. I'm down with that.

send me another email telling me I need to forward it again and give them my consent to send it to a vendor outside of Canada. This is not cool. Dealing correctly with this one questionable email has now multiplied into four emails, two that they've sent me and two that I've had to send back. Now, legal may have a different opinion of this, I don't see any reason why this couldn't have been consolidated into one email stating that forwarding the email constitutes consent to sending to a vendor outside of Canada. So ask your users, is there anything that seems unnecessarily difficult or tedious? And do what you can to streamline that. And that may mean that you have

to push back on other people's policies and decisions. It's your job to advocate for your users, right? And to improve the outcomes. So, Chester has a bunch of other examples. I'm going to try to give you some application of the concepts that Tierney's been sharing with you. I want to apologize on behalf of Sophos in that we filter UBC's email. But that part is not your fault. No, and the irony is the mail goes to our lab in Canada. in Vancouver where we process those. But without going down that line, I did realize at the beginning, I forgot to mention, all of this is premised on not the malicious insider, right? You will have people who are intentionally not compliant and that you may need to actually

put the fear of God into with HR and their manager and they may need to be dismissed if the activity continues. That happens and this is not meant to address that scenario. It's much more meant to address the scenario that Tierney was talking about how it's people's, in general, most of us when we feel part of a group want to help in protecting. We care about what we're doing, we care about our organization, we care about our customers, and we're often not empowered to help or we don't know how to help, and all it takes is asking, just like the Fishers do, and we'll help. So we need to be able to leverage that on

our behalf. Maybe in the future we'll do a talk on what to do about those malicious people, but... It's not this. Now, one of the most effective ways of getting through to people that these are real risks is through storytelling and modeling. And I think there's a lot of challenges here in that most organizations, let alone most people, think that's not going to happen to us. No one's going to, you know, trick our CFO into wiring $300,000 to the wrong account in Nigeria. And yet it's happening to thousands of companies just in Canada every month. Everybody thinks it's not going to happen to them until it happens to them. The smoker doesn't think they'll get lung cancer, on and on and on. Take whatever example

you want. And it's particularly true in cybersecurity. So one of the things that can be done that is really helpful and effective is sharing stories within the organization about things that are happening when they happen on a semi-regular basis so that people understand that this isn't a theoretical risk. Depends how big your organization is, how effective that can be. But certainly some of the most powerful stories I've heard inside of Sophos have been when our people have been the victims of these things happening. We had a couple years ago an incident where somebody was trying to get information from our help desk people and we're spoofing the telephone numbers of the executives on the caller

ID and calling in with the correct phone numbers of some of our high-level people. do password resets and get other information from our help desk staff. We were lucky that what we think was the first person who received one of those calls recognized that the voice wasn't the person claiming it for who it was supposed to be because they actually knew that person and reported it. But after they reported it, we were able to put the message out to our entire forward-facing staff and help desk and support and these types of things and get the word out. what this is about, we don't know who's behind it, but we know something's going on, and this is happening to us right now. Be aware, whether you're in sales, support, we

don't know what they're after, so we don't know who they're gonna call next, but if you're someone in an outward facing position in the organization, you may want to be extra cautious and maybe hear some advice on things you can do. And then we detected several more incidents over the next few weeks, and I suspect we either wouldn't have, or we would have detected fewer of them. context sensitive, it was people that needed to know, and it was something that affected them in a way that they understood, and it happened to somebody they knew that sits with them. And we've shared those similar stories when our executives have been phished and other things that sometimes

at company meetings or, you know, these types of things. And when we have incidents ongoing that we feel are a threat, we often send out company-wide emails. saying, by the way, we've been seeing things impersonating this, this, and this that are trying to trick our staff and disclosing passwords. The fact that they're happening to us and there's a screenshot of an example in the name of a person you work with or you've heard about because they're a senior manager or they're a director, obviously only sharing those stories when you have permission from those people, especially if they were actually victimized. And those are the most powerful ones is when somebody actually did plug in the

thumb drive. open the attachment and are willing to allow themselves to abuse as an example for the company because suddenly your eyes open up when it's like, oh wow, if it happened to Sally, it could happen to me. This is not virtual anymore. This isn't something I only hear about on television. People have this idea that it's only spies and they're only going after the people making missiles and

talk, but I have a great example of this around a spa that was compromised because the executive assistants all went to that spa for somebody who makes missiles, and that was how they got into the missile company was compromising the executive assistants who go to the spa, through the spa. So the spa didn't think, we don't make missiles, but the spa still got compromised. So these things can really help when they're personal, they're much more powerful. I'm a big fan of the Lunch and Learn. I haven't done one in a while within the organization, but when you're trying to find out who those people are that care in your company about security and privacy, you're trying to start building a program, you want to go from having a security

program to a security culture. Well, a culture means the whole organization, right? You've got to have people from all the different departments. You have to have not just executive buy-in, but you've got to have the people, the regular everyday people got to buy-in. So, a great way to start that is with Lunch and Learn, right? We're going to run a seminar on privacy stuff this afternoon in the break room for lunch. And, oh, by the way, there's going to be free pizza. So, you know, you're going to have a little bit of casualties on the side. There's going to be a few people that come just for the free pizza. But a lot of people

are like, oh, I've been hearing a lot about this stuff. I'd like to know more. I'd like to know more about privacy and password security because I want to protect my bank account better. I want to protect my family. Maybe I'll go to this to learn more. running these things on a semi-regular basis, maybe once a quarter, once a month, you start identifying and you're like, wow, Jan from tech support comes every time. Steve from finance comes every time. These people clearly either really like the free pizza or they're clearly invested and interested in this topic and while by profession they might be an accountant or they might be a tech support person, at home they're probably a gamer and a mom or a dad and

all kinds of other things going on in their lives, and one of the things is they're technical people, and they're interested in learning more about privacy or encryption or security or whatever these things are, and they want to know how these things work. And once you start to be able to do that, that's when, you know, as Tierney called them, security ambassadors, whether you're deputizing security sheriffs or security ambassadors or champions or whatever kind of terminology you want to use, now you've got an idea. Now I can figure out for the 10 departments in our office in this, I'm in Vancouver, I'm people I can pick to kind of be the security point person for finance, for accounting, for human resources, for tech support, for engineering, for

labs, you know, I'm thinking in our world, you know, in my office, but we have those people. And that really helps, right? Because now there's a person on that team that has a direct relationship with the security team and isn't afraid to call ask, invite you out for a beer so they can pick your brain on something, whatever it might be. You also have the opportunity to start investing in them and go, well, we've decided that we're going to, we really want to improve our privacy stuff with relation to our customers' information. Since you've been participating in this other thing we've been doing, we're willing to send the five of you to a one-week course on security basics or privacy certification or this kind of

thing. So now you've got people you know, they got something to put on their resume maybe, right? You're improving their LinkedIn profile. You're investing something in them to help them become more of an expert. And you've also sort of commandeered their entire team in that all the people in support know Jan. Now Jan's becoming a bit more of a security expert because she's interested in it. If those people get a potential phishing email or a suspicious document or whatever kind of thing, They may not be willing to call the support team or the security team. They may not even know who they are, which is a whole separate thing we don't have a slide on.

It should be really easy to contact you in really consistent ways. But they probably would go, I know Jan knows a lot about this. This could be important. I'll ask her what she thinks. And of course, if she's not sure, she already has a relationship. She's been getting free pizza for a year. So she's like, oh, no problem. You know, I'll call Chet and just ask QUICK WHAT TO DO. BUT YOU KNOW, IT'S TRYING TO COPY FROM FACEBOOK. YOU'RE TRYING TO CREATE A FRICTIONLESS EXPERIENCE THAT'S BASED ON TRUST. IT'S TRANSITIVE TRUST, RIGHT? THE PEOPLE, MOST OF THE PEOPLE DON'T KNOW WHO YOU ARE OR MAY NOT TRUST YOU OR KNOW YOU, BUT THEY TRUST THE PEOPLE THEY WORK WITH. IF YOU CAN PUT PEOPLE THAT, YOU KNOW, SPREAD THROUGHOUT

THE ORGANIZATION THAT THEY WORK WITH, THAT THEY HAVE TRUST IN, AND THEY TRUST YOU, YOU GET THAT TRANSITIVE TRUST THROUGHOUT THE ORGANIZATION. YOU'RE STARTING TO BUILD THAT COMMUNITY OF PEOPLE UNDERSTANDING role they play in this and what they should do if they're not sure what to do when they need to make a decision.

You obviously want to encourage people to report things. It's easy to throw away information you don't want, but it's hard to get information you can't see. I'd rather have too much data and throw away what I don't need than not get the pieces I need to know about. And there's a lot of challenges with that. I mean, you probably need to, you to automate that if you've got a large organization and you're getting a lot of phishing reports into the SOC, because clearly you're not going to manually go through a thousand reports a day of, you know, is this link bad? But you do want to know about those things and often even forensically you

may not look at them right now, you may want to keep them and look at them later if you do have an incident, so you can try to determine the origin of that incident and not have that data go lost. one of the more successful things I've seen is the mild bribery, as she calls it. It's not a big prize, but enough of a prize to get people to kind of send some psychological messages to them, right? A $100 gift card to the keg will be auctioned off every month or every quarter to anybody that puts their name in the fishbowl by submitting a security incident. And all of security, physical security. I saw somebody allow the FedEx man into the building without checking his

ID and didn't escort him to the front desk. You know, I saw the smokers all come in and didn't badge in when they came back into the building. I mean, you don't want to turn people into rats, but whatever your policies are, no, but I mean, it's around what your policies are, right? What is your policy? How secure does your organization need to be? And say, like, any security incident, whether it's one of these physical things, whether it's the phishing attack that I think this could be, you know, for one, It was the first time ever I received a phishing attack that was specifically tailored to Sophos that was not a phishing test. Like, it

was a copy of our login portal that we used for some of our web-based assets, and it was criminal. That's one I, you know, with my expertise, I decided I need to flag this up right away with security team because I'm probably, if somebody bothered creating a fake Sophos login portal, they're not just phishing me. But that's because, of course, I know a lot more about this. have to report all of them even if it's just the O365 fish and the DocuSign fish and the other ones that we all see every day that are somewhat automated that may not be as important because if most people can't discern between the important one and the unimportant one, right? So you just want to encourage that and the real

message when you're putting that $100 gift card out there aside from the fact that it's the most productive $100 of InfoSec spend you'll ever get is you're also kind of psychologically demonstrating that the door is open. We are not going to We're never going to be annoyed with you for coming to us when you have a security question. If you're not sure if something you're doing is going to make us non-privacy compliant or that it may be a phishing attack or you're not sure if this document is poisoned, don't open it up to find out. Dial 4444 and one of our people will help you. Or forward it to security at organization.org and somebody will get back to you in whatever our... response time is and make sure

you stick to those things. Do things to demonstrate the doors open so that the information can start to flow because what you really want to do is you want to go from having 800 potential vulnerabilities that are clicking attachments and convert that into 800 remotely deployed sensors that are going to give you a first warning when something's starting to happen so that you can get on it. And you may not prevent it, but boy, you can detect it and contain it 10 times faster with the help of your staff than or they're afraid to report things to you or they're not aware of how to participate in the program or they don't feel welcome. Tools is another critical thing. If you give people advice and you don't

give them the tools to enact that advice, they will ignore you, as will I and anyone else. Back to adding 2019 to your password if you make me change it every year or these types of things, right? You create predictable bad behaviors instead of better behaviors. And, you know, giving people tools that they can use at work and at home is part of that. Back to passwords. One, don't have bad password policies and I've written a lot on this. If you want to know more about password policies, I'll talk later. But don't force people to change them every day and all these types of things. Also, if you're giving them advice to have the 32 character password and da-da-da-da-da, that's wholly unrealistic for most people.

Don't tell them not to write them down. There's nothing wrong with writing them. Nobody's mugged me for my password paper in my wallet. Not yet.

When I do change my password, I write it down for the first month because I do choose a complex one and I stick it in my wallet. I haven't lost a $20 bill. I'm not losing my password. Or if I have to write it down and hide it at home in case I forget it in a month, that's okay. Nobody's burgling my house for my passwords, especially if I don't write my super important corporate password on it. So give them a password manager. Buy them LastPass. Buy them one password. Give THE TOOL THEY NEED TO HAVE A CHANCE AT BEING COMPLIANT WITH THE RIDICULOUS THING YOU WANT THEM TO DO. WE'VE ONLY GOT 45 DIFFERENT PASSWARDS AT OUR ORGANIZATION WE NEED YOU TO SIGN INTO IN YOUR DAILY JOB,

BUT YOU MUST MAKE ALL OF THEM 32 CHARACTERS LONG. I MEAN, NOBODY'S GOING TO DO IT. HELP THEM DO IT. AND HELP THEM DO IT IN A WAY MAYBE THAT THEY CAN DO IT AT HOME AS WELL, BECAUSE ANY HABIT THAT HELPS their families and themselves, they'll bring back to work with them too. It becomes familiar. Well, this is just how I do. When I log in in the morning, I log into my password manager and I stick in my YubiKey or my token or I get the text message or whatever process you have. And that's how I pay my bills and sign my kids up for their school classes. And then that's what I

do when I get to work when I have to log into the web management portal for the marketing campaign I'm running. We like recipes and templates to stop shadow IT. We see a lot of shadow IT. Every marketing person has a credit card and can set up an Amazon instance in three minutes. And they will because they want to launch a marketing campaign in three days and it takes you two months to provision a server. So, find ways to make that easier so they don't break the rules. One of the more clever ones I'm seeing large companies do now is pre-configured Amazon and Azure instances that already have all of the security monitoring tools installed, the AND

THEY JUST TURN ONE ON AND GIVE IT TO YOU. AND IT GETS BUILD TO THE CORPORATE AWS ACCOUNT OR THE CORPORATE AZURE ACCOUNT. IT ALLOWS THE SPEED THAT THE CUSTOMER WANTS. IT ALLOWS THE COMPLIANCE THAT YOU WANT. AND IT'S ONLY SLIGHTLY MORE DIFFICULT THAN GIVING THEIR OWN MASTERCARD TO AMAZON. AND PEOPLE WILL DO THE RIGHT THING IF IT'S NOT MUCH HARDER THAN THE WRONG THING. can make it easy for them to do it. The other one that was rather clever that a big company I was talking to is doing now on that same front is they have all corporate Amex cards throughout the organization. So they have Amex send them a report of anybody who

bought anything from Amazon, any EC2 instances from Amazon that are billed to the corporate credit cards so they can go find people and take their instances away and move them into the program. Not everybody can potentially do that, but I thought it was rather clever. But that's what we're getting at. Be creative and think about why is this happening? Marketing is going around the process because they can't launch a campaign fast enough, and that's really important to be nimble in marketing for our organization. Can't blame them for that. What can I do to make sure that I can expedite it so that they stop going around it, but I still get the compliance stuff that I need to provide protection to the organization? And on

I'll just leave that note with one minute left according to my timekeeper. Thank you. I hope that was helpful and at least thought-provoking. And I guess we could possibly take an A question, but if not, I will defer questions to Tierney afterwards because I'm going to go home and take a fever nap.

Great. Thank you.

have a really good sense, even though identities are always the target of virtually everything, there seems to be this sense that human identities are well under control. I am teams are pretty constant, I'll say, in lots of conversations that I've had. But yet, machine identities, there's less confidence. When you go and you ask them, hey, are all your machine identities protected? The first thing they'll say is, well, what do you mean by machines, and then what do you mean by machine identities? So like I said, I'm backing up a little bit. Let me talk about what I consider machines, and we'll start pulling all these pieces together. Obviously devices, right? So my laptop and my

phone, both of those are machines. Like we can all agree on that. And that includes servers and F5 load balancers that have literally are marshaling hundreds if not thousands of different machine identities as they do load balancing as they perform their tasks, right? So servers, IIS servers, Apache servers, those are all devices and we get that. I really want you to either agree or disagree but get thinking around the fact that code is fundamentally a machine. Code by itself is a machine. Does anybody fundamentally disagree? I love to start an argument like, right, as soon as I just jump into it. So especially with containerization, we should look at code as being fundamentally a machine

and having the same requirements to be, have an identity and have that identity be protected. But then we've got services and APIs, database services. And then in the future, we're gonna see algorithms and blockchains as being fundamentally machines as well. I'm not gonna talk about blockchains. I'm done. I'm done with it. But we need to think about this constant evolution of machine identities and what constitutes a machine. So how do we establish machine identities? This is where we get into the subject of this talk. Obviously, probably the most common, the way that comes to mind for most people is through SSL TLS, right? And if I say, how do you identify a machine? You go,

oh, TLS. Yeah. Or if you've been around forever like me, you go, oh, SSL and that other thing that we call it. So SSL TLS certificates, SSH keys, right?

to identify to establish a machine identity. Increasingly code signing certificates. Does anybody have in there where they work today? Actually, I should ask. Are all you guys are in InfoSec? Probably, right? Does anybody have direct responsibility for code signing? Code signing operations? Anybody? Rapidly growing. Almost all of our customers are saying, hey, you need to help us sign code for internal use in the development streams as well as for their customers that are downloading apps, right? So, code signing is another way of establishing machine identities and then mobile and IoT certificates, which kind of blows the mind because it takes the whole notion of X509 certificates used there and dramatically changes them, changes the way that those are

identified, how we establish them, how persistent, how ephemeral those are. But we're going to focus most of this talk on SSH keys. That's what we're really here to talk about today. And I'm going to jump to another metaphor. And this encompasses everything around machine identities. I hit the blank button. I love metaphors. So icebergs, like in, I have a middle schooler, actually she's a high schooler now, but in middle school she learned that like what percentage of an iceberg is below water. Anybody remember? Thank you, 90%. Everybody seems to know that. 90% of the iceberg is below water and 10% of the iceberg is above water. That's how we're starting to look at identities, all right? The part above, the iceberg above

the water is kind of the number of people that are in those networks. And you think about all of the machines that you connect to and the identities that need to be established and validated as you say, go online banking from your phone to even just look at a balance transaction. That's a lot of machines between code devices, your phone and everything else that are part of that system. And what we're seeing is that the network part is growing exponentially, right? Relative to human growth. In fact, humans are pretty flat between right in the next five years there's gonna be a marginal increase in the number of people actually establishing and using identities in networks.

There's gonna be a significant increase in the number of devices, almost fourfold. And we can all see that, that's readily apparent because I see phones and laptops all over the place. What makes it even more challenging as you think about this exponential growth in machine identities and how well we're protecting them and how we're securing those identities is when we also add in the layer of applications. The whole scale changes, the scale on the left changes and if we think of an application that requires, that has an identity and that requires protection, things get really crazy. It's about 30 times the number of, in the next six years, the number of people when you start

looking at just applications. And when we talk about machines and machine identities and SSH being a component of that, this is what we're talking about. This group here on the right, which is all concept of applications as well as devices working in concert and working separately. Not difficult concepts, right? Everybody with me so far? It all kind of makes sense. I have a really quick, simple poll, okay? participation. Everybody has to participate. Sorry. So if you can spell SSH raise your hand. 100% alright right. So now leave your hand up for a second. Leave your hand up for a second. Alright so if you're an occasional SSH user leave your hand up. Just leave it up. If you're a frequent SSH user creating SSH connections, keys,

pairs, leave your hand up. like a server, SSH server administrator, leave your hand. Okay, you guys need to do this presentation. So this is where all the tough questions are gonna come from. So part of the purpose of this is for me to help map what we go through in this. This can be a pretty dense topic and I've got some great visuals that show how SSH keys work, also shows where the risks are around managing SSH keys. to find the right path through this given the different levels of experience and commentary. If I go too far afield, please feel free to ask a question or stop me dead in my tracks too. So background around SSH. So kind of the protocol

that replaced Telnet, it's been around for a long time. It's not quite, but almost as old as I am. Secure Shell, right? Cryptographic protocol for operating network services and devices. to almost connect to anything within a network environment. A few basic things about it. Typically, standard TCP port is port 22, highly configurable, okay? And most people who are using SSH don't, I say most without empirical data to tell me how many, but in most conversations I've had, don't understand the configurations and parameters available around SSH. Or even the fact that it's got versions, that older versions have vulnerabilities that don't exist in, you know, most recent versions of SSH. Often originally used to access Linux and Unix systems, but now of course

it's in Windows, right? Since Windows 10, there's open SSH is used in Windows 10 to provide SSH connections. Which basically means that every system that you have, you can create SSH connections with, right? Machine to machine connections. A lot of similarities with SSL TLS, which is sort of the predominant type of machine identities. I'm not gonna deal with the things on the left too much. But a few significant differences and to me, the biggest difference apart from anything that it says up there, SSH is very very powerful relative to say SSL TLS. It comes with protocols that do things like SFTP, the secure file transfer protocol and secure copy protocol. These are embedded in SSH. When you use open SSH or something else you

get these capabilities with it which makes it potentially more powerful I think than TLS or SSL TLS. Second key thing is SSL TLS certificates as they come bound a whole bunch of metadata around the key itself and how it's used. And one of those things is when it expired. When do SSH keys expire? Never. SSH keys don't expire. You make them and they live forever. And then you start thinking, oh, well that's where the security risks start coming from. That's one path that comes from. So I'm gonna lay out some data that's available and actually at the end of this presentation I've got a slide share link that you can go and you can get to all these slides and

you can also see the references. This is from a survey that was done in 2017 that we leverage frequently. But when we look at SSH usage in the wild, especially in large organizations, global 5,000 organizations is incredible. Almost 30% of organizations, large organizations, SSH to manage a thousand or more systems. Now I'm starting to see that as actually a small number when we start looking at some of the big environments that we go into. It easily goes into millions. I've got a really interesting story to share with you in a little bit. I was gonna say funny, but it's not really funny to them. Enterprises with large SSH environments are three times more likely to use automated connections between all their devices

and systems. So that just makes us start thinking about machine identities and their connections. and taking people out of the process, right? Taking people out of the process. The challenge typically with SSH, and especially if you're SSH admins, I'll throw this question out to the audience, is that typically, again, sort of no empirical data, but in most of the conversations we have, nobody owns it. It's not like the CISO goes out and says, I'm gonna buy an SSH solution, right? There's SSH and it comes with most of the platforms and most of the operating systems that you're using. just use it. So nobody owns it and it's free and it makes it really easy to overlook its important or overlook its risks. Any questions so far or commentary about

that? Everybody's like, last session of the day, I just want to get through. I feel the same way, I feel you. So here's my story, so one of my stories. This is a true story, I learned this about six months ago. This guy whose name has been changed to protect the innocent is actually the CISO. Jim is CISO of a company that's got a two-letter initial and it's global and they make, you probably have appliances they make in your house and they make aircraft engines. I'm probably giving away too much and they make turbines. So other than that though, you probably don't know who they are. So he's the CISO and he's part of our

CAB, our CISO advisory board that we meet with two or three a year to talk about okay what's really bugging you like what's keeping you awake at night and what kind of things should we be working on and a couple years ago going back on four years now he was presented with a problem and he had to go to his head of infoset Chris and said hey they're changing audits so our PWC audits were kind of making up their own security constraints as they go along their requirements is changing it and now we're to manage, we have to show accounting for all the SSH keys used in our critical infrastructure, originally just critical infrastructure. So

Chris said, I have no idea. I can do some, we don't track that. Like everybody gets to make their own keys and manage their own keys. What do you, there's no system to go to. So he does some back of the napkin work and he said, okay, look the number of development environments in the people was probably 500, 400, 500,000 key pairs. So it went back to Jim and Jim said, okay, well we've got to know in 60 days and that's where this phrase actually came from originally. So 60 days go by, Jim comes back in their regular meetings and he says, okay, okay, what did the assessment say? And Chris says, okay, did I say half a million pairs? True story.

digging their own manual work weekends, they found six million key pairs in the different estates that they have. Okay, six million SSH key pairs in their environments. Everybody was effectively blown away. It changed the calculus completely. Everything went to the COO and the chief financial officer in order to pass the audit. And they basically said, okay, we need a plan like ASAP to get this down to two million key pairs. And they have to be justifiable use and they have to be accounted for and we have to track them. do that? They did it because of accountability. They did it through actuarial analysis to figure out what the risk was. What is the risk if you've got six million key pairs out there and you don't know who actually

has them or what they control? Well, we can justify two millions is how they kind of backed into it. It's not bottom up from the number of projects, it's top down in an actuarial analysis, which kind of freaked me out actually. But they looked at how much risk they could actually afford. So how did this even happen? Let's talk real briefly about how SSH is used, especially, so this is the primer on SSH. Guys that are really experienced in SSH, check your mail right now, we'll be right back to you in a second. So where's SSH used? If TLS is used on the front end, right, of your systems to access applications, web services, and

so on, SSH is effectively used on the back end. So your system administrators, everybody that's administering on your teams, they're using SSH typically to make those connections, machines like I said secure file transfer protocol, secure copy protocol, other machines in the environment and all of the people that you rely on in your ecosystem, all of our partners, people that we work with, they're all using SSH to make connections, not all of them, most of them are using SSH, they have some SSH connections to make their systems work and work together, okay? So you could view that, if you look at this and you think okay well that's interesting, go back to the question about who's

actually responsible for these in that same survey this is the dimensional research study 59%

of the organizations that we talked to basically said the admins are managing their keys so they're making these keys to make those connections these three guys here whoever sets up the other adjacent systems they're managing those keys they're keeping tracking storing all those keys okay so how does that actually work and this is the part where it's long since I've done this that I tend to state this wrong, but it's a cool diagram, right? So bear with me. There's a server. Bob owns this server. Sally has a client that she's using to connect to this server. She's providing information. She's running processes, okay? She actually has an account on the server. They've set her up.

Bob has set Sally up on the server using privilege access management or some other access control. We'll come back to that too. She's still tired of entering her passwords or her credentials all the time. She's got to get in there like 10, 12, 15 times a day, right? So Bob says, well, or actually a peer of Sally's, let's call him Jim, says, hey, why not if you can SSH into that system, okay? You can bypass those credentials. Now SSH can have its own credentials and a password, but the whole goal is to avoid those, right? That's actually what she's trying to do is avoid those. hey Bob I want to do that and he goes

well there's an SSH directory it's SSH d underscore config and I'll let you know that it's there we'll set it up and then you just contact it you download the public key and things get started we're often running it creating an SSH connection okay her client by the way and she makes SSH underscore config which is the client sort of the client SSH configuration this is It always says hey, we just got this public key, is this the right public key for the server that you're trying to connect to? What does Sally say? Of course she said, yeah, does she ever say no? No, she never says no, I love this. It's not even a security flaw at all. She says yes, and then she actually creates the connection

between these, then she creates her own public private key pair without going into a whole lot of detail, it's stored in the office, her. public key is sort of the authorized keys file on the server, she keeps that. Private key and obviously then when she goes to make a connection it compares to the private key and public key and boom, Sally's in without having to use a username and password or any login credentials. If that's what she wants to do, right? You can create SSH connections that actually require a password as well, most people don't. Because you're kind of obviating the whole reason why you're trying to create an SSH connection in the first place,

especially if it's automated. goes, hey, I have this other system that I use too, and I need that system to run a script against the first server probably at least once a day, sometimes twice a day. So why can't I do the same thing? So there's an SSH D underscore config on that. She gets it set out, she downloads the public key, it asks the same question, hey, is this the right public key? And of course she says yes. So she goes through the same process, takes her private key and stores it, I'm sorry, her public key and stores it in the authorized keys files, talks confusing, it's hard for me to keep up. And

then she, on server two, basically creates a client connection. Same thing that she did on her system. And that's where things get really interesting. She makes that SSH config on server two, which connects to server one. She now has the ability to take that key, sort in the authorized keys files. She has an automated connection from her client to server one and to server two. Life is really, really good for Sally at this point. All this is automated and she doesn't need any human intervention. Okay? it a really condensed, simplified way of what SSH looks like. But this is why people like it, right? I no longer have to log into it. I can have

relatively good security without oversight and control, which is the question of how good the security is. But, assuming you've got some kind of oversight and control. But what we start to see is the beginning of how did my friend in the two letter company get to six million key pairs? And there's some other significant drivers that are actually pushing that business too. And it all started with a desire to simplify access, to actually make this easier. So one more diagram to go through, because I want to now start talking about, okay, we've done this. One of the worst things about cybersecurity is we try to like scare people all the time. You guys noticed that? So I'm gonna avoid that. But it's important to talk about risks.

And not all risks are, as risks. Not all risks are like everything has to change, throw up our hands risks. But there's a lot of risks to enumerate in SSH, right? So you can, if you don't have the right controls in place, setting up unauthorized SSH servers, again, that's a risk. SSH obviously is software. Software has vulnerabilities and software has versions. Software also has configurations. You can have configuration vulnerabilities, right? Again, I'll go back to, does everybody that SSH version 1 has significant vulnerabilities, right? Hence SSH 1.1 and 1.2. So what is actually running in that software? Obviously the ability to have unauthorized access is a problem. Taking keys themselves. Keys is a digital asset so I can copy

it easily and that leads to little things like a terminated employee access and having backdoor keys. Those are significant risks. like just on the edge of five o'clock so that's why I'm talking fast so I skipped a piece in here but if you guys want a really interesting story about this terminated employee access Google Hyatt Regency Queensland breach no Hyatt Regency Queensland hack at some point in your life okay and that talks about what a really embittered former employee does with SSH keys that they've copied and reused and this it was like reversing sewage systems into public grounds. He ended up in jail for two years, but it's still a funny story to tell. Was that why? Oh, wow, yeah,

sewage. Yeah, well, he was, and he tried like 20 different ways to like really get at them, and that was the one that was easiest to, it was the hardest to clean up, too. So, Backdoor keys, root compromise, privilege escalation, all of these are sort of ongoing things. Weak or guessable keys. Go back to Sally. Sally is a system administrator or developer. Does she know anything about PKI or what the difference is between strong and weak encryption or anything? No, we don't expect her to either. Yet she has the power to create the keys. So is she creating weaker guessable keys? Oh, this is a favorite, port forwarding, right? friend of Sally's says, I wish she had to go through a lot to get access

to that server, right? So just sort of describe her rationale. Well, I need to load stuff. I need to connect to it too. She goes, well, I will just create a backdoor for you basically. That little forwarding over there. You just send it to me and I redirect your request. The request will come from me, but it will actually have them coming from you. Another security risk. And then I think, oh, yeah. So again, we like to do breach shaming and talk about stuff. We all know that Equifax was like an Apache stretch of vulnerability, right? And everybody like fully stops right there. Of course then they pivoted probably through key compromise to end up

compromising 53 different databases, right? So pivoting is a significant concern. A lot of it starts with the security of your SSH controls. And then man in the middle attacks. So if you take all of these up and you put them together, there's 13 key risks around SSH control. and controls and their related vulnerabilities. 13 key risks. There's probably more than that, but for the sake of conversation, let's start talking now about how we get in front of this. Because the challenge with this is that going back to the dimensional research study, most organizations don't have a really good inventory. Less than 10% have a good inventory of how many systems are SSH enabled on the server side, how many SSH clients

are running, how many people are actually involved in that, how many people are using that. worse if we look at privileged access and how that's actually controlled, right? Removing keys when employees are terminated or when they leave basically. So yes, almost 60% of them do. That means 40% don't. They don't have a plan in place to actually rescind or remove key or even rotate a key. Or rotating those keys, the second in there when an administrator leaves or they're reassigned to something else, so 61% say no, they don't rotate or change out keys, right, when that happens. So again, lots of risk being uncovered there. We end up with what I would just call the typical state of

SSH, right? So no inventory, we have no key rotation. Everybody knows what I mean by key rotation, right? Especially people using SSH. How often should we rotate SSH keys according to Ms. Let's say?

We have a funny scary story. We have discovered keys in environments that are literally six years old. A key that was created six years ago, SSH V1, never been replicated. And if you don't have visibility, that's the kind of thing that happens. So how can you make accommodations for weak keys, meaning how it was actually encrypted or guessable keys or no key location? So, solution now in ways that we can sort of get to visibility, get some intelligence around the types of configurations on those keys and then get to some way to automate, okay? Before that, everybody at some point also says, hey, what about PAM systems? I'm like, Pam, Pam, what are you talking about? So obviously, privilege access

management, right? Not that Pam. interesting thing. Privileged Act, who actually in their organization has a privileged access management system? Phycotic, ArcSight, I'm sorry not ArcSight, why did I say that? CyberArk. You have CyberArk? Who else has got CyberArk? CyberArk's great. We love working with CyberArk. The challenge is that that's about what I expect. About 10% of the population is actually invested in a PAM solution. Now PAM's solution are made to manage privileged access for individual users. And essentially an SSH connection is a user, it's a machine user if you will. Some people know that I'm working? That's probably because they haven't gotten that picture yet and they're like, you're not actually working. And machines are

fundamentally different. So there is a notion, and I want everybody to carry this away, that there's privileged people and there are privileged machines and they're not the same. not the same things. When we do a discovery or anybody does a discovery of SSH keys in their environment or SSH servers and you start looking at relative risks, how many of those are routed, how many of those connects to other systems that potentially have route access and the mapping, doing a heat map of just SSH connections is shocking and it's always very different than the individuals and the people. So, the last note on here is even if we're doing it, we're trying to keep those maps

together, but typically when Benafy, for instance, goes into agreement with CyberArk, they've got one piece of the puzzle, you've got another, and they'll say, well, hey, we do entitlement reviews, but typically, we only do them like, half of them only do it annually. And if you think about the frequency of your connection to other machines and other databases, especially in DevOps and fast IT environments, you can't keep up with those kind of connections. in terms of looking at a way to get to it and I'm going to talk about a process more than anything but how do you achieve visibility and get some intelligence on it and get some automation done? Visibility basically means how

do we find all of the keys, how do we find those six million key pairs in our environment as I have them go to the expense and pain that that company went through. How do we apply intelligence for that? Like intelligence is I have an audit of all my keys, how are they configured, what are the What's the expiry on those keys? What systems applications are actually using those keys? What systems would break if we rotated that key? That's a great question, right? And the reason that keys don't get rotated very often. And leading to the goal of automation. Automation is hard because of that last point. Very often you have to do a lot of intense listening to systems and SSH sessions, if you will, to see

ones are actually active before you can start enforcing things like rotating keys. But what we'll recommend to people is like, start with a risk assessment. Do a risk assessment and do some discovery and inventory. And at every point along this, you have a choice. You have a choice to make where I can either say, okay, this is so bad that I have to put a tourniquet on it, which is effectively stop the bleeding, right? Everything stops. We've had customers just say, okay, we're gonna stop allowing anybody to issue SSH keys and connections as of like today. gotten out of control that's what the two letter company did when they discovered six million compares. It didn't last for very long because of course that stops everything. Or you can clean

up the mess. You can say okay so we know there's a mess, we know there's problems but we're gonna go clean it up. The next thing in terms of intelligence is establishing standards. Can we create a policy that says what SSH connections and keys should actually look like? What is the key strength? What encryption do we use? Who's author to make those SSH keys? What is the standard expiry? Do we do six months on every key pair? Do we ensure that that's happening on all those? Do some sort of continuous monitoring of what those standards actually look like, okay? And then ultimately bring it around to this goal of doing some kind of automated remediation, which scares the bejesus out of everybody, and it should because things

can break. But again, automation is the path to sanity. It really is. I'll use another example that's not SSHP related, it's TLS related. But

again, this is not breach shaming. This is just, there's these in the world. They happen to be a customer of ours. Equifax is a customer of ours, right? So yes, Apache Struts vulnerability that was exploited. Why didn't we see that? Because they have SSL traffic inspection devices from Symantec, really good equipment in there. Why didn't they see the attackers in the SSL traffic? a inspection device requires a certificate. The certificate expired 10 months before the attack, right? Had never been replaced. They basically had a very expensive, dumb device that was sitting there, right? So this is my argument for remediation, finding a way to do the automation, because we can swap that out. There's lots of smart tools out in the world that could go in

and swap out a certificate. You can do the same thing with SSH keys. The challenge is, and this is what scares everybody is, I don't know what I'm gonna break because they're not well documented. create this system where you're constantly listening, again, look at SSH session traffic and say, hey, this device, it's got a key pair, it's got a key at least, maybe it's an orphan key, but it's not connecting to anything. So let me show you sort of graphically what that looks like when we put it together and the kind of results you can get out of it. So we'll, this is, again, I'm not gonna say we, but this is anybody can actually do this. Start doing a discovery. Find out

where all of the SSH servers are in your environment, right? So that's gonna identify the pieces on the right, basically. And it's gonna go to a centralized repository. It's gonna identify all of those SSH servers and then using a combination of agent and agentless technology, depending on what you've got available and you can leverage. Start mapping all those relationships out. Find all of the keys related to all those SSH servers. This is not easy and it takes a while to do. Eventually you get to a point where you get to report this up and it becomes part of your compliance rule and you can start talking about how you're gonna go from six million to

two million compares, et cetera. But this is the key right here. You've gotta get visibility into it. You've gotta start seeing actually what it looks like in your environment. This is a off the shelf system, but there's tools out there and this can be done. This can be put together, especially in smaller or mid-sized environments.

that are out there that will help you label and identify all these SSH connections. This is what we're really looking for right here though on the right. How many of these are SSH v1? How many are using password authentication? So why is that in red? Why is password authentication bad? It's not actually bad. The reason it's marked red in this particular report, it tends to be weaker and more exploitable actually than encryption that's using SSH keys. SSH if they're set up to use either just a password or combination of password and it's digital key, there's questions about ownership of it, responsibilities for it and everything else. There's also a bunch of open interesting attack vectors. But you get to look at things like, okay, how

many keys do we actually have? How many orphans do we have? So a key that has no representative pair basically, where did the other pair go? Post orphans are particularly Another thing that's worrisome is duplicate host keys, right? So, like I've got a bunch of servers that all actually have the same host key on them. And there might be tactical reasons why you wanna do that. From a security perspective, obviously if I get one, get all, you see how that works, right? It's that snowball thing. You'd be surprised, possibly shocked at how often we actually see that, whether it's duplicate host keys and sometimes high numbers of them.

what the path looks like. So whether you're using a off-the-shelf tool or not, get some kind of discovery and inventory of all the SSH enabled servers first, find out what they're connected to and then get to clients that are using them. Provide some kind of visibility to them. If you're under some compliance mandate, could be PCI, GLBA, GDPR, whatever it is, actually all of those have encryption requirements that touch heavily on but on SSH keys and key sets specifically. If you need armor to go in and have an argument with your boss about why you need to spend money on tools to map SSH keys, there is a NIST document, IR 7966, I think is the name, or you could just search NIST SSH. That's about four years

old, fantastic parameter on like why you should go in and establish policies and rules around your SSH configuration. How do you establish policies you're not If you don't have to have a compliance policy, it's better to be the one that brings it rather than be the one that has to comply with it. And then get monitoring and reporting going, get notifications out. Some systems can just do syslog export. This is interesting too. We, let me step back and tell us, I'm sorry, I'm gonna step into an FI mode and not make it a commercial. We have this fund, it's called the Machine Identity Protection Development Fund, where we fund developers to go things to connect different systems platforms and environments in our ecosystem, right? Very cool, right? Like I

said, SSH connections are particularly problematic. So we worked with a company called Open Credo in London. Has anybody ever heard of Open Credo? Really cool developer that's built a lot of connectors and tools. So we funded them to build a Kafka connector that basically says, okay, here's all of your SSH keys in your environment, your servers, and that that sort of histogram of what the connections look like, what the relationships look like with potentially vulnerable servers or critical systems and infrastructure. And they actually map that out now using this CAPTA connector. It all comes from that monitoring and reporting and getting some output, getting some really simple output of the system of inventory. Entitlements report,

which you can usually get from your PAM system. Map those up, say PAM says this about users. Now what of those users are also using SSH connections in systems? And you start doing their reduction. And then ultimately you get to the remediation and the management of it, okay? I can't stress this enough. This is a really big deal. I have never, never is such a strong word, I've almost never been into a customer environment where they have a strong policy for rotating key sets and actually follow that. And when we sit down and look at it, look at the length of time since keys have been rotated and what that means just from a pure risk perspective that people usually are somewhat shocked. So rotate

key sets, even though it's problematic. Why rotate key sets? Okay, two things. I have two concepts I wanna leave you with and then we'll open up for questions and then maybe they'll give out free drinks, I doubt it.

Butter. This is really recent too. It's like the old becomes new again. You guys familiar with the butter attack? that's actually an attack on SSH authentication. It's a brute force attack on SSH authentication. Grint Butter is fascinating. If you get a chance to look at it, Butter's like five years old. The attacks only come from four IP addresses in Singapore, Southeast Asia. It's like Singapore, maybe, not China actually, but in that area, very limited,

where the IPs come from, very, very smart, but they basically brute force SSH connections that are user credential authentication rather than using digital keys. Brute force those connections and then they establish remote access Progen. They just sell time on that basically. And they'll do some crypto mining on that. Sometimes they'll engage in other DDoS attacks and things like that. So the first thing about this is, we wanna use digital keys instead of user credentials. Especially if you have a, again, if you're responsible for creating a policy and the policy can be, you might say a policy is both, you need digital keys and you need a credential, but you'll get pushback because that slows people around. But this is why digital keys are almost

always better than using user credentials. You can go in and rotate the keys. You can create, enforce a policy for key rotation and go in and make sure the keys are rotated and you're less susceptible to things like that I wanna bring up. If I go back to my story about two letter company, how did they get from 500 million to six million? Wild guess. Anybody know how they got to that so fast? It's just a two syllable one thing called DevOps. So for the last three years, I love DevOps. DevOps is our friends, right? Really do. For the last three, four years, this movement among devops practitioners, because this is what they're trying to do. They're trying to

get fast access. And so they've been sort of self-educating themselves as they go along about how to get SSH access into systems. And it's really cool. It's kind of, it's like when I ask people about how they learned about SSH and managing it, they're very often say, well, in the devops team, actually, that's where I learned how to do it. That's why it came out, right? So that's one of the reasons why it's been proliferating so much, but they talk about it a lot, right? So, and there's lots of of great examples and they'll give you specific guides online about how to actually set up SSH connections to speed up and enable DevOps pipelines and how to make that work, right? Not a bad thing, there's nothing wrong with

that at all. But as security practitioners, we just need to be aware and we have to have a conversation, okay? So we've got, now we're down to probably four minutes, hopefully we'll have some time for questions. I wanna give you five things that you could do literally tomorrow, that you could go back office and start thinking about relative to SSH security configuration and risks, okay? Unless you know already, find out if your organization has a machine identity protection plan of some kind. They might have rules for, say, SSL TLS, but they might be ignoring code signing and they might say, well, it's just developers signing code, just let them do it. They might not have anything on SSH, just find out. the question,

should SSH be part of that plan? I'm gonna say yes. If you have a governing policy, right, NIST is great because they're very strong on very prescriptive standards for how to use and configure SSH controls. PCI less so, GLBA almost, you have to kind of read a lot between the lines, but if you have a governing policy that your organization is responsible for, say, hey, SSH is a part of that. SSH encryption as part of that, how do we make those actually work together? And then you start gaining visibility. So again, there's tools and off-the-shelf solutions that can do that, or you could do it the way that the two-letter company did is you just get everybody involved in it and start making an inventory. You

start going basically system to the system, developer to developer, and start. I would not recommend that anybody build a massive ugly six million row spreadsheet, but that's actually the path they started going down. Now they have a much more automated process. Gain and

Understanding as you're building that out of actually the intelligence around those keys, right? So insight into how those keys, it's one thing to know that you've got a key, but you've gotta know, okay great, how was it encrypted? When was that key made? What is expiring on that key? What does that key give us access to? Which systems, what applications? That's the intelligence around it. Start a discussion about automation, because again, everybody's afraid of it, especially with relative to SSA keys.

In other words, find out who in operations is gonna be most concerned, who's got the most angst around it. It's amazing how much you can shrink attack surface if you just get your arms around SSH and what it exposes. So you just start having the conversation. And the bonus six thing that you can do, this is very, very, very sensitive. The bonus six thing is talk nicely to DevOps about their usage of SSH, right? Explain it. who's actually using it and how are you using it and how are you, when you make keys, how do you make those keys? You have standards for making those keys. Obviously we cannot slow DevOps down. We don't want to slow DevOps down, but you gotta start with a conversation. Like what is

your reliance? Hey, if somebody put controls around your SSH key usage today, how much would that screw up your CICD pipeline in the work that you're doing? It's a great conversation to have, because when you scrape them off the ceiling, they start talking about, you know, reasons that they're actually using it and what it means and they start being very conversational about ways to you know ways to get their arms around that and solve that problem so that's the wrap-up signal I actually did go almost exactly to five o'clock but I have time for questions if you guys have time and yes sir

That's a great question. He talked about Uber and their recommendations on creating an SSH CA. So there is a movement of foot. Like the reason that SSL TLS is, well, there's a lot of reasons. It's fundamentally different. But like I said, you can attach a lot of metadata that includes expiry. It includes just a lot of information about the certificate strength, where it comes from, chain of trust, right? There's a movement that started by Google and others to start making SSH certificates. It seems to be well, well thought out. interested in identify our customers are not because the two things are like oil and water they go wait that's I say you mean like I'm gonna have to wait from a for like a

CA like I have to for SSL certificates and like no no no that's not what we mean but as we have the conversations there's a lot of friction in our customer base and it's starting to get a lot more airplay Paul Turner on our board who's also with this talks about that subject a lot

getting kicked out. So thank you.