Reducing operational toil when responding to your next critical CVE

I'll hand over to our next speaker who is Ben Robertson um he's speaking on reducing operational toil when responding to your next critical cve thank you Ben big round of applause for Ben hello everyone I'm here to speak to you about how to reduce operational toil when responding to your next critical CVA uh first for we get started I'll just touch on briefly what a cve is uh so cve stands for common vulnerabilities and exposure and it's run by the MIT Corporation and cves are assigned a number by cve numbering Authority so when a vendor or has a vulnerability they will apply to CV number in authority to get a cve number which consists of cve plus a year that was

released plus some arbitary digits at the end and cve come out with a severity rating uh zero being the lowest which don't think you any of them have and 10 being the highest these are made up of a combination of um how easy it is to exploit the vulnerability whether it's remote or local unauthenticated or authenticated and what the impact of it is uh so who am I I'm Ben I work at puppet as a Professional Services engineer advising customers on dev um devops practices in a post sales role uh many of the customer engagements focus on OS hardening and process automation previously I've worked in commercial and government gateways and had a lot of

exposure to Blue Team activi ities so OS hardening perimeter defense setting up logging aggregation IPS and things like that and also fixing cves when they come out so I'm just going to touch on a few older vulnerabilities a slightly new one and then talk about how we can reduce operational toil while trying to solve one of these as a as an example so here's some vulnerabilities from the past shell shock heart bed and log for Shell uh shell shock and heart bed were both from 2014 these were quite big they both made mainstream news like ABC News New York Times And if you're working in it operations or security you would remember these vulnerabilities from

these days and log forell was 2021 so much more recently so just got to be close uh so just to briefly touch on shell shock that was a vulnerability introduced into bash in 1989 so it had existed for about 25 years before it was discovered uh consisted of of six CPS in total uh the initial was rated 10 uh the reason for the five additional cves is once our security researchers started looking into um the vulnerability in bash they found a lot more similar vulnerabilities um and why was this a problem this primarily affected um web servers that were using CGI bash Handler programs so the web server would take the input from a user and pass it

through to A bash program the problem was um when it's saved that variable is an environment variable if you um formatted your input as such uh with the brackets and more brackets and uh semicolons uh bash would actually execute the command that was run after it uh it was actively exploited in the wild um cloudfare offered free protection for their customers and published some useful stats on the attacks there was seeing around 10 to 15 attacks per second at one stage and that was legitimately one of the attacks that was seeing that users were trying to eject CD drives from servers which is quite funny um but most of the attacks were using used for

information gathering purposes like CAD Etc password who am I try to see who the web server was running as try to see if they could um connect out to their own servers and on top of that 8% of attacks are used to gain control the web server so they try and set up a remote access program and some of these were quite smart as in they would remove the program from disk once it was run in

memory one two sorry uh in the real world there it's hard to get information about what organizations are affected by these vulnerabilities they don't usually publish it but uh we know that a a server used to run the elections in the state of Georgia in the US was compromised in December 2014 and this wasn't discovered until 2017 and that server was used to transfer electoral uh voters registration files across the state so yeah these effects are real uh the next vulnerability we're going to quickly touch on is heart bed which was a vulnerability in open SSL open SSL can be found in uh pretty much most products things like Cisco products F5 in Linux AWS Android

smartphones Wi-Fi routers so it's very widely used uh the vulnerability got its name from the heartbeat request of the TLs protocol that was used to keep connections alive and it was this heartbeat that was exploited uh so a client was able to send 64 KOB of data and they would send a length attached to it but what it turned out you open cell wasn't checking the length that was being sent in so an attacker could send 1 kiloby of data say it was 64 Koby and get 63k data back from the server's memory um and the user the attacker could run this again again and again to keep getting more memory from the web server um so this improper input

validation allowed um attackers to read things like private key material username and passwords or anything in memory on the web server unfortunately with so many products and companies relying on open SSL you'd assume the product was well maintained and supported back in 2014 uh however unfortunately I had a single full-time developer working on it three part-time Developers in terms of funding to the project they're getting around $2,000 per year um I think it did get a bit more funding after this obviously with the focus on it but back then it did not g on further sorry um so with the so few resources that I M eyes maintain in this project uh code that was the heart bed exploit

uh was accidentally included in February 2012 and it was released to the public and it wasn't discovered until April 2014 the solution for heart bed is the same with shell shock was to apply an updated package to the system however with the heart bed uh private key material could have already been stolen so it resulted in a lot of uh websites and online services need to roll passwords and private key material a few notable organizations that were affected was Yahoo who actually advised users not to log in for I think it was about 6 hours and to change their passwords after the Patch had been done and the Canadian Revenue Agency uh lost 900 social insurance numbers on a web server

in a 6-hour window and they had to end up shutting the site down so the last exploit we're going to briefly touch on is a log for Shell uh this was also this was rated 10 for criticality as well um it was a vulnerability in the log for J Java Library these up it's better um if the application logged used the input using log for J so it was a web server and it was taking user user provided input you were able to exploit that web server so attacker could load and execute malicious Java code orchestrate secret environment variables from the web server uh this could easily be achieved by placing the malicious string within

the HTTP request as lucky web URL and the tenable CEO labeled H log for Shell log for Shell as a far the single biggest most critical vulnerability ever it affected many vendors uh both cloud and on premise and Ernest and Young estimated 93% of cloud environments are vulnerable and after 10 days after the Patch release average orgs had only patched 45% of vulnerable systems um in terms of attacks that we know of um they believe Iranian government sponsored actors compromised the US uh government Department through an unpatched V Horizon server and were able to move laterally through that department um quite easily but uh why was uh log for Shell so much worse than the

others it was difficult to scan uh for log log forj uh because it runs within a Java application and it's not managed by uh system package management so things like your standard sort of patching tools um for Young dead packages wouldn't be able to have visibility of it uh Java libraries are maintained by the application vendor so therefore you need to they he needed to publish a new version of the application for you to get the patched version of log for J so it makes identifying the vulnerable applications difficult cuz not all Java applications were vulnerable used and Used Log forj or may not have been configured in a way that it was vulnerable you need to know which ones

were vulnerable and the versions of that software to actually know um fortunately there was a workaround for this one you could um disable J&D lookups or remove that class from the jar file um yeah next one so luckily with log for J because it was hard to find Google released an open- Source um scanner and it's compiled for Windows and Linux and it was quite simple to use you simply downloaded the executable extracted it and then and then ran it and with a path argument of which where you wanted to scan on the system so you could limit it to program files or the whole root dis uh it also came with an option to

rewrite so I could actually remove that vulnerable class file so it was quite useful in that way um the unfortunate part was uh you had to SS H onto every single system or or IDP if you had Windows to run it and with thousands or tens of thousands of servers that's a huge daing task if you're not using any sort of automation um even if you had 1 th000 servers 10 minutes per server you'd be looking at multiple weeks with um One tech doing working on it and that's not even to remediate that's just to identify ify them uh so CV patching costs organizations millions of dollars per year and they still miss and leave

systems unpatched for long periods of time um and that results in losses from old cves which really shouldn't be hanging around anymore so what what is the solution love say's a better way to do this so we want to automate the scanning so we don't want to put our operation stuff through this manual repetitive um n numbing work so we want to use automation where it makes sense and just one thing plagiarism is bad at school and University but at work it's okay as long as you're not breaking copyright so we don't want to start writing something from scratch so let's have a look at what we've got here so we've got the Google scanner which is a binary that we

can ship around and run and in now case I work for puppet puppet build bolt which is a free open source tool which allows you to automate BK scale across a wide range of servers so let's just quickly touch on the two different types of puppet um so puppet is a configuration management tool for primarily windows and Linux systems and we have two types of puppet so decorative puppet where we specify the desired state of our system and puppet figures out how to achieve that um it's it and potent so it won't continually keep running the same action again and again so if you say install package X or write this file it checks see if the file the package is installed

and doesn't continually Do It um there's also on demand puppet where we specify the exact steps for what we're doing so we might say uh user ad Ben uh yum install this shod this file Etc so um puppet can run with an agent or agentless however bolt uh does not use agents and pushes configuration changes and updates via SSH and winrm so we're able to run this in an environment with where we don't have an existing installation you can use bolt to uh automate simple tasks you can use it to run a python script or bash script or Powershell script doesn't really mind what you're going to run in um so I've developed a very simple bolt plan which

uses the puppet language as I mentioned earlier you could have done this in any language and yeah um I'll leave that QR code up for a second and then I'll jump on to the next slide oh hang on this is where I need to switch and that didn't work

sorry okay uh so this is the uh V plan I was talking about so it's not particularly long 100 lines long and we have some parameters at the top so we take a a list of targets is that font large enough or try to make it larger like it's okay guess I can't really see it from here um bigger okay that bigger yeah probably go there okay um so we take a list of targets we have a couple of other parameters that we don't need to set like where we want to install it so C or for/ temp on Linux so you can feel free to move it around if you have disc space requirements and

then we have a scan path so obviously for Windows scan C but if you had other drives you can change that uh Linux we just scan route nice and easy um so we uh run the plan so we collect the facts about our our devices our servers so we get a list of a variable that has the Linux systems and a variable that has the Windows systems um so the fact is just information about it like how how many discs it's got what's his IP address and we're just checking the kernel version um and then for Linux we simply extract uh the targ gz file which is provided by Google on that GitHub Reaper I shared earlier uh we extract it and we

just make sure the scanner is mode 755 so it's executable these are just checking the results to see if any servers failed and then for Windows um even easier we just have to extract to the zip file uh there's no need to set uh log for J scanner to be executable cuz it's exe uh we then run the runability scanner um so just running the command and then the scan path as required and then across the targets that were successful and same with Windows we then get those results we check to see um which ones so with the log for J scanner if there's no output on standard out when you run it it didn't find anything vulnerable so if

there's anything we see any standard out that's greater than one we know it's a vulnerable system or it's found something at least and I won't go through the bottom of this that we just present the results to the user so let's see if I can get back to this PowerPoint okay and we can skip those cuz we already went through that uh so we have a demo as well running it so please work okay that's working so we've got 150 Linux servers here just running AWS and some of them are vulnerable log for J so we've got a Linux system here we running bolt which you can run on Windows as well um and on this one and

sorry I can't increase the font this if it's too small um I've installed um the uh scanner um it has instructions on the read me how to do that and we can print out information about the plan so you can see the variables that we saw before bolt has an inventory file where we specify the servers that we want to connect to so in here I've specified the 150 IP addresses for the Linux systems U we're just using SSH keys so we run the plan so Bol plan run name of the plan and the targets being uh yuntu which was the name of that group um I think it's took about 5 minutes to run I've chopped the video up so it's

obviously way quicker but we can see it stepping through the stages like um checking the facts checking the system uh right extracting the archive and actually running the scanner and once it completes we'll get a result set um of all the systems that were vulnerable or not vulnerable um I'm only reporting on vulnerable systems so if it's not vulnerable I just leave it out of the output and there it comes back so at the bottom you can see there was 79 of those 150 systems were vulnerable and then it's actually printed out the standard out from that came from the scanner so we can see which jar files were vulnerable and usually by path you could probably tell

what application that was um uh there's also another demonstration which is just running it on Windows which is basally exactly the same um 40 Windows systems if that's plain yeah it is uh 40 Windows systems the only difference with Windows is I able to zip into this uh we specify a longer connection time out cuz for some reason windows times's out when I run this at 15 second time out so uh that should yeah so Windows obviously winrm username and password uh no keys here unfortunately oh there we go so the connection time out specified for 45 seconds for uh the Windows systems other we're getting some timeouts so run that and the results will will be the same as the other one

just specifies the vulnerable systems um and yeah it saves time but I guess the point is um obviously this is an old exploit and if you scan your systems now hopefully this wouldn't find anything but if there's anything new that comes out or there's a way to scan or check it use any sort of automation tool in or script in to wrap it and run it at the scale so sort of the takeaways oh yeah 22 hosts on that one and then it also prints out the path uh where the vulnerable jar files found uh you could also run this with the rewrite flag that I mentioned earlier so it would actually remove the um vulnerable J file um so you could

also do that if you're confident that wouldn't cause any issues with your

application okay hang on this is doesn't not want to go to the next slide yeah thank you aome big round of applause [Applause]