DIY Patch Management

um yeah my name is Ingo thanks for joining our talk this is my colleague Florian we're both working a company called sinner Schrader and you know Schrader builds web applications all the time and we are part of the security team the local security team which tries to enable developers to build web applications with a reasonable security right so and today's talk kind of fits neatly with the previous talks because we are also talking about patch management or vulnerability management and we want to share some some insights how we kind of fix that topic at our company so maybe some of you asked yourself while patching isn't it a solved problem right because I mean there's a software you're running it has

a vulnerability there's a patch just install it right it turns out that's not that easy maybe or probably many of you remember the ransomware news especially from 2017 if you don't maybe this is a screenshot you have seen media yeah and these ransomware attacks were kind of successful when we where you get the malware it encrypts all your data and then you have to pay a ransom mostly in Bitcoin to get the data back so the screenshot was from a ransomware called want to cry and wanna cry hit the world on I think was May 12 maybe May 15th I'm not sure about that in 2017 and it kind of spread in a short amount of time kind of

virally by exploiting the eternal blue vulnerability it doesn't matter what eternal blue is what matters more is eternal blue was fixed by Microsoft on march 14 in 2017 and if you didn't get it already the patch was there for two months but nevertheless this want to cry ransom I'm not sure hundreds of thousands of pcs got affected by that so if you look at reward scenarios it turns out patching is not that easy it also told us and probably most two of us will have a common attack scenario we can we shoot from a spectator strategy strategy perspective very less about these days which are kind of exciting and get the patching done in the first place right so

patching is hard we've encountered it on our own at least if you have real-world scenarios but why right I mean another assumption if you work in IIT and spend the Easter holidays with your family chances are you have fixed their PCs and you can patch them too right so why shouldn't it work in an enterprise which has millions of dollars to spend on that stuff turns out there are some constraints three of them I will mention now which we have encountered first one is legacy if you have some sort of growing IT infrastructure chances are there are old systems like your CCTV servers or old file source or what have you and chances are they run on an

operating system which is just not patchable anymore I recently had to see the doctor because of my voice and guess what the windows was kind of XP I'm not sure if they got the extended support I'm not that confident but yeah you have legacy second one availability yeah you can't do patching and you can patch all your kernels and stuff like that and then you could just reboot the whole thing right this is easy if you have sort of single notes which come up pretty quickly again with their application running but try to reboot these hypervisor clusters which host like I don't know 20 30 VMs each people will notice that downtime and if you

have a web application which generates money it's not that easy to just boot it through which brings you to the third point money right patching causes time and effort and money and the application runs fine on its own so why bothering with it right so these were three constraints that we encountered and now I'm going to intentionally hand over to my colleague Florian who will just give you a glimpse into these scenarios system okay thanks it's not about rambling about our infrastructure and our Kuya as we said this is about the eye patch management and we want to share here our yeah past insight into what was going on and to let you people know that even if it's burning at your

infrastructure and it's okay we've been there and this is maybe one of the ways you may fix it and as Christiane put it out on the last talk you can use open source tools and we did that as well but before that what is our what does our infrastructure look like in a world where we have M tenants like we have big customers fortune 500 customers we've got electricity customers we've got customers which led to Z approaches we've got different devices not only in a setup like we've got the service but also in like okay we've got had to have testing devices which we need to manage so we have a big lot of infrastructure

not only in service but in client systems which means we have m10 flex for example and in the end we are focusing here on the product also used to like n dot n times M requirements but in the real world it's more like exponential like M exponentially to end requirements so so this is something you have to keep in mind like ok it's always worst in the thing about it it's always worse it's like ok it's very heterogeneous we at least on the hosting part focus on Ubuntu and Debian based systems but this is not true for all our customers we also have windows systems and Mac systems as well like the good thing is

that our UNIX based OS is packaged into VMs in containers so we are quite far at a dead shore and nowadays we also do server list which is quite cool because we are getting to enroll new furniture of infrastructure we can manage with all new methods and techniques but on the other side we have to maintain electricity part as well so this means in the end that we've got an inconsistent patch management so never - sometimes regularity to whoever was looking at the server No maybe you didn't tap get a parade or something like that but we can't be sure of for sure so do we can fix this problem for a grandpa's end with commercial scanners

like get ourselves a customer as a chef product like the big candy companies have it no we can't because we don't have to budget for that the customer pays for a product the customer pays for experience the customer pays for cross-functional teams for example but unfortunately telling him like ok we need more budget to fix things this is not a cool selling point so unfortunately if we are not in the present fortunately luckily we're not in the position to get ourselves some big servers and let them do the patching for us so lessons learnt so far then like ok installation is easy I mean we're development driven like our developers they are quickly to make an

NPM install and then we've got something on Ross instance which we don't know about but on the other side as I said various infrastructure then patching your heart and the worst thing about it is like ok you know about it you've got a gut feeling like okay what is going or what may be going on then you're looking at highs ad and you read about the surge the news like okay there's a couple eat stuff like that do we need to patch don't know don't know where our software is don't know what the state of the software is just it's not cool in the end and so we decided we want to fix that and we want

to fix that in an automatic way so our approach here is as every tech approach like the easy part is ok we can add some solution for that some automatic solution and this contains like scanning for CV entries do you know what CV entries is to have to explain that one good so then we skip that but the important part about the CV is is it's not about the information like the areas were nobility in that package valid anatomy lalala you've also got the CVS escrow like to come up with a scoring system so this gives you a number like how much bang for a buck an attacker made gain if he explores a certain

vulnerability so let me set okay we need to find out about the CVS we have like CVS means like affected packages we automatic ultimate with the skins then and we gather all the locks so this is our information base we have then and this is the starting point for the tricky part first you get a - part two things so people are in a better position to understand what you're talking about like you gave as I said there's a severe description like okay an attacker might gain we or might remote execute something if you go to the developer is like okay under what conditions no we went binary like okay do we have a patch or do we don't have

do we don't have a patch and the best thing about it is you can get to your higher levels to Amanda to us and say like okay this is what the infrastructure looks like this is our state of infrastructure maybe we need some more money to put on our infrastructure or maybe we need to need to buy new stuff then you can take action you can go to the admins for example and say I had ok please have a look at the system my dashboard says it's got 77 pictures with CVS and then you can all start all over again after leaning back for now so I hope you're in right mind so let you know where we went

and I'm gonna present to you know what our building blocks are we ourselves build a CV scanner two auditoriums this is just the surrounded lines Python script which talks to a web service I would present in a minute and it runs every night to check out for CBE packages which means it will get a dpkg list the install packages from control or Debian and will check for the packages as they are affected or not then we added integrated in our conflict management I will talk about that as well and an all essential logging and - boring our CV approach was like okay let's use one Ostrom who of you does know about Warner's okay that's not that

few who uses it actively ok cool then the guys will thank me because we will do a little bit of commercial here we're not sacrum is a we are not affected we are not affiliated with them but we find it found out that's a quite good people and a very good service it is Verna's calm is a premium database service with all the security feeds from the major Linux distributions from the BSD and it also has the CVE Nitra nvd the national vulnerability base in it so this is our authoritative source of information about CVE entries you can carry it via a REST API or via the web front-end if you're sitting in front of it in front of your

browser it even offers the ability to scan systems for you they've got an open-source project where they did something similar to us but it wasn't fitting to our needs so we reinvented it somehow but it's do a good thing and you also can subscribe to certain feeds like for example I want to know about the Apache Software stack okay please give me information about Apache and then you will get an email for it and the best thing is they're passing all these open information from the Linux distributions from the CVE so they don't charge as well and they're really nice people meaning in a way like if you tell them like ok I need this

information or would that IPA and point better then they're really keen to talk with you and figure things out and they are also quite a good and integrating bonus and for other partner projects for example there's an nmap parking you can use it for burp that it will check the javascript dependencies you have in your web pages get deployed it's a nice thing I wish I would have had in my diploma thesis like it will tell you read vulnerabilities in your software stack are you see it's similar to what we are presenting here and as I said we've got this nice API cool so we've got that our approach then as I said we get all the

install packages now resorted to our colleagues like when we were doing heavy cybersecurity stuff here and it's always better to have some candy and the construction workers hammock there to make a good approach and then when we cut the list of installed packages we all read them four CVEs entries and get the CBS s crores where rather yeah rather simple with that like we just get the top CBS s core if it's a package was affected we just need to know about the worst vulnerability we have besides that it's not in fact affected so we don't mind about it at all a team nobody go ok let's hope that works single-handedly so we want to show you how easy it is to

use that API because you don't need an account there are nothing so yeah I'm using HTTP and not curl because I cannot remember the cool stuff but basically you can just send a request there and you get a response saying okay I can you read that ok or should I make it make it bigger like so ok cool yeah you're seeing the response ok you're missing some parameters ok then maybe I put in OS Debian version 8 how about that ok looks good better now I need to pass in some packages I need to tell voters which packages I have on my system that I want to order then I want to learn about if there are

any CVS and depending on your operating system you just a ask DP kg or yum or what-have-you for that information and what I'm going to do now is to just pass in one package variable in Linux kernel and yeah then you lights get out but that's fine I guess then you get all this bunch of information in this bit of a second so all you had to do is in the post request with some JSON and then at the sudden you get yeah this is how to fix it and these are the CDs which are a bunch of them and you get a ton of information that you can react upon and put into your locks to give you more specific

view on that I'm now typing or that J'son through JQ in order to pass out to few that are of interest to me and then you get like for Debian the Debian security advisory notices and the CVS escrow and also the CBS s vector and yeah this is kind of nice you don't have to spend any money on commercial solutions you can just just use the API another thing we did and which I can show you kind of briefly we built a CLI to on our own and this does all the magic in the background scanning a host a special host that I have prepared for you and then you get just like a summary

of okay how about how bad is it on that house so it SSH into the host that's all the bonus magic and then you get the vulnerabilities okay so now we know what our vulnerabilities are we have some special truths in place to understand and eight and eight hours an hour approach but the problem is like okay we don't want to ssh into every house we need to centralize this so we went with our ops crew and said okay what kind of a configuration management you are using we are using salt stick does anybody of you use it you know solid stakes anybody of you using it okay so the other guys just together Forks

are just with puppet for example or okay cool anyways we do our construction with that which means you have got the server's they are the minions in this [Music] paradigmatic configuration from a central master and not only the confirmation that the nice part about a salt stick so you can also centrally send commands to the minions and as minions then we'll send the results to the master so you just have the orchestration cycle there you can directly from central points orchestrate your infrastructure and the best thing about it you can codify your update strategy in it like okay we know we need to update our systems when do we know then when do we need to update and is

the customer cool of updating for example Sundays or do we need to during the weekdays stuff like that so we need to have an update strategy which we can then codify I will go on later into that so the thing is about orchestration as I said you can codify update strategy and we have several requirements for that and so we can with saltstack match our infrastructure that certain aspects certain servers certain stacks only get a certain configuration here for example you can see that there is a generic templating for every system in the environment this was a star then you see that every system gets a whim and has some scripts on it and then you see

two additional that matches for web services and for database hosts that you can say ok on a database house I want to install MySQL for example and only Apache the Apache is installed on the web systems and the minions then will have a configuration like ok I belong to the web configuration system so they beget the Apache installed and stuff like that and you even have a templating engine because you know that on Ubuntu it's called HTTP VD or on a fedora it's called Apache for example so there's different names for different services for different systems and as I said we have several requirements regarding our update strategy and the OP screw then managed to say ok let's have three

models or three flavors of updating our systems the best thing about it is for example when you do a patch they actually have to manually manually look out for patches and have to install them yourselves well then you have a certain day in the month that you say like ok it's Tuesday we have to install the patches right now somebody has to take care of it somebody has to be on the lookout for results and then needs to check again if everything works as required this is for example this is why I put the picture of the Neapolitan in ice here this is the strawberry nobody wants the strawberry you want to go for the chocolate and that's the unattended

upgrades our system of unattended upgrades the other side says okay we do every day on 12:30 we do our upgrades be Y 12:30 because after 12:30 the Deaf's order up skies will be back from lunch and then they see if the systems are still living or not and not many scheduled meetings are around 12:30 so they might be able to work on the problem then just at that moment and in between there's the thing with vanilla thing like the orchestra it has updates this is like we have one certain system which gets the update and if everything is fine if everything is good then the update word the reboot cycle work then it will get a lot and

from the locks the other systems and that cluster will say okay our Franchi approach work or a frontal system it's still living it's still up it works as intended our checks are good with that so they get the update as well like then we have our guinea pig system and the other thing systems will look out for that and as I said we have a centralized looking as well it was mentioned by Christian earlier as well like we have our egg stack we lock everything Internet and the best thing about it is okay we've got it and the elasticsearch as a database the lock stage locks - components are on the system so they deliver the locks through our

centralized system and the Cabana this is all front-end so that we may show you four shiny dashboards and this is our dashboard for example looks like this is just a teaser to give you some mouth-watering pictures and now English has some live stuff okay cool let's keep our fingers crossed mmm it's still there which is good so as Lori I mentioned just one question how are you running timewise this is okay okay cool doing all the coding using villainess building all that stuff interacting with you Ops guys that was um not that easy but it was fun right but obviously you have to convince your management right so if you talk to your manager and was here your CFO SEO

and start about yeah there's a CVS escrow of 8.9 we should fix that that's probably not going to work fortunately we have kind of managers which are into security which I understand it's important but nevertheless we had to build some sort of - polling for them and for us as well so we and this turned out to be kind of the hot topic because we try to put a lot of detail on these - but but in the end one of the metrics that work the most you can see on the left side so you basically see okay you have flawless systems everything is good no pending patches cool and then you have tainted systems there's any patch missing we

don't care what it is because it's not fully patched and what you're seeing right seeing right now as a disclaimer you just see seeing our sort of internal zone so no clients that we work for our affiliated to that zone but nevertheless we wanted to fix that right and get more inside we also have you you kind of get the impression that we are not sure about what it's showing right because it's the circle of stuff yes that's this fancy stuff it basically displays all the zones in this case I only have one and then just says okay I'm going to split down all the findings that you have like okay which systems have our flawless which

are tainted and then you can just like kind of navigate through here to the tainted ones okay this is one which has the CVS course and point 5 this is 5 point 0 and then you can just sort of investigate what stuff you see there and to be perfectly honest some of these things actually work and give you some sort of information that you can use to make the situation better some - boring thing is kind of hard for us another interesting graph is this one here it shows the number sort of number of vulnerable packages over time and what you see here is that there is a line at the top at 1 which is a bad bad but

and these are a kind of two holes but they are on the same height and at the bottom you have all the rest when you look at other phones you will see these Alliance going up going down going up going down and if you have such measurements where they don't go up and down it means that the patch management is curve broken right because it should fix itself but it doesn't and the rest of the stuff is not that important to you if you want to look at it in more detail just grab us at the coffee bar and this is the big picture and we have to kind of like the circle of life as I

said we have the engineers they define the update flavor and they put it into a salt master which then orchestrate the minions the minions by themselves get the information from bornus directly and we put it into the lock stash then which will get then shown why I'd keep honor to us as security guys or to the customers who may work with that and the good thing is we also can carry the bonus then to check on me for further information about the vulnerabilities so and as we said okay you sometimes need to talk to your manager and say ok this is cool stuff we want to have it like you also can put it in terms like ok we

have our is m/s information security management cycle like we do plan-do-check-act so this is something he or she might better understand or if you are more like into audits you are afraid that isn't that an order tourists visiting you and he wants to find out if you're into ISO 27:1 or not then you also can say ok we've got a four-eyes principle for example we have our engineer and we have our security person they are thinking about the flavors and which system gets which flavor then we've got our configuration management database over here and we've got some reports we can automatically get out of Cabana so then you have into for compliance models as well and you

can salad in an agile way for that as well but the problem is we ourselves we are still in the progress in developing our solutions so we present some limitations to you and if good thing about it is as I said you can go into compliance you will know about these clerks we are not presenting then the compliance guys probably won't know that these are there for example gates on the kernel yeah actually I mean we wrote software it has bugs as well right but especially one thing that we encountered because we have thrown this stuff on like also on legacy infrastructure was one thing that is mentioned here if you look at the output I hope you can read it okay it

basically tells you that yeah there's Linux image Linux kernel installs not just one but two and the thing that broke our system was that on that box on Ubuntu these Linux kernels are installed explicitly with a version number if you look at the output it says Linux emerged 3:13 yeah they get the other and then the second column again 313 which means the full version number is part of a package name and on Ubuntu this cost on that system that first of all ones didn't report any vulnerabilities big which it should because it's an outdated kernel but then we figured ok this box was running the unintended upgrades from Ubuntu and it didn't get any updates so it was not we

shouldn't blame bonus for that and it's more like we should blame us for that because we installed a kernel a package which will never get an update anymore so we fixed that by kind of installing the appropriate package Ubuntu has these meta packages and you see it at the bottom looks image generic no version number anymore and the package name but then it points to the latest version of the support Condor right a second limitation reboot has oh it turns out that another system this time it's a Debian and this has been with us for quite some time in fact it was patched or distribution upgraded from Debian 6 to Debian 7 and then to Devin 8 I guess although I'm

talking I'm on the wrong track sorry this is again that Ubuntu system but nevertheless the Cotten's the distribution upgrade you can see it there in the third fourth line where it says till the precise one yeah but now it's running Ubuntu a support version but nevertheless there was a quirk it says that there are two chords installed 3:13 1 2 1 & 3 13 - 1 3 7 and the thing is you gotta check if you have an updated package if it's actually effective right because of that system still the old kernels running you see that in the kind of lower half the you name - a output it says the current running is 313 1 to 1 but there's a

newer current installed but it's not run so things you have to look at and that regard are that's absolutely correct the remark was on a bun - you have a system or package which will tell you if a reboot is required and that server has that package and it got told last 176 days yeah you should reboot but it failed so what you want to do is in order to fix or to detect this stuff Verna's won't help you you've got to work with your Ops guys and just monitor all that as a metric like more than 12 times of your service which used to be a kind of metric for ok this is a very

stable server it's running for decades nowadays it's more like doing the reverse uptime metric well this has been running for 30 days we've got a killer that quick and make it fresh again this is not our idea it's been I think the first one who wrote the blog post about it was from Yoko Monica from I think he's still with docker right so we're quite in the middle of development and we also have some metrics right now but there are also some next steps and the funny thing about it is yeah there's always some net steps like we need to fix some quirks but also like really digital agency that means we work with various tech specs and nowadays we are

also into the talker environment as well also as in serverless and this is something we want to achieve over the next time that we integrate our containers there we know container checking is a hassle there is some good enterprise solution for that but on the other side as I said we're not at enterprise so we've got a gob of Claire again which is an open-source tool to check container layers and gives you information about which layers affected by a CVE package they do the CVE checking by themselves and we just need to integrate the results from that as well as AWS integration we need to check our serverless scripts and for that we for example can use NSP this is an old

security project they've got this nice command-line tool that you can use to check your node dependencies and this is something which directly fits into our environment which our developers use and we also can integrate it into NCI process on the front so that the developers will get the feedback but we also own it in the back like another system representing here that we also have a second information stack that we know or cage was really fixed or what should we do with it but on the other side there's also some alternatives which you might use or might integrate into your environment on the one side we have open vas the passive uno - open source project this is something

something open and on the other side you have to solve scanner this does exactly what we do but they post the information directly to Jireh so you have a ticket or they put it into select so if you get a message from your friendly neighborhood scanning robot like your server needs to be fixed they don't put it into the egg stack but they also talked with learners and it's also if pison scripts are easy to install of course you can also use your typical enterprise environment solution I'm not into that word so maybe if you are like into big environments like for zoos and stuff like that you can use it as well and Ingo was already talking about you

Monica he's reverse uptime and growing image freshness metrics are the way you want to go nowadays if the system lives longer than a few days or few hours then kill it and on the other side don't fix it but get rid of the odd image you were using just update the image and then put this into your environment and yet it's everything open-source and everything support from furnace so we've got to thank Kira he was a great guy in communication with us he was open to our questions and as we said for us it's a freemium model so you can use it as well you can get into contact with them via Twitter or made them directly and for

the Nevada I got to thank Christoph Claude vine especially because he was the guy responsible for the orchestrated update model and the other Ops guys as well because they were open to us running into their offices and like ok we need to fix that we need to fix that we have a software here that can show you how to fix it or at least you can show you what is broken please fix it and they were like mmm ok cool and so far so good thanks for more sides we have some time left for questions I think so thanks for attending here and no yeah [Applause] thanks guys first question for me how long did it

take you to come up with this solution the solution the coming up was easy like we had every week every day we had this pain point like here we need to do something this went on for quite awhile I don't know how long the solution press in the technical terms was like we did it in two days entertaining information the hard part about after that was going to our management going to our colleagues going to the project manager and telling them yeah okay we need to do something convince one remark what kind of helped us to do that technical deep dive quickly was just do a pairing session so there was not one guy doing

the coding but two guys and they just we did it in two days together and so if we have up that system right setup but you want to go into that I highly highly recommend to grab a colleague who you like and do it together with him or her thanks for the ideas and for the presentation my question is you guys are using CBS s scores most likely based scores from CBS s v2 or v3 I don't know did you consider including other factors for this prioritization which bets which batch should be applied first besides CBS s course base scores so if I got your question correctly you are asking if we are using the CBS course in

order to schedule which patches are installed first for instance not only the CBS SBS base score but you can think you can use aspects from threat modeling from your environment and see okay this because sometimes see if SS 9 from two different vulnerabilities one of them is a big problem for me and the other one is not that's why I mean besides this course using something else like input from your threat modeling understood so actually we are not doing that right now because we don't have time to do that threat modeling and that death and that automated way so the remark is correct if there's a CBS 10 it may be in a software package

which is not running or exposing anything not used anymore but in that regard we just go the cheap way patch and we can we don't have a model through to the stuff that you are suggesting right just because I'm a couple of I think two or three years ago one company named I think a risk IO they were using not only the CVS course but they were considering for instance whether there is a Metasploit module for that for that vulnerability and so this is something that would be interesting for you to look at Thanks hi so you briefly mentioned that note security scanner at the end but most of the talk was about operating system package

mandates for Debian and Ubuntu so what about other package managers for the applications what about NPM PIP whatever you name it you have lots of applications and I think most developers will use their own package management and not the operating system package management right exactly this is why I talk about the CII approach we tell our developers to integrate these scanners into their CI so that they will get a check for that from our good lab system and that will tell them okay your part failed or not and we deliver them before some scripts like UK please integrate it into a gate lepsy iyammer and it will tell you which of your requirements need update or not and this works about quite

well on the fourth forefront like even the developers are developing but as I said on the other side we want to make sure as well that we have an overview about what is going on so they so we want to integrate the NSP check again into our system and we have it on the backside because some requirements say okay we've got for example outdated node modules but we can fix them right now then the CIA check will pass but on the other side we don't have a second look on the way I don't have a second opinion about that so we want to integrate it as well all right hey thanks for the good talk I have one question which might

lead to another fall question depending on your answer how exactly are you pushing these update commands through solid states like what do they look like exactly it's just I've got just upgrade or what exactly are you pushing and where do these commands come from as far as I can tell it's just up get just upgrade but is it the ones that are coming through the vulner sinf oh how to fix it or not no damn because we figured if you just submit a CV with a command on how to fix it then we could run code on your systems alright so thanks hi thank you for the talk here um I've just kind of discovered a lot of

things or problems that we had as well so like the Colonel's and all that stuff but we have a big issue with legacy software that is dropped into the file system why a zip or tart Lisette and comes with a lot of packaged libraries and stuff like that it's very hard to to discover what is in these packages but for example if when you look at hard leter something like that you would like to know if the application has a packaged openness as a library or something like that yeah these site loaded stuff so obviously we are correcting the package manager of the operating system anything which is like W get download extract use it we won't

see a common example that we have encountered this Oracle JDK this is something we have thought about but not fixed yet so in that case our strategy is currently to review our contact management configurations for these cases and then try to change that to use a proper package management system but you are correct these are the nitty-gritty details when you talk about patching thank you okay I think I hope we the questions for this session and if there's questions you will be around and after the next talk and we'll continue here with incident response in this strike and yeah we'll just take a few minutes for the other people to arrive from the room trick

[Applause]