VEXatious Vulnerabilities: CVE Management For The Overwhelmed Security Engineer

excellent thank you um so as Paul said I'm going to be talking about uh vulnerability management today um my name is Fram I work at Ive um which is a Cisco Company um and uh if you haven't heard of iant uh we work on ebpf B ebpf based software um if you work in the kubernetes space you might have heard of cium and tetragon um and we were the creators of both these uh both these platforms we ship a lot of containers so we ship about 20 containers and for various reasons they have very different flavors um from Alpine to disis to auntu um we ship pretty regularly as well we have a monthly patch release

Cadence uh so really a lot of my job is maintaining container security um and a big part of this is CDs as I'm sure a lot of you might have experienced uh you there are a lot of container vulnerability scanners at the moment and uh they they definitely give you a lot of lot of results to deal with so to get a sense of the scope of the problem uh here are some figures uh the number of CVS assigned every year has risen significantly uh about threefold over the last 10 years uh scanners tend to provide uh tend to prioritize completeness over accuracy so it's all about processing the largest number of programming languages and giving you the

most complete set of results rather than telling you accurately whether a set of results actually affects the software that uh that is contained in a particular package on the other side of the problem you have customers who want zero CVS for whatever reason either they're going to um high high security environment or they've just got a very restrictive policy set by their security team so I have examples of customers who have come to us and said we want zero CVS we don't want to see any CVS in this image so how do you solve that problem um you can try these things these are very reasonable things to do so you make your container images as small as possible uh

you try dish list images uh that really helps a lot you automate your dependency updates that helps too so that means that when there's a new package update available you take it as soon as it's possible you put it through your CI you see if you can bump that package and then you ship those packages as frequently as you reasonably can so so your customers always have something that's fresh and contains as few CVS as possible and all of these things are really good things to do I uh we do them and I would recommend that people people should start there um however all of this still leaves gaps um so you we still regularly

see that say we release a package on the 15th of the month by the 19th or the 20th you've picked up more CVS and your customers might not even have put the put your new packages through their pipelines yet so you're definitely going to get some questions regardless you will always have third party dependencies that are not under your control so there will be binaries that you are pulling in from somewhere that that you can't upgrade because the Upstream maintainers haven't upgraded upgraded them themselves and not everything is a trivial package Buton right so so there may be a new version of a package that is available but you might have functionality reasons you might not want to take it you might

have uh licensing reasons to not want to take it so really you the the the goal of zero CVS is very very hard to achieve so this leads to what I call the compliance Doom Loop um somewhere a vulnerability scan is run um the those results get compiled into a spreadsheet the spreadsheet results get emailed to me I feel very sad about it uh but eventually I have to get over it Tri the vulnerabilities and uh return the tri spreadsheet to where it came from at which point I will almost certainly receive within the next couple of weeks a new spreadsheet with new CVS there has to be a better way of doing this and that's what I'd like to

talk about today so I'm going to focus the talk around Vex documents um which are not a new technology they've been around for a while there are a few different formats of it but the idea is always the same it's a way of um adding some extra information to the results output by container scanners to say you know this package in this product is either affected or not affected for a particular CV as you can see here it's super simple this is the open open Vex uh Json format um it's uh you know you specify the vulnerability you specify the product you specify um whether you think it's affected or not and why you think that

way so this is going to be the core of my talk and I'd like to talk about some slightly more interesting things you can do with these documents once you have them so the conventional Vex use cases are uh to assign your container images and push them to container Registries um this is really useful to be able to tell the people who consume your uh who consume your container images that um you have considered certain CVS and you do or do not believe them to be affected um you can see that some major container scanners actually do support file processing so you can uh see the example output down there where um you have the same Vex document I showed on

the previous slide and at the very final line of the output you can see that one vulnerability has been ignored as a result of the package that as a result of the Json statement that we made in the previous slide so super simple stuff uh this was the classical use case as such this type of document but how do you go further with this so one of the things we do at ivean is we integrate it with our cicd pipelines so the main result the main reason why I've seen uh vulnerability scanners not being used in cicd pipelines is because of the high number of false positives uh development teams do not want to deal with uh 2550

75 false positives uh just for one CD that might be actually applicable the integration of Vex documents mean that the scans are mostly green about you know 85 to 90% of the time which means that I can now hand this over to the development teams and say you look after the Vex documents now and it's not a huge burden on them to do this um so this also helps greatly with shifting left and I'll talk about this a little bit more in the next slide so moving the care of the Vex documents over to the development teams means that they also now are the ones looking at the new CVS and uh triaging them for

applicability to the packages that they ship so you see here that this is a colleague who I work with and um he has noticed that the V scan is uh failing uh he has taken a look at the results of the vulnerability scan and his open p request here to update the Vex document he has tried the CV himself um and this is often the best way of doing things right because the developers have the most context about the things that they are trying to ship so he knows that it's it's a package that we do not use so we're not affected by this so we have added it to the exceptions list um we use code owners as

well so this has to go through the security team's approval before it gets merged um the document gets merged and then uh the scan goes green again and we have a documented reason as to why we whitelisted that within the codebase so it really helps with shifting left this is something we use across most of the teams and uh has been really successful in both lowering my workload and having more contextual information on um why we have whitelisted certain things for various different packages so once you have these um once you have these documents as well for the customers who don't want to run this as part of their pipeline uh or who just don't know what Vex documents are or

just can't consume them you can also a generate documentation so this is something that we do we just take the Vex documents and turn that into markdown and put that into a documentation portal and that means that you know for the customers who can't consume the Vex documents um you can point them at the documentation portal and say you know rather than come to me with your spreadsheet Maybe go and look at the documentation portal and then come to me with the gap between what you've seen in your scanner and moding the documentation portal again like a significant time saer for me and the team um so so far I've talked mostly about uh how this would help me but if

you consume software as well um Vex documents that show that an image is affected um like this second example here um are extremely useful because they're extremely high signal uh more than any container scanner this is a really high signal that we have looked at this and we think it is affected and you need to do something about it so I think um building that into your release pipeline could give you really good contextual information that you probably won't get anywhere else so I would like to end on a plea uh for the people who make software uh generate your own Vex documents uh Vex still isn't as widely used as it should be so there's a lot of Open Source

tooling that could do with some improvement and uh the community is really eager for contributions here uh so go ahead and start using it uh for the people who consume software uh it it would be really nice if the people who sent me these spreadsheets could consider themselves whether certain CVS are applicable or not before they send them my way um but in the event that that doesn't happen you should also ask your vendors for vex documents um instead of sending them spreadsheets um ask the people who provide you with your scanning Solutions if it's not trivial gripe um ask them to add Vex processing functionality Prisma Cloud for example really widely used doesn't have the Vex processing

functionality and doesn't have it on their road map at the moment um so yeah and finally uh include uh include V processing in your CI pipelines make sure that it um is something that you are taking into account before you deploy into production because it can really help the quality of your soft and that is it thank you very much um I'd be interested to take

questions are we doing questions all right okay are you happy to do a question yeah yeah [Music] yeah um you mentioned how um after four days yeah four days the ucbs can come along and M software is perfect but all my third party Li is terrible so tomorrow do I have to resan and is it a daily Cadence for um ingesting ucv metadata and applying it to framework yeah we scan daily uh because of the overhead we don't ask that people review the results daily but we do scan daily [Music] here another [Music] one so in your example you mentioned m security maybe for like guess is library and you put it into Vex to ignore that

because it's out of context but what if the team starts using mod mod that is that is a really good question I mean you have to have some sort of regular review process for the Vex documents as well you can't So currently that is something that I go through manually from time to time and refresh whether these things actually apply uh but there is no getting away from that process [Music] yeah uh hi I really like to talk um how so obviously this is Shifting left and uh you know those comments in the Vex uh fora how can you trust them because some of the comments seem quite just oh it's not it's not in the executable path yeah

how can we tress especially um is there any control that that you to add to that developer assessment because sometimes it can missed yeah yeah yeah absolutely so we don't uh we don't let developers merge those changes themselves um so there has to be an approval from someone within the security team before that those changes can be

merged minute so you were saying about getting the patches as quickly as possible and things like that my thought was instantly about like the XZ back door which happened a better going now like how would you then mitigate against those sorts of threats with so like I don't think be solves that like that level of problem to be quite honest with you yeah last question because we're on schedule now but this one does the um with the tooling that reads the files or the allow for um a a an exclusion or to time out because people might be tempted to leave it in there and say I'm not going to fit I can't be bothered to fix

that um so they could it could you could leave something vulnerable in there for months when they should should deal with it that's a great question I don't think the spec as it's currently written has a concept of an expiry date and it maybe it should actually um so yeah that's that's definitely something to take to the working group I think there is a working group that does do the improvements on this so yeah yeah yeah yeah absolutely right thank you very much indeed for us