Securing The Supply Chain: Practicality vs. Paranoia by Alan Mills

so my name is alan mills uh and this morning i'm going to talk to you about supply chain security uh with a focus on paranoia versus practicality a little bit about myself uh i'm a lecturer at the university of west of england look there's me giving another talk um and i've got some colleagues who are here today a couple times i will mention we my colleagues they were also lecturers at the same university so software supply chain security and attacks i'm sure you've seen in the last six months or a year there have been quite a few instances of software's protein and checks and they're becoming more popular and that's because of how popular the targets for these attacks are we've got some stats here looking at sort of the popularity of things like open source security not too long ago a paper was released which showed that you could intentionally introduce vulnerabilities and issues with the linux kernel for example packages and package managers unless you're writing your code all of your code from scratch the chances are you are going to be using open source packages and they're going to be susceptible and vulnerable to these sorts of attacks in containerization containerization is becoming very popular people are moving away from the monarch architecture towards microservices as the use of something becomes more popular it becomes a popular target malicious actors are going to try and take advantage of that now we could take 20 minutes or so to talk about each and every one of these subjects but we've only got one 20 minutes today so i'm going to focus on containers and containerization so the best place to start is by looking some of the attacks against containers now in 2019 uh container images were identified and they had crypto malware uh preloaded into them this is referred to as a poisoned image you're downloading a container with the micro service you expect and you get malware which you don't expect uh most of the time as say this is crypto jacking malware so you're running the service and mining crypto for someone else in one particular incident there was an account which had over 2 million pools which means container images from this account were downloaded over two million times one of the associated crypto wallets for this particular instance had earned over thirty six thousand dollars now that's not a huge bank breaking amount but considering all on which sakta had to do was put the image up there on the repository and then wait you know that's a pretty decent amount of money for them similar attacks were identified in 2020 and in 2021. now common tactic they use for these attacks is they'll name the image after something popular i keep moving away terrible of me tell me off when i do it um so that you will look for say open jdk or gowang you download it but it's not from the official repository it's from our malicious actors repository or typo squatting very popular again for packages just as popular now for containers you search for tensorflow you miss type and you're searching for tes northrow you aren't paying attention you get a match when you shouldn't you download it and again you're running that service you expect and that malware you don't now it's not just poisoned images as well there have been instances of crypto malware actually attacking and infecting running micro services so it could be that you download the actual tensorflow from the original legitimate repository it's a perfectly safe image but if it has security vulnerabilities in it then they can be exploited they can be used your micro service can be infected with malware or worse still up until that point your infrastructure might have been relatively secure and now there's a hole in that infrastructure it can be used as a foothold to pivot and attack so there are some issues with containers that's it we just don't use them no of course not we have security solutions now the best way to go about these security solutions is in the first instance identifying that we have a problem and the best way of doing that is scanning tools now there are a lot of scanning tools out there um there are some open source there are some paid but they can differ they can differ in their coverage by intent some of them might be designed to scan for vulnerabilities some of them might be designed to scan for malicious binaries and they might also use different vulnerability feeds so we might find that with two different scanners we're getting different vulnerabilities picked up on the same image so if we want a full picture maybe we need to use multiple scanners but if we're using multiple scanners what we might find is that when they both find the same vulnerability they're reporting different severity one of them might say it's high severity one of them might say it's medium severity we also get differences in the actual output itself normally with these scanners the output is very text based very text heavy but could be in different formats plain text json different levels of detail different verbosity as well and again if we're using multiple scanners because we want that full complete coverage and we've got multiple reports to go through they're quite heavy some of them are quite long lots of detail it's going to take time chance we're not doing this for one image as well so it can be a time consuming frustrating experience during this uh talk i'm gonna look at some case studies that sort of highlight these issues and we focused on using six scanners which are listed here now five of the six are open source cystic is a paid for scanning service full disclosure we did it all in 30 days and used their free trial so you can too so as i say we did a few different case studies we looked at a few different images and one image in particular influx db we noticed that five of the six scanners are moving by the way so you can take your picture uh five of the six scanners picked up these vulnerabilities listed here but we got that differing severity which was mentioned before from high down to low now if we're doing this as a security analyst and we're saying that maybe we use influx db in our stack well what do we do about this how do we treat this vulnerability do we go air on the side of caution this is a high level vulnerability do we go for the sort of middle ground approach and say well we'll call it medium [Music] and we also need to take other factors into account so for example these vulnerabilities actually have verified working exploits on exploit db so in that instance maybe we actually do want to err on the side of caution maybe we want to say well this is a high level vulnerability with exploits we really need to focus on this mitigate it what do we do so our realistic risk has increased because with those exploits here was the barrier for entry however that's not really the full picture because these vulnerabilities are local access vulnerabilities arguably if someone has got local access to our influx db container there's already been an issue and not only that but it's a privilege escalation vulnerability in both instances and not only not only that but the default user for that container and most containers is root so suddenly our high level vulnerability with an exploit that we were panicking about because it was all high level and all the rest of it is actually negligible because at the point where someone's got local access to our container and they're at an area where they could exploit this vulnerability they don't need to they're already root you've done it you've escalated the privileges so we go from high level to negligible but it can be hard to identify all that information to to get the full picture especially if we're just going off our scanner output these are the two scanners which showed this is a high-level attack attack vulnerability and this is the sort of default output we're getting from them it's pretty minimal information we get the binary that's impacted we get the version that's impacted we get the vulnerability itself of course and we get the severity with one of them uh trivi we don't get anything in the fixed version that's just empty we do get some detail in the title a little bit of a description where it tells us actually not a huge amount i don't know if you can read that but it's basically a short thing that says set your id set group id binary and a link and with the other one gripe we do get something in the fixed version uh fixed in which is won't fix but we don't get any title any description so if we were just going off of these we're probably edging towards that paranoia level we've got high level severity not a huge amount of detail panic panic what do we do we're not getting the key bits of information which allow us to make those practical real world risk assessments the attack vector was missing we don't get anything about next weight the existence or lack thereof it's just simply not included or touched on and with one of them we don't get any description or details with the other we get a little bit but not a huge amount and we also have issues with cvs which are listed as won't fix again go back to that gripe output won't fix what is a won't fix vulnerability right if there's a vulnerability here cve why aren't we fixing it sounds like something we should do well that's a maintainer a maintainer-based decision um they might decide that actually they don't think it's a vulnerability they don't think it's a security issue at all they might decide that actually they think it's trivial and that is the instance with these vulnerabilities by the way if you go and have a look on the debian page they have this as a minor issue for debian 10. they might decide they are going to fix it but they're not going to do it in a patch they're going to do it in a stable release so with debian based cve feeds won't fix cvs are classified as open so they're still included in all your scanner outputs with red hat based red hat based cve feeds they're classified as closed so you don't even see them so arguably should you really be seeing them for debian based cbe feeds now you can have your scanners set to ignore them but often not by default and part of the issue is if you don't know what a won't fix cve is or you don't know about the existence of them for example trivi doesn't tell you it won't fix it's just empty then the chances are you're not going to look for the option to ignore it because you don't even know that you need to you don't know that there's something there to be ignored so you end up going away doing all this cross-referencing all this research looking at vulnerabilities very time consuming and so and this is where we come to the we and the colleagues we wanted to create a framework that would help address these issues amongst others and what we wanted as well was something with a nice visual report and output we wanted to use multiple scanners for that complete coverage as we mentioned before sometimes if you use one or two scanners you're not going to get the full picture and we wanted it to be easy to disseminate and we wanted the data sort of out of the back to include all the relevant real-world risk assessment pieces that you're going to need things like the attack vector the attack severity if an exploit exists on exploit db and we wanted to flip the the won't fix sort of situation so by default we actually don't show i won't fix vulnerability but you have the option there to show and highlight them and we'll just have a quick look at some visual outputs from these frameworks so the lighting here isn't great you probably can't see this too well but nonetheless if you could you would be able to see a node diagram around the outside are vulnerabilities with lots of different shapes and covers the shapes indicate the attack vector across is remote a diamond is local a hash is reserved and a triangle is something that's not a cv so github security advisory red hat security advisory something like that different colors as well so we have a color scale for the severity gray is negligible green is low yellow is mid uh middle medium even orange is high and red is critical in the middle we have circles uh the circles are the packages the big circle in the middle is the container image itself now the vulnerability markers also can either be open which indicates the vulnerability exists but we haven't found an exploit on xydb filled in which means the vulnerability exists and an x-y exists on exploit db or and as is the instance with this particular output when you're showing those won't fix cves you have a little dot in the middle of your marker so you've chosen to show them you've made that decision but you still want them to be visually different from the other vulnerabilities so you can still easily identify them and that's how we do it so in this instance we've got the input from gripe for another influx db scanner it sounds like i'm picking on them i'm not we did lots of different ones and we can see there are 112 vulnerabilities and this includes all of those won't fix and it looks it looks like we've got a quite a few issues here this is the default output where we don't show those one fix vulnerabilities and we can see a drastic reduction in the number of vulnerabilities and a drastic reduction as well in our overall severity picture uh previously it was multi-colored now vast majority of these are negligible as well and we had a particular binary highlighted in both of these we're just going to focus in on that and that was lib c when we're showing those won't fix we had 22 vulnerabilities from negligible all the way up to critical it looked awful liberty was so broken but now when we remove those one fix we've just got seven and they're all negligible suddenly we probably don't even need to worry about this and that reduction was not just that one binary but overall for the image a reduction of around 53 cves and we can see the breakdown of severity there as well now the framework we used to put these together part of it is called ogma and that is what produced the visualizations we've seen today it's open source it's available on github and it takes in multiple scanner feeds but specifically the six scanners that we looked at today supply chain attacks are a growing concern and as any sort of security area or part needs to be there has to be a degree of healthy paranoia but we don't want to go too far with that we need to balance that practicality and that real world risk if you're just using scanning tools and the default scanning output it can take time you've got to go through multiple outputs if there's not a huge amount of information you've got to go and check those vulnerabilities this high level vulnerability all i've got is the vulnerability id and the fact that it's high severity go away check it find out it's go quick access it won't fix you're doing that multiple times over and over again possibly for multiple images it takes a lot of time and you get that alert fatigue as well you might miss something which is an actual vulnerability that's where visualization tools like augma really come into their own they can help to highlight that key detail that you need at a glance you can see the attack severity the attack vector you can see the existence of an exploit and if you want to see the full picture including those won't fix cvs you can see them but everything is visually shown in a way that makes it easy to separate identify and triage and suddenly you don't need to go away do all that research on your own look into things on your own you can just look at the visualization at a glance and say all right that's what i need to focus on i don't really care about high-level local access vulnerabilities especially ones i'm going to be fixed now a bit of a shameless plug here um i've only touched uh very briefly on augma and i'll be presenting this at the ie csr at the end of the month along with its counterpart borvo which is an automated remediation tool so not only are we identifying the vulnerabilities we're trying to fix them as well if you already signed up for the conference fantastic if not and you do sign up uh maybe tell them about the fact that you heard it here maybe they'll send me some t-shirts i don't know um but i will also have the paper in the conference proceedings as well so you can read about the full ogma and volvo setup we've got some case studies in there as well now hopefully oh almost and there are links of course there are links for all of the attacks and the stats i mentioned at the start and now hopefully if i've timed this right i've got two minutes or so for questions so thank you very much for your time and your attention and if you do have questions please feel free [Music] oh she was first i'm sorry [Music] so in an entirely so uh and the question for those it's a pretty small room but anyway um the question was who's your end user and how do i see them using this in a slightly selfish way i was in part the end user as well as being a lecturer i also work for a company in their cyber security sector the product is very heavily containerized and i got very tired of going through multiple text based outputs for the containers so if you are for example involved in a company and your product uses a lot of micro services maybe a cloud in a box style product that sort of end user and using it in a way to just provide that sanity and that health checking especially when it comes to government agencies they're very interested in making sure the containers you give them aren't full of high level critical level one abilities often they don't actually care if it's local or remote if it can or can't reasonably be exploited they just simply have these sort of tick line things that say well there's a high level vulnerability there fix it uh so that's the sort of end user i'm envisaged the gentleman who had a question yes so if you're if you're building your own container um then chances are you're putting it together from a base image right you're probably not going away and writing the base image yourself [Music] if you're using your own image making process and you're doing everything from scratch then chances are you've got oversight on everything anyway you can put it through ogma just as a safety check to make sure you've not missed anything if you're building an image and say you're using a debian base operating system you're adding in functionality in a micro service maybe you want to scan that operating system at the base level before you start making changes to it so you can go away and say actually this has got a couple of vulnerabilities uh i might use the alpine one instead if it's not for you that's fine i'm not selling it anyway it's open source so i get nothing from this hello yep yes uh so at the moment one of the limitations is that it's largely based around identifying vulnerabilities based off of scanners that are given to it um and the only scanner that we listed here today which scans for malware and i've got to finish this and then i'm done was dagda and dagda if anyone's aware of it or uses it is open source lots of issues isn't actually stable so that is something we need to consider i'm told that's it time's up no more questions but feel free to grab me after this [Applause]