BG - Hiding in Plain Sight - The Untold Story of Hidden Vulnerabilities

all right everyone for our final speaker of the day this is besides Las Vegas in the breaking ground sorry it's been a long day in the breaking ground track uh for our final speaker we're going to have uh yotam uh yotam peral uh he's going to be talking about hidden in plain sight a couple reminders before we start the talk first off we'd like to thank our sponsors that includes Adobe our Diamond sponsor and a couple of our gold sponsors including uh bluecat Plex track and conductor one it's their support along with the support of our volunteers staff and everyone else who makes this event possible additionally this talk is going to be both recorded and live streamed so

we ask that uh all of our audience ensure that their cell phones are silenced so we don't interrupt the talk uh without further ado

your so hi everyone uh I'm excited to be here um I actually uh just came back from a trip with my family uh this Saturday uh to the Black Forest uh in Germany so uh I thought more accur to say my wife thought that it would be a good idea to have like a theme throughout the presentation uh of the forest and you know uh the wife is always right um and she also helped with the design so so at least the the presentation will be pretty that I can guarantee uh we'll see about the content so um let's uh dive in uh so my name is uh yotam I currently lead vulnerability research at a startup called resilient

uh we do applications security poster management before that uh I did uh thread Intel vulnerability management and Insider threat research at PayPal for a while um I'm also involved in several of the open ssf working groups around uh vulnerability management and we have a presentation in the audience um and supply chain uh security as well the as the cisa working groups around sbom and vex uh and I also help organize one of the organizers of the Pyon ilil uh conference in Israel um okay uh I'll briefly go over what we'll discuss today so first uh I will cover the importance on the or the role of vulnerability scanners in SCA tool as part of the sdlc the software

development life cycle um we'll we'll touch a bit on how they work uh and then we'll dive into the actual research uh that we did around uh basically quality evaluation of Open Source and Commercial scanners um and then the kind of the meat of the talk which is hidden vulnerabilities in uh Docker containers we'll show a quick demo and conclude um so first um vulnerability scanners and SCA tools are an integral part of the S software development life cycle um and um basically in the the current landscape that we leave when we rely a lot on third party code code that we didn't write whether it's open source or commercial um then uh this uh on the

one hand allows us to focus on the Core Business logic release faster um but on the other hand it's come it comes with inherent risk in the form of uh non vulnerabilities so um that's where kind of the vulnerability scanners and SCA tools um um play in but um not a lot of people give too much thought into how they actually work um so before we kind of dive in I want to to touch on that a bit so basically you could think of every uh scanner uh or SCA tool it has two basic operation that it has to perform one it has to identify all of the relevant software components in the environment that it scans and then it takes that

information and map that against security advisories in order to determine what vulnerabilities exist in the packages that it identified so suboptimal performance in any one of those uh stages would lead would lead to um basically um inaccuracy or misidentifications in the results so you can think of it basically every scanner is an s bomb tool um and it's the while it wasn't maybe designed to do that if it doesn't do that s bomb Discovery phase well um then it won't be able to identify uh the relevant vulnerability so that's an important point to uh to remember uh by the way I said as bomb um is anyone not familiar with the term maybe I'll what's okay so I I'll do I'll do a

quick uh overview so an as bomb software build of materials basically it's an inventory of all the software components in an environment um and it has a lot of other uh metadata as well uh but for the sake of this the this context we'll we'll leave it at that we had I had a a talk in the morning about that that kind of dove into deeper into this topic uh and I love to uh expand on it more after the talk if anyone wants to um okay so with that mindset in mind we kind of uh went out to the to do the research this is the the talk combines basically insights from two separate uh research

initiatives that we did the first one uh is what what you see called scanning the scanners we did uh basically a benchmark of um four uh commercial and two open source vulnerability scanners uh and we wanted to evaluate the quality of their results and more importantly we wanted to identify the root causes for misidentifications um and we later on reached out to those tools informed them of the findings and hopefully we wanted to drive Improvement across the board because a lot of times we see we hear people talk about the the problem with relying on these uh scanners results and this something that we wanted to address and the second research on the right hidden in plain sight which is more of

the topic of this talk um is basically a deep dive into one of those root causes that we identified in the first uh research um which is packages that are installed not via the package manager and obviously we'll we'll expand on that later and yeah you have this uh the QR codes for the uh links to the research reports uh if you want to dig a bit deeper okay so uh what we did with the the first research the uh The Benchmark evaluation we basically deployed 20 Docker containers uh as you can see this is the list it's uh I hope you can see it well but um we try to make it as diverse as possible in terms of the run

times in terms of the applications so that we'll get a a broader scope as possible uh but these are containers from the most popular containers on dockerhub as you can see uh some with billions other with hundreds of millions of downloads um and we um uh we uh scan them using as I said four commercial scanners two open source scanners and then we compare the results um and this is um I'll I'll I'll show what we found but before that I want to uh explain a few basic concept that are important to understand before we uh we dive in because this this is the evaluation method that we use is based on on these metrics so first

Precision Precision uh as you can see in the formula is the TP is a true positive divided by the true positive and false positive so in plain terms this uh tells you how many of of the uh reported results were accurate so out of all of the things that the scanner said that exist in this container all the V abilities how many of them were true so that's that's Precision um and uh the second term is is called recall and this this as you can see in the formula also takes into account the false negative aspect so um in this case we divide the true positive so what was identified accurately uh by the amount of true positive and

the false negative so what was supposed to be identified but missed so out of everything that the scanner should have been should have identified what it got right um and that's an important aspect the false negative because a lot of times we hear talk about the scanners false positive false positive but uh which is it's a problem because it it causes um a waste of time and resources triaging vulnerabilities are not relevant which is fine but in my perspective I think the false negative aspect is is worse because it's a false sense of security uh if you you don't know something is there then you can't tend to it um so that's the recall and then we have the

last one which is uh the F1 score which is basically a single metric that takes into account those two metrics uh together if you want one number to compare like the scanners bu you can use that so those are the three metrics um questions maybe about that part before I

yeah all like yeah what's the the ground truth so that's that's that's a great question so it's a combination of one um I'll I'll I'll I'll uh show it in a bit but uh you know what it's it's a good segue to the next uh to the next uh slides of what it we find and but I'll I'll address the question so these are the results broken down by containers so to your question as you can see there's a huge variability in the results so no two scanners agreed on the vulnerabilities that exist in those uh containers and you know even if we got the same number it won't it w it wasn't the same vulnerabilities um so so that

was the the first uh way that we did we went about it so if a scanner says something and others didn't find it then then then we have a a discrepancy and and we um we analyze that determine what the ground truth is what's what should what's the truth and then uh work based on that so it was a lot of manual analysis involved um but we started off based on the the scanner's uh um output um enrich that with with uh manual analysis so uh as you can see a huge variance in results uh we discovered over four 450 high in critical severity vulnerabilities that were misidentified across the 20 containers some of them

are actually in the cisa non- exploited vulnerability list um a lot of them with known exploits um and um on average you there were like 16 vulnerabilities per container that weren't detected uh or misidentified I would say by at least one of the scanners um and this is the basically the same results but broken down by scanner so as you can see I didn't put the scanner's name and this is uh intentional because I didn't want it to become like a uh blaming game or uh um uh pointing fingers the the purpose is really to just to surface the problems and hopefully uh Drive Improvement across the board um but again you can see that that uh on

average the the Precision is 82% so you can say that one in in every five vulnerabilities is not true in terms of the false positive obviously uh some scanners in this metric performed pretty well but as I said what's more interesting in my perspective is the recall which takes into account the false negatives and here you can see that even the best scanners the the best scanner uh in this category uh misses one in in every 10 vulnerabilities uh so fails to identify will not tell you that you have a vulnerability um so and and these are the results which you see isn't optimal like this is not what I would expect uh from leading tools in the industry uh

but that's what we have um so on average you see 27% of the existing vulnerabilities that should have been identified uh remain undetected um and then as I said we try to analyze the root causes to understand why that is um and we'll dive into that so the first thing I want to discuss is the variant ecosystem support so there's uh the support metrics of this these scanners is uh is vastly different um and um I'll I'll I'll explain so um going back to the force motiv uh so um you have in the forest you have a lot of trees um or in our analogy uh package managers oh I missed the um so app dpkg RPM APK and you have

a lot of uh like OS package managers you have the uh runtime package manager so a lot and for every runtime you have multiple ones so and the support for the scanners isn't always the same another issue that that the documentation isn't great so you can't really understand what is supporting and and what's not um but that's just one layer because even if we pinpoint to one uh runtime one package manager let's say maven then even if M Maven is supported and it it's supported in jars a jar file can have several artifacts that that basically describe what's in it so you can have you have a p XML a manifest mfile P properties and even with those files the

support varies between uh scanners so some look at uh the pom XML some look at pom property some look at both um and again no transparency you don't really know uh what they do and this is also just a part of the problem because uh apart from that uh even the if the scanner supports jar there are several ways to package uh a Java binary so you can package it as a war or ear file or a car file and and the that aspect also lack of support varies um so this is this is a uh huge problem uh yeah inside the yes that's a good point so so it's also important to uh to take into

account where in the software development life cycle the the scan uh actually happens and to look for the the relevant artifact in at that point in time where the scan happens so that's true but this is just to to kind of highlight the point another good example for for the same point is we saw with the goang so some scanners support only go mode files so you'll you'll only uh get vulnerabilities in go modules but they you won't get vulnerabilities for the actual goang the actual runtime which is a lot of times more critical if you run an old Go version you will have uh far more ver vulnerabilities than in a single uh go module um yeah and again

these are just a few examples but this is a main issue that we saw that that led to the results that we saw earlier um apart from that there are other issues so failure to take into account the context of the vulnerability so uh as you probably know the um let's take for example the OS advisory so for every OS um a vulnerability can be relevant or Irrelevant for that specific OS because of configuration because the way they package it because of various aspect and uh usually those advisor those os's uh issue advisories that tell you uh whether in their specific context the vulnerability is applicable or not and not a lot not all the scanners take that

context into account um so that that is also something that we uh show and uh that we identified and also uh something that I want to highlight is data issue so not everything is the the uh the scanners aren't the only corporates uh because the data isn't great as well and the advisories don't don't always agree between themselves on the ground Truth uh so a lot of the misidentifications also stem from uh from various uh data issues um and security advisory issues uh also relying on CPE tends to scanners that rely on CPE uh or mainly rely on CP for identification tend to have a lot of false positives um uh not uh taking into account the

curral version etc etc and but uh again I won't have time to go into everything here but the thing I do want to highlight and focus on is the last one which is packages that are installed not via package managers um and this is kind of what we'll we'll try to um to uh pull the thread on in the remainder of this uh talk and this this brings us to the second research that we conducted and um we took this this issue and focused on Docker containers and wanted to kind of show uh the implic ations of of this thing in uh dock environments um so again the forest when you look from above um everything looks great um let's

say you have uh and similar let's say you have a forest rangers that tries to rehabilitate some kind of area in the woods after a fire and he plants these trees um and uh you can uh uh in on our analogy this uh forest ranger is the package manager um so from above if we look from a high level it looks like that uh everything is managed orderly and all the trees are the same but if we dive a bit deeper into the the forest we can see that the forest floor is full with other things that weren't planned by our um carrying Ranger um and as a way of nature as time goes by the uh the

amount of stuff that the rent didn't put there or the amount of code in production that we see that isn't uh managed by the package manager uh grows um and this is this is again something that we we wanted to to focus on so um in the context of Docker containers the seeds for these kind of uh Rogue trees are these four commands or mostly these commands so uh add from copy and run so I'll just give a brief overview so so uh copy and add are pretty much the same uh functionality you can add a resource from your file system into the doc container file system um and then you have uh run that allows you to run uh uh

basically bash commands or uh execute a command in a shell during the the build process of the container um and from which is uh usually the first line that you see in a Docker file which is inherits uh um a base image uh for that uh specific container um and just to give kind of um um a view of how common these things are so for example if we take the Run command a lot a lot of times we use the uh grap app and get up code search to to find occurrences in Docker files where these commands are being used to pull in resources or deploy resources not through package manager so this case

there's the W get and curl command that are that can be used to to pull um external resources and just uh put them place them in the file system or extract them and then um uh manually install them um so as you can see tens of thousands of different Docker files like different containers that that deploy employ this practice uh add and copy as well for zip files for jars for tar files for tgz and this is again just a sub set of files also uh Python scripts bash scripts that you don't have visibility into so people run bash script as part of the container build process that bash script can install things by himself or if if it installs

them it's okay then the scanner will see it but usually it's not an installation via the package manager it's simply again pulling stuff pulling them and putting them somewhere in the file system and the from command is basically it applies to everything here because uh um it's kind of a you know a recursive thing so depending on the base image that base image can also have the same kind of anti patterns um that that uh that we discussed with the other commands so overall we identified over 100,000 Docker files different Docker files that have this this these uh practices in place again it's a common practice it's not but but we'll we'll just we'll show the implication of it in

in a bit so some examples from uh from you know real world scenarios so I again not sure you can see okay I'll I'll I'll talk it but uh so in the first case we see that the the main application of the container in this case uh mongodb isn't installed via package manager so if you run a scanner or most scanners and again we we'll see in a bit but most scanners won't report on vulnerabilities that are related to mongodb in a mongodb container so you'll get all the vulnerabilities for the other stuff that were installed via package manager but the the actual application of the container you won't get any any vulnerabilities on regardless to whether

it's vulnerable or not so and this the important Point here is the like the the concept because you could you could scan it today and get no vulnerabilities because there are none but tomorrow morning there can be a critical vulnerability in mongodb and you still won't see it because of the way that it was deployed in that container um so uh that was true for elastic search NX nexcloud rabbit and q h proxy console mcash htpd redist and many more containers that again the main application uh is installed in a way that doesn't allow the package manager to know that it's there and hence uh the scanner won't identify any vulnerabilities in it um the hidden

component can be uh run times required for the operation of the application that's a second example you can see here go uh using uh version 1.13 which is has a lot of vulnerabilities but since it's just you know uh the binary that that is being placed in the file system um and moved again this most scanners won't identify it um uh the the hidden component can be a dependency required for the application to work uh in this case uh they have uh both gosu and JS yaml that they pull um the comment I think I I Tred to enlarge it let's see yeah um so you see uh there's a comment grab gosu for easy uh step down from Route

and then grab JSL for parsing mon Gob yl config file so that it's a conveniency they pull it in again if these things become vulnerable or have vulnerabilities the scanners won't know about them um The Hidden component can be also a requirement for or a dependency for the build process itself in this case we see it with cake um and we also saw situation in which uh the actual binaries are being pulled in um again not something that the the the the scanners will will be aware of uh because they rely on the package managers in order to get their data or mostly uh or on these artifacts that we saw earlier so in this case it's toxic

proxy um and then they're just um uh either executing it directly or sometimes we saw the source code this is a second example here being pulled again from a remote source using this time curl and then uh using make uh they build it locally um again the same problem in a different way obviously scripting is a problem uh we saw various installations that are using um various python script uh to deploy stuff the same for bash scripts um and you get the picture um these these um and this is again it's it's more common than than it's not like most containers have some kind of uh dis practice in them um and just to again to

get a a more kind of sense of the the scope of the problem so we took the 15 most popular container and Docker Hub um again uh in this case every one of them downloaded over a billion times um and the the vast majority of them again contain the same uh bad Co U you know anti patterns so uh engine X the container application uh mongodb is the example we saw earlier with the the uh run times uh the dependency sorry that goes to in jcm the conveniency function uh nodejs the same also installed the YN uh redis uh the so again the gosu and the application itself buy box the same um in the python container the python

container the the actual again python the runtime you won't get vulnerabilities on because of that the fact that it's not installed via the package manager um as well as the the uh tools like the peep wheel and setup tools they St in the same container um postgress httpd M cach MySQL they all suffer from the same uh issues uh and even the docker container um for Docker does the same thing um with various installations um so um so that's kind of the scope of the problem now I want to emphasize um the the impact or the potenti impact so we we'll look at uh the the Pachi httpd container um it has uh in versions uh

2.49 and 2.50 two cves the ones that you see here that are um uh actively exploited in while they are they appear in the cev catalog um they are path Travers path traversal vulnerabilities uh one of them was actually um like they both they the first one existed uh at the 2449 container and then it was fixed but it wasn't fixed properly and then that's the second CV so 2.4 49 suffers from both and then 250 suffers from only the the the second one which is the the improper or incomplete fix for the first vulnerability um and um as you can see here the the we have the same issue uh the the docker file downloads the HTTP

tar gz uh using WG then it extracts it to a local folder and then it builds it uh from source using the make Command um so uh that's the the same problem that we kind of highlighted and um as you can see in this screenshot um and we can see it live in in a bit um if I run htpd minus V to get the version obviously it's installed it's there it's I won't say installed it's deployed uh it's there it's running uh or it it is uh yeah in this case it's running because it's a live container um but if I do up list minus a for httpd if I try to query the package manager for it it

has no idea that it's there so I get an empty list um so okay so that's that's kind of the the scenario and now I'll stop the presentation and I'll show you how it looks like live hopefully if the demo Gods will be on our side you can see the screen awesome let me just okay everyone can can yeah perfect so um we are now um within an environment where we have let's do this just so that you'll see uh we have these two containers each for you know what I'll maybe okay yeah so everything will fit nicely in one line so we have a container for the two uh for two versions vulnerable versions that we

discussed uh for htpd um and what I'll do now is I'll run a a small script that we wrote um that I'll let it kick basically you'll see here three stages the first stage uh what it does it it scans these containers using three uh open source scanners in this case so it's trivy gripe and sneak CLI um so that's the first stage then the second stage will be running a tool that is called mix or Mi exploitable it's an open source tool that we wrote that uh scans but it it also um depicts the entire flow of the it doesn't just look for the vulnerable version it looks for everything that has to be in place

in order to exploit the vulnerability uh and it also can it describes the logic behind it uh behind the decision you'll see it in a bit and then uh the third stage will be I'll I'll show the actual exploitation uh just to just to make sure that I'm that you'll be sure that I'm not just uh messing with you and this is actually exploitable um so let's go over the results okay so it's finished so as you can see first section as I said scanning so we have here uh trivy uh found 17 27 vulnerabilities but none of the two vulnerabilities that we wanted it to find in this case um same was sneak but it gave a vulnerability

extra just for on the house what's you called uh and then uh gripe also 177 vulnerabilities but uh not none of the ones that we identified and this is an interesting and important aspect as well you see here uh with gripe 64 this is the version number so now gripe when we conducted This research grip didn't uh attend to this situation but uh since then they they uh they support this at least in some scenarios and and as you can see they did identify both cves uh which is good and it shows that we're we're progressing but still most scanners are still not in a place that uh that they can identify it and I hope

that this is one of the things that will change following uh this talk um so uh then we did the same thing for the 240 uh 2450 container and again same results just this time the uh only one varability that that's good that's what we expect but again the three scanners um triv sneak and the old gripe uh don't recognize it and uh as you can see just another point the difference here in the number of vulnerabilities is strictly because of that this phenomena the hidden vulnerabilities because now now that they know they scan also things that were not installed via package manager then they they they uh report on more vulnerabilities um okay so the second

stage is as I said the M exploitable tool by the way uh Shameless plug I'll be presenting this tool at the Defcon demo Labs on Saturday so if you if you're around uh swing by uh so it displays I won't go into the tool come Saturday but uh it displays the flow and and as you can see it did it did flag the uh the relevant vulnerability in the newest version and the two vulnerabilities in the oldest version um so that's that and now just to again for the sake of completeness I'll run the the actual exploitation commands for each of the uh containers uh just that you'll see okay so the basically what it does is uses the past reversal

vulnerability just to print the Etc password file so you see that it works on the 2449 and then we have the same one uh with the other vulnerability like the because it wasn't fixed properly this is kind of the the payload that you have to run to uh run it uh wait wait wait what are you wrong yes ah at the beginning and the end yes you don't have on it true ah yeah I didn't take the whole line okay okay okay okay okay that's yeah okay run it again okay works and then we have uh for the 250 container the same thing that's not the line that I want sorry we'll just finish in a

bit okay so here the first one doesn't work because again it was patched uh but the second one should and hopefully will work um and we'll go back to the presentation again we need the curl okay okay okay got it okay and it works okay so again pretty simple but uh effective way to exploit the vulnerability is that a scanner won't detect uh or most scanners at least um cool so let's go back to the pre presentation okay so what what we saw here is give it a sec um is again two actively exploited with known patches vulnerabilities uh from the cisf catalog that are undetected by most vulnerability scanners needless to say but I'll say it anyway uh that the fact

that the scanners don't identify it doesn't mean anything in terms of exploitation um and so so that's that so uh This research the like the when I look at both researches uh research initiatives uh we found um this this is the amount of issues that we open for the different scanners uh or tickets uh for different um uh vendors um and usually most of them were were pretty responsive I can't say that everything was fixed as as you can see there's still the same problem like we used the newest versions of the uh both triv and sneak as well um but uh some were less responsive as you can see in this message so this case Falls outside of

the covered support plan therefore I'm I will be closing the issues so okay if it's whatever uh but we tried at least um so that's yeah that's it so um I think in terms of takeaways there are there are several layers of take takeaways from this uh from this uh phenomena or or problem uh one is for in the scope in the uh for developers or maintainers for the scanners and also for from a security practitioner perspective which is most of you so uh this is how I try to break it down so from a developer perspective for the folks who were building these containers um it's important to to try to AO first be aware

of this this Gap because a lot of times the developer when it uses this commands it doesn't know that what it is effectively creating a blind spot for security tooling it does it because again it's what they usually do uh it's convenient uh so just awareness is key just making sure and if you can avoid it then avoid it so that's one um and two from a security practitioner perspective so uh again awareness of this detection Gap um and make sure that you have whe whether it's tooling or processes in place to account for this Gap until uh hopefully this will be fixed um and um um again not not taking things at face value be inquisitive make

sure the first issue that we discussed with um uh the support variance it it's crucial we saw that it's it's difficult there's no transparency also for the quality of the results but that's understandable no scanner will tell you hi this is what I don't know how to handle and this is my false positive rate and false negative rate so that but but it from your perspective it's important to be aware and again not trust those results as you know uh uh I would say the the the in Hebrew the the Torah from the the the Bible from but yeah won't get to religious uh uh metaphors but anyway don't trust it at fast value take everything with a grain

of salt um and when uh you're evaluating different tools if you're trying to purchase these tools then then make sure that you take into these things into account and more importantly Focus your evaluation process on the um on your environment what's important to you so if you're developing in goang and your scanner doesn't fully support that uh then then your that's a problem so build your testing scenario in evaluation process so that you'll take these things into account uh and specifically in the context of your uh business goals and and uh environment um and again from the scanners or tools perspective so we try to raise awareness to these to these gaps and we hope that that this will

drive Improvement um and also this also comes from from down up like from security perspective if they'll get demands from customers look I won't buy a product until you fix this thing because others we we see Improvement in that space so that's good but there's still work to be done um um yeah so that's that's basically it from me um before question this is a picture actually that that we took at the black Force in Germany so highly recommended uh with even if you have small kids it's a great trip um and uh one more note I it's important for me to mention while I'm presenting here this was a combined effort from my team and

we have a free and Kya uh in my team that help with the analysis and uh uh the results obviously and uh unfortunately couldn't be here today um and lastly I will say that again this is more to emphasize the phenomena more of the specific vulnerabilities it's more it's it's wider than that uh and it's not only relevant for Docker containers it's easy to show in Docker containers but it applies to whatever environment you have an OS and you deploy something which is uh not through the common uh you know practices not through P package managers you could be creating a blind spot for your security tooling so again be aware of that um and um yeah that's it um and and I

think again uh as long as the tooling won't step up and and and uh improve that aspect it's important for uh the practitioners side to to be aware of it and and and have the the mitigating controls in place to to account for this this risk um so thank you very much for listening if you have any questions yeah I'll be happy to answer them thank [Applause] you if you have any questions please step up to the mic

awesome thank you very much oh yeah uh you had mentioned that you used I mean you saw there's Docker containers you're using are there other environments where you can build the software it can pull in all these dependencies and then you can scan them with various tools or is that functional functionality does not work in Industry right now so um I'm I'm not sure I follow the question you said you had showed examples like where the software pulls in like WG or curl pulls in like all the additional packages and additional software that's not being scanned by the code scanners because you're only scanning theit so okay so so I'll I'll I'll highlight the I'll try so

the the the issue is uh not the pull itself is the the the what's being pulled so if you pull a binary and it's not it's not something that you installed via the package manager because most uh scanner rely on the package managers they query the package managers and say okay what's installed and then they take that and and map that to vulnerabilities so if you do that not via the package manager then they won't know that it's there and then they won't do the part that Maps it to vulnerability and they they won't identify so it can be in a Docker container it can be in in your average Ubuntu host right right so there's no

way to scan what's being pulled after because the binaries can change on the Fly is what you're saying like they they could change in their dynamic yeah so you can you can uh some of the scanners support like Flags you can say scan this thing scan this file but again you have to be aware that your developers are doing this and where it sits and and and point the scanner at the right place in order for for it to to give you these results you won't get that out of the box right thank you my pleasure hi um hey so I think a lot of the reasons that those Docker containers that you were showing do with what they

do is that they they are upstream and they are sorry uh these packages are upstream and they are maybe months ahead of the package managers themselves so is there any expectation that we would really get to a state in which a maybe an ha proxy container would be deployed from like an RPM when the RPM's going to be 6 months after the ha proxy Upstream does a release yeah so yeah so this problem can be tackled from various direction did this is one approach by the way chainu guard uh takes that approach they build the images in advance in a way that that don't have these these uh blind spots because they use like uh they take the tarballs they

use APK and the APK package managers allows you to uh pretty easily um um build the same things like like do the same process that you need to do and then package it in a tgz file and at least it'll let you know that it's there so so that's one that's one approach to tackle the problem and I think but it has to come from both uh ways I don't expect Docker to to fix this I I expect the scanners to be aware of that this is the state of the world and and and address it like gripe is already doing it so there is a solution um you can obviously it's it requires some

engineering but it's it's not unsolvable uh so so I think that and again awareness from the the Developers side the practitioner side to uh to push and drive this this Improvement thank you great talk yeah thank you hey hey I'm really I'm really short yeah uh thank you so much this is great so uh springboarding off of this I think maybe both of the questions before me are going to tie into kind of where I'm going to lean in and that is with what you've identified here identifying anti-patterns can you not then maybe orchestrate something simple as a set of gith hooks that would be the responsible place at which you would do this inspection and at least alert like there

there's components that are hitting the image okay did you could you talk about whether you played with that or yeah so so that's definitely an option we we actually do uh some of those things as part of our uh product but um but yeah that's definitely an option because as as you saw also in the example A lot of the things kind of are uh repeat them yourself so there's again there's several ways to handle the problem um but but that's definitely also an option um I know for example you can look gripe is open source you can look at their implementation so for the from command they just they say okay this is from

this container I already know what's inside that and then I I'll I'll let you know um I I'll add that to the analysis um again it's it's not it's it's an engineering problem it can be solved many ways but it's not unsolvable there there are definitely uh ways to do it even as you highlighted you don't have to wait for the tooling like you can do you can once you aware of it you can do especially for in-house images for your images you that you need to uh uh do you want to be extra vigilant of you can definitely uh do something with uh through your CI process or through run and visibility that will give you the

same um uh like complete this current Gap okay thank you all