Blackbox Containers: Container Security in the Enterprise

foreign

[Music] awesome well thank you all for coming today uh my name is I'm gonna drop that chord so I'll keep making noise my name is Kenny Parsons um welcome to black box containers container security for the Enterprise so we'll try to keep this high level from a container fundamentals perspective and then get into how that affects Enterprise so run through this real quick um first a little bit about me uh quick aside I'm not the uh who am I guy that has all the who am I slides so I went a little different break the mold I did a tree at my home directory uh I am a senior Regional engineer for set Solutions now a trace three company I am

actually based out of Dallas um so a nice flight up here I am into devsecops obviously I enjoy the architecting and designing of proper microservice Solutions and I am also the chapter leader for devset con that's a global developer uh Community uh devsecops we just launched a chapter in Dallas and actually have our first meeting uh here this coming week and I'm a bit of a container nut so um I will get on the soapbox about that um I have my own ideas about philosophy and again how microservice architecture should be and how you should set things up but anything containers I'm in so in fact on my our company slack I have kubernetes Kate's container and all that

flag so I get messages if anybody's saying all those in any channels so uh I'm a Nut I will admit that uh so like I said we'll try to keep a high level with the container Basics but just kind of cover our bases here uh we'll talk what our containers what are they not and the types of containers and or run times that you all might have heard of so first what are containers um well simply put it solves the problem of it works on my machine it works on my machine not yours but straight from docker's website and again docker's just one of the container engines you can work with but they have a really solid underlying philosophy

it's a standard unit of software packages up all your code all the dependencies and it all runs reliably inside a container everything you need to run your application is inside that container or multiple containers that work together and the thing about containers is that they're independently scalable okay so uh some applications break that mold again it's part of the docker philosophy or microservice architecture but in general they independently scale or work together to scale um a monolithic or traditional monolithic application into micro Services which we'll get into that here in a second what are they not it's not a VM please please stop saying it's a VM um we will use the term it's like a VM

you can exec into it you can run commands you can do just about anything you can do on the surface level with the VM but it's not a VM so containers or VMS abstract Hardware right so you got your esxi proxbox all your virtualization they virtualize the hardware containers containers virtualize or abstract the software through the use of an engine so they share a kernel underground kernel on the underlying host containers don't have their own so that's there's a big difference it's not a VM so just keep in mind of that but really that is important when you're talking about proper architectures how do you build applications inside containers you know if you're used to

nnd or systemd or things like that it just won't work so the way you use your applications the way you build them is just fundamentally a little bit different so some advantages these are just my top three that I like to talk about with people when I'm just introducing the topic and I'm not wanting to scare them off uh number one be resource efficiency again they don't have their own kernel uh they're sharing the host kernel so you're actually saving on resources um so if you have a host like a type one hypervisor and you've got several VMS one for each application you've got some overhead there even though all those applications or excuse me all those VMS

may be the exact same operating system so there's extra overhead so you do gain some resource efficiency by using containers and sharing a kernel obviously there's some security concerns with that we'll get into that number two on my list would be portability so all the dependencies for the application are bundled in the container like we said this means they can be easily moved between different deployments you can have a you know test stage or a server environment uh trial production so on so forth developer and you're able to move that image between all those environments and you don't have to change anything all you have to do is just make sure that it has a Docker engine or any kind of container

runtime on the host and you're good to go on that note uh continuous deployment and testing so containers are really helpful here again because it solves that problem of I don't have to reset up the environment and make it the exact same um between like I said like a Dev environment and production environment um so everything you have is right there you just move the image over or pull the image down your container and you're good to go there's obviously a lot more benefits but these are my top three uh for using them so some popular container run times these are probably the three most popular Lexi is a little bit different than the other two it has a little bit of

different of a philosophy than Docker I will take it because I think we'll have time I'll point it out lxc um is most closely like a VM you can run systemd you can run in at D and all these things but it's still a container it has the entire operating system basically in a file so it's a little bit different than Docker wears Docker and in podman being an alternative to Docker um it's not like that you don't have system D without some really major modifications uh it's just a different philosophy of how you run things Docker is kind of your one service micro container that had it runs just one Service uh same thing with podman being

the alternative so a little bit different philosophy but it's still a different it's a type of container it's a different type of runtime I wanted to now that we've kind of got a surface level understanding of containers and I'm actually I'll take quick pull who here uses containers on a semi-regular basis actually wow so I'm I'm with Kindred people um I love it uh normally I'm the only person um at work everybody else hates me talking about containers um yeah every time we work with a vendor in it there's a container or kubernetes aspect they all start messaging me saying hold your horses um so anyway some common uses a lot of them you may use if you don't use these

maybe they give you some good ideas we'll look at runtimes versus orchestrators just make sure we're level set there and then we'll take a look at how we develop them so some common uses OS virtualization again it's not a VM but you can use it to virtualize an OS to figure out how an application will behave in a particular type of environment you know different OS has behave differently they have different versions of packages by default within the package manager maybe it's a busy box versus gnu things like that so you could kind of understand in a test environment how an application is going to run I actually just went through this exercise exercise this last

week upgrading a container to a different base image or upgrading the application and we just use the different base image and found out hey there's some things that won't work because it's just a different version so really good for that and then obviously microservices and distributed applications Docker is the most popular for microservices oh by the way lxc is my choice for OS virtualization it's really easy to spin up it's very fast and you get like a persistent shell you don't have to like do some finagling to get a shell in there but for microservices and distributed architectures that's Docker again podman being a more security conscious version of that um so microservice again level set on the

terminology it's a monolith that's broken down into smaller independently scalable Services each container runs a portion of the monolith hence micro microservice this allows you to distribute the core components of an application across multiple environments maybe even multiple hosts um and they can still act as a single cohesive application okay uh microservices microservicing allows developers to independently focus on each particular part of the application so in development when you're trying to update a monolith you usually have to update and redeploy the entire thing whereas a microservice you can focus on just the one part that you're working with and because it's independent of all the other parts and independently scalable you don't have to touch

everything else all you have to make sure is that it connects there are a lot of products out there that do service discovery that as soon as you bring it up online it just comes right back into the application that's a little bit more complicated but you get the idea they're all independent so you can just work on and redeploy just the particular part you are concerned about that allows us to do things faster release faster and it's just a better model I can't say better maybe a little bit more flexible if your environment can handle it there we go um and I've mentioned kubernetes a couple times orchestrators just wanted to take a quick pause here it's not an

alternative to Docker okay you know when I was first learning about this it's been like five six years ago I saw this you know kubernetes being thrown around a lot and I almost thought of it as like an alternative to Docker a different way to deploy containers and that's it's just not true uh kubernetes used this Docker under the hood in fact for its own components sometimes you can run the components as containers but kubernetes is not an alternative to Docker it runs Docker containers if you so choose but kubernetes is essentially just an API server okay it's a big fancy API server it's very complex um from the outside but not an alternative to a Docker it's what we

call an orchestrator the API engine and the API server orchestrates all the various deployments and Creations of containers or what kubernetes calls pods which have multiple containers inside of them so not an alternative and so from here on when I mentioned containers I'm specific or Docker I'm specifically talking about an individual container if I want to talk about a pod I'll specifically mention a pod but those are obviously not one of the same where are they best used again these are my top three that work for me just some ideas that may work for you this is not an exhaustive list but um obviously services that can be easily segmented if you're running a web server

um and you are running a database or a python application on your web server or something like that those are usually independent of each other they can work independently you know if you're running a web server everything runs over maybe a TCP stack and so it's very easy to segment that service um ephemeral short life workloads so who here uses like Lambda or Azure functions okay so Lambda Azure functions along that line serverless to be I guess exact along that lines but I'm just I need to run a workload I need to run an application or a script um one time actually I'll give you a good example of this working with a company and they used a very very

outdated version of ansible and that was not the version they had on my machine I did not want to have two different versions of ansible one being very very old so instead I just put it in a container I installed the correct version the old version in the container and it was completely independent my host and I can just run all the workloads from in there okay a lot of jobs that I run a lot of scripts that have different um dependencies I'll just run them in a container it's completely isolated again it's kind of like a VM that's why people think oh it's a VM but it's not it's not but you can use it like that you can use

it for that function like workload um ad hoc tooling another great way to do it you know I just want to run this script or I want to run this particular version of an application put it in a container you're good to go um here a great example of this I was working on a project for certificate management for a company and I could put all of its dependencies in that application we could deploy it inside of a Docker container and then we could just deploy that anywhere in our Jenkins server and any of our other icicd pipelines Azure devops whatever and it would work the exact same because all it needs is the docker engine

so I just figured some of the tools that I was using to create or manage these certificates inside a container so that's where you can also use it but your primary is going to be you know microservice segmentation that's probably the most important and world changing to be honest part of microservice architecture excuse me part of uh Docker is its ability to do microservice architectures so how do we make a container how do we get into the development first before you like take a picture and crucify me um this is not a well-optimized container totally understand that this was just a quick proof of concept that I was trying to do I was working with the

Azure CLI and I needed a particular version of something and I didn't want to have to mess with that on my house so I threw it in here but it gives you a good idea of what we could do so first sorry my hands are a little shaky here I guess just the distance but from that's our base image so I'm saying I want to use Debian their base image that they provide to me which is in and of itself just a bunch of layers of files so like just like this it's a bunch of layers of files that I don't have access to I'm just importing them basically very much like uh like an import in

Python okay I'm saying I want to use their layers and I start building on top of that these layers are now also um read only after the layer is finished so in this layer I run some updates I install some applications I do a bunch of stuff okay cool I'm done and I move on and I copy in my actual script that I want in the red and I'm just trying to get a proof of concept done here once I do that the layer before it becomes read-only becomes hashed it's Unique so anytime you change anything in this layer a version updates a file updates whatever that is now a new hash this layer get

rebuilt okay um so anytime you move on to another layer this becomes read-only and now you're just building on top of that okay it's a like a cache file system on the overlay file system is technically the term through and I just I change the user app and then I run my script cool okay pretty simple but this is uh to start from the beginning without this from statement it's just an empty canvas you got nothing uh so you can either choose to import and do all this stuff or you can import exactly what you want it's called a scratch image you can basically build your own Linux distribution inside of container okay or you can choose another

Linux distribution to build off of um there's some really cool things that this allows us to do I mentioned hashing that's a security thing that you can use you can look at hashes to make sure nothing has been changed so it's immutable you can analyze those hashes and everything changes or it's not what you expect you can you know assume there's been a problem or a breach or whatever it also allows us to exactly what we did with this from this is basically just a list of hashes from Debian I can instead of rebuilding everything that they do I again I just copy it in and I copy that hash in or if I took this let's say I dropped out this

entry point I could now use all of this stuff as a base image for another image that I want to create you can basically chain that and you do that with image caching image hashing in this layering idea but there's some security implications there and we'll talk about that so security considerations I want to talk through four main areas again this is not necessarily an exhaustive list but four main areas uh that you typically want to look at at the you know the lifetime of a container um our images where you store your images the actual hosts themselves or the orchestrators that are running these images and then the underlying runtimes like the actual Daemon or the docker

Damon in our images um so some security considerations here number one this I think this is the most important uh the Upstream vulnerabilities so as you remember in that previous image when I showed you the my ad hoc tool that I built I was trusting debian's base image I trusted that they had an updated version that if there's a package in there that there's you know CV cve and it's really high or CVSs that's really high by the way if you missed the talk over there that was really good where she talked about CDs and CVSs but let's say I'm trusting Debian to not have a ton of vulnerabilities in their image or I'm trusting them to consistently update and

making sure that they're handling that I need to establish what I call this is me registered trademark whatever base image chain of trust so I want to tr I have to go back and follow the trail because if I import let's go back if I import instead of from Debbie and let's say I import from someone else that they built an image with some custom packages in it there's a bunch of them out there I have to go back and figure out what did he import what did he build off of did he build off of an insecure or maybe an old version of Debian or whatever I'm building on so I have to go back and

establish that chain of trust figure out what do I have in my environment inventory my actual images and follow the bread the bread comes back um misconfiguration this is probably man it's super important but this is probably the majority of problems you'll find or security issues that you'll find in a containerized environment the big one being is incorrect permissions principles of least privilege obviously we all know that term but we don't use it and it's it's easy to miss when I'm building this image I'll go back here actually when I'm building this image this line right here is a huge deal when I switch to user app that's a non-root user it's a standard user no permissions

other event itself if I missed that which it's really easy to miss because a lot of times oh something permissions didn't work I'll just remove user app I'll run as root just to get rid of the problem okay that's a this is a proof of concept so it works and then how many times do we take a proof of concept right into Devon production and so it's really easy to miss that making sure we have correct permissions within the container itself we're not running things as root we're not mounting uh or providing file systems to that container that it doesn't need to have access to we're only giving it exactly what it needs so user switching mentioned that don't run

things as uh sudo Sudo whatever you want to call it do not run that unless you absolutely have to usually there's a way and podman's a good alternative where they have their rootless mode it's a really good alternative it a little bit more security conscious and a little bit extras that they provide but don't run things as a super user unless absolutely necessary most of the times it's not it's usually you just need to be a little bit more thorough when setting permissions within your container or with the files that that container is accessing so it just takes a little bit more work but you're going to save yourself a lot of grief then the docker socket so this is a file

system socket and this is how containers um interact with the kernel essentially how the docker Daemon interacts with the kernel some containers request that you mount this socket within a container so that that one container that you mounted it to can then control other containers so for example if I have an application there's a popular one that y'all may have called called Watchdog I think it's called it basically watches all your containers and if there's an update or a new image available it'll automatically pull that and then rebuild the inventory and redeploy it well it needs to have access to the docker Daemon itself and it runs as a container so you mount the socket

in there it's a bad bad idea I stopped using that a long time ago because if that Watchdog container ever gets compromised it now has full access to everything else that's running because it has access to the socket so don't Mount the socket unless you absolutely absolutely need to and most of the time you don't um there are other ways to do what you're trying to do obviously making sure we're not including vulnerable binaries making sure that you have an accurate inventory system or a product sneaks a great example there's a lot of them out there but they're probably the most popular where you're analyzing your repository you're analyzing your Docker file uh that thing that had all the layers up

forgot to mention that by the way it's called a Docker file you need to make sure you're scanning that and making sure hey there's no vulnerable buyers in there making sure I'm not installing a version that's old that has a new CD with a high cdss or something like that so that's pretty basic and build Secrets sometimes whenever you're building an image and you're copying your company's you know code repository you got to have a access key and API key what have use client secret ID um something like that and a lot of people just hard code that right into the docker file and they think oh it's good because the repo is private where

the docker file lives only people in our company have access to it and we have good you know our bot controls on the repo I'm that well you're not really because again if you remember that every line every layer in that image that I've built is cached and it's read only so theoretically you can peel back the onion not theoretically practically you can peel back the onion and if someone gets a shell access to that container they could find that build secret and then obviously use that for other things so don't hard code build Secrets have good Secrets management using something like hashicorp other products like that where you can inject build secrets into your pipeline even Jenkins you know

Jenkins has their built-in users or credential store that's even a great that's an easy alternative to injecting these build Secrets directly into or hard coding them instead of injecting them so into the docker file then in our registry um making sure we have good arbot controls who has access to push and pull and I'll explain why that's important obviously who has access to delete you don't want people just deleting your stuff um you know they don't need that permission don't give it to them principles of leaves privilege but the reason why I say push and pull is specifically we'll talk about this here in a second because I'll move on and I'll come back to that encryption at

wrestling and Transit that's an obvious one don't send things over clear texts don't store them in clear text encrypt everything at rest and in transit but the push and pull um I think I'll go to the next slide here's why it's so important um unit 42 this is back actually from 2020 this report but there's an updated report and um Aqua security I don't know if you all are familiar with aqua they actually just released earlier this week I didn't even have time to put it in the slide their findings on registry security what they found out in the wild and the reason why I mentioned your push and pull operation as being important for

your registry um if you don't give proper permissions to push and pull let's say you leave the pull operation you think oh it's no big deal it's a compiled image no one can inject a secret or inject malware into my thing they can just pull it the problem is you give them a chance to analyze your environment better you've given them access to your system indirectly because they now can peel back the ending of the container and look where a vulnerability might be okay so having correct Autobot controls on the pole but especially the push now we're talking about supply chain attacks someone if they if you have an unauthenticated push uh operation like you just allow push operations anybody

could pull that image or even let's say they don't pull it they just build their own malicious container that does you know just nukes everything well you've now allowed them to implement supply chain attacks and obviously we've heard a lot about this in the last few weeks with 3cx supply chain uh hack and there was another one that was they were a victim of another supply chain attack it was like the first double supply chain attack in history but with containers it's really easy people just miss this um and it's not getting any better again this is from 2020 I hope maybe it'll get better it's not um Aqua security proved that with their findings actually it was just like

Monday or something it came out but here's the stats but I'll read you a little excerpt from the uh from the article that they published so now research identifies 941 Registries exposed to the internet and 117 Registries accessible without Authentication there are a total of 2956 repositories and 15 887 tags so whenever you build an image it tags it so a bunch of Unique Images or versions of images out of the 117 unsecured Registries 80 of them allowed the pull operation 80 of them allowed you to just pull down their code without any Authentication the 92 Registries allowed the push operation 92 allowed people to inject their own containers into the supply chain with zero Authentication

and seven registers allow the delete operation I mean you know denial of service I mean can that's huge without looking into image content so they didn't actually start looking at it we could attribute about 25 of the unsecured Registries by reverse DNS lookup or scene in TLS certificates meaning they just found this with public data that's it the owners of these Registries range from research Institutes retailers news media organizations technology companies some exposed Registries have more than 50 repositories and 100 tax exposed with all the source code and historical tabs malicious actors can design tailored exploits to compromise the systems if the push operation is allowed benign application images may be replaced with images with back doors these Registries

may also be used for hosting malware if the delete operational is allowed hackers could encrypt or delete images and ask for ransom to file a service as each registry is typically accessed by multiple clients here's a big thing all the clients who pull and run images from those compromised Registries immediately become vulnerable you can't trust that those the hosts that are pulling and using those images are now secure because they've pulled unvetted unauthorized code potentially it's a big deal and how can we secure our container runtime again this is Docker itself the actual binary and that's running the Daemon and all the underlying API and all that stuff um obviously there's cves for the runtime make sure you're updating Docker

itself you can't just assume well I've updated everything in the image I'm secure now you still got to make sure your underlying engine is updated because there could be an exploit for Docker itself and then the configuration the runtime again configuration is usually where you have the most mistakes and expose yourself to a lot of vulnerabilities running it as rootless I remember I remember I said podman has a lot of rootless features that you can Implement easier than Docker having a read-only file system so the underlying host that's running your container everything should be read only except exactly what's necessary to run docker um in fact I'll take a quick pause on this um yeah let's talk about 20 minutes okay

um I'm working on a project called ponecube it's a kubernetes exploit path um and what I actually figured out was I was able to exploit an Azure kubernetes service so AKs cluster I was able to get an exploit in there and they don't have root only file systems I was able to get a shell on one of the hosts the worker nodes of the kubernetes deployment or the kubernetes cluster and install a Cron job that pinged out to my C2 server I just needed a proof of concept there but it was not read only I can just access it okay so even as the even if you're like oh everything's locked down the route not good enough

you need to make sure it's completely locked down and read only um if you're running Docker itself like you're running it individually drop Linux capabilities that the actual engine shouldn't need you know uh you usually want to give Docker certain Linux capabilities like net admin things like that because it might need to give a container a certain privilege well whatever you don't need to give your containers make sure Docker doesn't have access to either because if Docker ever becomes compromised it now has more than it needs more permissions than it needs and then don't run containers as privileged make sure you're running them as unprivileged pretty simple there and then the orchesters and hosts again

kind of similar to the other one this is really kind of hand in hand secure your host root only file systems things like that follow basic host hardening um I mean it's really simple protect yourself against container Breakout making sure you do security scanning within an orchestrator again now we're talking about kubernetes things like that openshift make sure you're scanning that environment itself the actual orchestrator the API environment check how your networking is configured how is your storage configured these are all done via like drivers and plugins how is your networking configured is it encrypted at rest is there something encrypting it in transit same thing with um or excuse me storage networking making sure you're

you know having good uh like firewall rules things like that they still apply even though it's a container environment or an API environment basic rules still apply here um and then your API access making sure you have good Arbok controls on the actual kubernetes API okay and so now having seen all that let's look at the entire container security life cycle um again this is kind of my own from building images deploying images and securing them this is kind of my own um idea of what a container lifecycle should look like in our build stage this is where we apply the concept of shift left okay you've all heard that term Buzz term but it's important pushing

security operations further left or for you guys left in the development cycle are we scanning these things at build time or are we waiting till they're at you know hey we're ready to run this can we scan it and figure out if there's any vulnerabilities no no you should do that way further around in the process uh earlier on in process excuse me establish your image chain of trust like I said before what base images are you using you know for an Enterprise application you may have heard of using like the Alpine base image just a really small really purpose-built you know like five megabytes but it's got a package manager so it's really easy to use

um if you're using that great but if you're not and you're using somebody else's image or even Alpine actually you shouldn't exclude Alpine just because they're really you know known to be secure you still got to establish That Base image chain of trust know what you have in your environment and what you're introducing because let's say they have a supply chain attack are you monitoring their stuff before you use it you need to be doing that and then using proper build Secrets mention that make sure you're injecting those not hard coding those uh and then shipping and storage secure the registry and we talked about that Palo Alto Aqua they've published some stuff on that please please please I

mean it's just getting worse and worse people spin up their own Registries or they use Docker and they just don't configure it properly at good R bot controls so cure your registry uh secure the image source so wherever you're building it from you can't assume that just because I scanned it and everything and I pushed it up to the cloud that's secure maybe your actual host has some malware on it that's injecting something before you actually push you don't know so secure your host make sure you have good endpoint detection things like that that's a talk for another day security registry host so if you're not using like a SAS very um registry like Google or Azure container registry

GitHub container registry things like that if you're hosting it yourself follow host basic host hardening things like that making sure you're encrypting your storage and all that most people don't do that anymore they still use or they use uh like SAS the ones I mentioned before but it's still very easy to misconfigure those so just make sure you're checking everything out and actually runtime this is the real fun stuff so at runtime um you can do a lot of image scanning and you can do a lot of security operations at runtime that you just wouldn't get in you know like a static application scan something like um sneak or other products that look at your source code or look at your

Registries that's all static it's not running anything unless they have a dynamic component where they're executing it in like a private environment that's all static and so in your actual run time you can do more you can do more security and get more insights I'll mention a few products I've mentioned a couple so far but for like run time you can use admission controllers or you can use a service that does all this for you um one good one or two good ones aqua and systig have a really awesome uh runtime security offering where they will monitor the actual system costs the actual like kernel ebpf system calls like as low as you can go to the kernel

they're going to monitor that and figure out what's happening in that container what files are being touched what permissions were used what TCP calls did it did it make um pretty much everything I mean you're looking at you're watching the kernels you're seeing everything or you can use a little bit higher up in the stack you can use what are called admission controllers where you're monitoring the kubernetes API so theoretically everything that's happening within kubernetes is going through the API and so if you're monitoring the API you should be able to catch just about everything again specific to the uh to the orchestrator so the admission controller says hey I'm going to evaluate every API that API call it

comes through and evaluate it against a policy if I have a policy disallowing maybe new files to be written or certain permissions to be used I'm going to catch that I'm stop it I'm going to learn on it I'm going to kill it whatever you can set that policy so admission controllers that's a great thing to use or you know use a SAS program or application that will help you automate all that and deploy that scale like in a big like any kind of press image pinning um so again if you remember when you build an image or even a layer it's all hashed and then when you build the final image it's also oh basically a

collective a cumulative hash if anything changes in that image there's a new hash so when I'm building typically you see like latest tag that's a default tag that's used in Docker when you build an image um that is what's called a mutable tag if you change something and you retack it as latest just overwrite the old one an immutable tag is the hash if anything changes in that image there's a new hash the old hash cannot exist or it's not valid if something changes in there it's immutable and so by using uh image the process of image pinning it's a very hard very complex process and it's not it's not easy it's not for a lot of

teams to be honest but this is like next level you're saying instead of saying I want to pull you know whatever app name latest I'm going to say I want app name and then you give it the long you know whatever bit hash that's got now you're saying it doesn't matter if someone has a supply chain attack on me and they push something and or they pull it and they figure out how to reverse engineer my application inject malware it doesn't matter because I'm saying I've scanned that hash and I know it cannot be changed and before I run I can use the mission controller to say hey is there a new hash out there then I'm

going to try that's I'm being I'm running if there is there a scam is there is it in compliance and if not don't let it run so you can kind of combine a lot of these things within like in this one's in particular kubernetes using the admission controller but with image pending you can use that with Dr podman or anything like that host security obvious make sure you've secured the host uh and then network security again basic rules still apply making sure you have good firewalls uh good rules on that so some tooling considerations were kind of nearing the end of the presentation um this is not an exhausted list but these are some really really awesome

ones uh systig I mentioned them they have an open source tool called Falco Falco again is the tool that sits at the evpf layer of the kernel and is watching everything that happens and you set up your policies and your alerts but you can just log everything and then you can have some third-party uh tool like Splunk elastic whatever monitoring for particular events that's Falco is catching systig also sponsors Cube shark um I think the founder of systig is one of the co-founders of Wireshark the code for the original code developers and the person now in charge of cube shark was the other developer of Wireshark when they first started so they've kind of come back together and so this is

basically Wireshark for kubernetes it's a really awesome tool you get super detailed information and you can combine that with Falco alerts to get insights like hey in avert an event fired off what actually happened well now I can go into Cube shark and look at the actual Network calls look at the file calls um the system calls anything like that or you can just roll up and use systicks you know SAS offering and they do all that under the hood for you uh same thing with aqua they have an open source tool called trivi uh both of these companies by the way have been Avid supporters of Open Source as well as sneak uh huge supportive open source

software and the community so definitely check them out but trivia is kind of your static scanner that you can run in your build pipeline say hey do I have any vulnerabilities in here it looks at everything in the image looks at all the versions and checks that out and they also have supply chain much like systig they have supply chain runtime Security in their SAS offering so if you're in an Enterprise definitely check out these top three these are probably the biggest when you're considering security from a developer or devsecop's point of view not from like a cloud throughout in other terms the cspn perspective that's from That's from the other end the angle so you got your developers coming into

the cloud this way and then you got your Cloud administrators your Cloud Engineers they're concerned about the actual Cloud components the actual setup of you know the kubernetes costume the actual setup of your VMS and things like that prismacloud they are pretty big in the cspm as well honestly as well as uh the developer aspect but if you're really coming from the devsecops perspective these are probably your top three um and again I mentioned they're huge and open source sharing of knowledge he actually sponsors devsecops um they have a division of their company that all they do is they pay them to manage these communities so they're really giving back and making sure that people are getting educated on these

um let's see so this is one that you can just use no tooling very much like trivia this is open source uh Claire will just look at your image and it'll spit out a bunch of CVS hey you got all this stuff here's the actual description what's going on what can people do um really great static application scanning that's called Claire and you can run that locally yeah

yeah this is so here um clear scanner and then you are looking at the actual image tag so I'm looking at PHP 7.2 so if I don't have it Claire's going to download it locally just like Docker will and it'll scan it so that's scanning from a registry you can scan your own Docker file so it doesn't have to be in registry but this one in particular I'm scanning something from Docker hub great question and if anybody has any questions feel free to interrupt me and we'll have time at the end too uh dockle this one was probably one of the first tools that I started using for gosh about four years ago dockle is what

we can use to help look at uh best practices it can look at the image and say let's see if we can get through here uh app credentials.json yeah that's probably not a good one uh let's see remove the APT lists kind of shrink your image size making sure you don't leave things in your image um or it may say another thing I forgot to mention before about your caching is it's going to scan each image even though it's read only so let's say I include that credential file and then in a subsequent layer because I needed that credential file and subsequently are deleted I think no harm no file it's gone by the time the image is built the

problem is it still exists in the previous layers it was just masked over so daco is going to go hey I found that file in one of the layers which if I can anybody else can too uh let's see what's another good one uh don't use no password uh confirm your permissions on these things uh I found the MySQL password is hard coded in a EnV uh environment variable um things like that you last user should not be root you know so best practice assessment yeah

say that one more time yeah yeah so yeah this will check the image that I didn't scan didn't have anything running so because I was running at his root so this didn't exist and this again is um let's see this was a test image that someone that I pulled from a tutorial where they intentionally left in a bunch of this stuff um but yeah if you're running a pseudo Docker will catch that or other tools like it will catch your best practices and then kind of help you do a health check on your image am I doing something that's against best practices again a lot of your Enterprise tools will have this baked into their products right out

of the box um all the I mentioned in the previous slide will do that um Docker content trust this is another one saying hey you haven't enabled content trust which is another layer of immutability you're explicitly saying who I trust things like that hey you should do that uh so that's dockle um I have a few more tools I could talk through but obviously I want I want to leave a few minutes for questions and all that but hopefully um this will help containers to you not be a black box because without using the right tools without using and knowing the fundamentals of how containers are built their black boxes traditional Security Professionals so with some of these new

tools and the idea of the container life cycle containers won't be a black box anymore so I do have some time for questions if anybody would like to ask yes

yeah so the question was any particular opinion or uh I guess opinion would be a good word on the difference between like Azure container Registries or security issues with Azure container registry Docker things like that the answer is no um as long again we're we're assuming that they are patching their own systems if we can assume that obviously if they're not and something is disclosed then yeah that's a big deal

yeah okay great question so any security configurations that are common in like Azure Google Docker the biggest one is our bot controls so if I'm using Azure devops for my code repositories my build pipelines and I'm using Azure container registry to store my actual images am I giving the right people access to that Repository how am I auditing those users and groups is probably what you would use like Azure security groups and things like that that's probably your biggest misconfiguration that you'll see is not limiting the permissions that they have to the Azure API or things like that to introduce a supply chain attack for example but the underlying SAS application itself no I mean they're

all pretty much the same as long as again they're patching their systems and obviously some will have issues of it more than others but for the most part they usually do a pretty good job of staying up to date because a lot of people rely on

yeah so I would like to say well I have two answers one the host is probably easy to miss because that's not what your developers are like working out of they're working out like let's say they're using kubernetes they're more concerned about kubernetes itself and making sure that it stays updated but honestly I would love to say that developers are being secure and constantly scanning their images and doing best practices here everything that we've talked about within knot so probably evenly split but containers are a little bit more complex they introduce a lot more um potential issues misconfigurations than a traditional hosting and that's more traditional security host security and so I would say containers is probably

the bigger threat because of the amount of complexity however that doesn't mean the host is any less important and it's super important yeah

yeah so most of the time the question was the different types of like socket or the communications of the docker engine the socket is just one type that's usually the most popular since it is a file socket like a Unix socket for example it is by far the fastest your latency is going to be almost none versus you can run Docker for example with a TCP Daemon socket you can access it over a network which is highly highly highly unrecommended Docker does provide best practices for that our bot controls network security controls things like that but that's not recommended so the Unix socket is the most popular you can do TCP socket as well yes I

can you say that one more time an immutable run time

right right yeah yeah so the question was about um like an immutable I would say like the underlying host or the the run time what does it have access to like

so so maybe I could talk to you after just better understand your question but um having root only file systems uh running your Damon or your engine I should say uh dropping Linux capabilities making sure you're running it as rootless or um without roots without elevated permissions but maybe I could talk to you after get a little bit better understanding what you're talking about yeah yeah I'm sorry say that one time I can make them available if you want to come up to you afterwards I'll get them available and then sorry I can make them available and then if you want to come up after uh I could talk to you but they'll also be on the YouTube recording

I believe uh this everything's being recorded so you can find it there too I can't see any videos yeah lxc just for um where I'm trying to get as close as I can to a virtual machine I don't prefer it for all workloads but like at home for example because I have cheap hardware and I don't want to virtualize everything in a VM I use lxc because it's lightweight enough but it still is most like a VM in that I don't have to have constantly running processes like with Docker when the process quits the container quits unless you automatically restart that and for a lot of the stuff that I'm running I don't want that so

that's why I prefer lxc for it when I want to get as close as I can to a VM without actually having all that overhead yeah thank you

say that last part one more time

yeah um I may need to talk to you after that might be a little bit longer of an answer and also we are out of time um but yes I I want to talk to you about that any other quick questions no all right well I'll be up here if you have any questions I appreciate you guys thank you [Applause]