A Minimal Talk On Distroless Containers - Will Dollman

hi everyone my name is will I'm a security engineer at source graph so about 18 months ago at source craft we were having a sort of recurring issue with containers and I think it's an issue that probably a lot of people here are also having vulnerability management so Source CRA is a code search tool and it lets you index source code from wherever it's stored run queries against it and because Source codes is such like a sensitive asset some of our customers want to run our product on their own infrastructure and these security conscious customers they also want to scan the containers and then some of them the very picky ones if there are any higher critical vulnerabilities they

won't even deploy a new version so this was causing us a lot of a lot of trouble internally um and there's a whole lot of talk about vulnerability scanning and all that but in general we don't want to ship you know containers with vulnerabilities in them and having this issue with customers meant that we had a really solid business reason for of like put a lot of time and effort into trying to figure out a solution to this and seeing how we can reduce this paino so one of the things we started looking into was dist containers and we found a lot of benefits there but what we didn't find was really many success stories of

anyone actually adopting them successfully so after doing some research we decided to make a bit of a bet on it and I L the product to to migrate all of source craft to drers and it it wasn't a straightforward project it took a while um but we did see a bunch of benefits and you know one of these was fewer vulnerabilities so before this project we were averaging 30 high in critical CVS in each release obviously these weren't actual CVS um we triaged them we made sure they didn't affect us but that's a huge burden on security some customer support you know maybe reduces customer trust if they see that many cves um and switching to disas

after the project we got down to zero CVS obviously it started going up again immediately but it was at a slower rate uh getting down to zero meant we had less toil um we were able to deploy some consistent automation for you know oneclick patching things when they came up and we also saw an increase in customer happiness that's a hard metric for security to influence normally um and it's quite hard to measure but one data point is that we had a huge customer where every release we had weeks of back and forth where you we would cut a release we would give it to them a week later they would scan it they would find new CVS that come up in

that week then send then send us an Excel file of vulnerabilities we would review them we T the release and around around you go and after we switched that just disappeared that work just disappeared so we estimate maybe 10 hours of time saved per release and as we release every two weeks that adds up pretty fast and we saw a few side benefits as well so there's a reduced attack surface because there's less stuff in the containers and once we use some additional tooling we managed to get reproducible builds as well which is a nice supply chain feature so I kind of think dist is really interesting because you know there are a lot of security benefits to

it but there isn't much information out there and it's not very well adopted so the kind of goal of this talk is to help you understand what it is um how it can be a powerful security tool and you know whether you might actually want to use it you know as with everything there were trade-offs so it's not the right solution for everyone but even if it's not it's kind of an interesting way of building containers that looks at the problem a bit differently so I think there are definitely some lessons there so in regular containers you have your application but you also have a bunch of tools libraries other packages that come as part of your base image and dist

containers take a different approach where they don't include anything by default and the idea is you just install the base dependencies just bare minimum of what you need for your app to run and that extends not having a package manager in there not having a shell in there so it sounds pretty extreme but there are a bunch of benefits you know the obvious run is a attack surface say the worst happens and someone gets remote code execution in your app it's harder much harder to get a reverse shell from that if there is no shell in the container still possible but harder um and it also makes your containers more secure and it has a really outsid

impact on vulnerability management the biggest benefit we saw is there is just less stuff to accumulate vulnerabilities so the amount of work you have to do just slows right down so to kind of put it into context I want to start with a really simple example so I wrote a very straightforward hello world go binary um and I started by shipping in a Ubuntu container you know Ubuntu is a great Dro I expect some people here use Ubuntu um and that ended up being 105 megabytes so for hello world that's that's pretty big there's a lot of stuff there we don't need um and the next step I expect most of you would probably say Alpine let's

look at Alpine and that gave us a container that was 10 megabytes so that's that's doing a lot better but the binary is you know 1.4 megab so that's still 9 Mega of stuff that we just we don't need um so kind of approaching it from a dist Direction you know because it's a go binary it's entirely self-contained we can just use an empty container and Shi it in that and that gave us a container that's 1.4 M so there's literally nothing else in there but this binary and to prove it works that little hell world so you know it's important to mention that having stuff sitting around in the container doesn't make it suddenly vulnerable you

know the attack surface in this container is Tiny so it's probably not going to be any less vulnerable but you know as security engine is were kind of beholden to vulnerability scanning so kind of to illustrate this I built these about 4 weeks ago and I scanned them a few days ago and the iunu container had 15 vulnerabilities in it the Alpine container had two vulnerabilities in it and the scratch the dis container had zero and that kind of just indicates that even though they're doing the exact same thing some of them are just accumulating vulnerabilities at a much higher rate so the first time it kind of worked and maybe people are think thinking oh

this dis stuff sounds pretty good um but the real world is a little bit Messier so you know say we update our application to fetch a web page or something simp like that and we use the same empty distress containers before so our app goes to the server the server says here's my TLS certificate and our application goes uh what do I do with that so the problem is that go relies on the the um systems root store for TLS so because it's an empty container we don't have any root certificates there so it doesn't know what to trust um and there's another example like Source graph makes um sourcc makes codes uh code searchable so it needs git

and git needs it needs Pearl it needs libal it needs all these other dependencies so although we kind have said a few slides ago we don't want a package manager we kind of need something like that to manage this dependencies for us so let's kind of move on to how we s about so you're probably familiar with how you build a normal container you know you take a Docker file you the from line that pulls in an image that someone else has built and then you install packages you copy in your own files and then you're onoc build and you get a container out the other end and what's different with discus is that instead of

using this Upstream image that someone else has built you build it yourself and it's customized to exactly what you [Music] need so kind of the way you do that is you start with an image blueprint and you basically list any packages you need any users directories things like that and then you run it through some tooling a bit of dist tooling and then you get a base image as the output so for example in the previous um example we probably want the ca certificates package because we need to verify TLS connections and for Source craft we might want to install the git package and some other things so when I say dist tooling there are kind of two main tools out here so

Google discus is a really good starting point they provide pre-u images um for different run times so go Java python all that stuff um and if you got really simple needs if you just want to run a python application with in as minimal container as possible it's a really good way to get started the problem with that is if you want to then customize those containers you I said Source CRA needs get if you want to add stuff into it you need to start using the basil build system which is has a very steep learning curve so that's kind of a problem you'll run into and the second piece of tooling is app code and that's what we landed on at

source graph so that's much easier to get started with you basically write a simple yamamo file run it through appco and it will give you a image that's going to pre-built and entirely customized to what you [Music] need so this is a really simple example of an app C file that describes a dis image which our um our go binary can use to fetch web pages um and you can hear it's pulling pulling packages from the Alpine repo it supports any any Linux dist that supports the APK formats and here we're just installing the ca certificates bundle that's that's all you need really for this particular case so if you're thinking this sounds interesting uh I'm going to share a few

kind of best practices that we came up with um so at source craft we migrated around 30 containers sourcc is kind of like a fairly complicated SAS application and we did that in kind of one big bang so basically we One release we had all the old images and the next release we have shipped everything toess and that worked quite well for us because uh Source graph is like a single product I think if I were approaching it from a different perspective you know if you've got maybe some Java um containers and python ones and go ones kind of pick an ecosystem and start targeting those start targeting them one at a time to sort of focus your

attention and the next one is to kind of make it simple for devs to use you know most devs know how to use Docker and if you take that away they might get a bit grumpy um and dis kind of changes how you think about dog files you do less work in the dock file and you do more in the base image previously and people people won't know how this works straight away so you need to kind of make sure you've got good documentation and it can be self serve as well you expect a few rough edges at the start but over time developers will kind of get used to it and we going go

smoothly and drers kind of enforces best practices and I think if you look through your company ster files you'll probably find a few that are definitely not best practice so maybe you're fetching a binary from GitHub maybe you're download loing some source code building something in the container um we found out a lot of things like that when we were doing this so the way we Sol that was actually following good practice so we set up a package repository um ourselves so we then packaged these dependencies and could just include them in this build process and not on the slide but at source craft we also adopted basil um and we're using an APPA rule set which

makes it easier for kind simplifies the build process for us and it lets us build fully reproducable images which is a nice thing you can get when you're using appco so the main trade-off with this is you're kind of taking on additional complexity and you're trading that for security and compliance security and convenience so it's sort of like you're slowing down a bit but the idea is you will go faster because adopting distress has The Upfront work but over time you'll have easier vulnerability management and then there is a technical switching cost you know if you're switching to a new Linux drro that obviously takes some time to migrate over into test and you need to unpick

any sort of like bad practices that the CPS in and because you are now effectively managing your own base images you're not relying on Alpine or YouTu to keep things patched for you you need to write a bit of tooling to do that kind of stuff for you so in terms of like changing Dev mindset I find most Dev developers are usually on board with security improvements so long as it doesn't block them um so if you've got clear documentation about how this works and you know developers can make their own changes and they're not blocked they're pretty on board with this kind of stuff and then debugging complexity is kind of the last thing because you're

removing a lot of tools from the containers it can make it harder to debug the containers you if you don't have a shell how do I get in how do I sh in and uh bundle around so there are some solutions here you know kubernetes has ephemeral containers which lets you attach a side car with debugging Tools in it and that's very convenient um we're still working on the trade-offs ourselves um you know we have some containers that have debugging tools just because it's worked out that we kind of need them it just makes life easier so if you're kind of wondering should I adop dist lus there are a couple of prerequisites I'd recommend

one is having a really mature continuous integration pipeline so building things automatically all the time um I work really closely with our Dev infer team to build up some of the tooling we needed for this because essentially you you are maintaining your own base images um so that requires a bit more work and the other is really that this isn't particularly low hanging fruit so if there's other low hanging fruit you should probably take a look at that first and kind of figure out is there anything we can do to improve fundability management but if you're happy that you kind of TI both those boxes then it's really a question of um what security requirements you have you

know do they outweigh the switching costs if you're in a regulated industry or if your customers run your containers and then scan themselves maybe that answer is yes for us it certainly was and we've seen a lot of benefits cool um I've said the GitHub repo with the demos from this talk um and a few more examples as well so if you're interested in kind of playing around dist lus or getting started um the QR code goes straight to that repo um happy to take questions if there's any time like to thank Greg my mentor it's been very helpful froming this presentation together thank you

Applause would anyone like to ask a question don't be shy un Bo everyone I think you just gave a very thorough presentation oh we

do that's a good question um how long did it take us to convince the developers it was so scec we have a sort of internal RFC process where we submit proposal um it didn't take as long as I was expecting we had some quite good I think developers knew that we having trouble in this area and they were kind of keen to not be blockers in this um and we did a lot of the heavy lifting for them so in terms of developer changes there wasn't that much to adapt to another one over [Music] there were there any uh non security related benefits Beyond obviously size yeah I mean size size is one um I think for for us we had we saw

quite a lot of benefits because we started a project to adopt basil at the same time which is a sort of build system that can build anything so you can now for example you can say build me this container and it will go from absolute scratch it won't you know install the packages build a container build our binaries put them on top that was a kind of nice Advantage we saw because that gives us a really secure supply chain where everything's hashed um hermetically um sort of EX sealed and like immutable so you can run the same command twice a week later you get the exact same thing which is something it's kind of harder with Docker because you

run it and then you might get a different set of packages installed or all sorts of

things I think that's all the questions thank you so much give another round of applause