Kubernetes Software Supply (Kill) Chains

just want to get the energy up real high um welcome to my talk kubernetes software supply kill chains the most clickbaity title i could come up with um first of all i appreciate uh besides buffalo number one it's the first b-side buffalo ever so give some appreciation for [Applause] this i want you to check out my my swag i got from the organizers this is hack the plan i've never seen swag like this this is amazing this is one of a kind not just for buffalo but i don't i've literally never seen hacker swag that has any of this stuff so okay let me uh let's dive into it this is what i want to talk about today

um software supply chain case studies i guess um what has happened to software supply chain attacks in the past uh we'll go through an actual exploit a case study of kubernetes compromising a container registry back throwing a vault image what actually ends up happening we'll do all this stuff and then i'll go into more defense um i was just mentioning like i guess kind of i'm in the phase of my career where i'm doing a lot more defensive stuff like how do we solve this stuff as an industry so we'll go into uh you know how to defend yourself from getting solarwinded uh this year and why you should be specifically concerned so my name is mark manning i go by

auntie tree on twitter if anybody uses twitter anymore i work at snowflake right now doing a bunch of everything sandbox kubernetes stuff um i used to work at a company called ncc group where i ran a container practice just exploiting um anything that had containers in it if it's kubernetes or docker or if it was a car literally running kubernetes in the dashboard we could work on that so i've got a bias towards like this is kind of my career in the last five years or just be like containers are bad and people like yeah i know do something else i'm like oh [ __ ] kubernetes is bad uh i am from rochester anybody from

rochester actually yeah okay um i i started the security besides rochester with which a bunch of other people um we have russia 2600 so i like this idea of like this western new york uh linking up communities here so this has been great so this is your software supply chain i hope um in the best case scenario you've got developers that send your organization a bunch of code code goes into a place like i don't know github gitlab whatever you've got some maybe you're using like svn still or mercurial i don't know the code gets stored there we build the code maybe you've got some golang thing that gets compiled during that build phase there's a bunch of dependencies that get

resolved right like these are the third-party libraries and these are the other like third-party developers that are doing stuff that you don't really know but it's getting baked into your to your build process and then the packaging refers to like any kind of delivery of like maybe it's a packaging of a container image to a you know a docker image registry or maybe it's just dumping it to an s3 bucket that you trust or you know packaging an rpm into a yum repo it's like that's this like where do the artifacts get stored and then consumers are uh if it's if it's internal it's your internal consumers or your internal production that's running your code or

if you've got an open source software stuff it might be a whole bunch of other people so why are we going after software supply chain in the last couple years like why are attackers doing it and why are pentesters doing it now too the old way would be like hey let's compromise an ec2 instance and that kind of was cool but things are getting more ephemeral and we can't really maintain state we don't keep those back doors as long as we used to and i used to get messages all the time about like hey i've got a shell in a in a kubernetes pod and first of all i'll be like why are you sending this message over sms this

is totally illegal you're going to get me in trouble second of all it'll be like good luck what are you going to do from there that pod is going to turn over really quick like probably lives for like an hour if you're lucky and from there it's not like you can easily pivot and escalate and take over take over the whole box so uh attackers are going for more persistent stuff they're going after your supply chain uh they can be persistent for years they're not exploitable with odays you're not dropping some like you know crazy jenkins o day you're just going like oh it's jenkins i exploited it because it's jenkins and it's not just like your organization

could be all the other organizations that you're a dependency for right so this is really what your supply chain looks like it's all these opportunities for different exploits at different phases and then as you scale with the different dependencies and so many different authors that kind of increases the the uh the risk to your organization because i think a lot of you probably taking that sam's course that says hey when my aws account gets compromised here's what we do we go into the battle room and we we do all this plan and like okay let's do this guys and we're gonna have an instant response and then like what do you do when you compromise

jenkins you just go i guess this too early for suicide jokes sorry uh uh einsa is saying that there's 400 increase in supply chain threats i mean you can just look at the news and kind of feel that vibe 650 percent increase year-over-year supply chain threats if you don't feel like that's that's accurate you're wrong it's all we're seeing is like different supply chain threats we will say that like this is biased because we've been calling things different stuff back in 2015 than we have now so i wouldn't say that like things are getting labeled as supply chain threats when they weren't before i used to try to make a slide of like hey let's look at the timeline of all

these different supply chain threats back to 2017 and and let's just do like in the past like two months because it's easier to consume um in april we had people that were protesting uh the russian war and they would they would on purpose uh backdoor their own code so node ipc event source they added in a malicious stuff that would say like i'm gonna look at the ips that are using my uh my code and i'm going to like either kill it or i'm going to do some back door into it and do some kind of protest the execution the fix crash stuff there was this one dependency in npm called fixed crash infected like over 500

different uh packages that would try to specifically steal discord creds we removed it eventually after it was there for like a year uh then there's all these typo squatting right we know typo squatting it's just the idea of like i've got something named like dist utils is a legitimate one and someone goes out there and like i'm gonna make dist util if anybody ever looks for disk util it's gonna run mine but that was running in uh pi pi for for like at least six months that would back door all of your code vmware had a dependency confusion where uh the internal name that they're using for the packaging was also hosted on github they accidentally resolved it to

the external maliciously hosted version of the package and they uh backdoored vsphere there's a bunch of kafka stuff that happened in the last couple months um the somebody took the so the the cara cara pace whatever package was just non-existent in pi pi there was a legitimate one on github and then they said well i'm just going to take the code from github and i'm going to host in pi pi but i'm going to modify it and back door it and there's no way to say like okay this is the code that was on github this must be the legitimate code that's on pipi so someone just hosted the malicious version on pi pi and said it was hosted

on on github so that's just like literally a clone speedy tts compiler this one was funny because it was like an ethical hacker quote unquote that would just when it ran it would pull down another package as a dependency and that package was empty it was blank and you go like what was they what were they doing and all they were doing is trying to steal your ip address so they could they could write like a cool like presentation slide like look at how many ips i've are running my malicious code um i i don't feel like that's very ethical i don't know why i guess he didn't do anything malicious but but what a dick move

um ctx ph pass this was uh one was python one was php obviously the entire repos got compromised somebody was targeting specific users just stole the repos and blocked them out from being able to modify their own code and and they ran them to steal credentials when anyone used this stuff same thing with um pi kafka that was fun because they they put in a full cobalt strike backdoor when you did a typo squad of pie kafka instead of pie mafka which just seems i don't know who's who's typo squad in that one and then one of my favorite ones is it's because it's named there's a typo squatted one for rust decimal that was hosted on crate i o

but they named it great depression like i just love that there's a name or like exploit just called create depression vendors can't solve this unfortunately they might tell you that they can there's uh stuff out there that can help but you have to kind of lean into solving this yourself you're not going to accept a pill right so here's a poll we're trying to figure out like how many people are investing in supply chain security stuff this year so there's three options the first one is like how many of you are your company or the people that you work for um plans to invest in software supply chain threat mitigation this year well that is dire okay couple in the

back all right how many are like your work is concerned about it and they're thinking about it but they're probably not gonna do anything this year okay and then how many are just like npm install yolo security stuff yes i know you don't don't raise your hand it's fine i know it's everybody else um there's like six categories of attack if you follow cncf when you're trying to talk to like the business of like what what do we defend from i like that it broken down this way of here's here's where previous attacks have happened here's how we can defend against them so the dev tooling is is is one i like where your like development environment

is completely compromised they steal your credentials they own the box that you're writing code on or like they own the tools that you use to write the code like vs code you know um source code just means like anytime you compromise anything related to the source code the source code repo modifying the source code like while it's being built can i upload malicious stuff into the source code uh and then negligence is just the bucket of everything else like it's almost always negligence in some way of just like there was some probably some best practice that you were supposed to do but you didn't do it um trust in signing refers to stuff like okay hey i'm doing signing i'm like

signing this artifact i'm doing this thing i'm putting effort into this thing and then i've got this you know private public key pair and then you go oops my private key was exposed in the whole thing because i don't really understand how to you know protect my signing stuff and there's been a lot of mistakes there like if you can imagine you've signed your commits but then you just keep your your you know commit key on your desktop that is you know unprotected so there's not a lot of value in that stuff the infrastructure itself will show some demos today of like what if you just compromise like the s3 bucket what if you get rewrite access to the the

publishing infrastructure when you do the the actual release of the code can you compromise that and then the most interesting one of course is like the malicious maintainer somebody that just becomes malicious and like gets turned over or like for 20 bucks just sells you his project right and then you can just use that uh to do supply chain attacks has anybody heard of salsa slsa okay this is this is like if you take nothing away from the presentation this is the thing that's going to be the metric for how secure your supply chain environment is and all it does is give you these maturity levels so it's a softer supply or that it's the supply chain levels for

software artifacts and all it is is four levels that say here's how secure my software supply chain is and it's not specifically like nist guidance that says like go do this you know configure jenkins this way it's just like here's at a high level what we expect from supply chain good luck going to solve these things where level one is just literally you get level one by just knowing how things execute and like that is a big task actually for for companies that have been around for a while just like what how do things get built in a consistent way build things that are tamper resistant can i sign things can i actually be sure

that the things built that wasn't been modified or is it just doing like apt-get install from random repos and curl bashing everything three is like audibility of having a log of everything that happened having provenance like going back and proving that the binary that i'm running over here in production matches the source code that i started with that is a a a large feat to to try to work on and then level four is like you know the rest of the owl it's like i need a two-party review i need hermetic builds i need a reproducible build don't over index on level four i don't know anybody that's doing it besides the people that got actually

solar winded like solar wind um they invested so much resources into having all the extra stuff to uh to defend against this stuff i like salsa because it helps you rationalize it to business it can help you talk to the business in a consistent way it helps you measure over time because the thing you're going to have problems with is okay i've secured ci or i've secured my bill i've secured uh my source code but each one of them are different projects and if you're if your organization is big enough they're probably different teams and to try to measure how all the teams are doing in a consistent way like there's no point in just saying like oh i secured source and

then my supply chain is good all right secured packaging and my supply chain is good you have to get them all kind of moving at the at the same time and that's the game so it boils down to this like these are the four this is the table this is this is all salsas right there um you can match up like what you have to the level that you're required at it's also one two three and four and for each one of their different categories where like the north star is for source code um your commits are signed you've got two-party review you've got a full history of your source code that's it for build it's like you need to build

service if it's jenkins i'm sorry i'm going to beat up a lot on jenkins so just get used to it if it's jenkins that's what you got to do if it's a github actions it's whatever you have a build service you can reproduce the same artifact when you run through a job twice can you get the same you know checksums after the whole thing it shouldn't have access to the internet it should be isolated from everything that's the goal providence is is again like all the dependencies that you have that you have to bake in you know where they came from you know the version that they came from they're not just you know you

know the latest tag and they can be traced throughout the whole source code from beginning of your your source all the way to uh the compilation of your artifact and then there's just this common bucket that says like do security access control you know make sure there's no super users like yeah okay we need to do all this stuff and hopefully you're already doing the best practices but it's kind of hand wavy at the end but it's also also important to say like jenkins needs to be secure and needs to be configured correctly so we'll go through a couple case studies um this is your environment this is like the the first step of this case

study and at the end i'm going to describe this thing and then i'm going to ask you if you can figure out like which one i'm talking about if anybody paints into the latest attacks so this one was uh an attacker owned a docker file because static credentials were baked into the docker file it was using uh intermediary layers to build the docker file so you can set up a docker file to say like okay i'm building nginx but like before that or after that i can kind of have a separate container that goes down like pulls credentials or does some like previous stage and then you can copy things in between them but what they found was like there was a

there was a stage and any intermediary stage that had all the credentials for the bucket and when you execute the the build you can go back up and say hey give me the credentials so long story short they stole the credentials to the packaging of the bucket so it gave them read write access because these are the credentials that after the thing was created after the thing was was compiled the image was compiled it would upload to um like just an s3 bucket or google bucket with that they modified the installer script which was just a bash script and everybody and then they told all their users to install this software just curl bash uh

this command just copy and paste this into your terminal everything will be fine it goes out to this bucket and installs everything runs like this installer script this also back doors the update script um so compromise everybody that's using this software does anybody know what software that was what compromise that was this is um uh one called code cov codecov was a develop a dev tooling thing that a bunch of uh vs code users were using that was like an ide thing you know just run in the background saying oh this is this thing that would resolve a whole bunch of like code for me so this impacted over 2 million users of codecov folks that if they were running

this code all of it was getting backdoored and all of it was getting was getting owned so how do you fix this it's easy to say like we stole credentials and we just compromised the stuff so like between like tactical solutions and and strategy solutions you go tactical stuff is just like don't curl bash don't tell your developers to curl bash and don't put secrets in your docker files obviously but we know developers we know that they're going to do that how do we solve this from the actual supply chain level the the kind of goal here is artifacts authenticity signed artifacts proving that like i've got a thing that i build my image is build my installer script

was built from this environment i sign it so wherever i put it and whatever credentials uh that somebody might have to put the thing there they still don't have the key that signs that so there should be a separate like we'll talk about this in a second what cosine means but there should be a signing process that just says like i can verify the integrity of this thing and tell you tell your users to do that and then there's build providence of just uh okay so there's there's a signature but a signature just says like i had the key at one time and i i i signed the thing but the build provenance says here's what happened in

the build here's where the build came from when someone just downloads install at sh they have no idea where it previously came from they can't really say like oh it went through this build environment they went through these users so if you take nothing else away take this one away like you could drive home right now you could download cosign and you could verify all of your code all of your container images in like less than an hour cosine is a project from sigstor it lets you specifically do oci images but it can really handle any kind of binary that you want you can do like just basically build an image generate a key pair

and then sign the thing you can bake it into github actions like if you go into the the marketplace right now you can get a github action that has all this stuff baked in where you just install it you load the private key into your your github action source you run the thing all it does is say like okay here's the cosign key sign the thing and everything is done everything else is kind of taken care of for you there's a lot of hand wavy stuff where i'm going like how's the password store in the private key but like let's just say like you can sign the thing and you get authenticity like out of the box

and what i'm saying is like an hour it's probably even less like if you go to github actions and you say install the cosign installer today you can get everything signed you can have all of your releases signed without like even paying attention to how you're configuring it so this other case study is um there's the goal here is i want to do some like typo squatting and what i'm going to do is first go find some famous developer and i'm going to spoof their github profile so it looks like i'm actually their developer and i've done this before like fishing exercises you go find someone famous everyone trusts spoof spoof their account make it look like you're

actually the developer and then go out and typo squat whatever package you want and just host it there and say like hey this is from this legitimate developer and in this case they went so far as as as saying like here's the stars and my repos and they're copying over like the resume of the developer and what was interesting about this is not just a normal type of spot the the typo backdoor type of squat backdoor had a check for the environment variables that when it ran it would try to see am i in a build environment am i in like get lab specifically and it checks to say like okay it looks like i'm going through a build process

now is when i'm going to inject a back door so it says like okay environment variables say like git lab underscore i'm going to go up to a c2 i'm going to download an extra stage and i'm going to inject a malicious stage into whatever you're building it only did this like if you're doing a static analysis all you're seeing is that like there's code that's checking the environment variable and there's like an if statement that says if this then you know make a url request which makes it much harder to analyze than as opposed to like just baking the entire stage into the execution so this back doors all the builds and note that

like it's back doing all the builds that are using this as a dependency it's not like their builds get back to orbits whoever uses it uses this as a dependency their back their environment gets backdoor somebody found this just by accidentally looking at like the names of the packages and the people that were there instead of like mark it was marked with a k it was like mark with a c and somebody just dove into it and found that this has been going on for the past like uh six months so this is another another recent one does anybody uh created a payload to execute it does anybody know what this one was from this one i mentioned

before yeah this is like literally two weeks ago it's not really fair um uh this was the uh the crate depression one this is where somebody actually uh if you're not from the rust community you're like i don't care this is uh somebody made something malicious in create that io was hosting all the stuff doing all those other things so how do you solve this going back to salsa it's hermetic builds hermetic means like can't connect to the internet during build like why does your build have to go and curl something down why is it even going up to yum like all that stuff should be centrally managed um having approved dependencies like is so

critical like these two things are are really simple to do relatively if you buy artifactory if you buy nexus if you buy like whatever like an s3 bucket that just says here's the approved dependencies that we we support as a company you mitigate this risk of typo squatting by like you know 80 percent the other thing that helps this is like having an isolated build between release and between uh development or testing whatever you want to do we should not be like have the opportunity to just blur the two like if i compromise the build environment you shouldn't be able to compromise my production release you could compromise my testing stuff and staging it shouldn't affect uh release

tactically if you just want to do this stuff there's companies that support this right sneak x-ray from artifactory somebody use phylum her to file them phylum is a uh a threat intelligence company that's just going back and doing like dependency analysis on like github uh open source projects so these would be like you give them a pile of dependencies and they'll just go do research on like how sketchy their github looks like did they suddenly get a whole bunch of uh people that are our contributors from china or from russia in the last two weeks okay that's a red flag but kind of an interesting product so how do you pull some of this stuff

off um when we say isolated builds it means from like every every angle how about anybody heard of spiffy inspire you heard that one this is used under istio if you're using kubernetes it's a way of doing like workload identity so you could say okay when when my build executes it gets a service token that it can only access this s3 bucket for an hour you can only push to these things for like 15 minutes it can only perform these functions and you can use this to to isolate at the secrets level at the network level the identity level the cloud level and this ends up being like uh ways to isolate builds from because

the litmus test for me is like can a pen tester escalate from your dev pipeline into your release pipeline you're going to hire somebody this year and you're going to point them out your jenkins or your github actions and they're going to completely own your infrastructure because you don't have strong tenancy controls and you're probably reusing secrets across these two accounts this happens time and time again the other one is for salsa 3 they want you to have strong dependency controls so i mentioned like artifactory but just having an agnostic policy that you can say okay these are the dependencies that i want um i don't want these ones these are the specific versions open policy agent is a way of just

making up a policy that says here's the list of approved vendors that i want to support and this can be used for anything really this can be used for dependencies this can be used for a control to say like i don't want my um my dev environment to accidentally get sent into production like that that can totally plausibly happen and oppa can kind of make that control other solutions uh artifactory can do this sneak kind of has a built-in github can do this then there's these last two i'm going to talk about like tekton tekton and tekton chains tekton is a ci system that is built on kubernetes of course and it's all uh container isolated so when you run a

single job like you know compile this code it's its own container it gets burned it gets deleted you just you know it's just a container for that for that lifetime there's a lot of cool security properties that you get out of that and then techdon chains takes that idea and it says like okay for each of the different containers that run i'm going to have a log of what happened in the previous stage in the previous container and and this is how i'm going to kind of control what happened so you can say okay this is just like a golang binary let's compile it in this image it comes out with this artifact here's the shot of

the artifact here's the shot of the container that ran the image so now like you go to ir and you say like hey i think we've got an incident you can go back and say like well here's the shot of all the things that were used to compromise this thing and you have an actual like you know chain in your supply chain this is i'm trying to uh describe like the isolation that decton chains provide once it's in kubernetes you can start making containers for every separate thing here and you can start mounting the storage in between i can't like again if if you can if you haven't chosen a ci today uh tecton chains from

from google is probably going to be the future so this is your your your last i guess case study that we're going to go through the step one is you own a developer not that that would ever happen like we have so much security and our developers are so smart but we compromised their credentials and they have access to a dev and staging environment we didn't give them access to prod but they have access to a dev staging credentials to access specifically like a container image registry turns out though that we have actually created only one single container registry we use that for dev and for broad so when we give rewrite access to the container registry we actually mean

you get read write access to the production container registry so now an attacker has access to write something into the container registry do something malicious and they just need to build a back door of a malicious container and push it up to the package a repo replace a known good image and back to all the deployments so this one's cheating this doesn't actually exist this is one i made up because we're going to kind of like go into exactly what this means so this is a kubernetes kind of supply chain attack i wanted to show you how many people here use kubernetes okay okay some i can never tell who's here for this watching who's here for

the kubernetes um kubernetes is like the one that won the orchestration war right like everybody's using kubernetes we just accept it um it's most cloud native environments are using kubernetes in some way probably most of them are using a bunch of stuff that they don't know about like they don't know that they need like do they need high availability stuff like no it's like sometimes it just ends up you get thrown at kubernetes this is just for this talk it's just like the output of a supply chain attack we're not going to go too deep into kubernetes and if you're interested in securing kubernetes on the left production kubernetes and on the right like breaking and

hacking into kubernetes uh you'll see me in that book actually but for the sake of discussion like this is your kubernetes cluster it's it's nodes the blue things are supposed to be pods the orange things are supposed to be like the control plane right so the the nodes in the in the pods are just managing your workloads but each one of those pods contains a container and for our discussion it's a container image so maybe you're running like a web server with nginx it's got istio sidecar it's got a vault sidecar but there's like three containers inside of your pod whenever you want to start a node or sorry whenever you want to start a pod it's going to go out and say okay

i need this container image i'm going to need to pull this down from some place and it goes back to a docker container registry now let's see what happens when we own the the container registry so the first step of this was we need to compromise the dev let's just say like we've gotten on there on their box we now have rewrite credentials and we can use these credentials to work across dev and prod why this is like a realistic attack scenario is because this happens like 50 percent of the time to me when i was on pen testing there's a bunch of reasons that you get image registry credentials it could be you gave developers access and their

accounts were compromised okay that's kind of what's happening here the creds were exposed because they got uploaded to some source repo because you go like well i need this build job i'm just going to bake in the credentials into there um the secrets get stored directly in a build job you've got some gradle script that just has the straight you know credentials exposed in there and but my favorite one and this is the one that happens most often is you've got a kubernetes environment that says oh well we want to use a private registry i'm going to give them credentials to access my registry i'm going to use this image pull secrets thing and then you accidentally said like well

they only need pull but 9 times out of 10 when a developer set this up they've always given me push and pull they've given me rewrite access so this was always been like a great pivot point that you could just push secrets back up there so we've got registry creds and we're going to go and build a back door um we need to figure out like what kind of malicious backdoor we should build um anybody use vault anybody heard of all okay yeah faults like the the kind of standard um the cloud agnostic or or however you want to think about it secrets engine by default kubernetes has this you know they call them secrets but there's

base64 encoded strings that they hold for you in an etsy database vault secrets are a lot more configurable you can kind of control around policies and when you get the secrets and it's the more secure solution between the two so i've seen this a lot uh just used as like the secrets engine that everybody uses what's fun you don't really need to know about vault but just understand this is vault has a whole bunch of modes that it runs in there's a in it container that starts up with a pod there's a sup there's like a side car that can run with your pod there's a server that actually stores all your secrets but it doesn't matter it's all the same

image so if we were back during one single image we get actually three different functions that we can back door so that's what we're going to invest our time in so back during secrets the dan kaminsky method i'm back during using dns i'm taking the hashicorp vault source code and i'm recompiling it with this exodus library which is just a custom dns client and a custom dns server so you make a resolution and it'll send out it'll exfil all this data back to your custom dns server which is legit if anybody wants to go and do a dns resolution on xfil.io.cz that's my server you'll get some interesting responses if you want random ip addresses that's what

it'll return so the whole point is that what i'm trying to configure this is when vault access is a secret i'm going to steal it and i'm going to make a dns request to exfol the data back out to me why am i doing it this way um number one the kubernetes always has a dns server built into it and it's usually named like cube dash dns so you get kubernetes you get a dns server out of the box it has to be there it's just always going to be there it ends up being like a critical service so you have to run this thing you know you know where it is you know how to

access it so you can even statically bake in like do a dns resolution to this server the other nice thing about this is if you've got like an egress control like okay i'm preventing this service from going out and accessing uh the internet directly um you i guarantee you're not preventing dns x fill or at least you're not logging in enough to prevent this stuff so this is this is the code uh it just basically says like read a certain path where the secrets are and then send it back home so we've got we've compromised the dev account we've built a back door now we're on to the stage of like let's replace the actual

vault image on the repo which should be straightforward um you can do this with just docker push you can just say like okay i've got credentials docker push it's done but i want to highlight this tool for anybody doing pen testing and red team stuff this is a statically compiled binary that lets you interface just with container registries so the best case scenario is like okay you've got remote code execution you've got a shell on the kubernetes pod and you go well i'm not installing docker to be able to push an image to a container registry i just want to curl down some binary that's statically compiled that lets me interface with the registry and that's what scopio is it's

not designed to be a malicious tool it's just like the most useful tool for this stuff so you can say okay container registry give me all the lists of images that are on your uh inside the registry uh copy down whatever image that you want the more interesting one of course is like scopio uh forget tls copy down or copy up the whatever image i've created and overwrite another existing tag so this is the crux of it it's like a container registry is an extremely simple beast uh it's just like block storage and an api it does no verification of like what you're giving it or like where the thing is that that you want to put it

so when you overwrite something and you just say like hey i'm gonna build nginx on my local box and i'm going to overwrite apache on the remote box the container registry doesn't care at all it's just going to be like yeah sure that's fine it's blocks i'll just i'll just hold it for you and you can get it later so this is this is uh the crux of what i'm about to do because all we can do all we need to do is um build my uh vault binary uh that has a back door with the exodus dns stuff in there rebuild the docker container and take that binary that is that i just had

and bake it back in and then you do a scopio copy take the solarwinds creds that i've got there and then send it back up and overwrite the existing one that's named vault cates and it's going to say sure i don't care you've got credentials to to do whatever you want i'm going to allow you to overwrite so we're at the phase of okay we've compromised the dev compromise build the back door and we can bypass the ci cd because we just directly wrote to the container registry without having to go through cid and now we're going to inject the sidecar so it looks kind of like this if we summarize it it's like vault's

backdoor we push to the registry it's stored in the cluster and the registry it's accessible there and then new deployments whenever they execute they're going to take down this new vault image every single time they go to deployment because you're using the latest tag and uh all the secrets as soon as they're accessed gets sent back to my server so let me just show you kind of how that works oh before i do that side note like yes i wrote all this stuff and yes it's currently hosted on docker hub so like in itself is a supply chain attack that no one has taken down yet so i'm just kind of waiting for that so

kind of a meta it's a supply chain talk and it's actually an attack happening right now so this is i'm just going to run like here's the web server that i'm running i've got this web app that if i curl the web app what it's going to do is access a secret when i access it like the web page like because the in the back end nginx needs to read the secret and at the time that it gets accessed vault will say like oh hey i see that you're trying to access the secret oh by the way i'm going to hold i'm going to do a whole bunch of dns resolution and send it back to my

dns server that's at the bottom so this is the way to back door all of uh vault server all vaults side cars anytime anyone accesses or touches any secret inside of your kubernetes cluster it'll now phone home to me how do you fix this um we talked about cosign we can sign images if there was a signed image i would have been able to you wouldn't have been able to like to run the thing you would have been able to to execute the the image in the first place i would have been able to make the back door because i wouldn't have had the the signed uh uh key i wouldn't have had the key to sign

uh and it doesn't have any providence that we don't have a build log of how the image was built there's no way for like the kubernetes cluster to say like oh where did this image come from you just kind of said like hey it came from this container registry that my cluster trusts i'm going to use that because i don't know i trust that that container registry but you don't know what's actually on that registry street again we should isolate dev and release that that would help but this one is kind of most important we have to trust artifacts not where they're from we a lot of times we'll trust like an s3 bucket because we own the s3 bucket or

we'll trust the image registry because it's the one that we configured but we have no idea what's actually being stored there at the time of like pulling down the thing this is the kind of crux of the xero trust models like we need to be able to verify what happened in all the previous stages right at the point of runtime and not care about the thing that it came from or their ip address or the host name like those are really weak uh trust models and then there's like tactical stuff right like yell at your administrators for even having like this thing set up in the first place so that you can have like dev and prod mixed together

there's an important one of like blocking public dns resolution so let me just talk about that one really quick because i don't know any organization that's doing this today but this is how you can do it right now you can do dns x fill on any kubernetes cluster and i would recommend it because this is the cleanest uh way to exfol stuff out even if there's some kind of egress control no one's paying attention to this stuff by default today if you look in your kubernetes cluster it's going to have that line right there it says forward dot to etsy resolve that conf so like use whatever the host has for units resolve that conf to to

do dns resolutions i mean that's super useful sometimes it needs to have access to the internet that's reasonable but you can configure this to say like okay i don't want external resolution or i want to uh like stub out exactly the upstream servers so that i can log them extra and i can have like some you know when it goes to the host when it constantly resolves like x fill that ioio cz it kind of feels like a red flag so having some log and some record that you can kind of do threat detections on would be nice so this is how you do it you can stub out just make a config map inside of kubernetes

tell it which dub domains you want to use or set an upstream to name server and i was having a conversation with the threat detection folks last week and they kind of gave me the idea that you could also set an upstream name server that's kind of like a canary dns server or a honey honey pot where like if anything ends up resolving with this thing it should be a red flag because why is why is your kubernetes cluster suddenly doing external dns resolution when it doesn't need to so securing this thing i i told you about there's cosign which will sign an image we'll sign up whatever you want but i didn't talk about how to verify

the thing and this is the piece that inside of your kubernetes cluster will go in and say okay i see that your image is signed i need to be signed by this uh by this key and i need to be stored from this registry and then i need to verify that at the last second so when this creates an admission controller in kubernetes that says like when i pull down an image i'm going to check its signature and i'm going to say yes or no whether or not i should go in this is a really simple thing that again is like super easy to run if you just wanted to do this command like helm

install connoisseur helm atomic blah blah create its own name space in your cluster it'll set up all the admission controllers for you that you need to do and it'll just start verifying not it won't verify like in this command like which signature it'll just verify that a signature exists on the on the image so you can go back and set a policy on like which signatures you want so this is just proof that like if you take the official hello world image all of them are signed by docker now as opposed to like some sketchy one that i built it'll say like i've got no key for that and i haven't been signing that one

so putting this all together i've talked about like a bunch of case studies i've talked about like okay here's how we can kind of solve some of this stuff but it's really about drawing the whole picture of the supply chain um because this is your supply chain but this is actually all the different vectors of the supply chain there's there's there's so many ways that an attacker can get in so again red teaming was so much easier job than trying to defend us against this stuff but before i start coming in and saying like here's what you need to do um i thought i should step back and say like um security b-sides was designed to be

like an interactive thing so it's not designed to be like i'm gonna dictate and go like tell you what you should do how many people here this is their first b-sides at all awesome oh wow so so matt was saying in the beginning like the difference between b-sides and rsa and like why b-sides existed was like rsa was like this vendor conference it was just selling like snake oil they would do these super short presentations of like 15 minutes long and they just said like hey go install my product and that was the talk and people started getting frustrated and they're just like well screw this i'm going to go build my own conference and we're going to be focused on

community stuff focus on the attendees facilitate this discussion so they'll do like longer talks like this and then just like have more interaction and and get technical and get vendor agnostic and just solve things like as an industry so um i wanted to like break out just like how are you all solving supply chain or what are the threats that you're concerned about and what can we talk about like right now that that would would actually change the industry do you think i know it's scary but i need let's let's get some interaction like what are the what are the supply chain threats that that we are actually worried about in our day-to-day jobs

yeah an old container that just hasn't been updated exactly it's just riddled with vulnerabilities you've got criticals that have just been there from from five years ago they haven't been patched you've got no visibility you don't you don't know how to do it yeah i mean how do we solve that how have other people solve that this is like vulnerability management i know i know some of the people in this room do vulnerability management so let's let's hear someone daring enough to to pipe in what's that firewall virtual vulnerability management of the containers okay anybody else

yeah it's tough to hear people but i think you're saying like get passes for the replacement to replace

exactly you need infrastructure to be able to patch and replace you need an inventory probably of like tell me all the cves for my product and then like how do i actually go and patch them maybe the developers aren't going to know how maybe the developers in the container world they they can they just need to get guidance on like which ones are outdated what other like how do you how do you manage the issue of like okay i've got a container image it's like uh using some node thing it's got like 10 000 dependencies and you go well there's 500 cvs for this thing but i have no idea if they're impacting me and also there's not like it's already

running the latest version i i can't patch all of these things right how do how do people handle the risk of of running this thing

how do you accept inside like okay i don't want these five critical cvs but i guess these four critical cvs are okay how do you tell the business

yeah i think he's saying like yeah analyze it look at look at the situation make it kind of case by case or like do risk analysis of the thing yeah you can't have a unilateral policy just says like no cvs like that doesn't exist so yeah so have a case by case risk analysis let the business understand the risk okay what are what are you investing in right now like where you've got you've got ten thousand dollars where do you spend that money to solve supply chain

right sure i i think you're saying you know invest in templates and invest in like standardization like best practices first you developer training i assume like going that direction of like here's how we want you to do it we'll add in controls eventually but we don't want to have like you know a cost and developer experience to just say like here's the super restrictive stuff that just pisses everybody off let's just tell them the right way to do it and then kind of audit maybe like the when they're doing it wrong and kind of help them yeah

is

sure

sure

i think it's a great takeaway like the developers want to do the right thing they don't necessarily know how you know but if you can guide them in the right direction or give them feedback when they're screwing something up as opposed to blocking them i think that that makes a lot of sense

i appreciate the conversation i i ask you all to go ask your neighbors what they're doing for supply chain stuff and and continue the conversation because the industry doesn't know how to solve this and if i stand up here in front of you and say like this is how you do it i'm going to be wrong and i definitely am going to be wrong about your environment so push it out um

i totally agree gold and blessed containers yeah i'm not with you about like trust the government on it part but i get the i get the intent yeah but yeah i totally agree like organizations could be thinking about like building containers from scratch or coming through which is something so we go through a couple of these just in the interest of time um so you go like okay i've talked about a bunch of stuff where do i start which one's like the the most bang for my buck i just want to point out like signed images for containers like it's so much value and just to say like i just have a signed uh uh a signed image this is how

it works i use cosine we haven't talked about s-bomb stuff today we haven't talked about in toto because we're not there yet like most organizations are just like i should start having like sign artifacts right but i i wanna i wanna have those reproducible build conversations like uh uh in the future this is kind of the tech stock that i've been summarizing like here's here's your existing uh you know software supply chain environment maybe you've bought sneak maybe you've bought github maybe you've got like chain garden force uh maybe you've got open policy engine but here's here's kind of the tech stack that you can you should be thinking about specifically uh cosign if you want

more information this is super useful the cncf has two documents one is like software supply chain best practices cool it's like a vast thing to go over the other one is this one that just came out uh last week which is they've built a like fake software a secure software factory saying like here's how you should produce uh uh secure software here's the best case scenario they make you know it's kind of like a case study in this fake organization that they built and how they solve the problem so again tactically cosign document how everything works have centralized uh dependencies a place like where all your apprentices come from as opposed to like random places on

the internet and then harden your ci right you if you you if you just need to solve the threats today that's what you do strategically it's salsa salsa salsa salsa like that's that's kind of the core goal provenance of like proving what built what when how did it come from where to come from proving that like the binary that you that you just ran in your production environment actually came from things some things that you trust and then just isolate everything like that it's just like basically my career it's just been like isolate it make it ephemeral and then stop caring about it because it can't can't impact you anymore so talked about case studies talked

about how kubernetes is kind of a substantial portion of the supply chain i'm trying to paint a picture of like long-term efforts into sign artifacts policy gates salsa maturity framework attestation stuff the way that my presentations work is if you give me feedback on my talks i give you these slides so if you want to scan this or go to this form give me some feedback on what you thought about this talk so i can kind of iterate and improve upon it you get a copy of the slides questions and comments i think that's my time anyways so thanks very much for joining [Applause]