Whose Pipeline Is It Anyway?: Attacks and Defenses in the World of CI/CD

um I think we're going to begin yeah cool all right uh yes that's correct we are going to begin um good afternoon everyone welcome back bsides uh whose pipeline is it anyway um if you're too young to get the reference I'm not sorry um improv it's like AI can't spell um that might be like the new Turing test we're not sure if you're too old to get that joke I'm still not sorry so hey I'm Matt and I'm Zach everyone um we are staff security engineers at a company called duy Inc um I focus primarily on application security and my focus is infrastructure and Cloud security um but we work really closely together in kind of all aspects

and the ins and outs of our day jobs um so we do these things together um uh quick disclaimer our views are our own and now representative of any current or former employer oh we got that out of the way uh great so why are we here why are you guys here what are we here to talk about um so we actually do do a lot of threat modeling um and in kind of preparing for this like I focus a lot in the Weeds on application security and Zach was like hey and like it's time that we actually do like we need to take a look at our infrastructure and look at cicd and like look at our pipelines and

things like that and I said okay you're right and then we saw a lot of the other research that was coming out and around in the space and we and we looked at each other and we were like oh yeah we do need to look at this um so because of those emerging Trends and because we wanted to transition and start looking at our infrastructure and do a threat model of that that's basically why we're here um so to emphasize on that um the core feature of this talk really is threat modeling um and then threat modeling of cicd and so then to do that we will build a cicd pipeline kind of walk through the structure of it and its

purpose um and then we'll start to formulate like the attack trees and threat models um some of our attacks will be based on some of that research that we saw and it will be like real world scenarios or try to get you know we're threat modeling so it's a paper exercise but we're going to try to bring in as much like applicability to it as possible um so with that uh we're doing the cicd of threat models um this talk is not specifically how to threat model it's not a threat modeling Workshop um but if you've never done a threat model before there still should be something here for you and if you're like oh man I'm threat modeling cicd

next week um there also should be something here for you um so the last thing is this different from the body of work that's out there I I don't know but I like to believe that it is um and we hope we can kind of Workshop that with you um some of the things that we're doing that we appear to be different are like we're we have the we weaknesses that manifest into attacks um on the threat model in the attack trees um and so you know that might be something that's of interest to you to take back to some of the things that um so before we get into anything else that's threat modeling related

let's talk together about what is cicd yeah so like Matt said um before you even start to try to threat model environment you first need to conceptually understand what a cicd pipeline is and essentially there's two converging ideas that are taking place that conceptually start to form a pipeline and there's two lines of thinking two ideas one is continuous integration and the other is continuous delivery or continuous deployment depending on how mature your processes are so you can those are best encapsulated via the uh orange highlighted areas around the workflow here and then if you click sure um so continuous integration is really more so followed as like a practice or a process really more so than like this is the

system or the tool that you're going to use so you kind of keep that in mind and that also makes this like more applicable to what you might be doing in cicd but there's different parts that you can like plug and play and bring in and use um but ultimately the the CI The Continuous integration portion is getting your developers and their code into production as seamlessly as possible the CI part is like the source code repositories the building application process of that um where it's packaged up and then handed over to the delivery enployment yeah so the next phase is like I said either continuous delivery or continuous deployment uh once there's actually a functional

release candidate um this is the next phase the of the pipeline is focuses on automating the the release of approved code changes to a repository the goal is to ensure rapid reliable and repeatable deployment process um there are a fairly exhaustive list of cicd benefits out there three of which are here on the right the quick feedback loop the enhanced collaboration between developers and also the increased engineering velocity uh the ability to push code frequently and often is often the term that's used um great so here is a uh we're going to build this cicd pipeline together um this is a pretty accurate representation of what you might be using or your teams might be using or

someone somewhere in the universe might be using regardless this is a function cic CD um so you know going back to what the CI components are with this um so we have the developers and essentially in like an Ideal World um a customer whoever that is whether uh like it's someone that pays you or it's like a development team that you're trying to get tools to um they would manifest an idea in their brain then the developer would think it and then it would just be running in production that's not actually how it happens we have to like build the thing together to get there um and so we have the CI pipeline to do

that so it starts with like your idees your developers what the the tools that they're using um here this then gets pushed into a VCS Version Control System um and then this is where the code lives uh and then the code lives in its various different forms you have your applications and your products um and then you have your infrastructure as code you have manifests and scripts that are helping the build process and kind of instructing the other facets of the CI uh of the CI getting the build and your artifacts together um so there's triggers in there so these things can happen automatically they can be responsive to what type of code you have

in there um but it also can be something manual you can have like a gated flow process of hey great we're in the source control system but I don't want you to build yet and I certainly don't want you to deploy yet um so you know we have these configurations and Rule sets to talk about what are the next steps to take uh or you know you can just automate all of that and forget about it um your secrets live here and then we talked about your yamal and your manifest like other instructions on how to build so once you do get to that point you say hey let's turn this code code into the binary or the container um

then it gets passed off to the build server so in the build system um we're talking about GitHub actions here but this could be your Jenkins it could be your circle CI um you go into the build and so now we're actually talking about compilation we're talking about building we're talking about getting a binary and an image together um that's ready to be handed into the delivery phas um GitHub actions will also allow you to like start doing your static code analysis um and you're creating the artifacts that ultimately become the things that get pushed up so now we're getting passed over into the artifacts this is kind of similar to your source code repository but it's a container

registry or it's an image registry or an artifactory for your dependencies and your binaries um you put your image up there you give it a tag you give it a label maybe put some rule sets around it um and then ultimately we're going to talk about like the environments but you have a development image you have a production image and then there's processes that can happen that promote the container into being like hey can congrats you're not in Dev anymore you're PR and so uh this gets built into that build um so now that the image exists it's packaged up you have your container and your binary now we start doing delivery and deployment yeah so the continuous

delivery portion you can click next one um it's highlighted in The Orange Box here um like we were saying um you know we have one physical Pipeline with two converging ideas one with continuous integration one other with continuous delivery um that doesn't mean there's like a perfect segue from where one things end and where the next begins so um once there's a functional release candidate and you're sitting there with artifacts that have been tested by the development team uh it's ready for orchestration which sets up to get ready to deploy your application onto a modern environment such as kubernetes so that's best dictated by the Argo CD uh um layer as well and then it essentially controls

the applications uh in the lower environments as well as the production environments So within the Argo CD environment uh it leverages uh a two uh a principle called giops which essentially leverages GitHub as the source of Truth um and then from there there are many pods that are sitting there in clusters holding your application it does an app diff against the GitHub repository that's actually storing the code and that's best uh dictated by the repository server that you could see the flow goes into GitHub the Source Control Management suite and then feeds back and then from there it moves Upstream to the application controller which then is able to deploy your application to various environments

um so that is the actual orchestration but with that said there's also a production environment that hosts your infrastructure as well um and that's best indicated by the the sub bullets under each respective area such as networking infra oidc for anything uh uh authentication um and the eks and storage environments as well um benefits of gitops um it's auditable it's version controlled um it's increased reliability um it Ena you enables you for roll back so you can deploy a a service if something quickly breaks you can essentially just use what what's essentially a git revert command and then you can move back to the previous known stable build um so it's very scalable at to that point as well um

many different developers once you abstract your application are able to iterate on their code um independently of other pieces of your application cool uh how many people threat model here yeah great amount all right we're doing stuff um we threat model 2 and so we have our approach to threat modeling um ideally from art perspective threat modeling is a highly interactive and highly iterative process um I think for the this exercise uh Zach and I did really well in the highly iterative part um I think we could be a little better with our interactive part um but anyway when you're kind of starting to sketch this out of like what you're going to do um you know I'm going

I'm going to laugh at myself when I say this because like I'm sure everyone here in this room hears shift left a lot and you're [Music] like but small scope user stories is like the shifting left of threat modeling it's like hey there's only so many people in this room that can do threat modeling we need to get our developers threat modeling and if you're developing user stories and like having agile Frameworks and doing stuff like that if you just give them a few tools and the mindset of like attacking their own little user story um you know that's shifting left and like getting them to kind of have Buy in and help us out and

what we're doing U and then there's large scope so that's like hey we have a giant framework there's this huge architecture pipeline we need to kind of threat model this whole thing you could do it in like each stage of the pipeline but it also kind of makes sense that while you have the engineers and the Developers and their attention and the resources to do the large thing um and so you know we're we're balancing in these worlds um and then all of this is not to be demanding on engineering I know that you know we're building cycles for them to work on backlogs to do threat modeling um but they have jobs to do

also uh and so you know it's important um in having that highly interactive space to have development and Engineering bought in early and just to to believe what you're doing and in getting them to believe in what you're doing is to have a framework and to have some structure around it that they get and understand we're going to come in we're going to bring the security expertise and guidance but they're bringing the expertise in their system and this has to be collaborative because you know I you know before starting this I was doing application security I wasn't going to be able to work on a cicd pipeline so we got input and buying from them to help us build this out um

and kind of what that bubbles into is like there's all these overlapping layers and so you kind of build this framework C you push your mod model through it and outcomes you know the outputs of your threat model you choose the pieces that work for you and you get rid of the pieces that don't um you know still keep the core functionality and components up so ultimately when you're doing a threat model you want to build the system you want to kind of come to terms and agreement on like hey what is the thing that we're doing are we speaking the same language are we looking at the same architecture so do that make sure

you have that um and then bring in the perspective of like hey now we're going to attack this what is the reason for attacking this what are the goals what is my motivation what excites me about attacking this system is going to drive someone to coming up with attacks to get through it um and then that also kind of feeds into like where do we get the entry points how do I get Ingress into the system and then how do I propagate through it uh so you know the last thing is that um I know a lot in threat modeling it is theoretical maybe there isn't even a system maybe it's just like an idea and so like you can't do POC or

we can't PC everything we come up with because there's not enough time to do that um but have the resources have the references do validation when you can when they start building it validate kind of the attack trees don't exist or the mitigations that you've put in uh have been built correctly um and do all these things together to kind of come up with the proof of concept of like hey the threat model actually achieved something here for security um so with that being our approach to threat modeling um we can talk about other pieces of the framework yeah so you before you start to create tack graphs and start to actually threat model your application

you first need to you know find some sort of uh authoritative framework that you'll be able to map against it um this isn't a one siiz fits-all approach um what you should do is first understand your actual environment and then from there be able to map it against like some sort of authoritative framework to be able to follow the uh follow the pipeline from the beginning to end all the way through the deployment so with that out there there's a lot of Frameworks they all have their each their own benefits for the purpose of our threat model we chose to start our root node on our attack graphs with The Oaf cicd top 10 um I would be remiss if

I didn't bring up cider security they're one of the actual leaders in this area they were probably one of the the main contributors to the OAS cicd top 10 that we're choosing to start our attack graphs with um but you know there was a lot of other architecture Frameworks out there that went into our threat model and helped to understand like the full scope and how we want to attack our environment um and with that said we found that rccd environment most closely uh mapped to the AWS deployment reference architecture and so um like you'll see here pretty soon when we start to actually Define our threat models um we have them defined as our

stage um to make sure we have proper coverage great thanks Zach um so you're welcome like hey whoa uh oh no we just had that other cicd pipeline what is this cicd pipeline well it's yet another cicd pipeline there's lots of them there's flavors of them but they all kind of boil down and ultimately end up doing the same thing and then to Zach's Point um we want to have this as like an open source reference framework that we can use so we're going to Pivot and the threat modeling is going to be around this specific framework in cicd good news is they map exactly the same um so it kind of rolls up there's uh if you

can't see this it's okay the legend is uh a required component of this and the uh the Orange is the required and the light orange is kind of hey you should do this but you don't have to so when you see something like this you're like oh that's required this is like free enumeration for a hacker like I know that these components are in the pipeline so now let me go and kind of unpack what this actually means to think of other ways to start attacking it's also free information for a security architect as well for when you're trying to define a secure cic environment because it's required and these are are excellent components so once you roll

all of this stuff up and if you just roll up all the other things from the other cicd pipeline you're kind of left with this which is going to um be our pipeline moving forward we're looking at this from like the high level structural element um so you still have local Dev your ID you still have your uh VCS Version Control you still have packages you still have scripts that are running this there's still build servers um we get into uh release artifact ories which is the continuous delivery of this uh we have the deployment aspect continuous deployment the orchestration that's happening and then all of the pipeline ends up jumping out into the ocean and

you're running in production um and so that's our Pipeline and don't worry about The Dumping out into the ocean part it's uh it's a freshwater pipeline um and so uh that will be kind of the framework that we go back to for the pipeline um and here's the framework that we're using for attacks yeah so like I said um our rot node in our attack graphs we need to start somewhere so we wanted to be able to frame the attack um so what better place to start then oasp uh specifically the cic top 10 I won't bother going into the actual definitions here um but you know at a high level there are obviously 10 of

them and they all contribute to a different piece of the pipeline um there is a bit of overlap and that's okay in fact that's even beneficial because it'll help frame your mind when you're starting a new attack graph like you'll see shortly um but starting at the top and working my way right then going down there's for those in the back that can't see um the the cicd uh the first one is the insufficient flow control mechanisms uh which leads us into inadequate identity and access management ccd3 which is critical there'll be an attack graph here pretty soon about dependency chain abuse um poison pipeline execution insufficient pipeline Access Control insufficient credential hygiene insecure system configuration ungoverned usage of

third party Services improper artifact integrity and lastly insufficient logging and monitoring which should not be forgotten about um so I started with the root node um obviously there's an attacker objective in in every single hacking attempt right so um what are the objectives this will mostly stay the same but can change uh for the purposes of this talk we made it somewhat generic um however if you are threat modeling your own environment make it as specific as you could possibly do but at the end of the day um steal compute resources is one such example would be a crypto Miner um gain a competitive Advantage um stealing source code and being able to reverse engineer and understand what's

going on to set you apart from competition exfiltrating data that could be anything from customer pii to Secrets um denial of service uh essentially resource exhaustion attacks uh stealing of intellectual property and then lastly extortion such as ransomware and so then um keeping with the theme of whose pipeline Is It Anyway scenes from a hat so Drew Cary he has chose uh these four sticky notes on attack trees that we will be mapping and defending against and um yeah great so scenes from a hat so um there's the OAS top 10 we're using that um in our threat modeling approach we have our entries uh we have kind of this guidance system that is the cicd top 10

um and then we have our goals that we've gone over um and so what's going to happen now um is that we're going to have the scenario that's kind of pulled out onto the card a lot of these have overlap amongst them um that's not a bad thing but don't feel like you're like oh man you only did four um we can do 10 come hang out with us afterwards we we have more of them uh we can Workshop them with you but we'll talk about the four that we have so great Drew has selected uh cicd SEC 4 um poisoned pipeline execution and and we want to use the pipeline build system to run

elicit code uh and then the attack simulation is that a path is accessible that is called to run within the build server all right let's see what we got so just a tiny bit more framing before we move forward if it's not blatantly obvious uh you've seen the framework that we're working on um the red notes that are here are actual attack steps that are happening later on there's going to be some orange stickies that are like manifestations like cwes weakness enumerations um and then the green component onon to this are the defenses that we would want to build into the pipeline um to uh obviously enhance security and stop these attacks from happening um so poison pipeline

execution is like we have a lot of compute resources that are available in this pipeline we're talking about building like really complex applications that are going to live and run somewhere and so you know we're sending artifacts in we're getting artifacts out and while they're inside of there there's a ton of CPU processing that's happening which means there's a ton of resources that are located for us to kind of take advantage of if we want to so how do we get in there how do we run this script what what's going on here um obviously you know Insider threat like fine that happens it's actually pretty legitimate and we all know that um but there's you know you

would want to create a branch and then you would inject the script to that there could also be leaked tokens that would get you there um but ultimately there's going to be code that you're going to have and you're going to push it up into the repository that lives somewhere it can be kind of your own repo it can be a new branch that you have um it could be uh kind of a separate thing it could actually be a call back to a script that already exists you're just overwriting it there um so I've gotten this script up into the pipeline and then what I want to do is actually have it called from the

build flow so in our in the flow that we're talking about we're using GitHub actions um and kind of this is like a script that's going to run down and say hey pull me in this source code pull me in this script that I want to use and I've overwritten the path and I have this script that's going to dump to the log the credentials to then log into the production kubernetes system and I can execute things there um another interesting thing that happens is that if you have workflows that are in your main branch as a new developer I might not have access to that main branch at all what I can do is

kind of uh Branch off of that and then what happens when I do that is that these workflows are set up to run automatically just on branching and so I can kind of Branch um the main repo it's accessing the secrets from there and now I have access access to the secrets to dump into the logs and anyone with read access to a repository can actually look at the logs that are happening um so I'm writing now in the script something that just dumps the credentials from the environment variables into the log that I have access to um this is again happening on kind of trivial actions that almost anyone can do because you're branching the

repository and then either creating a PR um or just kind of pushing something into your own branch that then executes the build functionality um and moves forward um so when we start looking at like Hey how do I defend against this from happening like yes you can do code scanning you can do cat uh static code analysis but the two things where that kind of falls apart um is like if you're not explicitly looking at the path where the script has come in you're not going to catch anything and you know a lot of scripts have a lot of good common regular functionality uh that is going to run in the rest of your application

so why would I flag that now like there's not a rule set in sass that's like looking for or this malicious functionality whatever that is um so the way that we can kind of go around that is that you actually do now start to hardcode uh like paths that you're looking at I know hard coding paths is bad but we're looking at specific instances in the repository of where scripts are allowed to live and where scripts are allowed to run from uh and so that actually moves forward into approved actions um so another way to actually inject into this is that I can call actions from anywhere in GitHub so I can say hey this is a

really cool action that just takes my container and pushes it into the artifactory for orchestration to happen um so I could either write that as an attacker and then say hey look this is a really cool thing um that does orchestration for you try to use it uh depending on where you are like your developers might be like oh yeah this script's really bad this is really easy but no one's actually doing code reviews on GitHub actions like you look at you look at it you see it in the workflow you're like oh you're calling this action in that's great um the engineers that are doing PRS and code reviews aren't going to catch that this is a

malicious action it's kind of another example of like how you might be able to um inject onto this um but what you can do is then you can also Access Control actions and now I have a source repository at my company these are the actions that we're allowed to use we have the scripts here you can only call actions from this repository and so I wrapped an ACL around that to prevent you from calling actions from all over the interfet um ultimately this is a little bit more of a function uh a question actually around dast it's like so can I look for that um I'm not looking for a specific weakness or like a

manifestation of a vulnerability but can I identify if there's Now new unplanned for activity that's happening in my application so uh that's more of a thought that we could um kind of leave out there like is there something that we can do uh to validate or build that into what we're doing in the test phase of things uh so you know ultimately we were able to get the credentials that gives us access to prod and then when you're in prod uh maybe too much hand waving but there's a lot that can happen when in Pride uh and so uh poison pipeline execution we've abused the pipeline against itself because of its like uh very nice complex robust

computational resources um yeah there it is all right next concept SCS from a hat Zach your attack Vector is typo squatting your attack simulation a developer mistakenly Imports a malicious package into their project the cicd Falls victim to a supply chain attack via distributed malware so um you can see as uh dictated outside the pipeline with a little bit bigger of a square um there is a contingency that happens outside the pipeline that you happen to consume um too bad so sad something happens outside your pipeline that you now need to deal with so starting the local Dev environment following through the red attack tree um a developer imported a malicious package the as dictated the

orange weakness above it never guess it's uh a package name was the first issue that took place in this they then committed it to the Version Control System another weakness there is inadequate code review that's taking place here no one seemed to notice that pandas wasn't actually uh the actual Library they were trying to use um it then has some some unintended uh consequences of being included in the artifacts it was not discovered which then deployed to a forest environment and deployed to production um so taking a step back going back to the local development environment um and trying to map defenses against that specific weakness that happened in that stage um there are defense mechanisms

that you could do to protect yourself against a malicious import there are a lot of IDE plugins you can put in um there's a lot of security tools I don't need to name them but you can Implement them to actually stop before you even do any sort of uh committing to aversion control an alert comes up that PR you do not pass go um there are Integrity checks depending on what your language you're using npq is a is a common one for uh npm environments safeties and another library for python um continue down and you can find there's Integrity Checkers for just about every language um in the in the source stage um probably the same vendor you use for an

IDE plug-in also has reputation in trust scores and that is a number that changes based off Community feedback so HP for example had a compromise that happened early in 2023 that specific package number went from a trust score of something very high to very low and then you can establish thresholds based off the actual numbers to find there um so that will help you um in that respect um there are also similarly to uh IDE plugins um there are GI hooks you can install um that correlate against your software build materials that you have so there could be a diff that takes place and says hey we have these approved dependencies and this isn't part of that do not pass go you not

allowed to import this um until you reach out to security and say yo what's pandas with the Z this is weird um moving on to the uh build phase um there is dependency lock this is a balance of security uh on one hand um you don't automatically want to use the latest release candidate the latest stable version you should probably go a couple releases back that changes depending on other vulnerabilities that may be introduced but if you remember correct if you remember from earlier our attack graph is one specific attack graph and we are trying to defense against that not the entire company which is why we do many different attack graphs so in this case um you could do dependency

lock so um wait for other people to fall victim before you uh don't be the first one to automatically update and use the latest and greatest um version of whatever Happ happens to be um there's also container scanning that can take place that could actually identify the malware that's on the containers um that's a a defense that could uh you could leverage as well um and then like I previously mentioned um against the correlation there's a software build materials that you can use so you can actually understand what um approved libraries are allowed in your deployment pipeline um the not discovered attack graph there uh the defense mechanism essentially a firewall you can reduce the blast radius that may or may not

been happening um so how bad is your malware uh you're about to find out um moving down the line monitoring alerting security operations teams huge function of security they need to be able to have the insight to tell if there is compromise happening which can then kick off standard IR process so you should check to make sure that's also in place and then lastly the production you know like I was saying CD can either be continuous delivery or continuous deployment continuous deployment is the goal but also you want to be safe right and you want to protect production because there's a reason your company exists and with that there are gated approval checks that could be put in

place that can provide you additional layer of comfort in uh being able to push something to production code review as well um you know this kind of sitting before you push the production make sure this uh actual code review of subject matter are experts that are taking a look at what you're doing um and then approving on your poll requests and then there's an outbound traffic policy in kubernetes you're able to implement either through DNS or aoud list of ips which is um a daunting task but at the end of the day it is still a defense mechanism against this attack graph um and that's it for this attack tree so I'll go again I guess I'm Wayne

this time um my Wayne your attack Vector is an accidental public repo with public information your attack simulation this is a big one a mistakenly public repo containing infrastructure's code scripts and kubernetes manifests accidentally exposed Cloud credentials which provided access to resources of a lower environment the developer fixed their hard-coded credentials but forgot to sanitize their commit history an overly permissive KMS key discovered by the attacker allowed them to decrypt resources of all environments including production so this one's a bit a bit longer a bit more complicated and that's okay um but following the kind of the same thinking um starting with the attack graph um before the attacker even did anything there were some weaknesses

that took place as dictated in the orange there was no pre-commit blocking that said hey you committed a hardcoded credential this isn't good um additionally um once you get to that a lot of this skipped the local development stage um and the attacker was able to jump right to the source stage CU you had a publicly uh a public repository um which again this cicd is insecure system configuration so there's issue number one um so the attacker they discovered the public repo and inspected the contents um a weakness that took place here here um best practice for kubernetes manifest um you're supposed to separate your uh your the actual configurations of your kubernetes environments and the Manifest separately

in different repos so this was in one public repository so you gave all the keys to the kingdom early on following that same line of thread they discover the old commit history um from there they were able to access the cloud resources of a lower environment often times you'll find that production is nice and secure and beautiful looking but the lower environments are not they use outdated practices because it doesn't deserve as much attention um the weaknesses that were identified here less secure off of the lower environment there were no logically separated environments as well the KMS key worked for staging it worked for development it worked for production um and obviously there was an over overly permissive

access key was incorrectly scoped following the attack tree um the attacker was then a able to take that KMS policy and leverage that to decrypt the encrypted values uh that are stored in git if you remember correctly or if you remember from previously we were using gitops so most if not everything we are storing in git which speaks to the importance of securing your Source control uh management Suite um once they decrypted all the environments including production they were then a able to elevate the Privileges uh in discovering new Secrets was then it was a able them to follow the pipeline down to production and connect to the database and exfiltrate data um so already speaking on the

weaknesses and now speaking on defenses going back to local development um there's a lot that took that was wrong here they could have sanitized their G history both locally as well as when they pushed it to uh ver The Source Control Management GitHub in this case which is why there sanitized git history in both uh both stages of the pipeline um they uh there's gith hooks like I said previously that you can implement um secret scanning sign commits code review those are all part of your Source Control Management tool that are could serve as adquate defenses here um moving to the continuous delivery and deployment process here um there's a lot that could have taken place if there was

principle of least privilege that was being followed the um blast radius was not nearly as severe um kmsk keys for your own separate environment logging monitoring alerting is critical um the actual backend environment could be separated by different organizational units which would then prevent you from being you'd have to be much more thorough and intentional and creating an overly permissive KMS key if your environment is accurately architected from the beginning uh and then lastly there's a lot of cloud security posture management tools out there are you looking at them are you actually ingesting those findings and trying to fix them it's one thing to have the tools another thing to actually fix the things that need to be

fixed and that's my my tax three great nice thank you Zach well done I want you on my team um what did uh fan I like that what did the development team say when you told them to implement all of those things uh how much time do you have they're like hell yeah these are all great ideas yeah we'll get to it next year um okay great cicd St one insufficient flow control mechanisms in the pipeline how do I get something that's not running it's in my brain to running into Pride so attack Vector malicious source code injection and there's someone in the audience that goes wait you did script injection last time this is source code injection it's

different we're going to attack a different area of the pipeline the attack simulation is enter a custom code via any part of the pipeline that ends up running in PR okay great I I might be able to see like an argument be made that like maybe I kind of look like that guy but I'll also give 10 points to anyone who knows that guy's name 10 points don't count okay so the Crux of the issue here is basically like the core beauty of the CSD pipeline itself is that you're automating getting code to running into production without anyone intervening like that's what people want they want to get it there as fast as possible it's

kind of what agile is like let's do a th000 commits a day um and so uh that beauty of that Automation and getting things right up there is also so like the attacker's best friend it's like you're saying that I can get something at any point injected into this Pipeline and because it's automated it's going to kind of have residual uh like propagation into production at some point so that's great that's awesome um so the core attack on kind of this flow right now is the bypass Branch protection that's on here the things that end up manifesting around this the orange um is that you have over permissioned users I know we all hear this and say this a lot but

it's really a thing and it's like there's humans in the loop all over this we're assigning roles it's not automated we're bad at it we're bad at setting it up and we're bad at cleaning it up um we're bad at taking it down um and so that over permissioning of users uh is a challenge the other thing is just a misconfiguration and so you're like well wait a second you said that there was going to be overlap between some of these pipeline attacks well there you go Zach was just talking about system misconfiguration this is one of those things also um so in your uh git repository there's this great thing called Branch protections and it

prevents people like me the junior engineer it's my first day I want to be awesome and commit stuff and commit right to Main and that ends up running a production and you're like oh crap we we shouldn't have allowed that to happen so Branch protections kind of set rules around who can actually merge with Main and then push into production um but if it's not there or if it's set up uh incorrectly then you can get past them um and so the ways that I'm able to then get things into the source is fine I might be an insid I might write some code um that's just malicious and like somehow get it past someone well if

you're not doing code reviews if you're not doing pull request reviews um with peers then I can write anything I want and have that push too um so that's kind of like all right fine we need some manual human in the loop uh type stuff for that um the other thing write malicious dependency so someone might not even know that this exists at all I might write a something that my project wants to use like hey I'm going to do this really cool um string Sanden I'm going to write my own library for it and I'm going to put it into our own inside artifactory and then I'm going to call it later in the build stage but I'm

going to first like make my own Branch or my own uh GitHub repo that stores this and does this and no one's I'm not going to tell anyone about it we're going to go into the code review be like ah this thing's dope look at this sanitization that I'm doing here and just ignore the rest again just like we weren't doing um code reviews of actions we're not going to be doing code reviews of dependencies either um so that's how I'm kind of kind of injecting into all of this and then if these things are misconfigured or if my user has too much permissions I'm able to get it into source which then again then propagates

all the way through like once it's sitting there in main no one's looking at it again it's getting compiled and built and deployed with all the great things that we've talked about and how that happens um so then skipping there's actually a whole another side to this that like you can ignore that altogether if you're running code if you're using AWS lambdas and you just have code up there and I get access to that that means that I'm actually Hands-On just editing code that's running in prod so I've bypassed everything I really don't even need a pipeline at that point I'm just editing the code that's running on your system same type deal with tokens

of getting into kubernetes and pods I'm then live on a production system it's kind of anything goes at that point um so defending these things uh let's see audit enable protection configs so set them up set up your branch protections um and look at them regularly to make sure that the switch doesn't somehow you know single event phenomenon the the setting has changed um do peer reviews yes it's human but do it you learn uh we build trust and we're a team uh static analysis great lot of great stuff that does that um and then rules kind of hearkens back a little bit to like the configurations but it's rules specifically on a branch of like who can

push who can merge uh what level permissions do you need for that uh supply chain ACL um for the artifactory like use an internal one and review everything that's in there don't allow me to call to any GitHub repo on the internet don't allow me to call from the public Docker registry um depend can check things for the vulnerabilities es bombs man these are hot now right they've been hot for a couple years actually um but uh software build materials these like if you're using a VCS this is probably packaged and happens there like you have this this exists so just reference this check it uh use the es bomb and reference it back to what your ACL for what packages

you're allowed to use um and then tags and hashes set these up don't use latest because then I can can just overwrite something to latest and check them um if I have you know we have an ACL that you're using this package well that has a hash and so when you pull in all the packages to your build system check the hashes there also um and then check the hashes of the things that are running in production all this exists uh and we can use it and we will um so ultimately access to the cloud uh do some change control do you know have Cloud release management don't allow live editing of lambdas that automatically are running

um and then you know everyone's favorite uh user access control um great Zach let's get out of here yeah themes and takeaways um lots of I am uh lots of you user access reviews overly permissions service accounts we really didn't even reference them uh that falls into this as well um pretty critical SCM GitHub was already important that hosted all your code is becoming more and more important so be sure to actually take a look at GitHub and make sure it is properly secured not just Branch protections but also GitHub teams and provisioning you should Implement that until your oicd provide oidc providers such as OCTA or whoever whoever they OCTA chooses to purchase um know your CI

CD environment attackers are already there cool um yeah what are my takeaways uh so I think you know we're we're building everyone else is out there like building really cool things that they're really proud of and they're building really like advanced awesome cool widgets um that are cool but you don't need your pipeline to be the latest cool thing make your make your pipeline something that's reliable use the technology that exists use trusted Tex stacks and then use the documentation when you have an actual mature tool there's a section in there that says these are the security best practices when you're threat modeling read those and bring them in and say hey are we doing this thing these things and if

you're not you can create an attack Vector based off of that Gap um Integrity checking your favorite s bombs depend depencies doing access uh and like allow list these are huge the actions the dependencies the scripts um knowing your environment is part of that uh you know your environment you know what's allowed so you can control what's there um and then you know back to uh everyone is building Cool Stuff everyone's really good at building Cool Stuff engineers and developers honestly love security they're hyped when we talk to them about this um and they're the experts on the system I am not I can talk to you about security I I didn't know about Pipelines a couple months ago

um and so going to them using them as a reference building that relationship too kind of fostering that teamwork uh that's fun um so that also ends with threat models being mutually beneficial and if you aren't threat modeling you should be um there's some references for later looked at that on the recording uh and thank you very much

than