Is Your Approach To Pipeline Security Flawed? Rethinking CI/CD Security

hi everyone I hope you can hear me and if not raise your hands and yeah I'm Patricia really happy to be here talking about cicd security and it's good to see so many people willing willingly hopefully here to geek out on cicd security disclaimer I'm going to say cic a law um but hopefully you won't be annoyed with that so that's me uh I'm a class security engineer and a year ago a year or so ago I managed to get myself into this project that was all around Cloud tagging so tagging your resources that you are doing via your terraform deployments or even container images labeling them so I had one goal or we had one goal uh in my company and that

goal was to targ assets with very specific keys and set of allowed values so I found myself okay how can I do this how can I ensure that I guide users into using these keys and these values that's when I I was genuinely excited to find the the concepts around polics code and guard guard rails and that picture is honestly is really accurate I was so excited by when I was talking about these things because you're telling me I can define a high level policy in some sort of Json format and then give it some configuration some engine comes out with a yay or nay as in is it's compliant or it's not compliant I'm like tell me more there are many

tools that do this uh open source ones you have things like open policy agents and uh in in our case we were using check off because for us was kind of a a no-brainer we already we're using checko by Bridge crew prism Cloud um it's already in our pipelines let's go for and you think okay awesome right well slight issue there we need a little bit more than just the outof thee box policies which meant it would be more resource more time to put into this um this attempt to do policy as code to be honest with you I was like okay maybe it's not worth to invest the time here uh and it's just too much work we we

don't have the resources to do it that changed when um well you when you find yourself with a free weekend and a kind of cool ghub name project name idea a problem starts to sound more like an opportunity and that's when p on Rails uh came into into exist existence Fon rails nothing to do with um rubyan rails it stands for polic ES code on guard rails I thought that was funny somehow um I have another one it's called pack onlines I'll leave it up to you to try to understand what that means but essentially it got to I don't know Sunday or something and I got somewhat of a working minimum viable product so I gave

it some Jason like that one that said oh you need to use these keys and the set of allow values and then uh I was uh testing it and it was kind of working and you can see there the to test it I was using the first line which was essentially saying to check off which is the policy engine do the standard out of the box checks but also use the python code that I've developed to do this extra checks I was happy I was getting the the results that I wanted close my laptop go for a bike ride because I I I done it I'm happy to do it on Monday when I get to

work well I started to think wait a second this thing these policies that have just written in Python are going to be running in every single pipeline that I have uh or that we have in our organization and then I started to think wait a second what's that second um line saying it's saying I can simil similarly specify uh GitHub or any G server URL to go and get some policies which would mean well you're essentially running someone else's code that someone has written in your own pip and I started thinking slowly on in this because I I started think no it cannot be true because I've written this and I could have done what whatever I

would like with that python code so I decided to do some malicious checks so one of the things that I did was okay I have our arbitary code execution from someone that pulls the checks that I've written what would what else could I want to do well I could try to just exelate some data for example with Git which was what I did and now you're like patri that's not that's not that interesting because you just describe essentially open source because you're running someone else's code and I agree with you I would even go a step further and say well I just described any time you are running someone else's code that you haven't thoroughly reviewed and um that you're

running uh in in your machine the solution here to me is not to stop using things that you haven't written yourself because that's not very scalable but it really made me start thinking about the security of the cicd platform so whatever is running whatever is running those checks or whatever is running code that wasn't written by me um that thing we will be accessing whatever that pipeline for example has access to so that that's what let me here and that this this is kind of a long introduction to why why am I here now why did I start to look into this more and more so in this talk we're going to go over some definitions just to make sure we

are all talking about the same thing when I say for example cicd and cicd pipelines uh I would ask if uh is anyone not familiar with cicd but hopefully the uh the definitions that we're going to go over will cover that after the definitions we're going to do some observations on uh based on those definitions and trying to scheme puzzle together uh a few a few of them and then I'll end with a proposal that proposal W solve our all problems but hopefully it it makes it a little bit more digestible it has certainly helped me face this uh some of the of the challenges the first definition that I wanted to go over is on screen which is

cicd platform what do I mean when I say cicd platform uh it's fine if you disagree but to me in this talk cicd platform or cicd system whichever it's really anything that touches your cicd uh workflow that that means from the the repo the G repo that you're potentially using that is azure repos GitHub repos bit bucket anything to um pipelines uh things like Azure pipelines I'm a bit biased towards Azure devops because that's what I use but it could also be GitHub repos uh GitHub actions or Jenkins pipelines things like that all the way to more towards the right side things like argd flux um that will uh play a part in this cicd ecosystem as

well I'm tempted to include things like binary repository managers if you've heard of things like jrog artifactory or Nexus repositories I would include them as well or even things like container Registries out there that's one one definition uh the second definition hopefully uh more interesting uh pipeline what's a pipeline a pipeline uh when when I started to look really look at what what does it mean to be a pipeline it's just a set of instructions it's just saying that that's um that's quite of just seral code obviously it's not yo like what we would usually see in the the pipeline definition file but what that's saying is it's just it's just something that tells you to run

somewhere with some code as some user and potentially to use some sort of variables that are stored in this variable store for or environment uh but the core there is just that set of instruct it's that recipe without the resources that make up so that compute that user that repo it's really nothing you can really access anything there's even a poem I I'm not going to even try to uh to read it but but it's to capture that Essence the last definition is around resources cicd resources and it's the things that I was talking about is that compute that you're using is that user that you're using um it's the repo it's the the environment or variable store

that you're accessing all of them one way or another point to something and then something is an identity they are pointing to a certain ID more directly or or a little bit more farfetched they kind of point to some kind of access into something you care about for examp if we take comput if you're using things like workload identity Federation or some sort of access that is based on the machine you're working with well you're essentially getting access to the permissions that that machine under under uh that the machine has with a user if you're using certain things like service connections or certain credentials you are assuming the Privileges that that user has that's your pipeline as well if your

pipeline is using that that pipeline is going to have that those permissions repo is a little bit more indirect uh with repo you can think of the the current ways that you that you have to do um uh for continuous deployment systems where they are looking where there are binaries essentially looking at your uh repos and if there's any change to that repo the the changes are done in the dur runtime environment so essentially by by by messing with that repo you can um make changes to the live environment as well and variable Source again you're storing there you potentially storing there credentials usernames passwords and those credentials will grant you access to some sort of

identity some sort of access privilege and that's what it's that line says is the the pipeline itself doesn't have privileges the Privileges are granted by the resources that it accesses so to put it all together we have a pipeline a pipeline accesses some resources which ultimately are just a proxy into an identity and there's this book that uh I'm I I really like how the authors um phrased it so identity native infrastructure access management and their take is that identity identity in as a core concept is really immutable and it's non transferable but you you can't you can't use that in a digital world to use it in a digital world you need to create some sort of data format for

that identity and in this case U there their uh proposal is credentials they are calling it credentials so in a sense a cicd resource that is access by the pipeline is just a credential it's a proxy into an identity and why because a credential is U is essentially data it is transferable and um you can steal it they they do mention something that I really like and I subscribe to that ideology which is there's no such thing as identity theft what you have is credential theft and I'm I I really like that that idea okay we're we're done with uh definitions now we're moving on to observations and as I said I'm a bit biased towards Azure devops um that's

because I use it most and it's all I always use I'm pretty I I am almost sure that you can adapt it to any any other cicd platform or system that you that you might be using okay start with the blue dot the blue dot there represents just the high level pipeline concept the pipeline definition so that's uh that's just the recipe let's call it the recipe now imagine you create a pipeline in whatever um pipeline uh system you have and then you run the pipeline when you run the pipeline you create a pipeline execution the yellow Dot and if you press the button again you're going to create another pipeline execution associated with the same

pipeline definition that that I have there and now imagine you're about to click the button again to run but you haven't quite clicked it yet you have a potential P plan execution and in this case is associated with main because you're in the main branch if you're uh familiar with that I have a video just I mean this is what it looks like in aure devops uh I'm just creating a pipeline I'm selecting uh the source repo uh and I'm going to select uh the path to my yo file and and and that's it essentially that's how you create a pipeline in AER devops and when I press run I just created a pipeline execution to that

pipeline okay if I create another branch in this case I call it feature 197 I have another pipeling execution uh a another potential pipeline execution because it's another branch and it's just another potential version of that pipeline file so what I can show you is here in the in this version I am changing the pipeline definition I changed the pipeline definition and then I'm just going to commit to a new Branch so I ended up having two versions of the same pipeline but actually they they can be as different as we want okay we're done with the pipelines now we're going to introduce the other concept the concept of a resource and this resource if you remember is just a

proxy into an identity to make it easier we're going to say that this resource specifically is a a variable Group which is a concept in Azure devops to store your variables your usernames your credentials uh and any other variable you might want after defining the the resource what we need to do so that our pipelines can access it is Grant it um Grant the pipeline access into that resource so essentially we're going to just draw a line saying yeah this pipeline has permission into this resource and here you're going to you can see that my pipeline is not running and it's not running because it's waiting for me to granted access um and I I stopped at a very strategic Point

here in the video because what it says I don't know if you can can read it but it Ado does let us know that once you allow access to this resource this variable group that has some username and some credential to an identity that I really care about my prod environment for example all the resour all the pipelines from then on will have access to that resource and why why am I slowing down here well that means if I create another branch and that branch has some kind of twist in the the pipeline definition it can still access the the credentials that are in that uh resource that red resource there so now now we're gonna now as the

security seles might be like oh okay that doesn't seem quite right because what if I'm ex trading some some sort of credentials with that PIP plan definition and that's that's kind of the point that's kind of why it can be scary to to think about these things so how does this look uh I'm going to touch on two things uh for Thor um approvals and Branch protections with the approval look I I've fallen for this drop and I've seen other people falling for it it's not even a drop it's just you might be over relying on controls that are not that rigid and we'll see why I'm saying this so what I have here is I have a

pipeline and it's a terraform pipeline if you played with terraform loads of people do this and 100% it's something that that I've done as well terraform plan and then you have a stage for approval to see the plan is it good is it not good and then you apply it okay what if I am uh either not patient enough to get someone to approve it or I I'm potentially malicious person I can just go to this approval stage and just remove it and then commit it to the main branch and then run it again or no one's going to stop me from from from from doing that I'd say cuz what I can see is that I can save

it I'll save it I'll commit it to the main branch and when I run it again I won't have that approval

stage Tada no approval stage and the thing is I removed it's quite evident that I removed the approval stage you could do anything this is just to examp you can just mess with the pipeline definition file I could have just said you keep the approval stage but I'm going to do run whatever command I want before the the approval stage um because remember pipelines you're just essentially running arbitrary code you're giving that that that possibility to users which is good um in a controlled way but now you say Patricia you won't be able to do that in our felines because we have Branch prot protection rules okay is that a challenge well I'm going to add Branch protection

rules and uh you'll see that I'm adding them it's not it's not that U that hard at tting aure devops you just go there and you say for the main branch I'm going to say it's a protected branch and I'm going to say it if you want to push to the main branch you need certain users to approve it and that's fine except what are what are we really protecting with those um with those branch approvals because if I go um to that uh to the main branch and I remove again the approval stage okay I'm going to be blocked from pushing to the main branch as we can see here I am blocked but what what does that matter will I

still be able to do everything else that I was doing in the main branch but in some other Branch because something that that I'm doing now I'm creating another branch and then I'm executing the the the pipeline anyway and once I execute it in the other in the in the other branch and I press one no approval Sage uh again you do anything there what does this mean what what can we do about this well what it's going to be always about layering controls ensuring that the controls that you're applying are uh working together in order to meet whatever goal you want to to achieve uh because what we what we've seen with uh the approval and the fact that you can

just edit out approval stages from pipelines uh or manual validation um well first of all you shouldn't be relying on controls that you can just edit out from your pipeline file that's one but when it comes to the branch approvals more than thinking I need to apply this Branch protection rule to my main branch you also need to ensure that whatever you're doing in that main branch that I'm I won't be able to do it from a nonprotected branch else okay you're proing the main branch good for you I I can still use the the um any branch that I create and now he's telling me okay I'll uh make sure that my pipeline definitions are really strict and you

cannot I'll put them in some other repo and you won't ever be able to touch them that's fair that's an approach uh but then we have the this possibility that we are including other other things that we haven't written we are using templates we using um packages from other sources that we have no control over so external to that pipeline definition file and I'm going to examplify this with uh two uh again two uh sort of use cases the first one is about semantic versioning and I really like to talk about this because it it does um mess with uh with my brain a little bit because the security person in me is saying oh we need to use hashes because

that's the only way we can be sure we are using a specific version if you if you work with containers and potato versions um as security people don't really like to use latest because latest is always changing you never know what's actually in the latest uh pointing at latest in the latest version but who here uses hashes I I have rarely seen people using hashes usually engineers and developers SLE for a middle ground a semantic version things like oh this is version 1.0 1. one 1.2 two uh those kind of semantic version the thing is even that is um over I can overwrite it so here I'm being a good citizen I'm using templates that have been built by uh a platform

team for example and those templates are versioned and I'm using version 1.0 uh from that prle that is in mind but this works really with any kind of artifact and versioning and then I'm okay if I am as talker I go to the main branch and I change the the that version that and I change the repo so the templates this is these are the templates what I'm doing I'm changing the late test template and I'm adding a dragon cuz I thought that was that was cool what we're going to see is that okay I am committing uh an attacker for example was able to do some change to the template but if I still go

to the version 1.0 which is the version I am using in in my pipeline I I still won't see any dragon so that's cool however what happens if I go to the the version 1.0 that exists there now and I delete it and I recreate it but this time when I recreate it I will point at the dragon version what you think it will happen well I'm going to see the dragon taada because someone has just recreated T so um I've done this right I've done this in a non-malicious way even if it was malicious by I wouldn't be saying it was malicious but um it can happen that even if you're using version 1. zero and you

think oh it's never going to change it can change uh hashes are probably really the only way that you know what you're running in your pipeline okay you you're now say that's it you're going to always point out the latest version but I'm going to control really heavily how people access those templates or those uh versioning systems so it won't be a problem still have another example for you another use case which is all about what are what are the tools you're using in your pipelines are you using terraform are you using um things like uh policy as code engines like check off and opa because what what uh what you can do there so terraform has this really cool

resource which is called like the new resource and it allows us to specify provisioners and provisioners can be either for example local execution or remote execution remote execution is essentially saying run these commands in a Target machine I haven't play too much with it um but I I did play a little bit with the local execution which is essentially if you're running a pipeline is running terraform local execution is going to make the terraform execute commands on that uh runner that you have on the the the P the build agent the the compute that you have so what you can do is something like that U quite simply remember how I started the talk uh

talking about check off and how I use Python and git to exil trate data well here is the same concept uh same basic concept but using terraform to run uh these commands which are G commands um to push data out to a random recod that had created uh in big buckets and I was able to exfiltrate uh the environment variables that that pipeline Runner had access to so um if you if you had access to production credentials well no I have access to them as well there are other ways that could do ex CL more clever ways you can do this and uh Shelly Shelly uh Robin she has a really cool talk that talks about things

like DNS tunneling as well um so she used terraform and opa uh to exfiltrate data um through uh protocols that you usually would not have blocked within your pipeline because it's DNS and everyone kind of needs DNS um okay but what can we do what can we do about this uh well one Access Control access control to your pipeline definitions that's obvious uh but at the same time you need to allow some development activity to be happening look when when I do um when I'm developing some sort of pipeline I rarely get it right the first time the first 10 times it's just garbage so if you're going to lock it down you really need to think about the prod produc

productivity side of it track management also really important but again uh you do need that flexibility for people to to to De develop their own stuff you always need some sort of a sock use cases or detection rules so one detection rule that I thought about um would be if you detect some kind of activity where someone deletes a tag and right after recreates that tag with the same name something maybe maybe you should investigate why why would someone do that because it's supposed to have an incremental uh tack number again investigate don't don't just say it's malicious because it might not be but I've done it before as I said and it was more of an operational

decision but maybe you should investigate and again it's that layering controls concept we're all doing or are we well there's a bit of a proposal U that has helped me uh lot uh when thinking about this not to get overwhelmed and instead of looking at the pipelines themselves I try to look at the resources that are being accessed and but by do doing that you you you are focusing on what really matters to you if I don't care about this resource that is being that is giving me access to a Sandbox environment maybe I'll leave it to the end to deal with and I'm going to start with this really important production access credentials that are stored in

some sort of variable group of service connection so what I would say is okay one you need an inventory inventory collection a list of res cic resources you have and understand which ones are really important to you uh so that you can prioritize and so that you can also as a group because you're going to be a group it's not going to be you single- handly uh doing this but you need to agree on what what are the conditions of access that that you are comfortable with say this is not malicious access and this is get access and it's obeying all of the compliance and security and operational um requirements you have it could be things like business hours if

you know a certain critical service connection or variable group is only used from 10:00 a.m. to 12:00 p.m. well maybe you can do some sort of uh access uh restriction based on the time or you can also have a service connection or a variable group that always always always must have an approval from someone to be used and you apply that based on the resource itself not on the pipeline because I can just remove it from the pipeline so you you establish and you agree those conditions of access and then uh easy uh the easy part is just manage the the cic resources according to whatever conditions you have defined this is thear um it's really all around

focusing on the access that pipeline has instead of of trying to run around to and make every single pip planine secure you can do that I'm just saying if you it has helped me a lot just focusing on the the actual resources that are being accessed because since I started the talk probably you have 100 more pipelines that have been created and it's really hard to keep track of them so if you focus just on the resource which might be used by those 100 pipelines well you only need to focus on one on securing one thing this is the final slide and it's really a disclaimer um I'm not saying you're doing anything wrong or

everything wrong uh it's really just to raise awareness for some things that well might get you off guard or or maybe you know about this and you are doing everything okay um anyway I hope I hope it was helpful and if I had to finish on uh any kind of statement it would be it is really really hard to strike a balance between security and making it usable for developers and for engineers be reasonable and focus on the outcome and that's it thank you [Applause] thank you very much Patricia uh are there any questions we have time for some

questions great talk how was it taken at your organization when you tried to push this thing uh did the developers give you any push back hard time uh sorry can you repeat that did the developers when you try to put this balance in terms of restrictions to various resources how did you manage that uh we're still working for it actually so um I haven't implemented it yet but but how we we are uh managing to kind of get it into production is just to think about let's focus on what really matters and then apply those um those conditions we is not going to be taken lightly uh by by any teams because sometimes you just really need

to do it and is not it's not always malicious it's also just in terms of from an operational point of view you just need to get the work done so you really need to be flexible and work um it's also risk a risk decision and at the end if everyone agrees the risk you have to say you know it's you're in charge security can like can govern that that for you hey there um great talk I've hit all these problems before the sort of cicd which are Developer wild wests I'm sure you've seen them too um the only environment I've seen which is secure against this is when the cicd Tool itself is governed under terraform or

something like that and itself is changed managed whereas if you have developers of privileged roles they'll break versions change icds and you might as well give up at that point um would you would you agree with that or can you have developer having privileged access and secure this because I would argue you can't I would say the the privilege access it's on those uh on those resources again and then it might be if you have approvals for example and you have more than just a couple of eyes uh looking into it you you is that shared responsibility maybe you have a senior person as well there uh to to help you with that but but I I do I I

yeah any other

questions thanks Patricia um so you were talking about controlling access to these resources now in some organizations there's a lot of those resources um and it's not clear to security people what control should be put in place around them I wonder if you've got any thinking around that uh yeah I so uh you know I was saying I was a little bit biased towards Azure devops um what one of well I do use it a lot and I do think that a devops gives you a way to protect those resources in a quite efficient manner manner because it allows you to say to that service connection to that variable group to that uh agent pool so that

compute that you're using every time you're accessing it apply this uh this Rule and that rule can be an approval can be a bus uh that business hour um restriction can be things like calling an API which is gives you a lot of flexibility um so so it it's almost like having a a conditional access on the resource itself uh which would be something uh to do H how I would handle it is if you can if you're not Green Field so if you have loads of resources start by stopping uh for a second and anything that is new goes through this process where you look at the resource and you work out is this resource

um really important to me or not that much and if it's really important you you start any new resource that is really important is Broad environment you start adding certain conditions of access and once you you've got that life cycle managed for the new resources you then start working on the the resources that are there um and are you being used already and eventually you'll get to some sort of eventual consistency there and any more

questions right if there are no more questions then uh thank you very much to Patricia Rodriguez