DevOps or DevOops? Securing a Pipeline Without Losing Your Mind - Jonathon Everatt

hello everyone my name is Jonathan evat um if any of you know me you know my voice doesn't normally sound like this and I know it sounds like I'm sick but I'm actually Batman at night and I forgot to Swit my voice back so if I get a bit too raspy during the talk just let me know um so this talk that I did is devops or Dev oops securing a pipeline without losing your mind and off the bat we off the bat we have some spoilers already I may have lost my mind uh so just a quick who am I I'm that guy um I'm a proud dog dad uh she's about a month old with me and older than that

without me I'm also a senior cyber security consultant at MWR I'm the fluffiest bunny and I know my dog likes likes bunnies because she likes me I also think I'm funny and this talk is up to you to decide if I am funny um I'm also very clumsy and that falls into some other things as you can see I've already spilled coffee on myself and I've only worn the show for like 2 hours today so some background for this talk so I had to create a devops pipeline as part of a crisis simulation exercise so what is a crisis simulation exercise you may ask basically we create a network and the network is specifically vulnerable to a certain

attack path then we execute the attack the attack on that network with security tooling installed as it would be in the snapshot of a corporate environment and then the instant response team would try and kick us out basically it's a really good and really Safe Way for people to practice instant response in a way where if you lose you're not getting Ransom word across your entire environment so this network had to work firstly very important it had to be up and it was up in AWS and anyone who's worked with the cloud know that cloud bills stack high and Cloud bills stack fast so I had to be very cognizant of not doing that T3

times 9 large instance and go crypto mining for my salary that month because I wouldn't have a salary the next month um and then it has to look realistic right I can't have a gitlab repository with no commits in it I can't have a domain admin emailing me their credentials the attack path had had to be realistic and it had to work um fortunately it didn't have to be working at scale so what is this talking about so it's kind of a day in the life of creating the crisis simulations and my learnings um I made a huge amount of mistakes doing this in mistakes that other people have made and could have helped me not make this time um and I

didn't ask for that advice um but I think it's fine to make mistakes as long as you're learning uh it's also for a security professional like I am a look behind the curtain because it's very very easy sometimes to recommend a security control that will make the thing secure but doesn't think about how you're going to maintain it how you going to build it how you going to build onto it so it's looking around the corner that making it work making it work well and making it safe aren't necessarily the same thing right different Prof professions have different perspectives and every everyone optimizes to something something different from the other person obviously we're building a Dev a

devops pipeline so there is some devops in here and then also that security is hard right because security is a balance it's a balance between security and usability now as absec lead at MWR the most secure web app you'll ever build doesn't connect to the internet right and the most usable web app you ever build doesn't ask for credentials to log in but it it's just usable in the wrong types of way so it's very important to have a balance between security and usability and this is something that can be very difficult to think from one perspective to the other as a security professional why is the person against doing a third uh factor of authentication logging in why not

username password and phone and email right that's more secure slightly more secure um and as another person is like why do we have to do the third Factor we're going to lose users cuz it's not as usable so finding that balance is really where you become an effective security consultant and to find that balance you have to live both sides so to say and so I dove write in sorry sorry speak I dove WR in yeah so there are some some jokes in the slides not that many so some basics for the dev Ops um it's basically write code deploy code get feedback write more code deploy more code in an infinite Loop That Never Ends

there's a lot of advantages to it but it's also very very complex and I cannot call myself to be a professional at it at all because there's just so many Technologies and so many different things things that can go into it but the main end goal is that you can quickly push code to new production and maintain it so rather than being 6 months for that new DLC to come out for your favorite video game they can add a new tree or a new Bush every day or every minute or so so it's really speeding up that development life cycle and there are a huge amount of ways you can do it there are lots of technologies

that exist for it there are lots of technologies that exist for it there really is a lot of them but I had to choose three I had to it had to be a snapshot of a realistic corporate environment and so I chose these three right I chose gitlab as my source code repository because I couldn't install the code on GitHub um and then on merge it used Jenkins uh to build the code into a workable application hopefully um and deploy it onto a tomcat server and that use engine X to deploy that out to the internet right not in jinx as I thought it was for the first year that I read that word um we've all made that

mistake before so and why I chose it right well first ly this needs to represent a realistic corporate environment so these Technologies are common and that's one of the reasons I chose them I've interacted with them both before right I've used Jenkins and gitlab it's basically rce as a feature and I've you I've used Jenkins and ctfs and I've used gitlab in other places um and then I also had some config files on hand for both of them from other projects I was like oh this will be easy I've done some of the prep work those config files were useless so we're going to start at the beginning with um

gitlab so I had a plan and as everyone knows when you make a plan you definitely stick to all of the steps and don't forget any part of it so install a resource constraint gitlab instance with Community Edition configure it securely um create uses and repositories make many commit so it looks realistic and we all know there's always a typo in there so make sure to include only one typo because I like having no typos in my slides and Cheryl sitting over there hates it when I have any typos anywhere then install everything else then come back and make it work but one does simp does not simply get technology to just work so first things first download and

install gitlab Community Edition so as I said there was a guide for creating a resource constraint gitlab instance and I went through the guide and basically it said install uh Community Edition and it should work so I had a T3 medium instance um and I ran through the commands and see anything weird there where it says gitlab that's a typo in the documentation I think because if you just change that to CE you download uh Community Edition you keep it at e it becomes Enterprise Edition so yeah they're trying to upgrade you even from documentation times and it failed immediately out the gates and I didn't know why right I had configured the host I installed the

gitlab installation failed there was enough space I didn't know what the problem was and then I did something crazy I'm not sure anyone's done this before I read the error and what the error said is it said uh Port H for HTTP traffic was closed obviously because my AWS Security Group only allowed access from my AP IP um so I opened that up and it worked right but at what cost did it work right it did not fail it did install it was technically usable it was just unusably slow right so when I said this has to work work it also has to work within a 4-Hour exercise I can't wait 3 minutes for every page to load and that's what

was happening here right and I was really confused because I had done everything i' mostly read the guide and it just said install gitlab and I didn't know why it worked so I went back to the guide and I realized did I read the manual first I skimmed it we've all skimmed it um and I missed the only important step and we all know we've done the same thing right so just before I get into what that step was I have a reminder for everyone and that's you can always save yourself 10 minutes of reading with five hours of debugging so I did not configure a swap file and because of that I did not have

enough memory to actually run it properly and it was the only thing in the guide that you actually needed apart from the installing part um and the manual actually has to be read to configure the swap file but it's a common thing on Ubuntu that I actually didn't know of until this point and without a swap file in my life I can have no swappiness so the swap file is a certain amount of space on a disc it's about 8 gigs I think I made it which is basically really really slow memory now you don't want this on a desktop because your CSO latency will be 200 before you send any packets to the network but on a

server it does the job and for my gitlab instance it do it did the job great it went down from 3 minutes to loading a page to 5 seconds and I can survive 5 seconds and remember this just has to work network doesn't have to go vom and then swappiness which is my favorite word going forward um is the percentage of memory that is used before it starts going over to the swap file so once I had a swap file and I brought the swappiness into my life everything was a lot better right now everything works is at a usable speed and I've installed gitlab and that only took me two days not not that much time um so time for

the next steps right we have to create users and we have to create repositories and I only have access as the root user and obviously I would never create all my users admin just to make my life easier so when creating the users there's some interesting stuff I saw firstly who validates their gitlab account with a credit card that feels like your admin's just trying to skim your card details from you um but also when creating an external user you had to click external which I saw after creating a few external users uh before clicking the button reading is not a strong theme for me I see from this and then I can proudly say I created all

my users as regular users I only made them admin on the repos so now um you can see here it's funky coders with an N um I have my external users I have my internal users of Globex um but they still need to be given permissions and added to repositories I think we ended up creating 20 something users at the end and there's only one admin so yay security apart from the fact that I'm using the root account we don't talk about that lot so I needed to create a few repositories because I've never seen a gitlab application with only one repository everyone starts that Pet Project works at it really hard for about 3 hours

gives up on that pet project and it sits in your gitlab for the next 5 years until you delete the application you start again um they need to have commits in them and the commit should have at least one typo you know have to keep the theme going um and the merge requests should acire require approval because we are implementing security best practices here because of the merge request don't uh require approval if you compromise a single user you can push code to production which may be fun from a thread actor perspective but remember making a mostly secure network here and then the different users should have different differing levels of access Because by the principle of lease

privilege we want to give them the least amount of privilege that they need to do their jobs so I created my very safe projects and immediately got to the problem where it's not public I don't need it to be public and I don't want it to be private because I don't feel like inviting everyone so I can do it internally right that's a great idea except by default and gitlab even says this in a documentation if you have access to the network you can sign up for a gitlab account so if you make it internal you're giving access to anyone who has access to your network which we all hope is our only our employees but we all

know is can be a bit more than that so I made this public because it needed to be and I initialized my repository then I added users and I clicked I clicked roles to add and it gave me this now I can see a guest I know what a guest is and I know what an owner is so no Privileges and all of the Privileges but I don't know where the Spectrum in between is right and I don't have a high level explanation of the difference between a developer and a maintainer so where is the line drawn and what do I need so I went over to the documentation and this time I actually read um and I immediately gave up on

reading because it's terrible this is not usable for me I don't know if it's usable for you guys if I'm looking for something specific I love this but nothing gives me a oneliner of a developer can generally do this a maintainer can generally do this it's all these check marks trying to work out exactly what privileges they have so I gave everyone maintainer and noer I understand why people have authorization problems now because I didn't want to go back and fix this later so I said oh no I'll push it down the line and it's funny how that even as a security professional when it's a deadline when you're working hard when you're not thinking a lot it's really

easy easy to just push that down the line and say oh no it's not going to production yet so I can deal with that later right so there was technically a typo in the in the commits I couldn't find a real one so I'm very proud of that um but knowing my luck is probably in the big screenshot um also we never merged uh any branches we just pushed straight to main I know this probably causes a lot of pain for a lot of people and I'm sorry but it's just easier um and we all have to accept that if you don't have a technical control in forcing a branch to be created and someone else to approve it before it's

being merged into main it's not really a security control because threat actors are the lucky ones because they're the ones who can skip the bureaucracy of everything so take takeways read the manual um because I already read the errors principle of lease privilege is a great security control but it can slow people down and I need to understand as a security professional why people don't do it they're not necessarily being malicious they're just not necessarily thinking and I will admit that at times I may not have been thinking best practices can only be supported by technical controls if it's there in theory or through bureaucracy it's not a security control it's a hope um default

configurations are not the best configurations everything always needed needs to be configured properly this includes your security products and any options without explanations on them kind of becomes roulette um which for which one I go for and then it's kind of funny how even when a security professional is building networks they end up not being the most secure networks you see because we then understand the issues everyone else has to uh deal with on a day-to-day basis but now I have now I have git lab working up working so I want to install Jenkins and that is apparently called Jenkins cat and I thought it was a really cool cat so the initial plan was to boot on a

Windows host and this one I did delegate um and basically they said no they came back and they said we're not doing it we're doing Linux there were a lot of problems we couldn't access it outside of the host we couldn't push the runner other places and let's be honest Linux is better so new plan we pivoted so why I pivoted Linux because Linux is better is a lot easier to configure you have a lot more control and I'm very comfortable with it because I use it on my day today if I used Windows on my day today it would be probably be a different story there right following the installation from there was quite

easy especially cuz we did the runner on the same host which is a bad idea don't do it and Linux just worked so does that mean Linux is just better yes it does so why should the runner be on a different host well security is a good reason if you compromise a single Runner you only compromise that host you're not compromising the entire Jenkins application but another one is for testing right fortunately my application didn't need a work at scale so I didn't need a test across multiple operating systems but in the in real life it does so if I'm deploying a thick client to a deskt toop I need test it on Windows 7 8

10 11 and that one person who still uses Vista um and yeah so that's why you need Runners on other hosts I Didn't Do It scalability cloud cloud bills a number of things laziness but it works now right wrong it's only installed um unfortunately I still need to do configuration but at least it's installed and this was ended up being a lot less of a painful process than gitlab was mostly because I delegated it um now we need to configure it right we need to pull code we need to build code and we need to push code and there also needs to be that golden threat of vulnerability right and I realize now that's a typo it's meant to be thread um

sorry Cheryl but we still need that uh chain of compromise coming through and that's just a theme I want everyone to remember as this goes also where's the web server so we went on a quick side quest um and we configur the web server now this is a really interesting thing as to how you can see it's a difference when you've done something before it's taken me five or six days at this point to actually um configure Jenkins configure gitlab get things working the web server took me 15 minutes I've done it twice before I knew exactly what I was doing I ran my scripts I ran my configs and everything was just working perfectly I had

pre-compiled code that I knew worked from a previous time so I had my uh web app up up and running within 30 minutes and that's why this is only a side quest yeah when you know what you're doing it's quick so back to the main Quest we need to configure a build script we're back to those three steps get code build code deploy code but where do I put it right I know Jenkins is rce as a feature but where is the rce to be a feature right so I saw something called Jenkins files and I have no idea what they are I have no idea how to configure them and I don't have any on

hand so I ignored them um the web application has an rce section sorry build config SE section and I'm more com comfortable with that so I just configured the commands in the rce section I know it is called build config I do promise that but in my head I've kind of done a match and replace rule for rce at this point so I'm just going with that and so now we get our rce going now I'm going to give everyone a couple seconds just to look over that there are a few problems there um and if you spotted the problems that's great we'll talk about them later so this seemed to work um seemed being the operative word there um we

would build it then I'd use SSH to deploy it to the web server why SSH you ask because I know how to use SSH and secures in the name so pretty much good there I also gave the admin of Jenkins control over the server because I push it with an SSH key to Ubuntu Ubuntu on ec2 instances for anyone who doesn't know is by default can run pseudo without a password because they don't have a password configured so if you gain access to Jenkins you gain control over the web server and as a side note the database just because I needed I was testing and the key now works there too um yeah but that's useful for lateral

movement so I'm going to leave that um and and it works and I need that vulnerable path now it's time for integration hell so anyone who's ever built things separately and then tried to put them together it becomes very very difficult um and it is a complete pain everything immediately stopped working right and that's what it felt like I tried to get the Jenkins build script to run and it just didn't run um gitlab I couldn't log in but that's cuz I was typing password incorrectly um yeah at least Jenkins could kill could pull the boat at least Jenkins could pull the code and build it um but it needed a prompt how did it know when to

pull the code right the web server was the only thing that was really tested and working and when I say tested I just kind of trusted that I'd done it correctly um so we're firstly going to do gitlab to Jenkins right so what do we have we have gitlab working it stores our source code we have source code in there I didn't take you guys through the 89 commits we did because I thought that would be a bit boring but if you really want I can sit and run through each commit with you guys if you're interested after this talk um Jenkins works we have our build script and we can push it to the web

server so that's great we have the pieces there we just need to put them together um and we need to build the puzzle so how do we know when to build right so we either need gitlab to notify Jenkins it needs to push the notification through or we need to Jenkins to pull the notification from gitlab it needs to pull for that information I.E pulling it now the person who done it before me tennis head of consultancy said hey when you start setting up your devops pipeline come talk to me I have some advice on how to make it easier and I didn't go talk to him and so I spent three days trying to

work this one out um much later we had the conversation about it and he put He put the issue to me in detail and I was like you're right that's exactly what happened um but we ended up pushing the notification from gitlab simply because there's nothing worse in my opinion than you have your malware you execut it and you're waiting for that 30 seconds at 40 seconds for something to finally pull and tell you you made a typo in the first word um yeah so we op to do the push was more reliable and then we use the first option of a web hook that Jenkins let us configure but wait we went home first

cuz we do sometimes go home from the office and we turned off the instances cuz Cloud Bell stuck High Cloud Bell stuck fast and we came back and everything was slow and when I say everything I mean Jenkins right and we spent the next day debugging and we had no idea what the issue was and we were all stuck so I delegated it off to our young proor Ethan um and for the first time ever somehow turning something off and on again actually made it worse and then we worked out the issue cuz it's an ec2 instance when you stop it and there's sometime and then you start again it's psycho's IP addresses jenin seems to do um calls out to its

own IP address and we had configured it using its external IP so by changing that IP it became very very very slow not gitlab initially slow but like one and a half minutes 2 minutes to even load a page and so Ethan eventually came back to us T us how to fix it but I am incredibly disappointed in him for using Nano to fix it because we both know and we all know Vim is the answer is the question and them is our solution but fortunately by fixing the command he sent me and updating the configuration file to the new IP IP address it worked so we can get back to the normal um issues we were having so

first we had to create a web hook so making a token is quite easy the problem with the token I now realize is I hadn't created an authentication material or credentials specifically for that token so we were getting authentication issues and like every good debugger should we ended up going all the way to Reddit where we saw this post by this person here saying Oh no just put your credentials in the URL it would be fine and so we put our credentials in the URL and it worked security the worst thing is that this actually works and remember we need that golden thread so I'm going to use it yeah now to prompt it right so we need a

prompt on merge requests which is great for my use case and terrible for security specifically CU I haven't configured it to be second party approval so any developer can go push code approve their own merge request and then push that mware straight to production and a small problem for testing I don't really know how to create branches at this point because I've been pushing all of my code directly to main um so I kind of just ended up hitting the web hook specifically to test it um but yeah so don't do this without the propess security controls in place um this is the kind of thing where you make it insecure but by dumb luck it's the wrong

insecure for me to actually exploit um and then deploying from Jenkins and I do apologize for white and green that looks terrible there we go um the code didn't work I don't know why everything else was fine the source code was pulled a new war file was created I even know this cuz I deleted the war file and then a new one was there um it was copied and deployed the application would go up and the connection to the database would go down and we don't know what went wrong and we debugged for days right so let's take a look at the problems here so this is where we were looking specifically at the build script and you can kind of see

the attitude we had we looked at the Bild script as a whole and we were like well firstly we're using root credentials but that can't be the problem because it means there's no authorization issues not a good idea just don't do it SSH does work because we can use it to laterally move move it's not a good idea to give um admin control of production to Jenkins but I mean it works and rm- is extremely stressful but it works so we can just leave it nothing here should be stopping the build and we were stuck right except for one problem we didn't actually compile everything now for those who don't know you go for Java you go from a DOT Java

file to a do class file to a war file and we then attack okay that joke I think is funny um no then we deploy the application our Focus was too small um we were we were compiling the class files to the war files but we never compiled the Java files to the class files and apparently the last time I had compiled the class files the application was broken so nothing was working and we had rabbit ho hold so specifically on the war file not working we never took the uh step back necessary to actually see the problem in a in the whole of looking at the whole compilation process of java to class to

war um yeah so we just need to recompile um Java files into class files literally two words and then from there we can make everything work and bring it together with a touch of malware because what really is a crisis simulation without some malware so what it needs to do remember we are in a crisis simulation and the kind of attack path we thought up is uh compromis developer third party some malware gets put into the gitlab instance which will then pull through into production right for production we had bizarre bank because everyone wants to attack bank.com for some reason I really feel sorry sorry for those guys um yeah so we were thinking of a type of

Mage carts right and what Mage cart is it's a card skimming type malware where it'll pull the card details from the user's front-end browser and exfiltrated it to an attacker controlled instance right and you can think of it as stored xsa yes just with the injection point being through the gitlab instance ironically this was the easiest part of the whole setup process because building the um attack path was actually building the environment code doesn't know good from bad and a lot of the time the difference between good from bad is use it's like a thread actor's favorite tool is the tool you use that is effective right so it was quite honestly one message to chat

GPT fix what they wrote to be our domain and then we just sent it through and it just worked is that playing oh cool um okay so you can see here we're logging in as our developer from funky code is Ryan Smith that is a very secure password which I'll tell anyone who's curious um so this is a maintainer on the repository and as a maintainer you can go to settings look at we Hooks and read our great very secure we hook um the web hook worked so we kept it um with that web hook he's obviously now able to get the IP address the internal IP address of the Jenkins host and the admin credentials um you are meant to

have credentials specifically for one user that can only build for that one um code you're doing right now we log in with those very secure credentials and you can see the mentality we had when we created that user cuz the password is installing Jenkins is hard um and once we signed in we now have rce on Jenkins um which is great because also in Jenkins has a root credentials for gitlab also in Jenkins has the SSH key um that we can use to transition over to the web server so we have everything here we need to go from VPN access to gitlab all the way to the production server through this golden threat of misconfigurations along the

way and you can see here now we just read the SSH key and this SSH key doesn't work but you're welcome to try copy it out from the video to your um specific device and then try ssh in let me know how it

goes and there's the key yeah so that worked that that was the lateral movement part of it we also obviously had the Mage card part of it we wanted the mware to come steal the data we needed um so that we can make fraudulent transactions against bank.com so here we have bringing it together with the malware so we had pre-embedded the malware here because idea was it's been compromised for a while so we're going to see here a nice um Choice people had to make is we embedded it when we fixed SQL injection in the app so if you want to roll back to before the malware was embedded you're making a vulnerable app to SQL injection which is

a really nice catch 22 people got stuck in and if you roll back to that no one ever expects a SQL injection attack to come for some reason but now you log in as your test account you go to the card endpoint and you see um internet analytics is getting all of your card information now I need I know analytics companies get a lot but I don't want them to have my card data and then with that we are able to um go make fraudulent transactions if it was real wait that's the same slide and it worked wait play okay I promise that's a video so can we stop this um not in the users

browser it's a bit too late um that's the terrifying thing about attacks like cross-side scripting now a prop a properly configured content security policy could work there's a small problem there I've never seen a properly configured content security policy so I've never actually tested it um the remedial actions need to be implemented sooner that's why it's important to have practices like Dev SEC Ops that's why we should disable sign up why we should not have me giving privileges to people um you need security controls everywhere so that when it gets to the point where the attack's being executed often that's too late to actually protect at that point and then just some takeaways from this talk and I promise no more

memes okay now no more memes oh sorry I need to click through um this stuff is hard right from all perspectives is hard everyone has challenges every day that we don't actually understand what they're going through and when I see something that I'm like that's so dumb why would you do that in this application I need to think back to my configured devops pipeline where I'm like that's so dumb why would I do that in that devops pipeline if it works it works and at some level that's a really good start for everyone right security workability and maintenance all optimized differently and I haven't even broughten the problem of scale yet when you get stuck and mess up it's

okay um you don't need to go and do a whole talk about all the mess ups you made but if you take a breath and you take a step back you'll probably find a solution and if you do get really stuck and someone has offered you advice probably take them up on the advice even stuff that should work out of the box requires configuration never trust a manual but do read it do read it reading is important trusting not so much trust that validates um nothing out of the box is optimal and that includes security tooling I will repeat it again um the best edrs I've seen are the configured EDR so please make sure you

are configuring your things corre correctly and for use cases that make sense for your business if you're a Dev if you're an architect a security professional or just anyone try consider the other side when making recommendations right we are all working together to have technology work and be secure but sometimes when you're making it secure you can forget about the working part and sometimes when you're making it work you can forget about the secure part right that's why it's so important that when we all work together we get the best Solutions rather than when we all work separately or against each other or here and hear me out on this one we can destroy all technology and cyber

security problem is solved and most importantly nothing is a waste of time if you're learning thank you for

listening any questions also that was like 75 slides so let's go

okay perfect thanks everyone across the attack surface scattered products and siloed views create blind spots that feel Unstoppable the deadliest risks are in these gaps where attackers move in it's time to unify fragmented snapshots into one allseeing view of risk and unleash a platform born with one intention isolate and eradicate your priority exposures from it infrastructure to Cloud environments to critical infrastructure and everywhere in between this is tenable your exposure ends here