CONTINUOUS INTEGRATION, CONTINUOUS COMPROMISE

[Music] all right there we go there we go all right oh I'm excited to be here this is my forest Vancouver B side also before thank you for besides a second I'm speaking at it so before I start I'm going to do an outline so if you're like getting coffee or whatever and you're like should I ditch this talk if I doubt so I'm going to start by going over what is continuous integration that's what my talk is focusing on I'm going to go over some common misconceptions then from the kind of attacker side of things I'm going to talk about the code execution you can get through the CI system I'm going to talk about pivoting between

slaves and masters within the CI system and then finally backdooring all the bills so quick intro to myself I'm Wesley Weinberg I used to work for a Vancouver company called world tech they eventually got acquired by GE I then went to work for a San Francisco company called cynic and then just in last year moved sell hopped the wall and went to work for Microsoft so currently I'm a member of the Microsoft Azure red team and so my focus is definitely on the offensive side of security but we work closely with the blue team and purple team and green team and whatever colors you got because offensive is really just one piece of your equation to get here

systems all secure and this talk is just kind of my own independent research no no company is behind it or anything like that thought I'd tell like a fun story when I was looking at my intro slide here I was like ah world Tech so many good memories so at one point we had we hired a sales engineer and he was going along things are going good and he put together a slide for GE actually they were going to become our customer were a customer trying to sell more stuff and he put like a slide of current customers put ge up there to be figured you would see it and like liked it but it turns out GE get super mad when you

put their logo up without asking them and so then Rotex like oh dude you screwed up we didn't tell you but you can't put their logo on and then they told them you can never speak to customers again so the fun story so here's some logos but for real don't like complaint to ge about that okay so to start with I'm going to go over build systems themselves as kind of a history before we actually go on to continuous integration systems this is just my unofficial history how I see the world so back in the day you had a code repository and you had a build server maybe even had like a few different build servers if you had lots of

products and you would do more or less iterative builds you'd get code checked in the thing would build your code eventually after a few hours and if you broke the build people would be mad cuz bill took a really long time and then after your build was all done in the system spat something out you could do some testing of that and file a bunch of bugs for anything that you found and kind of iterate keep going doing your cool waterfall model because that's what you did um but kind of importantly at the end of the whole build cycle deployment would be done by someone else you as a developer would it be really strange if you were the one pushing code

onto a server or packaging and releasing a product to a customer anything's like that developers kind of development and other teams did server management deployment that sort of thing so then kind of going back a little bit further but tying in to my talk the history of build systems from hacking perspective really ties into compilers themselves so back in the 70s apparently they had electricity and compilers and so some dudes that works for the Navy came up with the theory basically that at some point you have to trust the start of your chain of your software and to the start of your software chain really is a compiler itself if you can't trust your compiler then there's really

nothing that you can trust because even the compiler to some extent gets compiled by itself to create new versions later on then in 1984 Ken Thompson from Bell Labs I think published what was a popular paper and it's still kind of a popular paper to reference called reflections on trusting trust and he basically kind of just detailed that concept a little bit further and put it in a paper and people liked papers and so the theory of these attacks really hasn't changed since 1974 84 depending which one you like to reference you can still backdoor a compiler and have it fact or everything it builds and kind of ruin the chain of trust that way but practically speaking

there hasn't been a lot of real-world attacks using this methodology you've got way more email viruses and things like that things crawling websites than you do people trying to come trying to target compilers just because it doesn't scale is easily necessarily moving on to modern day though this all kind of starts to change so in modern day we've moved to the world of DevOps everyone's doing DevOps if you're here and you're not doing DevOps you're probably also not making software or things like that so DevOps comes with a bunch of different components in terms of build systems at least or build system pieces so you've got CI which is continuous integration you've got CD which is continuous

delivery and kind of a variation on that is continuous deployment so mostly depending whether or not you've got a system that produces the software application or whether you've got something that droning on a server like a website or something like that there's also been a lot of components around infrastructure automation so that your infrastructure itself is part of your DevOps process and get deployed by software instead of a human and kind of to glue all that together there's a lot more instrumentation for your code a lot more monitoring there's fun trends like a/b testing and things like that so when I talk to a few people and I said hey I'm going to talk about continuous integration systems

these are like developers or other people that know about these things they said oh what looks like a CI system so I figured I'd put up some logos because when I tell someone well you know like Jenkins it's a it's an integration system they say oh like Jenkins okay so I just picked some of the popular ones the ones that I recognize most there's a bunch that aren't here and probably a whole bunch of startups in San Francisco really sad they didn't make the slide but anyways so you kind of got three groups you get continuous integration and these often bleed over into continuous deployment which awesome bleeds over into infrastructure automation so a lot of these can be used

in more than one category but this is the general grouping and kind of to illustrate this a little bit more little on perspective and I don't think you can actually see this got like projectors everywhere but it's still hard to see but basically the top line is continuous integration that's the start of your evolves process the next line is continuous delivery and you'd use this if you've got a manual deployment approval process or if you're making software otherwise the next line down is continuous deployment and so at the bottom it shows you start from source control do a build over to staging and then finally production and so I've got I've got Clippy here to help us though

because there's going to be various levels of people in the audience some people are going to be like Wes I don't care just give me some exploits and other people are going to be like Wes what's an exploit I'm than management why do I care so I'm going to try and cover both so you don't have to get two words but basically the point for all of this so far is that if your company makes software or like deploys websites or even tries to secure companies that do that they're going to be doing DevOps and they're going to be following a lot of this process so moving over to an attackers perspective what do they think of when they see

devil this is what I think of and I'm going to generalize that so this is what all the attackers think is so when you hear that people are doing DevOps what I think is people love trends people just like rush into trends and do it because everyone's doing it they don't wanna get left behind and actually for DevOps that's a good thing if you're still doing waterfall deployment and you're ready to release an expert in your product and a couple years you're probably getting left in the dust but that doesn't mean that you're doing it securely continuous integration I think continuously compromised compilers continuous delivery software that you can't trust right from the beginning and continuous deployment this

is the one that I'm probably like really excited about now instead of me trying to hack one server air engineer and give it through a few servers and then maybe hack the next engineer it's all just automated through software which leads to the next point use this one cool trick to backdoor all the servers at once so DevOps has allowed companies to move a lot faster but it's given a lot of new cool opportunities for attackers and so this really looked like a talk about DevOps but I have a really hard time like talking about DevOps and not making fun of it when it comes to security so I just throw these in this is basically an illustration for people

that like is illustrations more than words of how DevOps and security work together and then we might have some more like management type people in here or a more concrete focused one so I've got like a pyramid of kind of wide DevOps is a challenge like why this picture is so here's our pyramid at the bottom of our pyramid we've got a source code that's full of vulnerabilities and Sullivan barely is because the developers got told build a feature they didn't get told make sure you're this feature is unhackable they got told build a feature that's their job they did it and it's full of all but that's okay cuz well test it and we'll test it in production because

that's the most realistic system to test it in and now that we've got this cool a/b testing methodology we can just kind of stage the vulnerable code to maybe one tenth of our viewers and only they'll be vulnerable until we notice on top of that we've got to put all these systems on the internet so you want to be successful you gotta look at successful companies and what they're doing so you take someone like Google they don't have a network boundary they just put everything on the internet so if we need to be successful we should probably put all our deficits DevOps system right on the internet and then finally we're going to sprinkle some AWS

keys or SSH keys or other other secret key material kind of like when you go to Olive Garden you get like cheese on your salad you just like see how much they'll put on that's that's what you do with the crisps so that's our DevOps pyramid of why it's hard to do DevOps security all right so of that pyramid we're targeting CI systems and the reason I thought it'd make a good talk is they're kind of the start of the chain of trust you can compromise a deployment system but deployment system just deploys whatever the CI system built your CI system typically involves a lot of test automation it's usually running your unit tests and things like that for you

and in kind of more advanced systems usually those do a whole kind of build in a staging environment there's often a lot of creds involved that gets shared over to other environments if the CI system is instead of flowing a website or something like that it's building a software application if there's any code signing often the code signing cert will get loaded into the CI system sometimes to the CI systems are actually used as a deployment system so you get a two-for-one deal if you hack to see I system and finally kind of like all areas of DevOps many of these systems haven't had very much security review of the specific software itself so I've been looking at random CI

systems deployment systems what have you over the last few years and every time I look at it I find some new vulnerabilities which kind of tells me that no one's really done a heavy security audit of any of these products so the the systems I chose I needed to choose some systems for this talk I didn't want to choose too many but I want to get a kind of a sampling so I went on Google and I came across the top eight continuous integration tools looked like this so that that screamed like legitimate Gartner quality result and so jenkins and team citi were the top two after that was Travis CI but that is kind of continuous integration

as a service you don't host it yourself didn't really match what I wanted and then next one was go CD but that has continuous deployment in its name I wanted CI so then bamboo was fifth so it made the list and Jenkins is made by someone King City is made by JetBrains bamboos made by Atlassian ok so that pretty lengthy intro but I think you need need to understand what all these systems aren't where they fit in understand why compromising them is good so the biggest issue with CI systems and really like I could do this slide and kind of stop here and it would give you enough to go on to either compromise or

spend a lot of time trying to fix your system and it's the fact that they're they're continuously misconfigured you generally don't even need any vulnerabilities in any of these three software packages to hack into one of these systems they're they're always going to be misconfigured and so I thought a little bit about why every time I come across one of these systems it's misconfigured and it kind of ties to what the continuous integration products have to be so to be successful they've got to be highly configurable and adaptable when companies have development and build environments there are always these systems that have evolved over time they've got all kinds of things Cluj together the build works

but just barely and so to tie that then into a CI product the CI product has to be very extensible has to be very flexible it has to be able to match all kinds of create the environments that companies have the issue with that or the challenge with that is that complexity and security are two offices you can either have a system or a function or communications protocol that is secure or you can have one that's complex as soon as it gets any level of complexity that's hard to understand there starts to be mistakes made and implementation and even the design really people stop to understand how it works so it's very difficult to audit

for security and as well go over soon for many of these continuous integration systems you're kind of at a disadvantage from day one when you do a fresh install a lot of the install default options actually pose either somewhat of a vulnerability or leave your system directly vulnerable this is actually kind of a trend within like DevOps and all the new software packages you got MongoDB or whatever elasticsearch and they all say well security is up to the user the implementer out of the box none of that's turned on five ten years ago that used to get you like a CVG like Oh install uses it will password just login with sa what could go wrong but we're

kind of swinging back that way and so CI systems are no different high level for this this slide though if you just wonder why does any of this matter your company has a CI system it's definitely going to be misconfigured let's hope it's not on the internet so you need to like go and find it okay so default configurations is what I'll go through first so I installed each of these CI systems just a couple weeks ago fresh install and took a look at how does it give you its configuration so Jenkins which previously was Hudson is almost a decade old it's actually a tool that's been around for a really long time and when it first started out it wasn't

designed to have really any security functionality the design for it was you have a trusted network somewhere ideally that's not even connected to the Internet you do your source code check-ins and your builds on this network and if a developer can get to the network then you're they're assumed trusted over the last four years it's become more and more of a mainstream tool and so people are starting to use in all sorts of environments so the Jenkins team has been adding security features and actually in the last couple years that made a lot of security improvements partly due to having a lot of security vulnerabilities reported to them that said if you have an install

that you did maybe four years ago chances are it's using some of those whole default config options so even if you've applied all the latest patches your configuration itself might be the vulnerability by default the server listens on port 8080 which isn't too significant it it's also not going to be something super easy to search for just don't internet lots of stuff runs on that a fresh install forces a user-defined password otherwise it generates an admin password for you that's pretty strong by default user registration is disabled but by default the authentication scheme that it chooses is just that any user is full admin so if you've done a fresh install and then just added five developers to

this all five developers are going to have full control of the system when you're doing install as well there's two options you're given one is no plugins another recommended option is they install a bunch of plugins there's when I did install 66 plugins that it recommends you're probably going to want and I'm going to talk about plugins briefly in a bit and then last thing to note is that it installed the build slave right on to the bill master server in the default configuration and this would be important as well later when we go over that side of things but overall if you just did a default installed in add anymore users the current configuration actually isn't too

bad it's kind of fun to look at the historical default config options that used to be there so they used to have a checkbox that said hey enable security and this man enable any sort of a fennec ation and I'm pretty sure if I remember right this just used to be by default not check and then eventually they did made they made this default that is checked another fun one is CSRF the requirement for that on any request to the web interface this just used to be something they didn't support and then eventually they added it in is optional and now it's a full default and then the last one is agent to master control

which I'll talk about briefly later as well but that used to never be there and instead they just documented an agent has full control over the master moving on to teamcity so when you do a fresh install the default server listens on port 8000 111 and when you do the install you're forced to choose both a username for that mint user as well as a password so it's kind of up to the installer how strong that is a big issue with this system though is that user registration is enabled by default and it's I don't know if I'd say it's hidden in the configuration menu but you got to go look for it to turn it off other

interesting piece of this is that by default all user accounts are put into the group project developer which chains up the group levels which I'll talk a bit more about later by default the build slaves use unidirectional communication and by default there's one build slave installed onto the master server next is bamboo this is the last of the three that I looked at so a default install of this listens on port 80 85 and I mentioned all these port numbers just because if you're on your network you go looking for one of these ports combine that with the page title of bamboo or teamcity or whatever it can be a really great way to just find what might be a

default install with bamboo they're forced to choose both a username and password as well just like team city and like team city user registration is also enabled by default when you register in your user they're put into the user a beer user group and that group really only has view access to the system so it's pretty limited but what is also really interesting is that by default you don't even need to log in to get Mew permission so unattentive kated users also inherit that view permission I don't have a bullet here for that but it's true by default there's only two groups bamboo admin so similar to Jenkins if you add five developers to your freshly

installed bamboo server you're going to put them in the admin group so that they can actually do things but that gives them full control of the server another another interesting option that's checked is resolved artifact contents by extension so this this option gives you a hyperlink and go read the help and it says this would lead to cross-site scripting attacks if someone checks in say a web page and then makes that a build artifact but that's a default and this one as well a build slave installed onto the build master server so I mentioned that you can go look on your own network for these ports but what what often happens and there was some

write-ups on this a couple years ago and people kind of started to go looking is numbers stage 3 of our DevOps pyramid you don't want to have boundaries anymore you don't want to have this on your internal network have you no teams around the world all accessing your build server which you've made online so I went on showdown and search for port 8080 and the word Jenkins and that came back with just under 20,000 hosts online I then did like a few more searches to make sure they were Jenkins and then I put that Jenkins needs to be in the title because it always is and that only ended up with 3,400 hosts online I then did a bunch

more queries and basically showed end 100% flaky on how it does its searching algorithms so anytime you see a presentation and people are referencing showdown the numbers are almost definitely wrong because showed in does really weird things with its search but I didn't stop me I kept on going next was team city showdown doesn't know about for 8111 or at least acclaimed it in but just searching on any port I found 3200 hosts and then for last name bamboo found a grand total of 2 so I guess it's not very popular or show them doesn't index it well or maybe both but all of these are not necessarily going to be defaults but I was trying to pair it up with the

port to go with the theory of if you're still running this on for 8080 maybe you're still using a lot of other defaults this leads to kind of a fun story because I love fun stories so you find yourself 14,000 Jenkins servers on showdown you think to yourself wouldn't be fun to see which one of these have user registrations enabled build myself a new project add a build command send myself a shell route the whole company I'll be a great hacker but just because you can doesn't mean that you should so whoever in the audience there's at least one person I'm sure he's like I'm gonna pull some Jenkin boxes right now I was the story here is I was doing a

pen test a couple years ago privately contracted company it wanted us to look at a number of things including their external footprint and I come across this Jenkins server and then it was basically configured so that anyone could authenticate to it with their Google account so you would redirect you to Google you descent acade it to it and then it would create you a user account basically it wasn't white listing it was just looking for any account drop you in the server so I'll go ahead and do this and as you can see there in yellow I blocked it out but that was my account I created and hey cool that that that's neat it dropped me into the server and

then I saw there was only six other users on or five other users on it so I took a quick look through them and three of them were employees and then I came across this other one named SEC and then I see the name then and whatever it is and I'm like hey this name looks familiar so I punched in Google and sure enough this guy does bug bounty testing all the time and I guess had just run a script to find all the Jenkins servers login to as many as I can or he was doing it by hand I don't know but he ended up making himself an account on this server so I was like oh that's not

cool but then like I really like to think of things from the hacker perspective and my perspective then would be well this company didn't like that they shouldn't have been able denne one to just login with Gmail so good for him but you gotta you gotta kind of like take one more step back and consider what's really going to happen in reality so it's really going to happen is you're going to go on Google you're gonna find someone Jenkins server you're going to log into it it's gonna make you an account I don't know why you'd use your real gmail account but he did and then you're going to look around and be like wow these guys are boned this servers

not configured right at all and then you're going to move on to acting other things and then six months is going to go by someone's going to come along and they're going to ransomware this server they're going to bring in some top-notch auditors from whatever Incident Response company you like they're going to come in they're going to look in the user list they're going to be like then let me use my threat intelligence ego goal to punch the same in they're going to come up with your name they're going to find your contact info and you're going to be trying to defend yourself from a lawsuit as to oh no I didn't ransomware the server I just logged in

and looked around and it was cool so a bit of a tangent but fun story find people in the wild hacking other companies that they probably shouldn't be and if you're going to do that you know go through six proxies and use a different name alright so what are some common misconceptions to look for so say you've done a default install but now you've let your developer operations team configure it a bit so there's still a ton of stuff that is going to go wrong I just put together a list of the most common things I've seen here but every system has its own set of issues and all you really need to do is look through

what can be configured and every second item will probably be configured wrong the first day is user registration so some systems take the theory that will let anyone register but they just have project viewing rights on pretty much all of these systems even a low permission user ends up being a disaster so even as you can just view a project you can often view the build logs and often view the build artifact and you can often view the test artifacts and one of those almost always includes credentials for talking to the next system the systems now the CI systems now at least the ones I looked at here have various scripts that try and guess

where credential is and output and mask it off for you but it it's mostly up to the user to configure that feature there's almost always creds that get through there's also going to be a lot of good techniques that you can try with just viewing I'll go over some more than later the next is anonymous access and this kind of follows with the first one a lot of companies take the approach of well developers can log in and trigger a build but maybe their manager needs to see it maybe everyone in the company needs to kind of see the current status so we'll make that open and same issue you start to be able to view artifacts

and logs and other item the next one I think is probably the most common is that developers don't just have access to trigger a build they have full access over at least a project within the system and so talk a little bit later about how you go from having access to a project having full control over the CI system but the two are basically the same if you have project admin access that's just as good as full admin access almost always kind of tying in with that larger companies even smaller companies really often they have more than one project on the go maybe you've got a software application maybe you've got a website that your

software application uses and you also have say your own corporate website all getting built by your same CI system within your company you've got different teams people wouldn't have access to login to one server compared to the other but you're all using the exact same build environment using the exact same build slaves and it makes it very possible to pivot from one person's project to another the next thing and this is this is kind of a second piece of what you do once you get access to one of these servers almost always the build systems have credentials with tons of access so that might be ss8 credentials to push the deployment out to a server that you've that's hosting a

production or staging environment it might be an ssh credential that just has access to every server because we don't know in advance what the build system is going to build for so let's just cover our bases there's often things like AWS keys if you're doing hosting in the cloud it could be as your management key it could be whatever whatever cloud provider you love and then in more corporate environments let's say startup and more bigger company I often see Active Directory accounts so that the build system can talk to various internal corporate resources that it needs these Active Directory accounts often have a lot of access a lot more than a normal user and then the last piece is

plugins to these systems so I think of CI systems like I think of WordPress like off the bat WordPress isn't great people find hacks in it all the time but it's actually not terrible I wouldn't necessarily expect to sit down right now download the WordPress source code and come up with a Volman a couple hours but the plug-ins for WordPress they're a disaster and I would expect to find a Volm in two hours same kind of thing with the CI system they had the plugins often haven't received as much reviewer as attention but they let a company get something done maybe they just need to integrate to their Active Directory environment so someone wrote together some LDAP

connector and it gets that job done kind of all right so expanding on that previous anair a little bit more so you have read-only access there's two great things that I'd always try the first is list of the users on the system chances are there's going to be an account called test or they're going to be account called dev test they're going to be count called build one build to build three and they probably have a super weak password so you might not have known those usernames before but since you have read access you can list those out and potentially just take over those accounts directly I've had a lot of success with this one the next thing is

kind of going back to those plugins a lot of the systems let you generate API or authentication keys to talk to other systems and they often expose these to anyone with Rita access because you need to know what key to pass to your secondary system so sometimes this is done through a plug-in sometimes it's built right in but often you can do something like say hey show me the API key for talking to the see is rest API and so even though you only have read access you've now got an API key with full access so the the Clippy notes on this is you need to tell your developers a there's always misconfigurations it's awesome guide besides he says they're

never configured right but further to that you need to get your developers to check regularly ci systems don't generally start off super vulnerable except those default config but they get more vulnerable over time every every system generally does the more you add to and change a system the more opportunity there is for incorrect configurations so monthly every two months every six months I mean yearly if you have to but regularly review your configuration because it's going to be changing over time and it doesn't matter that it was secure three years ago okay so that's assume that we've got some sort of access to a CI system each CI system takes credentials and it feels like it needs to protect the credentials

you gave it so you gave it FTP server credentials or your SSH key or your a to Bielski or something now I thought about this a bit like big picture why even protect these credentials because usually in these systems if you have permission to read the stored credential you've already got some level of admin access which means you can probably just tell it to SSH to a server and run whatever command you want you don't even need the key yourself but on the other side sometimes it's just a whole lot easier to collect all the credentials straight out of the credential file take them offline and look through them not leave any more traces or interactions on

the CI server so for whatever reason I basically the threat model doesn't make a ton of sense to me but everyone doesn't like credentials and plaintext I guess so what does Jenkins do so Jenkins used to up until some time in 2014/15 have a master key and have a secret key I don't think it was actually per project I think that was just like a thought at one point in time and then they take a sha-256 of one of them and they transform the other one a little bit combine them together to get an aes decryption key for the secrets that are stored in their credentials XML file so someone made a quick Python script

Tripta you can also just use the script console within jaykins to ask jenkins itself to decrypt it for you which is super handy once you have admin access to Jenkins I wanted to double check that this all still works and I took a look at it and they've definitely changed some parts of this so I'm still figuring out a Python script to do new decryption but I'll get that sorted out and I'll post one of those pretty soon because it doesn't look like anyone's fathered to reverse engineer the new format but Jenkins still does helpfully take all the credentials that you load into it and store them into one file so once you get file system access you can just grab

that for team city team city treats credentials in two different ways depending whether it's a string that you've entered or a file that you've uploaded so if you've upload an SSH key or something like that as a file it doesn't do any sort of encryption for that I don't know that that's necessarily a vulnerability but it makes it super easy to just through their web interface if you have project admin access to click export a backup of this project open up the zip file and you're going to have the unencrypted SSH keys right there credential strings though are actually encrypted and so what they do is that use triple death I'm told is not as broken as regular

desk so that doesn't really make sense but I don't do crypto like real crypto I just break it anyways and then that's basics t4 encoded so if you look in your config files you'll see all these base 64 encoded strings they've got a little header on him to kind of flag that there and encrypted basics before string so I took a look through their software all of these systems are written in Java for some reasons Jenkins teamcity and bamboo which makes it easy to reverse engineer because they didn't obfuscate it at all and so I came up with the decryption key here and I made a Python script to do the decryption I told teamcity about this a couple

years ago said hey you know you just start coded a encryption key in your application that did and they said they thought it was not a problem necessarily because you need admin access to get these files in the first place so I don't necessarily disagree with that but I asked them like so is it cool if I post this key and they said yes so here we are a couple years later got around to it bamboo takes credentials and it stores them in its database store them on the file system necessarily although the database might be on the file system depending how you configured it it takes a similar approaches teamcity it has well kind of like a bit of team seeding

a bit of Jenkins it AAS encrypts your keys and it uses a key file on disk that I've got listed oh there cipher key underscore zero and then it uses a key that's in its database it X ORS those two keys together takes the IV from the database and then can decrypt the secret so I've got a script for that at the end of my presentation as well moving on so you've got all the credentials that's great kind of backing up a bit maybe you don't have full access into these systems how are they possibly configured how might you be able to change that so Jenkins has six different authorization schemes you can use because these kind of like

follow from its legacy history so it has something called anyone can do anything which I just call no authentication it has legacy mode which if you turn it on gives you a whole bunch of additional options but I assume none of them are good it has anonymous users have read access so this means that you can have people who log in and have more access but if you don't log in you can still read everything it has logged in users can do everything this is the default and then it has two options majorek speed security and project based security and so both of those if you look at the very bottom of the fly if you kind of squint your eyes you can

see there's like a bajillion check boxes it it makes it so you can have super granular permission I have per project or just to all of Jenkins in terms of what a user can do and it's everything from like view build results - let's see here run scripts upload plugins and so on so chances are when someone's setting this up they're just going to keep checking boxes until the system works as they were hoping then they're going to stop that might mean that you want to try and get that view access see what users have what permissions and if you find someone with a upload plugins checked than your goal the other thing to look for is Jenkins is used pretty

heavily with custom authentication providers so instead of using one of those six you can tie it 200 or open ID or whatever the challenge with tying it to those is they often don't work with your matrix based security so it's an all-or-nothing once you login is one of those then you just become admin team City has I think probably actually the easiest to understand permission model they've got four default levels or groups viewer project you ER developer or project admin and then a system admin category with control over everything the one kind of trick or thing that I think people miss or is easy to miss with teamcity is that your permissions groups can nest so there's a global set

of permission groups that every project inherits and then per project permissions so by default everyone's put into the project developer group so even if you then add a user later pick a project add them as a viewer they'll still have developer permission to that group and the the second kind of piece of all of this is each of these groups can then be expanded into a more granular set of permissions so kind of like the Jenkins list of options at the ball if a company has attempted to customize this a bit they might have run into the same situation where they've given a little bit more permissions than it's necessary but even a project developer

which is kind of the most basic permissions you can give your developer they can view a project or start a build when they start a build they can supply parameters to that build one of the parameters they can just supply well one of but a set of the parameters they can supply is any of the environment variables that the build slave is going to use so they can supply a new path variable for example and if they are otherwise able to log in to that server they can just rewrite what something like GCC gets pointed to and take over the permissions of the build slave if you have project admin that basically is going to give you remote code execution

on the slaves and like I mentioned earlier you can get the project credential super easy straight through the web interface by just requesting a project back up so you don't necessarily need to compromise the whole build master you can just take over that whole project and last is bamboo this one really didn't have very much of a built-in permission system it looks like it's too defaults which is user that can only view can't actually do anything or admin that has full control so I imagine that the idea is that you would then set up which permission level you actually want and so there's only four that you can choose from access to view stuff create a new plan which is a project

create a repository which is a source code repository or admin so really the create plan or admin groups are going to be the two that let you actually get code running on the build slaves but by default everyone's just going to get made an admin in that system so what's the high level thought for all of these system permissions a level thought is going to take a lot of tuning with any of these to make sure that you've made the system usable for your developers so that when they check in code they can run a one-off build or things like that but it's going to be pretty difficult to prevent that developer from more or less having

effective admin permission because if they can run custom code on a build slave as we'll talk about next chances are they can take over the master server as well so a solution that you might want to look at is maybe your developers don't generally need access to the CI system maybe you've got a few build engineers or an engineering manager that has access to the system depends on your company or the second is that you probably at least want to segment your CI system so instead of adding every new project to the same system if you've got two teams that work on different projects that shouldn't trust each other put those on two different sets of CI

servers I got a quick note here starting to run at a time so basically as a quick example Jenkins had a bunch of plugins I looked at one of them I came across the server customers using used the Google authentication like I was mentioning before but it was tied to their company's domain I looked at the source code for the plug-in looked at what was happening behind the scenes and basically the plug-in developer I'd miss understand how the OAuth flow worked with Google so you could intercept the login that was taking place and instead of telling it to authenticate to the customers domain you could pick any domain you want Google would say yeah wet zones you know test domain com sure

he's authenticated stat I'll tell the server yeah Wes has authenticated the server assumes that I sent a kid it's their domain not my own and then lets me in so this kind of vulnerability is pretty typical I think for plugins to any of these systems not too Jenkins so it's always worth considering the plugins this next slide I think goes without saying but it's worth highlighting so the point of CI systems is to let administers administrators product administrators configure the source code repository the build step and then take off a build that basically means the point of these systems is to allow tasks to run a command so in Jenkins you had a build step called the

execute shell and it lets you configure what shell command to run teamcity and bamboo same kind of thing if you want a remote shell all you do is tell it run bash shell bash to pipe me back a remote connection to my IP import and redirect input so now you've got a remote bash shell if it's a Windows built slave I recommend some PowerShell magic but same idea so now you've got code running on your build slave what if you want to take over the whole system I'm calling it build slave here each system calls it something different node or agent what have you I'm calling it slave because that's fine for computers I think

probably not for people so if you can trigger a custom build you can get code execution and if nothing else say you can't pivot back anything else that gets built on that slave you can now compromise or mess with that might be good enough depending on how they've architected where builds go to what slaves in all the default configurations you end up with a build slave running on the build master and in the default configuration that's running is the same user as a build master so company has left this configuration in place you can simply run code on the slave that's on the master and take over the master if segmented you're still going to need

some pass back don't have a lot of notes on these but in Jenkins there's a bunch of different protocols for different protocols and two command-line protocols to talk from the slave to the master for old versions there was no control over that and so they just documented slave can basically use Java remoting call and tell the Jenkins master to execute whatever code you want on new ones they try and do function whitelisting and I'm looking into this but haven't really finished looking into it the theory at least is that slaves should really only be able to do limited things upload job results in that that type of thing for team city I actually found a number of remote code execution

vulnerabilities and how they did the slaves to master so I still need to tell them about it I guess and disclose that in some way but in the meantime you've kind of got two bottles for your slaves out of a unidirectional or bi-directional and they use a different protocol for each so how you might want to attack those is going to very XML one I tried like external entities attack and that didn't work so you don't have to try that but other opportunities might exist authentication is actually pretty neat with teamcity teamcity a slave just picks its name and tells the master hey I'm a new slave do you want to authorize me and there's a big list

of new slaves so if you want to just call it grief you can like script up a million new slaves and they'll all show up in the list using the admin does still have to approve them but you can mess with their system slaves are supposed to be pretty limited as to what they can access on master so when things are working by design they can really only upload job results King City doesn't use encryption by default for those links says just put in a secure environment bamboo uses the bandana program and the more slaves you have the more you have to pay for them so by default chances are good the build slave is on the build master I still haven't

really looked into the protocol and all but I'll be like that guy pretty soon hacking the bamboo so to kind of round things off that's now that we've taken over the build master and build slaves how do we do a back door obvious way is just add a new build stuff people probably aren't going to notice that something new is in there give it an instant name like static security analysis as Clippy says say you're a developer and you notice this do you even tell anyone I probably would a better way though is that a new plugin there's a million plugins no one knows what's going on and they have full control over the system plugins can be

triggered at any point in the build process and in most of the systems you can set it to the plugin runs against every project with no new changes to that project so I'm going to put some examples of these at the end but backdooring just a few jobs is good why not backdoor the whole system I think the best offensive is plausible deniability so you already know there's a million ways to do your configuration wrong once you've taken over one of these servers just make the configuration less secure turn off CSRF protection add some test account maybe add a new off provider generate a whole bunch api tokens this is possibilities are kind of endless and then and most of these no

one would ever know for sure was an attacker but I kind of wanted to have time to talk about like the next level compromise scenario so accusing backdoored compiler will backdoor all new versions of the compiler the backdoored CI system can do the same so you've got your CI server you've got some build slaves when you upgrade your CI server it pushes new updates out to the build slaves as well so say we took our scenario we compromised a project got our code running on a build slave we've got a compromise build slaves got some animation here it's pretty great so we've got our build slave it's it's infected and now they realized someone's been messing with our system we better

reinstall the CI server they reinstall the CI server it's all good but your build slave is anticipating this and since slaves have some level of permission back to the master slave checks his maps are still compromised no so it goes and reacts the master and it tells the master yo react all the other build slaves and then this cycle just continues for ever your changes are never pushed down to your build slaves fast enough for them to not react the CI server so I haven't implemented this yet but I totally want to so beyond the look of opposed some code so in summary I think it's probably impossible to fully secure one of these systems it's also probably

impossible to clean up once someone's hacked these just because the configurations are so complex so that means don't put your systems on the Internet and if you do at least put like an external authentication proxy in front of them don't just rely on the CI systems authentication and you kind of should consider how much do you trust output of one of these systems would you even know if it was back during your code I kind of ran through a lot of points on this talk there was another great talk given a black hat and the you a couple years ago so if you want to learn more about it I definitely recommend this talk I'd expected that a

bunch of the findings in that talk would have changed in the last couple years but I went back and read through it after I put together bunch of my findings and pretty much everything still the exact same so a little bit sad that not much has changed but a lot of good reference in that talk and otherwise I'm going to post these slides and all the tools I referenced on my website probably in a day or two which is there and if you have any questions you can find me here or that's my email and I don't know we might have time for questions right now I'm going to go with yes any questions just a quick question

now your thoughts about in-house versus staff CITV systems and you know the thread surface and management's of Juji controls yes so the question was thoughts on SAS versus in-house run I didn't really look at South Systems I would imagine it's kind of like any cloud hosted system security overall it's probably going to be a bit better but one vulnerability is going to affect a ton of customers but didn't really research that side of it I know the in-house ones pretty much all vulnerable to the things I described we got some more questions

hi so some projects these days tend to go towards what's called reproducible builds as in anyone can compile it and arrive at a file that's like hashable to the same hash just to make sure it hasn't been tampered anywhere in the chain how well do you think things like that might mitigate having a compromised CI system yeah so I think that's going to depend where and which CI system you compromised so that's like a great a great theory to have in place and probably something company should try for the challenge is how do they generate the very first hash so if you compromised the system that's generating the first ash people hashes aren't going to match after that and then there's

going to be a lot of confusion and anger and upset men so I think I think it kind of depends what you're trying to protect against but with the theory of say you and I both check out the same project and should be able to come to the same build that's actually going to be something that's a really good mitigation CI system normally can talk to the source code repository so there's also that challenge which is maybe the CI systems not just tacking on a backdoor but maybe it's specifically changing some of the source code right in the source code that might be another angle but no I think that that kind of thing sounds like a definitely a good

idea

[Music] so my question is it really seems that security is barely an afterthought with a lot of these systems so how would you recommend aside from kind of from an external security perspective that these vendors take responsibility for the integration of security into these tools because it seems to me like that's the best place for that to happen yeah it's a bit of a challenge like I said these these tools really have to be adaptable to very complex environments and that means by definition they're going to be very hard to fully secure and harden I think the more the more vendor is held accountable the better they're going to get I think Jenkins is probably a great example of

that two years ago they were a complete disaster in my opinion at least in terms of security and they started getting a lot of press and a lot of people complaining for lack of better word that their security had issues and so they then started actively looking at improving that I think it's going to be basically up to customers to demand the changes before the vendors are going to put them in place but I definitely agree the vendors are really the ones best suited to fix the security challenges all right any questions well thank you thank you all right oh my mic right there very much wet all right that great talk let's get ready

[Applause]