BSidesSF 2023 - Securing the Pipeline: Protecting Self-Hosted GitHub Runners (Adnan Khan)

hey everyone thank you for coming to my talk securing the pipeline protecting self-hosted GitHub Runners and probably your actions pipeline Secrets as well so about me my name is Adnan Khan I am a lead security engineer at praetorian I primarily work on praetorian's red team but I also do some web applications security assessments as well in my past life I was a software engineer and as a result I am intimately familiar with the conflict that comes between the need to get stuff done and the need to also not leave a lot of vulnerabilities along with that process so I was a speaker at shmukon 2023 along with two of my colleagues presenting an offensive focused stock called Phantom

of the pipeline abusing self-hosted CI CD Runners focusing mostly on GitHub and we open sourced a tool called GitHub attack toolkit Gatto for short in the mascots of cat you can see the play on words there and this is my second conference talk and I'm based in Baltimore Maryland now but originally I'm from the East Bay Area so it's kind of nice to be on some familiar soil so why this stock well because devops so devops there's a lot a lot of moving parts and a lot of places where things can go wrong and securing devops environments is pretty hard and I've seen that personally on a lot of red team engagements targeting organizations that use GitHub actions

and really there's there's a lot of chaos there's often a lot of repositories some organizations developers able to create their own repos you might have people creating individual dot files repositories it's things can be quite all over the place there are API tokens that might be added as repo Secrets as organization secrets and some organizations will use self-hosted Runners for deployment or other integration where they might not want to run it on githubs Azure hosted runners and compounding that there's a lot of permissions that can sometimes allow developers to make mistakes and not really know what they're allowing someone to do if there is a compromise so the way this presentation is going to be structured is I'm going to go give a

quick primer on GitHub action CI CD and some of the specifics that apply to that product then I'm going to cover some risks and attacks that exist as well as attacks that I've personally conducted on red teams or some colleagues have conducted and how certain types of attacks can be detected and perhaps even prevented and then talk about some future considerations and if there's time open it up for some questions so GitHub actions is a little bit unique in with how it plays into the larger part with GitHub so since GitHub started as a company focused on open source projects and smaller teams and recently after the acquisition well it's been a while now after that by Microsoft it's

continuously moving towards becoming an Enterprise SAS platform so there's a bit there's a bit of a conflict there the build agents themselves for kid of actions they can be self-hosted or hosted by GitHub and the secrets which are can be used for get up actions can be supplied at a fur repository or per organization level so there's a lot of places where Secrets could be left and then subsequently compromised and I also want to talk to you on some other CI CD solutions that are similar there are different attack paths that Target them they're are issues there with some of these other services as well so it's not just that I'm picking on GitHub so the way self-hosted Runners function

on GitHub is that there is a net core application which is the GitHub actions Runner agent and that supports a number of different operating systems and that particular Runner can then be attached at a per repository level per organization level or a per Enterprise level and each each of these levels it essentially functions as a worker pool so jobs workflow jobs will execute and a runner will pick up the job complete it and then that Runner pool Runner will be available again and return to the pool so the problem is that the default configuration leaves a lot of open attack opportunities for an attacker mainly because the runner is not ephemeral and there's no isolation between builds so

and I'm gonna really touch on that later in the talk I also want to introduce some get up personal access tokens so they these are API tokens that are associated with user accounts so GitHub is unique because a lot of times organizations will have developers join their personal GitHub accounts that they might also use for side projects to an organization and that will grant them access to their employers repositories and other assets but personal access tokens belong to the account itself so someone could create a token for personal usage but if the organization is not configured properly that token could provide access to a lot more and these tokens can also be used for traditional command line git operations

so that's your standard cloning you're pushing code that can be disabled and I'll position to cover that a little bit later as well finally the way that these tokens work is oauth scopes so the key thing about these Scopes are for classic tokens is that they really just determine what API operations that token's allowed to use just because the scope is present doesn't mean a user can actually do it because there are certain API operations that require the user to also be an administrator in your organization so creating a user token for a member that's has an org admin scope doesn't really allow that but that just but I do want to cover how these tokens work

because that also plays into where some of the over provisioning happens

so there are two types of tokens until last fall there were just classic get up personal access tokens for since about 2021 they've just started with the GHB underscore string and that denotes a GitHub personal access token but with fine-grained personal access tokens which start with a GitHub underscore fat string they they allow obviously as the nav says finer grained access controls however they're like with anything that requires more finding control it requires more upfront effort to provision them so people often take the the short way and finally going into github's API GitHub has a very extensive API that supports almost any operation that you could probably think of wanting to do with GitHub there's an

API endpoint for it there are some like for example API endpoint operations that allow you to directly interact with low level git blobs higher level than that there's ability to commit single files or pull single files from a repo using just that API so there's a lot of potential for enumeration and attack using just a token there and also there's some interesting things that you can do like if an organization restricts git clone git operations like clones or pushes to just users that have an authorized SSH key using the repository contents API to push single files or clone single files will allow you to bypass that restriction and get specific files from a repo and you can also list it so it's

slower but you can get the same thing or do the same thing there so that's a little interesting thing finally going into the gaps related to the GitHub API the vast majority of read requests for are not logged so the problem here is that attacker that compromises a developer token can enumerate everything they need to before they generate a single event that would trigger a detection or uh a compromise and actually start an incident response process there are some exceptions like the one where if you just download a zip file from a rip of a repository the associated API request is logged as is the zip download operation there is some good news though just

as of February of this year GitHub is finally released a private beta feature where someone an Enterprise can actually get a full fire hose of all the rest API events hitting their organization's assets so things are moving in the right direction but it'll be quite some time before this is made public beta and finally adopted and people actually write detections based on these events and then GitHub action secrets so this is something that is documented by GitHub it's not some not something unknown but some potentially people might not actually know this and because of how when you add a secret to a repository you need to be an administrator and to list it for from the GUI you need

to be an administrator however any user that can just push code to a repository can execute GitHub actions workflows that can access every single repository secret that's attached or even organization wide secrets that are in the default group and therefore are accessible to all repositories so there's that gap between who can look and monitor secrets from the GUI and then who can actually utilize them or hit the API endpoint to list Secrets which again also requires the just write access to a repo as a normal collaborator okay so now that I've introduced GitHub actions a CI CD at a higher level why would an attacker want to Target self-hosted Runners well a lot of reasons

the traditional thing when you think about GitHub is obviously source code theft that's a concern for a lot of organizations and one you obviously commonly think of but there's potential for taking secrets and then moving laterally to cloud or accounts there's artifact poisoning and trying to get malicious packages pushed where other people other systems other clients can pull them and now get malware on their systems and finally as we've all probably heard about the recent 3cx hat hack the potential for supply chain attacks that and if you're an attacker that wants to conduct supply chain attacks attacking organizations get up is probably the best place place to hit hit if you actually want to be

successful and finally there's the risk if you're in organizations using self-hosted Runners for broader Network intrusion and lateral movement if they are not properly secured the kind of attacks that an attacker can conduct from against self-hosted Runners so if Runners aren't configured properly then malicious code that started off from one build can persist beyond that build and get secrets that are used from other builds even if that attacker that kicked off the initial workflow might not have had access to those specific secrets that they're now seeking to exfiltrate there's the ability to conduct supply chain attacks by sitting on a runner and modifying code associated with other builds as they check out code before compilation or pushing to a artifact

repository and finally if you have just a compromise developer access token an attacker can knowing nothing else about what that token has access to get a foothold within an organization's Network by using that token to learn about the organization associated with and then conducting a attack at a high level this is kind of how the kind of the external to internal attack looks there's a compromised access token an attacker will push a malicious workflow to a code repository that is configured to have actions run on self-hosted Runners a self-hosted runner will then pick up the job only communicating there with GitHub and now that attacker has code running within a corporate Network again given that the runner isn't properly

configured and finally you also at least personally on red teams have seen that deployment of EDR and Antivirus Solutions on Southwest Runners a bit rare we've seen it but we've also seen absolutely nothing so you can just drop unofficated malware and you're good to go that's not good so is this particular kind of attack happening well not from what I know not yet specifically looking at GitHub self-hosted Runners and conducting an attack and gaining access to Network systems and conducting Supply chains to actually not yet however really there's been a lack of tooling to Target these self-hosted Runners as well as there's not been that much documentation and tradecraft techniques to for an attacker to get a personal

access token and don't know what they need to do with it and on however on red teams that praetorians has operated on we've yet to see a client that has had a secure self-hosted Runner implementation if they're using GitHub and in every single case that we've encountered that compromise developer access has allowed us to move laterally escalate privileges get prod AWS broad gcp accounts it's been access to crown jewels in every single case when we've seen organizations use GitHub and GitHub actions we also haven't been detected while doing any of these decks so this is a real world attack that was conducted on our as part of our one of our red team engagements Last Summer

and the way it worked was we had access to we this was we were from an external standpoint we used another social engineering attack to essentially gain access to logs where one of them had a personal access token so I'm going to start there what we did was we thoroughly enumerated what that token had access to with and just hit using github's API because again we didn't have access to that organization on github.com we couldn't just open up a browser and do and do that then once we learned that they were using self-hosted Runners we created and pushed a malicious workflow to a repository we set that workflow to run on self-hosted Runners via a tag

we then and then that then queued up the job a self-hosted runner picked up the workflow checked out the code and we were now on a self-hosted runner and that one had had Docker enabled so we what we did was we just spun up a detached Docker container that pulled down an implant and now we had persistence within their Network connecting back to our C2 that eventually then let us conduct privileged escalation lateral movement and we gained very high level privileges within that organization from that purely external standpoint and compromise access token okay all right and this is a similar attack path but the key difference here is instead of the goal being just attacks

on other systems is where we started off with again that compromised access token similar in this case it was a penetration test so it was uh we had that developer access to start with but very low privileged we pushed a malicious workflow but now instead of just we still implanted the runner but now we just sat on that Runner and waited now again this is a self-hosted runner so it's the our build is the main the build that we kicked off is finished there's another it's now able to execute other builds so in this case there's a sensitive infrastructure repository where there are secrets in use that have high privileges so here comes an infrastructure developer they commit a

code change maybe and now they want they push a change to to to to to run say integration tests now their workflows pushed that self-hosted Runner picks up the job and since it uses secrets and in this case it was secrets in a used in a shell script defined in the workflow which essentially on GitHub means it's just written out of the file system with the plain text secret since we're on that Runner we watch the file system the working directory of the self-hosted runner we now have access to those plain text secrets and we get the secret and we actually trade it so that's and again no detection and the developer has no idea that something bad happened because

the build succeeded properly nothing bad happened from their perspective so this leads to risks and that are present and then some of the mitigations that can be used to fix a lot a lot of these problems so for like footholds obviously you have your making sure that there's Network segmentation monitoring builds for Secrets there's Access Control lists and auditing compromise personal access tokens using fine grain tokens so it's like your perimeter security making educating developers and not over provisioning access tokens detection engineering and generally for the build modification just having per job ephemeral Runners kind of takes that entire attack class out of the picture and for malware deployment looking at Network segmentations and you know on on

self-hosted Runner mitigations and really to properly secure an organization's self-hosted Runners and actions pipelines against sophistic attackers really requires a layered Approach at every kind of step of the of being you know the external perimeter what do you do if something is compromised and properly responding to a conference if it does happen so for looking at Perimeter security one thing that we've seen some organizations do but some also don't is with GitHub you can require require saml SSO and what this basically does is when a developer creates a personal access token they will now need to specifically authorize that token to be able to access organization resources so this if you have this enabled in an organization

it prevents the situation where a developer creates a token for personal usage but it also has access to organization's resources requiring SSH authentication for git operations can also be helpful it could prevent an attacker that just gets a token and now it can clone every single repo so if say cloning repos isn't a part of a normal workflow then maybe an organization can disable the ability to do get clone get push operations with just a personal access token and there's also cases where there's certain defaults that are not enabled that you that organizations can enable to get better visibility into actions that are happening IP logging an example of that and I'll show where you need to go to

enable that setting and why it's off by default and keeping in mind that the point I touched on earlier where if someone has a right access right access to a repo they can access every single secret so having a secret in a repo and having someone that can write to it it's basically the same as giving them access to that plain text secret there's just a trivial exercise in pushing a workflow in between them and getting the plaintext value so it's important to understand what that means and limits secrets to only those who have really need to know to to of that plain text value so for and this is like the the IP logging setting which is disabled by

default I believe it's for privacy reasons and it's in public beta that it's disabled by default but if you have GitHub Enterprise and you have the audit log it's probably smart to enable this because then you can have geoip based detections on a potential malicious activity and so classic tokens as I mentioned earlier these are tied to an individual user and then those Scopes like say repo or workflow so a repo scope is what would be needed to clone repositories workflow scope is what would be needed to run GitHub actions workflows those are all assigned to Creation time and with how classic tokens work there's a lot of API endpoints that allow you to learn about what they have access to so

you can use a token to query what authenticated user it belongs to what organizations that user is a member of what repositories within that organization that user has access to so you see where I'm going where you eventually can map out everything you have access to and build an attack plan so to address this there is an option now to disable classic tokens completely but it's it's all or nothing so for an organization that might be new to GitHub actions and and personal access tokens are not ingrained in your current workflows perhaps taking those out of the equation now and dealing with the consequences is better than say years down the road where now you have a lot of tech debt and going

back and changing all those processes to not use classic tokens is so much work so guess what priorities it falls down the totem pole of priorities and it never happens so that's something that to consider if if your organization needs tokens as part of the normal workflow and again that saml SSO feature again this all requires the GitHub Enterprise plan so which is their most expensive plan but really the team's plan is I would say right now impossible to secure properly against a secure sophisticated attacker but making sure that that saml SSO setting is is enabled so to address some of the issues with classic tokens organizations can use fine-grained personal access tokens and as the name suggests it allows you to

have fine-grained controls over what repository the token can access and what specific kinds of operations that token can use to perform and Beyond just the limitation of the access organizations themselves can configure it so that if A fine grain token is created an auth and a user wants to authorize to an organization the organization can approve it before it has that access the current that's not the case with fine with classic tokens classic tokens developers authorize them even if there's there's SSO requirement for it developers authorize it and it's enabled the organizations can't go and stop that before it happens you can only do it after and it's just harder for attackers to use that and learn what I

can access it can access and conduct these attacks that I mentioned earlier so alerting on that for classic tokens organizations can monitor when they're authorized and what Scopes that that token has access to so an organization could monitor the organization credential Grant log event and then for fire off um potentially different apis operations against github's API to list tokens that have been previously granted and even revoked them so that could be made part of like an incidence response Playbook where if uh unusual activities detected with the user organization can quickly list and revoke tokens that have been created associated with that user and that API response does display the Scopes so it could also be used to detect over

provision tokens so it might not be a critical alert but it might be something where someone can message your developer say Hey you know this token's over provisioning can you see if you can configure provision in a more secure manner so that this something like this could help organizations move towards a better security posture with the kind of tokens that their developers are provisioning so all everything I've talked about so far is looking at preventing an attacker from getting access now say an attacker does get inside an organization with the right kind of access token and starts conducting these workflow attacks to attack self-hosted Runners as well as try to steal secrets well is there anything that can be done yeah

there's a couple things so again I keep stating this but this is important but since any user that has write access can get Secrets via malicious GitHub actions workflows so I want to emphasize you I'll I've seen a lot of organizations that use Branch restrictions so there might be a workflow that runs for the main branch and has a runs on Main Branch in the workflow yaml file as if that's the verse that restricts access to the secret it doesn't so any user can just create that has right access can create a feature branch and push a new workflow that they control entirely which is say take the secret base60 for it send it off to a attacker server congratulations

you now have the plain Tech secret value so that isn't is important to look at getting the who has access to secrets to make sure you limit that exposure from the beginning and the way that you can do this in a very in a programmatic manner you could probably write a quick python script to do this is essentially for for each repository in a given GitHub organization make a query to the secrets API endpoint which essentially lists all of the GitHub actions Secrets attached to that repository and then follow that up if there is at least one secret because otherwise there is no need to do that subsequent operation but if there is one secret then now you can

list every collaborator that has the push permission and that resulting mapping that's your list of users that has access to all the repository secrets for a given Repository I do want to note there is another way that you can use secrets and that's GitHub environment secrets and there's a little bit more security there but some of the similar problems exist but I don't want to spend too much time on that but that is some another thing that organizations consider consider is environments and using deployment Secrets there so I want to take a quick step back to talk about salsa and this is a a framework for software supply chain security and actually had to going up this slides because they

just released their version 1.0 on the 19th so I just and obviously so someone could spend a lot of time talking about salsa and probably a whole talk so I'm just gonna quickly talk about specific points that pertain to the build step and this is a very new framework but essentially for build there's four levels starting from build l0 to build L3 and build l0 obviously that's nothing no no security build L1 you have automated builds so that's just using something like GitHub actions to run builds and then there's that provenance of what what led to that build and what's the and tying that to the artifact build L2 essentially just take that Providence have that on a hosted build

service and it's signed but again the the if you're really interested in that link definitely read up on salsa I don't want to spend too much time on that but the thing that applies to self versus Runner type attacks is build L3 build service hardening and what what that is is that for an organization to comply with salsa build L3 they need to be using isolated builds that means that each build is isolated from the next build so there's no interbuilt interaction and that there's nothing in the build systems control plane that can allow a build to interact with with the subsequent build like say there's a cache and one build can poison the cache and the next build now uses a

poisoned tool so that would be something that shouldn't happen if an organization wants to comply with salsa L3 there's also protecting the signing Secrets used for the to sign the provenance but I'm not going to spend time there another risk so if a self-hosted Runners you should probably have antivirus or EDR present on these Runners attackers shouldn't just be able to go and run a malicious L file in the in the background and that's unobfuscated like it they should have something to at least catch the less sophisticated attackers there are if you're using containers for self-hosted runners there's a lot of EDR products and vendors that offer Solutions there but not going to name any but there's plenty of them out there

so there's also a risk of code modification and this kind of touches on that supply chain attack that an attacker might want to conduct in a very stealthy and hard to detect manner so what will happen in this case is you have a self-hosted runner and it's not ephemeral so attacker will create a malicious workflow push it and now start a build that build starts that background process and the build ends Brenner is now ready to pick up other tasks that background process now continues to run legitimate build comes out checks out code all right let's get to that step here that legitimate process will then check out code and the malicious project might already know which build it's targeting

so it looks for that file swaps it out with a malicious file that contains a back door and now that pipeline continues with its normal step so it compiles it and publishes it but now you have a backdoored artifact which is linked to a legitimate original commit so the only way in this case would be to go and reverse engineer that final build artifact to find out that there was some modification made because otherwise the original commit checks out is legitimate and it's all signed properly because it's following the normal pipeline in every other way so that's that's you know where the Decker could obviously conduct that 3cx style attack if they wanted to against

organizations that use GitHub actions and non-ephemeral self-polited Runners so the other risk is kind of that tool or cash poisoning and the way an attacker this again this is a more a theoretical example where the build will check if a tool is installed and if it is use it but if not go and install it so in this case if it's a non-federal runner that malicious initial workflow could just install the tool and then that second build will just say okay the Tool's already installed I'm going to use it and now your supply chain is compromised so another way an attacker could accomplish that same goal of modifying code but this is just presents in a different manner

all right and then there is also the risk of stealing build excuse me secrets from subsequent builds so that this is where you might have a container that's isolated from the network so there's no risk of lateral movement into other systems but if that container is maybe if not ephemeral maybe if it's only rotated every 24 hours then attacker can start a background process and if they can escalate to root then they can just dump the memory of the GitHub actions agent and get all the secrets that are sent associated with subsequent builds or use the technique of dumping secrets from Shell scripts in the workflow in that case is just looking at files on

the file system might be setting like an I notify watch and just pulling the shell scripts when they appear

to so to address those risks and this aligns with salsa build L3 is using Runners that have an isolated and ephemeral environment so it'll completely eliminate all of the risks associated with that build time code modification stealing of Secrets via monitoring stealing of Secrets via dumping memory but you know do you understand that some in some organizations it might not really be possible or it might be possible but the time and budget constraints prevent moving away from self-hosted Runners to a proper ephemeral setup there are other mitigations in differential controls some have outlined before like EDR that will then be very critical and if someone you are trying to move towards setting up ephemeral Runners I

want to note that the get up actions agent supports self-hosted Runner running in a self-hosted manner but really that just means that you pass the self-hosted flag and as soon as it runs the job deregisters itself but that infrastructure that's up to organizations to manage thankfully GitHub does recommend to open source Frameworks that handle all of the provisioning and deprovisioning configuration of I'll link both of these here here and also I want to I I have tweeted out the slides um for this so if any of these links or anything wants to see uh I can make make sure to so because I know I have a lot of content here Okay so touching on that risk of internal

network access and moving to other systems now this is one where an attacker could do on a ephemeral Runner given that network controls aren't set up properly so per github's documentation a single GitHub actions workflow can run for up to 35 days that's quite a while for a build and that's also quite a while for an attacker to do bad things in your network that's you definitely don't want that in your network for 35 days so the way they can handle this is looking at standard Network segmentation type controls making sure these Runners have limited internal access and if there is a need for say a runner that has more extensive access than look at limiting those Runners to

repository level self-hosted Runners and then auditing that create workflow run log event to make sure that no one's that shouldn't be running workflows is running workflows on those runners to kind of handle the the case where there's a build that's lasting for days and days and that's an attacker basically the run's just running it like an implant there really isn't a way to at a organization's scale to track the duration of Builds on a given a single self-hosted Runner so the way to kind of handle this one would be look at taking the logs off of the self-hosted runner itself so there's a directory the Diagnostics directory where the runner applications installed and they'll just be log files there so

setting up some tooling on the runner itself to take those files and send them off for perhaps some processing and then ingestion to a seam so that could help keeping keeping OverWatch of what's running on a self-hosted runner and how long these builds have been running to detect the case where uh run has forked off a process so this Runner tracking ID environment variable essentially for GitHub actions is it's a feature where if the a process is running if that's not set to zero then once the job completes the actions agent will reap any processes that it started so if the process if someone starts a process and sets that run a tracking ID variable to zero then that backgrounded

process will not be terminated after the build finishes again this is on a non-ephemeral runner so perhaps doing something where you check processes on the runner by the GitHub actions user and see if that variable is set to zero there may be some legitimate cases for this but if you don't have any leisurement cases for this have seeing that would be an immediate indication of odd or malicious activity so alert raise alert and investigate it

going into github's audit log I mentioned earlier in the talk that there's a couple gaps that exist so there's a few of these audit log events that can only be pulled in via the rest API meaning if you have it set up to just stream it's not going to be included by default so and these are the work the the audit log events like when associated with the creation of workflow as well as git operations so git clones or push operations associated with an organization those are not automatically uh present so that's important to ingest those and if you do need those to build certain detections and if you have that default setting of Ip logging changed then perhaps looking

at geoip based detections for activity associated with users so if you know where your user is and suddenly you have a bunch of a cluster of events coming from Russia that's probably not good so that's where you could start building some detections for malicious activities and this is more doubly more so for events associated with critical repositories and using the GitHub API you can actually filter audit log events associated with a given repository so for maybe critical info repos watch those very closely here's something that I've not seen personally but would be really cool for someone to start implementing is by adding a honey pot repository within a GitHub organization so a way I could see

someone going about this is making a repo that every user every developer can have every developer can write to name it something that seems like it's valuable add some juicy repository secrets add a self-hosted runner make sure it's isolated so make sure your Honeypot doesn't end up actually causing a breach don't want that and then closely monitor the API so this setup can allow someone to this setup can allow someone to pick up events associated with malicious activity and here's something again and I I recognize that detection engineering is a pretty complex thing and it's not just a you can't it's not a one-size-fits-all approach for every organization but I hope something like this could be a

start to for organizations to develop detections and start treating GitHub actions and Associated log events just as they would AWS IM events as an example so the kind of a cycle that could work for GitHub actions is taking the audit logs streaming and the rest API taking any filtering suspicious events and Taylor obviously tailoring that to how your organization works because every organization is going to have a different Baseline associate based on how developers work but taking looking at things like oh if you have a bot user and it's only doing like comment triage why why you're seeing events like associated with other events so that that's a good place to start because if a lot of times the tokens that are

compromised could be bot accounts looking again goips access to certain repos then you can move on to enriching these events so for workflows any malicious workflow even if the actions happen later are don't have a good trace it still starts off with that workflow yaml file that's associated with the commit so because that would need to be in a repo for it to actually run a workflow so having any events associated with GitHub actions workflows it'd be good to pull the workflow file and package that along with the event so if someone does need to look at it for manual triage that workflows right there because if it's say pulling down a shell script from an unknown domain that's probably

bad and good thing just alert on and start a potential incident response process and then this is these are the kind of API endpoints that you would use from GitHub API github's API to enrich workflow events so you can get workflow runs and get the workflow associated with a specific suspicious work created workflow run log event take the shot of the commit and the path and then use another API request to get the workflow reaml file itself so now you have the workflow yaml file associated with the potential potentially malicious event for and it's right there you have to go hunt down and spend 20 minutes figuring out where the where this workflow file lives it's part

of a triage process and like some if you did that and you saw something like this this would obviously be clear indication of malicious activity and if an organization can detect that just the fact that a workflow was executed that did this then already they're more secure than most organizations at least based on what we've seen on red teams with organizations that use GitHub actions so to conclude we're really starting to see developer compromises become the star source of really impactful breaches obviously you had Circle CI back in December and that one I really believe that if the attacker knew what they had they probably could have done a lot more damage to some of their clients LastPass

that started from a developer compromise and so for to protect your organization's GitHub action cicd like I had mentioned earlier there's that layered approach you want to apply the least permissible least privilege both the repository Secrets as well as how you have your self-hosted Runners configured you want to have secure Runner implementation in that you are using ephemeral runs if you can at all possibly do that and if you're not making sure you have proper EDR protections and honestly if you have a firmware one you should probably have EDR AV projections on it and then looking at starting detection engineering pulling in log events that could indicate malicious activity and starting there and trying to get

something that's better than what what what it is right now with a lot of organizations that use GitHub actions and just some food for thought for future research looking at like pivoting to Circle CI itself mostly Runners if someone's using those just by attacking those uh config yaml files given a malicious uh excuse me given a compromise personal access token and a couple of the resources and references I used again I really want to call out how good github's API document API documentation and just general documentation is okay and how it's been really helpful in putting together this stock there's a self-hosted Runners blog post that colleagues in pretorian and I really released back last summer as well

as our shmukon 2023 talk and I want to call it this blog post from from karimra Hall covering like reverse engineering the GitHub actions Runner and how it actually pulls secrets from GitHub that was really helpful in developing some attack tcps for dumping Secrets given a compromise token and I want to quick shout out my two colleagues at praetorian Mason and Matt they were my co-presenters shmukon and Crow developers of the gato tool and I want to also thank John and Jimmy also because he's a Victorian they were the first to use gotcha during that assessment where I covered that case where an infrastructure engineer pushed code and they were able to steal secrets and that was the first like real world

usage of that tool and that led to demonstrating a pretty cool attack path and all right thank you so much for everyone's I'm a glad to take any take any questions [Applause] all right if you've got questions please raise your hand I will come run to you with the microphone first question is about middle height on your right thank you uh so you mentioned that in the pen tests you've done you haven't seen uh organizations using ephemeral and isolated Runners but you did link out to a couple of tools that like help provide that functionality so why do you think though you haven't seen those adopted in practice are those like really hard to use or is it just an

awareness problem so yeah just to go so we actually have seen ephemeral and self-hosted Runners however they were not secured properly and that in one example the service account I was running was over provision so we then moved over to improvised within AWS so they had ephemeral Runners but they didn't provision them properly so they you they use them but we found issues any other questions I'm getting my steps in today for sure no you're good

hi so you talked about oh I'm up here okay probably with the lights so um yeah um so I'm I'm wondering what you think about using service accounts for personal access tokens um because you know uh people or developers can have their own personal access tokens that we can use for cicd builds and I wonder what your thoughts are on using service accounts instead of developer accounts in the instance that a developer switches teams or you know leaves a company that sort of thing yeah so in terms of the the question was using service account tokens instead of developers own personal access tokens for uh and it just to clarify this is it for use in

like secrets uh use of Secrets yeah yeah okay yeah so one way that would make it easier to secure and have more insight into what's happening is using those GitHub apps and having the install app installation tokens being used for the actual operations those allow more fine-grained controls and it's easier to monitor like which apps are installed so you could have a GitHub app that's doing service and and using GitHub apps as a service account because in GitHub when you have a service account it's just another account it's just like another user and compromising a token is listed with those is just as bad as compromising a developer we have time for one last question if we

have one last question from the audience raise your hand sorry one more time oh thank you coming down to you [Laughter]

haven't explored any attack vectors with third-party actions and if you have any recommendations on how to mitigate the risks there so not not personally however there are I've definitely read some blog posts and other techniques like looking at vulnerabilities within third-party actions and some of where those might come up is if you have a compromise token that doesn't say have the ability to modify a workflow then you can't modify the yaml file and therefore have to look at vulnerabilities and how that how a third-party action using the build is handling code that's in the repo that you can modify because if you compromise a token that has the workflow scope it doesn't really matter what the other actions are

doing because you can just edit the workflow itself and do whatever you want one last round of applause for Adam please thank you so much everyone [Applause]