GF - Actions have consequences: The overlooked Security Risks in 3rd party GitHub Actions

hello everyone good afternoon and welcome to bides Las Vegas I am Hara nikar and this talk is about actions have consequences the overlooked security risks in third party GitHub actions given by our speaker yon avel uh I have a few announcements before we begin we would like to thank our sponsors especially our Diamond sponsor Adobe and our go sponsor Prisma Cloud blue cat Toyota it's with their support along with our other sponsors donors and and volunteers that make this event possible these talks are being live streamed and as a courtesy to our speakers and audience we ask that you check to make sure your cell phones are set to silent if you have any questions please use the audience microphone so that YouTube can hear you as well and with that let's begin and get started please welcome our speaker hi guys my name isona vital uh thank you for coming to my talk really excited to be here uh I'm a security researcher at Palo Alto and I have about 15 years of experience in the cyber security space both as a developer and uh security researcher uh previously I worked at cyber security uh where uh we created we ref focused on sorry cicd security and we created a fairly popular Break the builds ICD go so if you want to learn how to hack pipelines this is the project for you you can search it up in Google or GitHub Palo acquired the cyber security about 6 months ago so I've been working since ever since now in today's talk we're going to talk about gab actions H and the problematic permissions model of Gab actions which in my mind encourage users to make mistakes security wise uh and uh what would have been a very profound mitigation that simply has a significant loople in it I'm going to show you surprising statistics from analyzing open source projects across the globe and a few takeaways so you can better uh defend your uh projects so gab back chance is a very popular cicd platform by gab which allows you to build test and deploy your code automatically and it has a very rich and extensive third party action Marketplace uh you can think of an action as a software package that instead of developing the logic developing the logic inside your workflow you can simply use the action from the market place now most of the actions are developed uh by the community and not by giab and everyone basically can write their own Cod that posted as an action in The Marketplace so what initiate This research is actually me analyzing going over build doogs of public projects uh it's a hobby of mine don't judge and I notice a phenomena where um benign action developed by unknown developer basically has full-blown permissions against the repository and I wanted to see exactly what is the root cause and why it is it is so widespread so let us begin with a quote by gab where it states there is significant risk in souring actions from third party repositories on GitHub now that sounds pretty severe to me uh let's see how the GitHub users handle this uh warning so the way things work in GitHub actions every time a workflow starts or a job to be precise GitHub creates a short live token called the GitHub token in order for the workflow to be able to interact with a repository now we can set permissions to the uh gab token in the a workflow yamel file either by specifying permissions for the entire workflow or for a specific job we can set uh permissions for the all of the Scopes or particular ones so GitHub does a pretty good job um providing options to um specifying permissions but if you don't set permissions inside your workflow the default permissions of the organization or the rep will kick in it used to be right on all Scopes up until February 2023 only then G up realiz is probably way too permissive and decided to change it to read only but they didn't change it for existing uh organizations only for new ones because I guess they didn't want to break builds across the globe uh which leaves the vast majority of projects on GitHub having right permission to this very day now let's review more of the sensitive Scopes um we can assign to gab actions contents being the obvious one allows the workflow to push new code to the Repository ID token which allows the work to authenticate to authenticate against a cloud provider like AWS or Azu on your behalf via the oidc protocol packages which allows the workfl to upload new packages um administrate that part of the repository packages that will be consumed later by the production environment or by your customers and lastly we have actions which allows the administration of the gab actions part of the Repository now let's review an attacker uh com trying to compromise a third party gab action it can happen in attacks like command injection or repoing or many other techniques out there and if you're thinking uh what are the chances I get compromised it rarely ever happens so uh my colleague AI who sits here in the audience just gave an awesome talk yesterday about how he managed to uh take over third party action by the thousands so these things happens all the time and our attack can push after the took over the third party get up action can push a new malicious code which will then be executed inside the pipeline which will in in fact poison the workflow and from there he can reach all the way to the repository or the production environment okay so now that we understand the risks let's try to reduce uh the attack s first by managing the permissions correctly so let's say I'm a developer I like to use a s gab action my boss told me uh it's a good idea to have more AI in my project because a is the best uh big thing so I'm heading down to the actions a Marketplace search something with AI and I found this open commit improve commits with AI sounds brilliant Brant exactly what I needed but there's no place here on the marketplace Place page or the repo page where I can see the permission this action requires uh so in order to set the permissions correctly uh now in gab ecosystem we have a concept called a gabab where I can approve the permission and I actually getting prompt to approve permissions but it's not the case here all I have to do to use this action is basically copy a snippet of the uh action name and I can run it inside my workflow without specifying anything so this was my exact reaction when I saw this like where the permission I need so I've decided to develop a tool that does exactly that called Piper it's a shortcut for Pipeline and permissions and what it does it can take any third party action do a static code analysis on the code and finally outputting the exact permission it needs no more nor less by analyzing the usage of the code against the repository let's see a few more examples here here we have the OCTA kit which is a client by gab very popular one abling which enables the repositor to interact with a repository with the workor to interact with a repository and we can see here that uses a pull request so I can derive from that what a permission it needs furthermore I can interact there are many ways to interact with the repositor you have AR rest API bash graphql so I've written a lot of static uh static analysis rules mapping all the usages again to the against the repository and I know I I know exactly the permission it needs so after I wrote Piper I decided I wanted to see how the major public repositories of major public open source projects are handling permissions in G AB uh so I took the 2,000 repos which had 6,000 workflows and one 1,000 unique actions from the top St open source G projects all the big names are here we will see in a moment and I wanted to see exactly how they manage permissions so I I've created a database uh of actions and their permissions using pipel and I've uh I um went over and analyzed the workflows and extracted the actions they use and the granted permissions by taking a look at the build logs you can see on the right uh each time workflow on it prints basically all the uh permission it needs in the build so once I have that I can compare granted permissions versus required permissions and I found this insane number of 93% of the uh open source projects the big ones doesn't even manage their permissions uh no one basically doing it right out there having at least one excessive workflow permissions also I found out that 50% of the action doesn't even require get a token in order to operate like uh if you take for example slack notify with the sole purpose of sending a slack message from the workflow it doesn't do any interaction with the repository it doesn't need permissions yet again in a lot of time having uh full Ro permissions against the repository same goes for seta Python and many other actions which does not require the gab token from the first place and all you have to do in order to mitigate the risk is simply set the permissions To None like this another interesting figure I found is that 94% of the action actually use three Scopes up to three Scopes out of 15 available leaving the more dangerous one one aside issues P request and qu read being the more popular ones and this is another slide that blew my mind this is actually the essence of the This research we can see here open source projects we all use uh Microsoft cloud flare and Airbnb granting right all permissions again on all Scopes to benign action nobody knows maintained by a private developer with low number of stars you can see the star on the right uh I'm sure Microsoft is putting a lot of time and effort to manage their employee permissions against uh scms but here we can see uh just an unknown developer on the internet sure let him have every permission he wants basically so to mitigate the risks we have a concept that is called action pinning which is recommended by GitHub and what GitHub actually means by that is that we pin against a certain commit hash of the action so that will ensure that I will always work with a safe version of the action that the way to do so is not by pinning to a branch as we can see on the first row or to our release tag the only proper way to do so is to use the full commit hash as we can see on the last row now going back to the drawing let's see uh another let's see a what happens if an attacker will try to overtake this setion this time so we have a workflow and we are pinned against certain commit hash in the green but our attacker can come push a new malicious code after he took over the St party get action of course but it won't have any effect on the workflow because we are always taking the uh commit with so this supposed to have a complete remediation over for an attacker compromise and third party action right well this is wrong as I've discovered doing my research actually uh and I want to introduce you a concept that is called the Imp pinable actions which I came up with uh actions that even if pinned can still have devastating results once compromised over the workflow so let's consider this uh workflow a very simple one it's a workflow named C and it uses an action you can see below called un pinable action with a full commit hush as it should however for the first time it ran it printed out hello world but in the second time I've got pawned so how it's even possible if the workflow hasn't changed and um pinned against the commit has so the actions code hasn't changed how is possible so I will take teach you the trick I will teach you the trick in a moment but uh for us to have a better understanding let's do a quick recap on the uh type of actions so just a moment so we have three type of actions we have the JavaScript action which allows me to write in JavaScript or typescript and the actions code it's very popular we have the octokit client that just show you now if require more heavy lifting uh certain operating system or dependencies I can use the docker container action and lastly we have the composite actions which allows me to write bash inside the yl file or uh simply use another action within the yl file so you can call another action and it's very um it's very good for short operations now going back to the workflow we just saw so here we have the commit has and moving on to the repositories code of the action uses the same hash so that mean it cannot change but what we have here is actually this action pulls the latest tag of a Docker image from container regy which completely voids the uh the pinning because lat St is mutable and it's the complete opposite of the concept so basically it means that every time this action will run it will pull the latest tag and the new code will be introduced inside my workflow so going back to the drawing one last time here we have the same workflow pinned against the commit hash but our attacker this time can take another approach and try to take over the dock container registry of this uh action and push a new H image image tag the latest image tag which will then run inside the third party gab action which all the way to the wflow and from there you know the drill to the repository or the production environment now what I just show you is not a one scenario I can show you in each of every type of action how to do so how to escape the pinning another way to write a Docker container action is to supply a Docker file file and here we can see unlock packages python packages in this example without re requirements txt uh which uh again violates the concept of pinning here we can see download and installation of third party binaries without check sum I'm moving on to the composite actions we can see here our action uses another action but in an unpinned way so again it's actually temper with the pinning mechanism you can see here uh another example of how can we escape pinning with uh pulling a mutable tag and here we have an npm installation again without locking uh JavaScript package and lastly we have JavaScript which uh action which the situation there is slightly better because there are no runtime installation of packages but we can still fetch resources scripts from the outside world and by that actually temper with the pinning so I love numbers so I took 1,000 actions again top start actions from the marketplace and found out that 30 32% of them actually are unpen what I would ick call them so it's not surprising to see that 67% of the uh walk for the duping directions they have done the uh have a lifting of calculating the check sum uh are pinning against an un pinable action that's a insane number basically a lot of workflows out there can be compromised uh few takeaways so you'll be able to defend yourself before we end this talk is uh basic this is very basic you can set the default work permissions inside your organization G organization to read instead of the permissive right specify exact permissions inside your work Prof file now GitHub just in time for this stock released tool it's still in public beta that enables you to calculate the uh actions the permission you needed for each action uh same like Piper but they took a different approach they installing a proxy a little bit more intrusive now this we haven't cover doing the stock but you can uh isolate actions to different jobs because uh job runs in containerized environment if one job get compromised it won't affect the other and lastly do pin tions uh it's not a perfect mechanism but it's better than nothing and uh you can use a cool open source project called ratchet to do the pinning automatically on your workflows and if you do found a third part action you would like to use with which is unpin you can always Fork it lock it and change it from there and uh to conclude this talk let's end with a quote by uh the late rald Reagan which said about get actions don't trust and verify thank you if you have any questions sure me there's a lot of similarities between GI actions dependencies and standard like node packages python packages which already have their own set problems you think that there's anything that GitHub should be learning from what we've already learned in those spaces poisoning to secure this supply chain yeah I think the GitHub should handle third party action as a as a whole not to do stuff maybe at run if it's possible I know it's I think it's in the road map to maybe have a binary instead of a third party action which will block for many runtime installations and which allow the escaping of the pin follow-up question so I know that I think it's Google or somebody has a like a group out there for like open source projects that are too big to fail which need like you know additional monitoring and security and whatnot do you think that these actions that are widely used should be a candidate for that pool of you know Express open source security evaluation right to make sure they don't get poisoned Yeah so basically there are a lot of mitigation you can do uh when you're using a third party giab action um starting from blocking Network to the outside world uh blocking the attackers from um interacting with the CNC and um um the obvious thing would be to vet the code of the action before you use it and try to see if it's really pinned or not thank you thank you for the great talk uh you you mentioned that you use static analysis to analyze necessary permissions for the actions yeah uh have you encountered any any tools that would use static analysis to just uh assess the risk of the third party action uh actually no because it's pretty specific to analyze third part gab action on its own I'm not looking for vulnerabilities in in the case of Piper the tool I wrote I specifically checked for usages so I can map those to permissions are are you aware of any uh exploitation that has uh occurred using this method yes of course uh basically my team and I are doing several um disclosures am mons for major projects uh by attacking uh the dependencies so it happens all the time yes anyone else okay thank you very much thank you it was a very good talk