Threat Modeling the Github Actions Ecosystem

yeah hi I'm Natalie we're here to talk about threat modeling GitHub actions and specifically about how GitHub actions is a a little bit different than some of your incumbent CI and and how you would have to change that threat model um this is not in in the next 50 minutes going to be a zero to fully in-depth threat model just places that you really want to spend some extra time with with GitHub action specifically a little bit about me uh I'm a squash Marge evangelist at GitHub um if I could like change my official title and work dat to that I would um and then I I joined GitHub but I led the real thing for years at Big defense

doing all the alphabet soup of compliance stuff kind of specific to developer life cycle in GitHub specifically so than uh nist cui cmmc defs itar socks all the the all the alphabet soup compliant standards um I was also involved in some of the 2021 code Club exploit uh clean up so lots and lots of fun there um and slides write up links Etc will all be available on website here shortly and I believe the recording is uploaded to YouTube later as well so I tried to put all the links on the slide so you could like do the the take a picture but if you miss something please don't be don't be alarmed it'll it'll come back

around and I want to just talk a little bit about why so why you're going to threat model you're really going to try to understand and consciously think through your systems all of them for security and stability as part of your software delivery and normally when we talk about threat modeling we're talking about the end product so you build widgets Fu and bar you're going to do Foo and bar we're not talking about the threat model for Foo and bar I figure you've got this covered we're talking about how you're building this um because this is an increasingly important part of of software craftsmanship and your fast feedback back and early feedback of your CI model

is is becoming more and more critical as you're trying to ship software faster and faster and faster and nowadays cicd is way more than your apps code um I started using a CI system called Hudson um is now known as as Jenkins um me yeah back when it was called Hudson and at that point like it was pretty remarkable that you could have a nightly build uh I think that's actually part of the Joel test is one of those things is do you have a nightly build and nowadays it is 2023 a nightly build having to wait a whole day for feedback that's pretty scary but now we really need to start thinking through okay if we're getting feedback 10 times

a day like what's the what's the problem here like we have so much more code in our CI pipelines now and this gets even more uh I would say a little scary once we start thinking about get Ops and infrastructure as code because it is a fabulous new way to learn how important repository permissions are um I learned that my very first experience with giops with a tool called salt stack um still around uh but not quite as uh not quite as popular as it once was and I uh I I might have taken down a couple thousand users in production because we had Windows new lion characters um in a in a repo that got

automatically pushed out to uh to an Apache server and uh reloaded and then all of Apache died all of it because of something that really probably should have been caught about 10 stages earlier and it's also gotten oos attention too now with the uh the oos top 10 there's a new one now for cicd security risks and you we talk an awful lot about the software supply chain and about the dependencies that go into your software and not so much about how that Chang gets made this is doubly important in the more regulated Industries because we are starting to talk through what we call a maturity model so you're always going to have threats you're always

going to make mistakes in your code you are always going to have some sort of problem somewhere even if you don't today there's nothing saying that you uh you won't have a a new vulnerability discovered tomorrow so a maturity model is really how do you deal with that and consciously writing down your risks and what your process is is a key part of that so I have two things that we're going to keep coming back through throughout this presentation the first is these four questions um I told you earlier you're not going to walk out of here with a comprehensive or complete threat model for your super special bespoke pipeline um but hopefully you'll have a really

good starting point to build it and I'll focus on highlighting things that I see folks Overlook uh my own struggles and having done this in a highly regulated environment and honestly highlighting some of the easier wins so the high impact low effort spot of of security that's just why are you not doing this um and I'll answer the first three I will never answer that top four as the great stage of love in life meatloaf has said he'd do anything for love but he won't do that and he never tells you what that is I can't tell you if you've done a good enough job um only you can you're good enough for a web app that's

handling some some basic credit card processing is going to be different than the software controlling the nuclear codes is going to be different than the MRI handling your your medical images like what is good enough is is really up to you and the next thing we're going to talk about is the super duper simple pipeline uh these can get very very complicated very quickly but I kind of wanted to tell you where in the sdlc that we are so we're going to talk a little bit about what happens here mostly in this build-in deploy step and here's 's a couple more things on that pipeline we're going to we're going to gloss over just for the sake of

time do include these topics in your in your threat model uh they are very very important but they're also kind of customed to you so I'm going to assume you know what git and GitHub is and some of the basics things like managing a poll request uh branching forking permissions how to check for merging Etc and if you don't see me later happy to happy to talk through through the only appropriate merge which which is a squash merge um um last uh next endpoint management so that that whole far left side where where the developer's laptop is uh that is its own can of worms uh patch your stuff but identity and access management is kind of its own thing so we're going

to assume that you have confidentiality integrity and access controls all kind of squared away and then at the very very right side of the pipeline how your how your environment is set up so like don't you know chamad 777 all the things don't put your passwords in an inii file for database um you all the the basic things and most of this is not specific to GitHub and what I mean by this is GitHub is super duper open about which parts of the platform you do and don't use so if you don't want to use a particular part of the GitHub platform like we that that's cool um here's the API if you want to integrate something

else with it um but don't have time to Choose Your Own Adventure so we're going to focus on GitHub actions uh github's built-in Secret store called GitHub Secrets um GitHub environments which is how you would SE segregate like development testing production Etc and GitHub packages but it's the same general idea when you swap in other tools too so you should be able to kind of copy paste and and mix and match as much as you need to um but I do want to call out like very very upfront I have a bias uh a lot of them actually but let's call out some of the relevant ones first I work at GitHub um I work

with the super D duper regulated crowd that I came from um next is having been experienced in that super duper regulatory environment before I I moved to GitHub uh user exper user friction is the leading cause of insecure things and what I mean by that is nothing is quite as dangerous as as a clever engineer working around said friction so if you block access to GitHub well guess what someone's going to write a script on some ec2 or or droplet or something to clone something from GitHub move it to a a shared location and then it's going to come into your environment um corollary to that is uh I have never ever done anything not once

have I disabled TLS verification to work around a corporate proxy nor have I ever run SSH over a TCP 443 or Abus web blob storage to do exactly what I said um so highlighting that that is my my bias of of experience there and then lastly is you know technology is constantly trying to make me Obsolete and what I mean by that is when I got started in it you know I talked about Hudson uh I also worked with Spam assassin and fail to ban and postfix and send mail and I don't really see those on like jobl listings anymore um not too many people like know how to set up postfix and send mail and all that good stuff anymore um

and I really believe that building and managing your own CI from scratch will go this way too so here's where we're going we're going to talk a little bit about what a GitHub action is we're going to dive deep into what goes into a GitHub action we're going to talk about secret management um because that is kind of like my second big pet peeve is uh putting secrets in your pipeline putting secrets in your code putting secrets in places that they can be read persistently All That Jazz we're going to talk about what happens when you do it yourself um so the your threat model is going to change drastically based on whether you're using github's hosted

compute versus hosting your own compute you can bring your own compute to GitHub actions we we don't care that's totally fine we support it 10,000% but it does change your model and then we're going to talk a little bit about governing the ungovernable or or how to manage this at an Enterprise scale all right so I I promise this is the last time I mention Hudson I I see someone in the back kind of twitch every time I say something but it was really really cool because you could click buttons and create a reusable reproducible is nightly build absolutely positively amazingly cool um and then the next kind of like wave of CI the Travis the circle the gitlab the

Azure devops are all fabulous and unique in their own ways the fundamental thing here is your pipeline now has become code so you can Version Control your pipeline and that is absolutely really really cool um but it's still a little bit of a problem for templating and efficiency because you have the same thing let's say check out a repo written by each team doing its own thing or each company doing that and it gets a little wasteful um it gets wasteful of compute time because you're checking things out you know every time but it also gets a little wasteful as far is now you have a hundred little more bits of scripts to keep up to date

and there's got to be a better way so what if our CI pipeline was a package ecosystem so let's take all of the amazing time and effort Savings of a package ecosystem and make CI awesome so instead of writing all your own JavaScript you can use npm and instead of writing all your own python you can use pipa same general idea at a high level so really cool idea because it can do all sorts of you know scripty scripty things and it also provides a place for for vendors and integrators and all that good stuff to to structure user interaction with their products so you know as a vendor you would know where all of the uh the dragons are in your

pro in your product how people want to use it and don't want to use it and you can say hey we're going to deploy to place we're going to scan code with settings with tool we're going to build code deploy code all that good stuff and it can provide a paved path for internal things tool so you can use GitHub actions to template internally um we use it internally ourself see kind of not sort of secret we uh it GitHub actions runs most of GitHub um we we don't like to do things manually and you can get those reusable module workflows from the open source Marketplace which is where we're going to kind of focus most of that

time your internal organization or you can Define them yourself in your own repo and again you can run on your manage compute or bring your own so this yellow box is where we're going to spend most of our time but we can kind of flex into these others as needed all right so what's a GitHub action there's one big file that kind of tells you what a GitHub action is and it's in a repo uh either the the repo of the the package that you're using or if you're defining it yourself it'll be in your own repo it's called action. yaml and it defines inputs outputs and then what what's what's doing the thing and under the hood ideally

completely invisible to your users it's one of three things it's some JavaScript it's a Docker container or it's a a mix so composite being mixes of all the things inline in repo remote script call other actions uh composits are kind of their own fun little thing all three of these are great and for whatever it is you're trying to do you usually have two good options um and one that's kind of like you could probably do it but it's really they all have their own use cases and tons of space for user preference so let's talk about JavaScript uh full disclosure I'm still a little bit scared of JavaScript I am the most backend person to ever backend

I will do go I will do python I will dive deep into all sorts of other things love shell scripting JavaScript scares me um so couple caveats about this pure JavaScript only uh node 16 uh we used to say node 16 and node 12 node 12 is is deprecated or being deprecated imminently um and it's been nagging at you for a good couple of months now to say hey you're using node 12 please stop this so let's talk a little bit about what really happens when a job is run so the compute is going to download the GitHub action so it's going to check out the repo at the shaw or or tag that you

tell it to um you can also do a branch but please don't do that unless it's something that you're developing yourself um it's going to run with a bundled version of nodejs so the runner agent has a bundled version of node and then all of the dependencies have to be in that repo and let's talk a little bit and dive right into the deep end of dependency resolution we'll do script injection like right at the end uh that deep pit of dependency resolution in JavaScript at least in GitHub actions it's really pretty shallow because for portability all of your dependencies in a a JavaScript action are vendored so uh that means you have to check in your package Json package lock

uh all of your node modules all that gets checked

in um another kind of side note here is that it means you can't really hide dependencies or require anything on a self-hosted environment so you can't require and call like any additional binaries or anything like that it like pure you do the JavaScript um and the other kind of nice thing about that is like a software composition analysis tool can see all the things so you have you're checking in your manifest and your static analysis tools can also see all the things so you're not hiding code from anything like it is a a reusable box uh I like to talk about containers I talk about containers a lot uh specifically though with GitHub actions we assume that you are using

Docker not podman not any of any of the other Ravensburger puzzle that is the cncf land ape um and really the the big thing here is that sometimes a container action can have multiple containers and we're going to assume that those containers can talk to each other or your developer will be assuming that you know if this action needs to to have three containers and all three need to talk like that'll be an assumption um I do see some people try to use like podman Docker and some of the other aliasing packages and sometimes it works sometimes it doesn't just at the end of the day kind of assume Docker and you can do one of two things

with containers you can build and run a container so your action yaml would have a a Docker file checked into that repo as well and on each and every run that container is built from that Docker file or you can say hey we're going to just run this container uh you need to be able to pull it and these are the inputs and these are the outputs and you can get the uh the a beautiful uh thing about containers which is you can write in any language you'd like you can do all the all the fun things um and it's really just an abstraction above the provided compute which is the whole promise of containers uh however image Providence

container hygiene all that good stuff is on you so let's talk about uh you know downloading from a sketchy URL uh it's normally not this obvious it's normally like a typo squatting uh image on Docker Hub or something like that but please please please read your Docker files um I'm going to highlight this little bit right here curl Tac k um specifically the T K part here which disables TLS verification um I see this way way way more than I ever want to see this I never want to understand that curl Tac K is is disabling TLS because it is 2023 um we are civilized human beings and we do not disable TLS encryption however

that's that's not the world we live in and we're going to pipe it straight to bash yes we're gonna Yello that um and then lastly we're going to run it as roote um so those are those are like my top three things that I I look at when I when I look at container files um but the tldr here is that uh your basic container hygiene is still important um and then a little bit about the container execution environment um your compute design for containers becomes way way more important here um because in general where we are with with the self-hosted like GitHub you're using github's hosted compute is H you if you escape from a a

container you're escaping into an ephemeral VM without a ton of extra goodies um so it's not really a big deal however I have absolutely walked into this bottom situation where uh somebody's using a really old version of Docker and they're running all their containers as root and it's a shared persistent environment and what I mean by that is we're providing compute say at a companywide level and projects a b c and d are uh are all sharing that compute but everyone knows that there's that one project that's kind of full of people who don't care um or people who just copy paste things from stack Overflow and not really read into what it is um and when

you have a persistent environment uh that means a compromise can be persistent and the runner agent does a lot to uh to to prevent this so I'm going to go ahead and do my first tiny little break and show and then ha way over here so here's when you actually build and run a container image so what actually gets run is this big Docker run statement so up here on line two we did the thing where we built the docker image so here's a Docker file so we're saying hey we're going to build this Docker file everything here is contained in this repo and we'll dive a little bit more into it here as well um so here's

the name here's the label uh here's a working directory and then we're going to throw in a bunch of environment variables in we're going to bind mount a whole bunch of directories but we're not going to just Docker run-- privileged notice Dash Das privileged is not here all right so let me move my mouse back over here and play for current slide and then lastly I want to talk a little bit about composits because this is kind of of like a weird catchall and it's what you really think about when you think about what your legacy CI stuff is as well so a composite is an action that is tons and tons of various parts and pieces so you

can chain actions together so you can have you can call one action that can then call three or four other things um so say you can have a JavaScript action leading to an inline script leading to a container all is a single reusable template um and you can o run scripts in line or in the repository and you can say hey we checked out this file run this file so lots and lots of slides here let's actually get to go into stuff all righty I'm gonna move my mouse all the way over here and try to look at the thing I can't so the first thing I wanted to like kind of walk through here is

actions checkout um I spend a little bit of time talking about how much I love this action uh end to end because I supported a bunch of different projects and each of them had their own thing that they wanted to do with the simplest task which is hey check out this repo or check out these three repos or check out all the subm modules for this repo too yes pull lfs don't pull lfs so this is a JavaScript action and let's go down here here's that action. yo file and it's a pretty simple uh yaml file so we're going to give it a name we're going to give it a description because we like you we're

going to make that description actually somewhat accurate here's our inputs with input descriptions and all the outputs as well and we can say hey it's using it runs using node 16 with this uh index.js here so can everyone see this screen I know this is kind of kind of a bigger room good awesome thank you especially you in the back because you guys are way far away from this all right so let's scroll down a tiny bit because here's what this looks like as far as usage so rather than having 30 different teams with 30 different scripts that say Hey run get clone blah CD here run get clone blah with all of the Arcane git

command line arguments that you need um it's yaml yeah it's just yaml and the fact that it's just yaml is really really nice because it means that I don't have to think too hard about like hey what do I what is that command to check out a specific ref uh what is the command to set up SSH on on Jenkins um I might have some trauma about setting up SSH keys on jenin um and all of these uh all of the the lovely different settings that you can have and then of course we have a ton more documentation here as well and then what this looks like at runtime is this so let's go through a job kind of based

on containers here real fast so on the setup job step we're actually going to print out a whole bunch of stuff to the console so you're you can see this is using github's hosted compute because I'm lazy and don't want to run a three-hour build job on my my desktop so I can see the operating system I can see the runner image I can see tons and tons more stuff I'm gonna talk about this a little bit more but GitHub has a built-in jit token and this is all the permissions that were given to that token when it was run and then we're going to download all the required actions so we're downloading actions checkout V3 and it's

actually going to print so V3 is the tag that I specified um tags in git are mutable which means that you can uh you can tag something as V3 you can commit change it and then move that tag to be the the newer commit however Shaws are not so it because I said I'm going to use a tag it's still going to print that sha out for me for what the code was that actually ran and then here's running that actions checkout so I with the repo uh it's not going to print the token because that would be a little little too obvious and it'll print some of the other kind of defaults here so strict

SSH we're going to persist the credentials we're going to do this as a clean checkout fetch depth of one so we're only going to get the latest version we're not going to get the whole history and then it's also going to print the Git Version in uh info so it'll print the the version history for you as well in the logs and the whole thing takes about one second so easy peasy and I more importantly the number of times that I have had to mess around with with trying to remember get credentials is nil all right so let's now go to where's my cursor there it is whoa uh let's talk about build it and run it so behind the

scenes here is that GitHub action that was in that repo so this is the the thing that got run so that action yo file is a lot smaller because this this GitHub action does a lot less using Docker file so it is on you to look at that Docker file all right couple problems here with this Docker file um deliberate on my part but um if you were trying to do this more ephemerally uh where is it pulling this just just you know as you're trying to review things where are you pulling the stalker file from um in this case uh because this is github's hosted compute it it assumes dockerhub uh so it assumes a reasonably

safe default uh but uh yeah just keep that in mind uh there are no pinned dependencies here uh so I just said update all clean all this is not an ident build in any way shape or form and then we're going to run an entrypoint script so very very easy to automate um little less easy to audit however here's that uh here's that entry point script and not much is going on here so we're running a uh RPM build however that was on me to figure out um and this file actually needed to be looked at and then here is the build step of that so run it build it uh pass in a whole bunch of stuff and

then there's thousands of lines of output for building an RPM that I'm not going to go through in any great depth and then the other option here is pulling and running a container so when you use GitHub actions to uh build your GitHub pages so there's kind of two models for for Pages sites now you can do the traditional build off a branch or you can define a GitHub action and when you use the GitHub action at actually is just using a Docker file and you would know that by looking at the action. yo file so you have inputs you have runs using Docker file and here we're telling you what uh what Docker image and tag is

getting used so it is now on you to to audit you know as as say a user requests this this action to use of hey it says we're running this uh this publicly available Docker image what's actually going into that image um you know for GitHub we're doing everything kind of in the open so it's it's pretty straightforward task but there are 19,000 uh items in the marketplace right now uh for for GitHub actions and not all of them are quite as as open and transparent and we'll get a we'll touch a little bit more on that in a bit we just went through that page and then here's using that in a pipeline so all of that so we're going

to check out the repo we're going to configure Pages we're going to build it and we're going to upload the artifact and then in practice what this does is on setup job it's going to download all the actions and then it's going to pull the images so it'll pull down the the container image that you need to run and then it'll do the thing where it actually builds it and does all the other stuff and then lastly composits and composits are kind of where things get a little tricky because you can have so much stuff in here but here is an example of a of a composite action so you have some inputs here and

runs using composite so as steps we're going to go away code navigation you're taking up valuable screen real estate at 200% zoom we're going to check out the repo we're going to run an inline bash script and we're going to pretend we all know what all the said Magics is going on here and then we're going to at the very end upload some of the files that that shell script created so it is on you and where I would really spend a lot of time on when a user requested an action was to to look at this action yaml file and you know like copy paste this script into something to to analyze it um because

normally what I would do is look at that and go oh actions checkout yeah whatever okay uh oh actions upload artifact okay whatever like that's not that's not where I would spend a lot of time but uh wunning an inline shell script that uh that does make me a little nervous sometimes and here's what that looks like um did I'm export Auto build click sorry the Wi-Fi is a little sluggish for me so on setup job we're going to to download the images that we need so there's we're we're downloading the action we're downloading checkout we're downloading upload artifact so it will recurse those dependencies down and then here's where we're actually going to run it so we're going

to run checkout we're going to set up off we're going to do all the things and then here's where we're actually going to upload that here oh that was not part of this yeah that's not part of this group okay awesome back to PowerPoint for a tiny bit sorry guys I do have a little bit more on the slides because I really really really want to talk about script injection I know it's everyone's favorite topic um because every system that has stuff in process going on stuff out uh needs to handle the stuff that's inbound safely so as you're looking through you know a user requested or user written GitHub action here's where I would like to think about

the data flow um like are the inputs like could they be unrusted uh specific Al GitHub can trigger on lots of things like issue titles or bodies you can pull in a lot of extra information um and do those get interpreted or executed directly or are you saying hey variable is going to be equal to this thing and then we're going to sanitize that variable um my other kind of like makes the hair on my the back of my neck stand up is like what are we opening Network calls of any sort are we downloading stuff are we uploading stuff uh Reading Writing to the disk in general guidelines on on this is like Hey we're

going to set this policy of we're going to prefer to use a pre-written action when we can so we're not going to write 30 different scripts to check out git we're going to use a pre-written thing um scan with some static analysis tooling if you can um when you can't you know store as an input as an intermediate variable uh sanitize it if you can and at the bottom is a whole security hardening guide for GitHub actions so if you want to spend another couple of hours on this uh and I highly recommend it uh do really dive deep into that link and here is an example of a script injection so in

Jo so down here so here's a a quick script where we're going to get a PR title and then we're going to just say we're just going to evaluate it with bash so if the title is this then we're going to do this thing otherwise we'll do this thing and on eval so this line right here we're going to uh to look at that title and then uh we're going to execute this so if this attacker creates a pull request that's escaping out of that and then running a shell command just to list the the files there like that just gets executed um and the good thing is here you with a a pull request type workflow

is is you can say hey we're not going to run GitHub actions and we won't by default on pull requests from a uh from a first-time contributor so normally it's not your you know your contributor that's you know 10th time 100th time on a public open source repo that's going to do something silly although you can require approvals on every pull request run so at least in this case this one's pretty straightforward to spot uh let's go back to PowerPoint real fast play from here uh all right no really play from current side thank you um and I just wanted to like put a little bit here on on trigger warnings um because GitHub action can trigger on lots of things

that are not pushes so not every CI system can can do that so if you like open an issue an action can run if you change a Wiki The an action can run uh deployment events discussion events release events uh anything you know a lot of things that aren't strictly pushing code can trigger automation uh reconsider running uh requests on a fork from a public repo and then not so much of a compromise situation but uh in my initial roll out of GitHub actions at scale um I accidentally got the entire company's IP address blocked from dockerhub so uh and that's a good like sixh hour lockout um so you know try to think through concurrency uh GitHub action

supports matrices that'll end up in up to 256 parallel runs so that's 16 by 16 um and that's a great way to tie up all your Runners uh have very very very peaky loads uh run way way more jobs than needed and if you you're using github's hosted compute that's kind of our problem um but if you're hosting it to yourself this is very very very much your problem um but all is not lost um the all of these problems that I've been talking about are true of all package ecosystems it's not a GitHub actions problem it's really thinking about your CI more as npm or pipa or nit Etc um what I would like to to Really

drive home is that get tags in and of themselves are mutable uh so pin to a sha so instead of saying at V3 you can do at sha um and for lower trust environments like that's really really important uh we're going to talk about a Company allow list more in a little bit require review with code owners and make it internal if you you want uh conditions apply there and basically everything in or sorry everything in the marketplace for for GitHub actions is a public repo um in order to publish your action in The Marketplace it has to be a publicly accessible act repository in GitHub and there's tons and tons and tons of tools

that are free for open source for security uh highlighting kind of R3 there for dependabot for for software composition analysis code ql for static analysis and secret scanning but but there's also a lot of other controls that you can put in place here I'm going to kind of keep moving a little bit because we're a little tight on time uh so let's talk about secret scanning or secrets you know this is secret information so you should look look at this Secrets uh it's a problem everywhere uh not going to lie uh as a developer on my laptop there is a text file full of uh personal access tokens and other such shenanigan re um and they

all have expiration dates of like 3 days but you know like it's there I copy paste I'm developing something uh but I can't commit it into GitHub um so we do we do have uh we block a lot of our known secrets on public repos you have lots and lots of options there on on blocking secrets from getting in your repo uh but really we want to talk about it you can store it on a persistent machine you can deploy it and log credentials in yeah um when I first got started in it I found jdbc URLs with password words written and plain text in a world readable directory um in production on user facing systems it happens um everyone

has a story on this but let's go a little bit deeper uh if there is nothing else you take from this talk please uh remove all credentials from all your repos and CI processes and rotate them GitHub has a built-in Secret store it is built on lib sodium so when I was calling a a GitHub token um I can also store a credential from something else and you can store and control this at the repo or or environment level um so it's a little bit of a different uh password management than a lot of your other CI uh systems a big important caveat here is for binary Secrets like your SSL certificates Etc or x509 SS or

structured data like yaml uh you need a base 64 to have a text string and then you'll decode it on the other side um and I have an example of that in just a sec but you can also bring your own uh just wanted to highlight the top four that I see all the time so Azure AWS Google cloud and hash Corp Vault so they all have official actions um and you can use those instead of trying to to write your own API calls here uh that built-in just in time off token is configurable it has really really granular permissions and it only lives on one token but let's make it really really hard to some do dumb

things because here's a questionable maneuver here um I'm echoing a secret B piping it to B 64 and writing it to dis not the best idea honestly um there's a little bit of a better way to do this here and here it is um in this case I'm deploying to Azure Azure has a pre-written action I can Define all my things and there's a lot fewer things that can go wrong in that but really doing questionable things with with credentials is a great place to look first second last uh but don't do that um and I'm GNA talk a tiny tiny bit about GitHub actions infrastructure so infrastructure is cool again uh we have hosted compute that's what I've been

kind of showing this whole time um but you can also bring your own uh completely free for you to bring your own compute uh but you of course are paying for your data center AWS Azure whatever um but all the security and operations are on you when you use our hosted compute you're getting ephemeral VMS like full stop you're getting a VM it's brand new it is clean at the end of your run pass fail whatever uh it is murdered heartlessly Rewritten um and there's tons and tons of dependencies pre-loaded on there so it weighs in at about 50 gigs uh lots and lots of stuff on there there's the re uh repo that defines all of them um and everything is

kind of magic it updates once a week or so scales caches uh but I say magic it's not running secure available and scalable compute is really really hard um as I learned trying to do this so here's a couple uh a very short list of decisions to make when you're self-hosting that will affect this this model so what type of compute that you're running on physical Hardware VMS or containers uh ephemerality your base operating system uh what software are you going to include who all can use it uh please whatever you do um log it send your logs to a seam uh update your stuff all that good stuff um and really really really think

through the blast radius of a compromised Runners um so ephemerality helps a lot has persistent compromise just is a lot more difficult um and VMS in physical compute can isolate themselves better than containers because containers are fundamentally just a Linux process with some special guard rails in the kernel um and then it's a a little bit harder within a self-hosted runner mostly because when you're self-hosting you're doing this because you have a trusted boundary and that Runner is inside that trusted boundary and then I am like four minutes left so I really wanted to talk about this so Charles deal famously said about France how can you govern a country which has 246 varieties of

cheese and uh I tried this at the Enterprise scale how do you govern a company with 19,000 pipeline steps available and that's you know currently roughly what's in the marketplace right now and the first thing you do is at the Enterprise organization repo level uh you can allow policies here so I don't do this for my personal projects because they're personal projects but uh what I do and don't use at work is definitely controlled in some ways but really the the biggest thing that I wanted to to drive here is that maturity doesn't have to be slow um you normally when you say hey you need to open a ticket to request something you're like yeah whatever I'll

get it done in the next like three months um but a maturity model really can shine here because you don't have to allow Marketplace actions at all you can have AOW list it's not a everything or nothing situation but defining that middle is where your threat model is fundamentally going to to Really shine and what I mean by this is here's a here's a reasonable process to do this so a user wants to say hey we have a GitHub action they're going to open a ticket um and then in this case there are three approvals so legal might want to take a look at say hey no uh no you know really copy left aggressive

licensing no do no evil licensing whatever whatever legal wants to do I we're in a law school building I genuinely do not know what those people do but I know it is fantastically important um appc infosec like is this safe are do we need to Fork it and scan it uh do we need to run some sort of code review like what you're what are you going to look for like please do look at the ingredients and then on the infrastructure side like we run this um if you are asking for a container action you need a runner with a container engine available on it to run a container but really what's what's super critical here is having that SLA of less

than a week because this is how you prevent people from uh forking a thing bringing it internal um but it's still doing something unsafe so really what we want to do is understand at every point in the pipeline like how the system can be circumvented by people and designing people processes that work here um because here are some really great traps to watch out for uh you brought it internal then you never updated it uh corporate proxies and TLS encryption I will die on this hill every single time uh every time you install a break and inspect a certificate if anyone notices it the first thing that's going to happen is someone's going to Google hey

I'm getting an SSL Error and the top answer on stack Overflow is disable SSL verification uh every single time uh but disabling SSL verification does just work uh maintenance maintenance of your systems uh internal and external like it's just maintenance isn't sexy maintenance doesn't get anyone promoted maintenance it is what it is um and then really what this picture is is all about alarm and ticket fatigue uh managing user requests can can kind of suck sometimes um scanning CI infrastructure is really really noisy especially when you have 10 different projects trying to share the same compute that all need 10 different things um I'm thrilled developer experience is getting so much time and attention but uh if every company is a

software company Talent is expensive and that really should extend to our operations in our security folks um having 10,000 alerts sucks and having 10,000 alerts on VMS or requesting Q sucks so let's also do developer experience but also maybe our security and infosec experience as well and here are some resources to learn more uh specifically I wanted to call out um GitHub Community which is how we We Gather feedback from the community uh we do have a public road map um so definitely want to call that out we have uh all of the security guys go if you want to spend hours and hours and hours more uh there's where we can and then lastly we have a bug Bounty program so

if you have found a thing please uh please visit that link all righty [Applause]

thanks we have time for maybe questions of course yeah totally [Applause] we have time for all the questions I well maybe not because we do have a another speaker but yeah does GitHub have or can can anybody point us to a any sort of tooling that has uh or that'll look at your actions look at uh your your pipeline in your repo and say hey you're doing stupid things here here and here uh yes through Professional Services um that a so we do have a Professional Services arm like a consultancy type thing um and they they can help you develop best practices and all that good stuff you know specific to your business

so so uh you you mentioned um that the the authentication token that jet token is configured both what it can do is there an interplay between like the user that triggered the actions and their permissions and how that configurable like how does that off kind of work uh H okay so this gets a little sticky so let's say I Natalie um have right access to a repo and I'm opening a poll request the fact that that poll request was opened by Natalie is handled by GitHub like not actions um the actual jit token is only scoped to to the the the thing that's defined in the workflow um so by default I believe it's read only um or read and write I'm not

sure but there's actually like 20 different um you know individual scope so you can say hey we can read to this but we can write to security events um and really what what I what I strongly suggest is at the org or Enterprise level that you set that to to read by default only and then in your pipeline have a discussion about uh saying hey we need the ability to write to security events so we need to ask for this token for this workflow and because your your workflow is code that discussion can happen on a pole request or at least it can be gated so that that's where we were kind of like out of scope a little bit on

that left-hand box of like your GitHub settings versus is like the actions compute cool okay let's thank Natalie again yeah [Applause] absolutely oh thank

you iing mic but