Tag, You're Leaked: Surviving the tj-actions Supply Chain Attack

[music]

Hello everyone. It's great to be here. Before we get started, uh I'm curious how many of you use GitHub actions or other CI/CD platforms? Okay, pretty awesome. So, let's get started with the presentation. It was a Friday afternoon on the 14th of March when my step security co-founder Vun and I we were wrapping up our last meeting for the week and suddenly we got a Slack notification from our automated detection system. We immediately started investigating and soon came to the conclusion that the TJ actions change files GitHub action had been compromised. We could hardly believe what we saw. We quickly created this GitHub issue informing the maintainers and everyone else that this GitHub action had been

compromised. Next day a high severity CVE was issued again confirming that this GitHub action had indeed been compromised. A few days later, CISA or the cyber security and infrastructure security agency issued uh a public cyber security advisory urging organizations to remediate this issue. This was no longer just an open-source security issue but something of national security importance. And this is because TJ actions was not any component but something that was being used by more than 23,000 repositories at that time including repositories from prominent organizations such as Meta, Microsoft, GitHub, Hugging Face and so on. Here is the agenda for our talk. Mark and I are here because we responded to this incident together. Step security

detected this incident through baseline event monitoring of CI/CD runners. Chain guard was one of the first responders and played a crucial role in coordinating the community response by um helping affected organizations recover from from the incident. My name is Ashish Kurmi and I'm a co-founder and CTO of Step Security, a cyber security startup focused on CI/CD and software supply chain security. Before founding Step Security, I spent more than a decade securing infrastructure at Plaid, Uber, and Microsoft. >> Hi, I'm Mark. Um, I'm a senior product uh security engineer at Changuard. Uh, where I focus on securing and hardening our supply chain. Uh, before Changguard, uh, I ran networks in academic spaces and later worked on iuntu security

Psert. Uh, and this really shaped uh, how I think about vulnerability triage and disclosure. Uh I've also uh co-authored the OpenSS compiler options hardening guide for C and C++ which is a community uh effort to help C and C++ projects adopt safer build and flag uh defaults. So it's Friday afternoon in Seattle March 5th 14th and most people are winding down for the weekend uh when my colleague uh Evan Gibler spots a LinkedIn post from step security uh something about a compromised GitHub action. Moments later, step security reaches out to us directly. What happened? Over 23,000 repositories are using TJ actions changed files uh in their GitHub workflows, including our own. Let's just say are we compromised

is not a great way to start the weekend. As soon as we learn about the compromise, we search our repositories for any workflows using the affected uh action with mutable tags. A mutable tag is a tag name like latest v42 or main that could later be moved to point to a different commit. Fortunately, every instance of TJ action change files in our codebase was pinned to a fulllength commit shot, not a mutable tag. Following our internal best practices, we always pin third party actions and we were saved from leaking credentials when the upstream was initially compromised. We were fortunate that dependabot uh had just uh updated our actions a few hours ago uh before the compromise occurred.

Still to be cautious, we disabled Dependabot um to prevent it from bumping DJ actions again or automatically updating to uh other potentially compromised actions. Internally, we found no evidence of uh compromise, but we knew that the upstream packages we build from might have been impacted. To stay safe, we temporarily disabled automated package builds for our distro until we could verify that our upstream supply chain was safe. So this slide shows a sample GitHub actions workflow. This GitHub action workflow job performs Terraform deployment anytime um there's a change either in the infrastructure or uh Terapform directories. This workflow has access to an AWS access key to perform the deployment operation and it uses four uh open-source actions by referencing them

by their release tags such as V4, V44 and so on. If you were able to take that workflow and run it again and again and monitor the outbound network calls being made by that workflow on CI/CD uh on the CICD runner, you would see something like this. This is a network baseline that has been created by running that workflow for more than 2,000 times. Now, this is a screenshot from step security harden runner, the product that detected this incident. But the focus of this talk is on the detection technique and not on the tool itself. And in the later part of the presentation, I'll also talk about how you can build a baseline driven monitoring system yourself using

a few opensource tools. Now what happened on the 14th of March was that there was a new call to gist. GitHub user content.com from the pipeline jobs that had never made that call before and this is what uh that and and this is how the detection was triggered. You could see that this call is coming from the TJ actions change files action step. When we looked at the process events, we could see that this was a curl call to download memdum. py. Next, we looked at the release tags for the action and we realized that just 3 hours prior, the latest release tag uh for this action had been updated to point to a malicious commit.

In fact, not just not just the latest tag. All the existing release tags were updated to point to the same malicious commit. When I opened this commit in GitHub, I noticed this message in the yellow text box. This commit does not belong to any branch on this repository and may belong to a fork outside of the repository. How is that possible? Enter impostor commit. Imposer commits are commits that do not exist in the original repository. Instead, they exist in a fork of the repository, but because of the way GitHub APIs work, these commits are accessible from the original repository. So, you know, in a way, it's like a ghost commit. It's visible but not not

really there. Now, let's look at the uh TJ actions impostor commit and understand what it was doing. Look at this B 64 uh encoded string. It's being decoded and executed as a script. Now, let's B 64 decode the string. What you're seeing here is an attack that only works on Linux runners. It downloads a file called mem dump. py from a very well-known uh public GitHub gist. Several security researchers have cited this public GitHub gist in their security research prior to the supply chain attack. Now let's look at the content of memdom. py. It goes through the list of running processes on the host and looks for a process named runner.worker. Once it finds that process, it opens it

and this is where it dumps it its entire memory. So this script is essentially hunting for the runner.worker process. Now why this process? Um when GitHub detects that a workflow needs secrets, it makes them available to runner.worker and mmdum. py is essentially dumping its entire memory. Now let's go back uh to the impostor commit. Uh look at the string format. This is exactly how runner.worker stores CI/CD secrets in memory. So the impostor commit is searching through the memory dump to find CI/CD secrets. And here is another clever technique the attackers used. See this double B 64 encoding? Uh that's not a mistake. Now why double B 64 encoding? It turns out GitHub automatically masks B 64

encoded CI/CD secrets in build locks. So single B 64 encoding uh exfiltrated secrets are masked. However, double B 64 encoding, GitHub no longer treats them as secrets and makes these exfiltrated secrets available in build logs as is. And finally, the output is printed to SD out which makes them available to GitHub action workflow logs. So now let us show you how this looks like in action. We have created an end toend demo of the attack. And for this demo, we have created a compromised clone of the change files action under a new rep under a new organization called TJ actions clone. What you are about to see uh is um is an actual credential theft happening in

real time. And please uh pay close attention to the build logs. When the workflow is executed, we see the encoded secrets and build logs. Now let's double B 64 decode this and you have every secret from the workflow AWS credential GitHub token everything. Now this is the network baseline after the last run and you would notice that the baseline is now unstable because the the platform observed a new endpoint to just.github usercontent.com uh that was seen for the first time in the last run. So, let's review how the attackers were clever and tried to hide what they were doing. The TJ actions impostor commit downloaded the exploit code from a github owned domain. They use

gist.githubcontent.com. This domain is a high domain has a high domain reputation rating. Almost all endpoint detection and response or edr agents and other runtime security solutions already trust this domain. On the surface, the commit history looked normal because they were using apostro commits. So commit activity in these repositories, even if during the incident, looked normal. And if you go through the branches on these repositories and look through the commit history, you will not find any of the malicious code due to the use of gist. The attacker used several GitHub users to compromise the repositories. In some cases, they tried to impersonate legitimate users. In the case of TJ actions uh imposter commits, they attempted to use the renovate bot um and

impersonate them. Many security tools would not flake any of this as suspicious. So by Saturday morning, we knew that our systems weren't directly affected, but the bigger questions were who was affected, is our supply chain uh impacted, and whose secrets are public on GitHub right now. uh to view uh supply chain our supply chain and the 2300 potentially affected repositories we needed automation to check for indicators of compromise or IOC's my colleague Evan Gibler uh created a tool called GHC scan at first it was simple but as we used it throughout the weekend GHC scan evolved into a more robust uh scanner it works by uh going through GitHub repositories fetching workflows either through the

GitHub web API or through the web UI when the API was unavailable and searching for these logs for IOC patterns. It includes straightforward features like setting target repositories, date ranges, and output formats written in Go. It supports concurrent scans and validates that the first round of B 64 decoding yields valid B 64 before attempting a second decode pass for the TJ actions compromise. Gcan looked for two key indicators. The malicious commit SHA 0E58ED8 and base 64 encoded data matching the exfiltration pattern. In this example, we're using Octo SDS for ephemeral credentials, but along live fat could have also been used. GHScand was designed to be extensible so that we can reuse it in the aftermath

of future uh incidents and we published it as an open-source tool um on March 18th, 4 days after the attack began. Through GHCAN, Evan was able to identify leak credentials across the GitHub ecosystem while I worked on reporting confirmed compromises to upstream uh projects throughout the weekend. We began by scanning and reporting leaks to all upstream source codes in the in the supply chain uh my DRO uses to build packages. Once that was complete, we expanded to every repository potentially impacted by the TJ actions compromise. By Sunday afternoon, here's what GH uh scan found. 465 repositories were affected with over 200 types of secrets leaked. Most of the secrets were ephemeral GitHub tokens which hopefully

expired before the leaks were discovered and potentially abused. But we also found log live credentials like PATS signing keys ads keys and cloudfare credentials. For the long live credentials we reached out to uh directly to organ organizations often through email. This entailed reporting leaks to over 60 organ organizations which included four major Linux distributions, Microsoft businesses, open source projects, and multiple governments. We also reached out to individuals with leaked keys. Several government agencies were impacted. Most leaks, like the one that affected NASA, involved ephemeral credentials, but GSA's Notify.gov was particularly concerning. Notify.gov was a shortlived GSA platform from 2023 to 2025 that allowed government programs to text the public to meet people where they're at. For

example, setting medicaid reminder renewal reminders to help families keep their coverage. Its admin and Terraform credentials were among those leaked which obviously could have caused significant damage if abused. We wanted to raise the severity of this directly to CISA urgently. Thanks to members uh in the open SSF slack uh I was connected to a CISA coordinator and after verification over signal we filed a report about leak government credentials uh Sunday evening between the TJ actions compromise and other boged happenings I can only speculate why notify.gov up with sunset. Unfortunately, as of now, no replacement service has emerged. I want to pause to emphasize that reporting vulnerabilities and compromises truly matters. Ashisha's public report enabled many others to

take action quickly after the compromise was disco was discovered. A perfect example of how transparency accelerates defense. The technical work of identifying compromises must be complemented by the social work of reporting and coordinating response to affect change. That's what uh turns isolated discoveries into real world impact. If you're interested in the human side of security communication coordination and disclosure, please find me after the talk. I'm happy to talk about coordinated disclosure. A great resource for this is Open SSF's vulnerability disclosure working group. They share practical guidance about coordination and they host a bi-weekly office hours like session where you can ask questions and seek advice. In the last section of this presentation, let's look at some of the

concrete recommendations and lessons we can learn from this incident. The first one is about um security monitoring for CI/CD runners. A lot of organizations have security agents for their desktops and laptops and similarly they use cloud EDR solutions to protect their cloud workloads but CI/CD runners or build servers typically have zero runtime security monitoring. In this talk, um, we saw how step security harden runner was able to identify this compromise. But you can also build a baseline driven monitoring system for CI/CD runners yourself using open-source EDR solutions such as Vazu, Falco, and Tetragon. These solutions provide runtime telemetry events such as network connections, process events, file events. And you can take all this information correlated with the the the

pipeline run, build a baseline, and then use this baseline for anomaly detection. In August, GitHub added a policy to block unpinned actions and enforce shot pinning in workflows. This lets orgs opt into requiring that every action reference a specific commit, preventing tampering with mutable takes. Two tools I recommend to help with pinning our Pinct and Zmore. Pinact automates making sure that uh pin that mutable tags are converted to proper pins. And Zizmore is a best practice tool. It scans CI workflows to spot unpinned actions, exposed credentials, and other risky patterns. GitHub also uh has immutable release feature for upstreams um that can lock a release tag so that they can't be changed or deleted later. This means

that downstream consumers can trust that a release tag always points to the same commit. As a rule of thumb, assume that all longive credentials will eventually be leaked. Avoid them whenever possible. GitHub actions already issues short-term short-lived tokens by default and services like Octos can extend this model further. Similarly, SixToro enables short-lived signing keys. So, even if something is leaked, the impact is minimal. So, thank you for attending our talk and and everyone's attention. [applause] Think you had your hand raised first. Do you think that these sort these actions that seem sort of simple that are used so widely that GitHub should uh think about incorporating some of this functionality into the core feature set so that these

widespread actions have less of a chance to affect as many repositories as they did. >> Yes, definitely. uh they have added some already but I'll let Ash add more to that. >> Yeah, I think u that's an interesting question because um and like like Mark mentioned we have seen GitHub sort of taking over some of these popular actions. There are a bunch of actions for setting up you know like common language environments like Go and uh and Python and I believe GitHub already owns this action but I mean regardless of how many actions GitHub ends up owning there will always be I think a big ecosystem of open-source reusable components for GitHub actions and that is one of the reasons why

GitHub actions is so popular compared to you know other CI/CD providers because it was sort of like the first CI/CD provider that built a native ative ecosystem of reusable open-source components. So for example, you know, um uh based on what I read last, it has more than 25,000 reusable components and you know, many of these GitHub actions are are highly popular. So regardless of what GitHub GitHub is doing, I think there will always be an ecosystem of these open-source actions. To your point, I I believe yeah, GitHub as a platform provider can definitely add more features to make them secure by default. and GitHub is already you know taking steps in that direction. >> So the talk uh kind of two particularly

I think you glossed over a little bit how the um threat actor had pretended to be renovated bot. I was wondering if you could speak a little bit more to that or how they got and then also is there any information known about the threat actor? >> So let me see we have a slide for that we could that we could not cover. So you know this was the first example of a chained CI/CD supply chain attack. So even though the attack was actually detected in the TJ actions change files repository, the attack actually originated in the spot bugs uh sonar find bugs repository. And this sort of you know shows the chain that the

attackers followed to eventually compromise the TJ actions change files u action and it basically started with a pawn pawn request vulnerability in um in the spot bugs repository. The attackers exploited that and through that they were able to steal a maintainer pad. Using that pad they actually added another GitHub user to the spotbugs/spotbugs repository. From there they stole another maintenance pad uh which led to the compromise of review do/action setup action and the TJ actions you know eslint change files action had a dep dependency on the review dog action and the TJ actions that change files had a dependency on you know the eslint change files action. So this was like a chain compromise and and based on the

available evidence it took attackers I think more than 10 days to you know travel through all these different repositories and they eventually got caught at the TJ actions change files repository. To answer your second question currently there is no public evidence to tie this breach to a particular threat actor. Obviously, I mean, as you can see, they put in a lot of time and effort and they were well familiar with the, you know, internals of GitHub actions and, you know, they managed to remain in this stealth mode for a while and then eventually got caught at the change files repository. But yeah, based on public publicly available information, we don't know who was behind these supply chain attacks.

>> Thank you very much. [applause] >> Thank you. [music]