Tales from the Trenches: Reacting to the Shai-Hulud NPM Supply-Chain Attack

There we go. >> Uh, this is how you know I'm a sock and not a speaker >> because I don't know how to use him actually. Uh, thank you so much for coming. Uh, another big shout out to Tyler Castell or how do I say his last name? >> Tyler C. >> Tyler C. Uh, thank you for all your threat hunting. You're a fellow defender. Great work. And hopefully you guys are not tired of threat hunting because uh we're going to do a little bit more. Uh this is reacting to the shyude npm worm. Uh if you were looking for AI, you're in the wrong spot. >> Yay. >> Uh slides are available at that uh QR

code. We're going to go pretty fast. Uh I do want you guys specifically to do stuff. Uh if you're going to take notes, I suggest taking notes. I promise to do give you things worth taking notes about. And it is my intention to start very uh excessively for beginners and newbies and continue to ramp up until we have something uh useful for every skill level. Uh show of hands, where my newbies at? Awesome. Where my Where are my red teamers at? Anybody red team? Penetration testing, adversarial emulation. Where where my defenders at? Blue team? Yeah. Yeah. Sweet. Okay, we're going to do a little less on the red team stuff. I'm going to skip through some of the um things that we

could have done to make this even nastier. Uh and I'm going to focus on actual objectives for defenders. Um oh uh where my humans at? Just make sure everybody's alive and awake. This is something we do in stock. Uh raise your hand if you are in fact three LLM agents in a trench coat. >> No smoke. Not one. Okay. Uh yeah. No. Not one sarcastic jerk. >> We're not we're not sure. >> Yeah, we're not sure. Uh oh, I skipped a couple things real quick. Uh so one of the reasons that I did this talk is that in my experience security operations uh sometimes gets criticized for operating like a submarine uh which is to say

running silent running deep. Uh information goes into the submarine and no information whatsoever goes out. Uh there is some good reason for this obscurity because it limits corporate liability and team liability. Uh but this silence has a penalty uh when vulnerability management secops or security engineering uh or governance risk and compliance even comes to developers who are trying to hit a deadline and says hey I need you to extend your deadline to do this security work uh we're asking them to pay a cost and if they don't understand our perspective as to why we're asking for those things um there's this mistrust that develops that slows the teamwork and the communication to get a large

organization or even a small organization to execute on the mission of serving customers and keeping our customers and companies safe. Um, I've actually had senior developers at a different company explain to me that hackers are unreal bookymen invented by infosc to justify our large paychecks. Uh, that's a real discussion of having a real human. Uh, it was not a long conversation. Uh, yeah. So when we had this reaction, we had an a opportunity where everything went right. There was no additional liability to nobody looked bad. Uh the malware we got identified the vulnerability and stopped it before it detonated. Uh it made the news. Uh of course we'll get into that in a little bit. Um, and so I went to my uh security

leads and said, "Hey, uh, what I really want to do is talk about this internally and build trust and help kind of break out of the sub submarine a little bit. Uh, I'm going to ask nicely if I can make this instead of traffic light protocol red, uh, if I can make this really I wanted to make it traffic light protocol green, but I figured I'd say traffic light protocol clear and go talk to everybody and then I was going to let them talk me down to uh, traffic light protocol green." And they looked at me and they said, "No, that was a great idea. Go tell everybody." And I said, "Huh?" And they were like, "Yeah, go

talk to the lawyers." I was like, "Oh, thank God. Okay, I'm going to go tell the lawyers that this area needs more research. Uh, and so I'm going to go tell the lawyers that I'm going to go tell the hacking community that they need to make more malware and I want to put our company logo on it and the lawyers will save me from having to go do scary speaking in front of people instead of computers." And I went to my lawyers if they would like. Oh, no. You seem like you know what you're talking about. No intellectual property. None of our private code, right? Oh, that private code. Oh, no. We can make an

exception for that. Yeah, go tell everybody to write malware. That's fine. My we can handle that. Go talk to people. Oh, go talk to marketing. That's okay. Marketing is going to tell me I can't go tell. And they're like, "No, that makes you sound smart. It makes us sound like like we're a pretty good company. Maybe maybe you should go do that." So, I do actually really want to thank uh it is not normal for a corporation to greenlight true transparency around a security event like this. Um, and I really appreciate F5 not only letting me do this, but actually sponsoring this for us. Um, it's really cool. We're hiring. Uh, it's awesome to work there. Uh, if you got

what it takes, join me. Um, I didn't get through all of this without the lawyers adding a little bit though. So, there's a couple of things we need to cover. Uh, first of all, the views expressed in this presentation are solely mine. Uh, and do not necessarily represent the views of the company as a whole. Um, this presentation is for information purposes only. Uh, that is actually really cool. That means that I can say things like uh my in my personal experience whether it was in this platform or another this particular platform solved the problem that there were instability issues and I don't have to worry about who's endorsing who and who's getting paid and blah blah bl like

I can just give you the straight dope. Um I also need to explain that none of my statements should be taken as legal financial or securities advice. uh if I say XYZ company is solving a particular problem or uh F5 has been under attack by cyber attack because we all have uh you should not take that as an indicator of who's going to do better in the stock market. Uh any bets you make are yours. Um similarly I disclaim any liability for protecting your environment in the ways that I advise. I will be going in depth in defenses against supply chain attacks uh for reasons that'll be very obvious in a bit. uh it is up to you as security

professionals and defenders to interpolate my suggestions and architectures for the environment that you guys are native to and you guys are experts in. Um and finally, I'm going to talk about I'm I'm going to name drop like eight or nine products. Uh none of these are endorsements. This is intended to beformational survey suggestions. Um they could release an update tomorrow that breaks the thing. This is where we were at when this happened. Uh just a quick review in case traffic light protocol is new to you. Um I'm going to be saying things like TLP red, TLP green, TLP clear. Um this is an industry standard. Um believe it's released by DISA. Um and basically what it means is

most security operations events start out as traffic light protocol red. If something bad is happening, only the people who are involved in remediation and detection uh and recovery are read into what's going on. Uh usually this is done for two main reasons. One being legal liability. We don't want the customer or uh an employee of the company suing us because something bad happened. Um and also we don't want anybody to make the problem worse uh in an event where there's compromised credentials. Like if you were here for the talk uh last hour um that malware spilled credentials in easy to access places. If you then go and discuss this with your co-workers uh you lose the

ability to forensically show that a bad actor was the one who then leveraged those credentials and not the developer that you then talked to like the insider threat skyrockets uh and it becomes a big problem. So that's why we do TL red stuff. Um, and then TLP green is uh things that are okay to talk about within your company uh or your larger organization. Uh, TLP clear is uh clear to talk about with the public world. Uh, so let's get a real quick brief who am I, why am I talking about this? Uh, I am Michael Hedges. Uh, I go. I am a senior security product security engineer uh at F5. I don't give any

distributed cloud product. Uh I uh actually interned with pick up uh with F5. I ran the ABSCAC R&D team at uh DocYsine for a little bit. I'm a black blog researcher. I'm a defcon sock goon. Uh I'm actually the youth director at Bides Seattle. So hello from Seattle. Uh it's really fun to be at one of our sister Bites. Uh and I want to just take a moment. um you guys have a really cool community here in Kansas City where you guys are very um generationally dispersed like there are young kids here, there are uh students, there are young adults, there are old adults. Um that doesn't happen by accident and that's not a given. That's

something you guys are doing really well. Keep that up. And if I may suggest uh the youth should have a program here. We need to teach the kids about Sepha before somebody else does. Uh when I was in school, we had a week or two of internet safety. And when we were starting the youth program at Bside, we of course did some advertising in local schools. And we found that although most second graders have access to chat GPT, there is not an hour of internet safety in the curriculum anywhere, private school or public school. Maybe you guys are doing a lot better here. Um but I I kind of doubt it. So somebody should teach the kids not just about internet

safety but about hacking breaking and understanding that the rules of the application and the business decisions that the programs provided to them don't have to be made that way. Uh and that things don't always work as intended and you shouldn't trust your users. Thank you for calling my TED talk. We'll get back to the real thing. Um, now I've done this for a while too. So, uh, we're going to break down Shag Loop. First, we're going to start on what supply chain attacks in 2026 looked like, what recent history, like I'm not going to go and break down, um, like physical supply chain attacks in World War II or anything, but uh, just to make sure

we're all on the same page. Once we're all on the same page, uh, we're going to pull apart the anatomy of Shy Hiloon specifically. We're going to look at specific code examples uh, of the worm. Then I'm going to walk us through a timeline of how it went down at F5. Uh because that's the cool fun part. Uh and then chapter four, I'm going to go into action items, things that you guys should go do. Uh maybe tomorrow or maybe Monday. Uh and then hopefully I'll leave time for questions. Uh I have taken two and a half hours to give this talk before. So uh speaker guys feel free to give me the hook. Uh just so we're all

on the same page, uh Shy Halude is in fact a sandworm from Frank Herbert's Dune. Uh the idea here is that um a computer worm is a self-replicating program like a virus except that a worm is completely independent of the program that it's infected. Uh, and I would actually argue partly because I could talk shade about a black hat's programming, uh, that this is actually a misborn because the shy hallude worm cannot propagate and cannot exist without npm. It has to infect uh, the known package manager. Um, which makes it a virus and not a worm. And the person who wrote this is not as cool as they think they are. And if you have a

problem with that, come explain that to me. me my law enforcement buddies would like to hear it. Uh so the first thing I want to talk about is shy haloop is not a surprise to anybody. Uh I think any hacker should have a healthy disrespect for uh frameworks and of the in the box thinking that comes with them. Uh at the same time you still need to know your frameworks. They're a very good place to start. uh you don't need to memorize or know this particular MITER attack technique. You should know that MITER is a framework of adversarial action. It's very helpful for um making sure that you're not centering your threat hunts where you're comfortable, that you're

actually looking where adversaries might be. Uh and in this particular case, uh MITER published this back in I believe it was uh 2010. uh and when they did they called out uh npm GitHub actions and that the malicious code adversaries might use would collect runtime credentials. Uh the emphasis added is mine but like they saw this coming. Um as of uh last writing miner still has not added Shy Halude nor Glassworm to their examples of this attacks. Uh and I think that somebody should go do that. it maybe not the highest priority for Matt uh writer right now but I think we should do it. Uh so there's also members of the hacking community and commercial

industry and academics that also saw this coming. Um if I'm sounding cool coming off talking about this, you need to understand I stand on the shoulder of giants. Um the first one I want to talk about is Aie Greenholds. Um asie talked about uh GitHub actions worms in besides Las Vegas in 2023. Uh his work is very interesting. He's a very compelling speaker. Highly recommend you go check him out if you defend an environment where you build software. Uh and he also talked about ways of initial compromise that might not involve specifically attacking an npm that might not involve fishing attacks or um bypassing 2FA. Um it may have been the initial vector for shyoon specifically. Um more recently

Maxim Shrek talked at Defcon 33. Um his supply chain attack was around uh S3 buckets and it was massive. Um another very interesting story. Highly recommend. Um and then there was a security incident involving GitHub uh during uh 2023. uh we call it the pastrama hodu actions or after the initial git pub uh repository um but this event was actually never documented academically or publicly uh and that's going to become important later uh and then more recently at a little bit of self-promotion shamelessly besides Seattle uh Jen Gile talked about uh an MPM account takeovers preventing the next shy lude um the recording is now available uh She has another great check. Uh so we're going to talk a little bit

about uh yeah uh we're going to talk about JavaScript uh a lot uh in npm. Uh shy loop targeted uh JavaScript. Glass worm not glass wing sorry glass worm also targeted JavaScript. Uh and the earlier um Aussie Greenhold's GitHub action worm was also targeting npm. So it would be very easy for somebody sitting and watching this to say okay screw JavaScript especially if you know you're prone to flame wars about which language is better. Um and I really don't want you to take that away from my talk. That's not what this is about. Um this is a pattern in the industry. Um, this could just have easily been uh cargo targeting Rust uh which would be

particularly terrifying because a lot of the new uh Linux drivers are being written in Rust. Uh this could have been big targeting C. It could have been in a bunch of the video games that you really appreciate and enjoy. Uh this could have been Golang package which is being used by commercial industry a lot. Uh it's one of the things that I'm very concerned about. Um this could have been either etc. DNF apt. It could have been targeting bash script. Um this could have been the other package manager that the minor attack framework technique talks about is pip. Um very very common. Um this absolutely could have been made with Java. Uh could have been Ruby gems. Uh quick show of

hands. Which ones am I forgetting?

anyone? >> There are more. Um, so this could happen anywhere. It just happens to keep happening in JavaScript. And when you're a defender, you should look at and uh translate in your mind from JavaScript to whichever stack you're defending. Um but that also kind of begs the question why does npm keep getting picked on? Uh and I think that there's three reasons. Uh one of which is uh the node uh open- source community has a real emphasis on doerp yourself. Uh they have very small Lego rigs, lots of them uh with incredibly good support. There's a lot of code for free. That's one of the reasons why JavaScript is so uh popular. Um and which

That means that there are many many nodes in the graft dependent dependency graph which means that there is a wider broader uh ecosystem for the worm to propagate through which means bigger blast radius for the attacker. Uh there's also JavaScript is interpreted code rather than compiled. uh which means that there isn't a compile step for a static code analysis tool to plug into. It's all just a time compiled. Uh also I suspect that Valer developers um one of the ways that Valer developers thrive is by understanding languages at a deeper technical level than the developers that they're trying to beat. And so they tend to go very deep on a smaller number of languages rather than

uh go shallow and picking dozens of dozens of languages. Uh and if you understand interpreted code, there are ways you can defeat the interpreter to get remote code execution. And while Shyoon didn't do this specifically, I I suspect that that may have been a factor in the authors of Shyeloon and Glassworm selecting and node as opposed to C. Um and then the final one is uh JavaScript is the most popular computer programming language period. Like if you go to uh Stack Overflows uh I think it's twice a decade survey um where they talk about the technologies that are the most interesting. For the last 5 years, JavaScript is far and away the most soughta language like it's everywhere.

It's on Mac. It's on Windows. It's on Linux. It's on iOS. It's on Android. It's on smartwatches. It's on smart devices. It's in routers. Like it's it's everywhere. Um which again means maximum impact for minimum effort for the developer me right here. Uh so we also need to talk about uh node and electronjs the framework. Uh electronjs is also incredibly ubiquitous. Uh it's in Microsoft teams, discord slack signal WhatsApp WordPress, Twitch, VS Code. Uh, and what ElectronJS does is it's a marriage of two of the most powerful frameworks, Chromium and Node, which takes all of the programming uh skill that you need to write a React web page or a node web page uh or Node server side code uh and

marries it with all the HTML development and all of the tools that work for all of those text stacks and allows you to write native user space applications for desktop. tops and then immediately port with no additional effort to mobile or to smart devices. This is incredibly powerful for a business case which is why it's got incredibly wide adoption. But there's a big problem here and that is that because node has such a a deep dependency tree and Chromium has such a deep dependency tree and both of these two are super ubiquitous and very general. Both of these are getting new CVEes almost every day and there's a lag time where if you're one of the people

dependent on one of the projects dependent on ElectronJS then you have to wait for Chrome to patch and then you have to wait for Electron to patch and then you can start to patch. And even if they're doing these at perfect machine speeds, it is impossible for the Electron Genesis project to patch with in the time that a uh adversary can then use that same uh CDE to then craft malware. It is it is impossible to keep it safe and it's everywhere. So that's kind of the state of supply chain attacks in 2026. They're everywhere. They're coming at this point. Hopefully you guys all understand. I think that a supply chain attack is coming to a stack near you.

It's going to happen again. Uh, I wish I had my article up when I first wrote this in November because literally 6 weeks later, Shy Halude came back and then there was the Trivia compromise and then there was the Aximus compromise like and we are still going. So, this is not one of those this is a cute story kick back and hear what an awesome hacker I am. You guys have some work to do. If you defend a place, this is coming to your place. And I'll tell you what you can do about it. If you are not defending a place, then this area needs more research because right now the black hats are getting there first. And

it is possible, we should all hope, we have homefield advantage. We have a brilliant community full of talented, hardworking people. We could develop white hat malware that explores these vulnerabilities and responsibly discloses them and patches them before people get hurt. To that end, let's talk about Shy Halude. Shy Halude has four I I've decided to break Shy Halude up into four place in four pieces. Um and that is the initial exploitation step uh the rep uh ex excuse me exploitation uh exfiltration of secrets, reproduction of the worm so that it continues to propagate through the Node.js environment and then the evasion steps that try to keep it from being noticed by us white hats. Uh this is the actual code execution

flow. Uh I did not I I pretty this up a little bit but the actual research was done by an excellent write up by step security. Um the Brian Krebs theorizes that the initial 20 repositories we believe were a spear fishing campaign uh for the npm credentials. Uh and with those credentials, the adversary then actually manually logged into the GitLab or sorry GitHub repositories and added shy worm to the github/workflows/shyhilude workflow.gygamel file. Um so shyude is the name that the malware developer gave to this worm. Uh it is not a cool fancy name that the security community then gave to the worm right afterward. So a couple of very interesting things. Uh the GitHub workflow hooks into the

npm install step. Uh this is a very typical uh place to put your malware because almost all uh node library development starts by installing all the depository or dependencies. Uh and remember npm likes lots of small little Lego bricks. So there's usually lots of those little dependencies to attach. Um, and so it says as part of your install step, run the malware. Uh, then it does something kind of interesting. It runs an operating system check and if it finds that it's on Linux, which makes sense because that's what most GitHub runners are doing, GitHub action runners. Uh, or if it's on Mac, then it will continue to detonate. Uh I believe the developer author uh

felt that most developers for node prefer Linux and not Windows. Um and if it's on Windows, it exits and tries to clean itself up. Uh that is interesting. We'll get to that in a little bit. Uh if it's deciding that it's going to detonate, it then uses a promise.all execution to run the exploit modules asynchronously. Uh we'll get into the exploit modules in a second. Uh and then once the exploit modules have collected all of the secrets and credentials that it can get its tendrils on, it packages those up into a secrets.json file. Uh and then it tries to excfiltrate those one of two ways. Uh the first is uh the way that the worm was actually caught is

it opens a new public repository with the doubleb 64 encoded secrets.json JSON file just publicly in the repository for everybody to see. Uh, and then it also makes a web hook post to the web hook. Uh, with the JSON file in the payload, which presumably is then bounced from the public web hooks site to the malware developer. Uh, notice that there's no C2 here. At no point does the adversary ever put uh hands on key. Uh which means this attack is happening completely at machine speed. Either your defenses were ready for it and you were going to stop it or you weren't at your bot.

Um so one of the first exploitation modules uh is using a tool called truffle hog. Uh and this is the actual uh I have a little laser laser. Um this is the actual call that it makes. Uh and this is not from that call. This is a human readable output of the typical output you would get from trufflehog. Uh what truffle hog does is it searches in this case the file system uh for all high entropy strings which are likely to be passwords, API keys, tokens uh etc. And it has some uh fairly mature logic. Uh truffle hop's been around for a long long time um to to detect and validate if what it got is potentially unusable.

Uh and discard cruff like uh another thing that's very high is uh hashes like file hashes. Um and so it it won't report those. Uh I love truffle hog. I use truffle hog. One of the first things that I exhort all of you guys who have any sort of code that you defend is on Monday go get the free open source truffle hog uh Docker container and run it against your source code because adversaries are doing that too. Uh and secret detection has come a long way in the last 10 years. Uh there's you know GitHub advanced security is pretty great. Uh Trippy I think also did this actually. Um, I would still run truffle hog because

there's something about the mathematics of if your password isn't easy to guess or easy to detect or easy to distinguish from other high entropy strings, it's easy to guess and it's easy to break. And so there's actually, in my opinion, serious value in having a human sort through the high entropy strings in a code repository and pull out dangerous physical secrets. So Shyoon used truffle hog for evil. It scraped all the high integrity strings out of the environment variables and all the file system on your GitLab runner which if you've provided your GitLab runner with the ability to check code back into the repository which is a very common pattern. You know you might have

your GitHub and action runner lint your code and rather than fail the build and complain and say hey go fix all your trailing white spaces. It's very convenient in the Berkeley P6 to have a uh robot automation, you know, just remove the trailing weight space and check that in for you. Um unfortunately, that was the convenience that uh Shia um exploited here. Uh and then the uh truffle hog method was actually named security scam. So it even though it was doing something sketchy and there are legitimate reasons to do security scanning with truffle hog don't trust what it says on the tin. Like obviously uh I'm not going to go through all of them but uh there was an AWS SDK, an

Azure SDK and a Google Cloud SDK. And all of them did basically this here which is uh they you know created a secrets map and then they just called all of the correctly normally configured secret vaults provided by those of uh public clouds. Um if you're here from Oracle I apologize. Uh I guess you guys just make the cut. Yeah. Uh and so then it collects all of the keys that your uh gitlab runner might or GitHub runner has access to um so that it could pop any integrations. Uh once it was done with that uh this is the actual expiltration code. Um you can see it ran curl on contents that contents is the secrets.json JSON file

and posted it to that web hook which was the initial uh reported indicator of compromise. Uh and presumably the adversary was at the other end of that web hook slurping up everybody's passwords and that gave them an ability to take the passwords that uh from runners and jobs that didn't have the opportunity to open uh the GitHub repository publicly. So whether or not they had uh GitHub credentials, they still got the passwords. Uh then it does double B 64 encryption or sorry encoding. Um thank you. Uh so that's a very bad slip of the tongue. I want to explain that for a second. Uh B 64 encoding is not a trap door function. It has no cryptographic strength. Uh

anybody can decode B 64 encoded. Um so it's only ever obfuscating the the values. Uh there is a very common misbehavior in developers who are trying to keep a secret secure but are also trying to not go to all of the trouble of properly storing a secret in a vault because that involves a bunch of integration and it's kind of messy. It takes extra effort. Um, and so they'll B 64 encoding or double B 64 encode it. >> Mhm. >> Um, that has never worked. That hasn't worked since the '9s. Uh, and pentesters for decades have been decoding B 64 secrets and getting your password. In the year 2026, this is worse because your AI agents natively understand B 64

and they as soon as they see a B 64 encoded object, we'll just decode it whether they're interested, whether they're malicious or not. Um, so this is not this is not working. Um, this is the same as posting in plain text with one key difference. GitHub advanced security now has uh credential protection and will stop you from pushing uh secrets into your source code in plain text. And because B 64 encoding is so common, it'll stop you from posting credentials that are B 64 encoded once. So they did it twice and they just posted it everywhere. Uh step security uh found more than 600 results. Now, some of these are false positives because it turns out hackers in general

in open source code uh contributors uh I guess are more frequent readers of Doom than the average populace. And so shyude is actually a pretty common name for things that have nothing to do with the shyude npm world. Uh but the lion share of those 600 total repositories were generated by discolor. Um, this is also I want to point out one of the ways that we understand well this is one of the dead giveaways that the developer of Shyolude is a black hat and um I will go so far as to say because this is the opposite of responsible disclosure. This is not you have a problem. I know where your problem is and I can help you fix it and

I'm going to tell you quietly and help you fix it before somebody gets hurt. This is I I broke you and I'm posting it publicly for anybody to use for any reason. Um that is a dick move that sets the hacking community back that harms the open source developers who are committing code on their free time to make the world a better place. Um and what it does the advantage the reason I suspect the reason behind it is this means that the malware developer now has some level of plausible deniability. If all they did was the quiet web hook and they got their credentials and got out, then if they use any of the stolen

credentials, then law enforcement could hang the entire Shyoot attack on them. But if they use one of the credentials that were publicly disclosed, now they could say, "Oh, Shy Halude wasn't me. I just got this one password. It was publicly uh available." In fact, I can argue I didn't even uh break SEIFA because there's no um expectation of privacy because all I did was use information publicly available. That legal argument I don't think would hold up, but it's a lot less risky for the adversary than if they had just quietly done it. Uh so there are additional expiltration techniques and I'm going to skip those for time. Uh if you are writing malware, please come find me. Uh we'll drink

coffee and we'll talk about ways that this could have been much worse. Um and then this is the actual replication code, the obuscated uh by step security. Um it asynchronously calls a package update. Uh it forces a patch version. So it's not a major version or a minor version. It's a patch. Uh and the all the patch is is adding uh the npm worm to this uh library that is being built as well. And then it's publishing it to npm.org. Uh which means that a new version is now available any web hooks trigger or any additional builds that that will run. Uh because npmrc by default is set to use the bleeding edge of every dependency

you have. the next run for all of those downstream dependencies will then also propagate the worm. Uh so there are a couple of evasion techniques here that I found very interesting. Uh we talked about the OS validation. Uh one of the reasons that it might do that is uh as an incident responder. One of the things that I do when I find a piece of malware is I remove it from uh the environment that I found it on. If I found it on Linux, if I move to Windows, if I found it on Windows, if I went to Linux, um, we talked about the base, sorry, not only was the uh, secrets.json B 64 encoded,

uh, but all the instructions were also B 64 encoded as well and were only decoded prior to being passed to the interpreter. Uh so any static code analysis tools that would catch sketchy stuff would just see a B 64 encoded mess and would say well no function calls here so exec is you know the use of exec is fine. Um we talked about truffle hog scan being labeled as a security scan. Uh it had a replication limit and I found this very interesting. uh it if in the event that it successfully poisoned a maintainer that had more than 20 npm packages uh it would only poison the first 20 and I to this day I've given

this talk a couple of times and I still don't really know why that is. My best guess is that if it had managed to poison um so like a massive contributor like uh Struts or you know Apache who have like a thousand packages and then all of a sudden it auto updated all 1,000 of them all at the same time. that might make it more likely to be caught, which is why I think that this is a um evasion technique, but there was no method of going through and finding whether or not it had already poisoned the 20. It would just have like poison the first 20 over and over and then it would be done. Um, so I don't really

know why they didn't just hit everything they could have hit. Um, and then we also talked about all this is happening at machine speed. Uh, there are additional division techniques. We're going to skip those for time. Uh, and then I'm going to take a short just a break real quick. Are there any questions before we go into how this broke down? Oh no, I'm getting the 10-minute limit. Uh, okay. So, we're going to go fast. Um, so, uh, one of the first, the way the story starts is Brian Krebs released a, uh, really good write up day of this is happening. Um, and I just woke up and did my regular daily check on my regular

like I hesitate to call this OSENT because it was just circuit being interested. Uh, and then when I got to work, I was like, "Hey, uh, this is interesting. Is anybody on this?" Um, and that's actually where most of the timeline went was trying to figure out at at F5 we have between six and seven security teams depending on who like how you count. Um, not whether or not somebody counts is how you draw circles. Um, and uh trying to figure out trying to make sure we weren't duplicating effort or sounding alarms that didn't sound or causing scroll uh was actually the lion share of the reaction time. uh we do have an automated threat

intelligence platform uh and it is wired up to automatically do IDS public protection and stuff uh but it wasn't tied in at the time to our supply chain uh tools and so seeing something and caring enough to say something is in fact how we ended up stopping this and one of the big takeaways I'm hoping you guys get is I I suspect as members of the security community uh you guys are more likely to care enough to see something and say something. Uh so keep doing that because almost every successful incident response event I have been a part of has started because somebody saw something and cared to say something. And also uh have extreme empathy in your

organizations and encourage people. always encourage people even when they say very dumb things because the person who says something dumb uh 3 months later might be the person who cares enough to find something. Uh so this this hit the real news um like a couple months later this hit like the normie news like CNN and stuff. Uh so this was a real thing. Um, and having found it, having gotten the clear to go look for it, um, we started looking for write writeups, doing research and looking for IoC's. Um, here again is, uh, Indoor Labs has a really cool, uh, write up and some interesting stuff to say. Um and our research led us to uh this is the actual issue uh where

Frankie47 also was just doing his regular JavaScript update stuff and found uh he describes it as a cryptocurrency stealer. That's not quite I mean he was concerned about it stealing cryptocurrency wallet credentials. Uh it was not concerned with what credentials it wanted. It just wanted everything. Um, so I wouldn't characterize it as a cryptocurrency stealer per se. U, but again, this is somebody caring enough to see something and say something and he disclosed it to, uh, the owners of net supporters, contributors of Tiny Color, and that's what kicked off all of this. Um, so, uh, we started sweeping for indicators of compromise. Tyler covered this really well last hour, which is um what you don't really want to do long

term is like we had a file hash of a specific example of the malware. We had the specific IP address. Uh and the step write up actually suggests sweeping for usage of curl. Uh but the uh the F5 platform actually uses curl built in. Um and so if we had alerted on that, we would have set that alert off uh more than 90 times a second. uh and that is a recipe for extremely exhausted IP uh incident responders. And if you're always looking for the last known definition of the malware, you're always stopping n minus one and we want to stop n + one. So uh we did actually uh do a quick sweep smoke test, you

know, um just because it was very easy. We literally just entered the file hash into Splunk and was like hit during the timeline where we popped by the gnome version, but you never stop there because otherwise you're playing security whack-a-ole. Um, big shout out to Mash Kermy who actually did the write up and oh, good job. That was very helpful. Uh another thing we didn't do is uh the day that Shy Haloop came out uh there was a very conveniently made uh ElectronJS uh application called the Chris knife. Uh for those of you not familiar with the books as I wasn't a Chris knife is a knife used to kill a sandworm and it made from a tooth in a sandworm blah

blah blah blah blah. And so this uh more than 2,000line application uh strongly encouraged us to put our credentials into it to see if they had been popped. >> And again, it was written in the same language as shy hallude. So I don't actually know. We did not use this. uh it might be fine, but I can't do a 2,000line code review even with the help of AI uh in order to verify whether or not this is actually just going to be like trying to put a bonfire out with a tub full of gasoline um or whether or not this was actually going to be helpful. Uh so we didn't do that. Uh so what we did do is what we call

behavioral threat hunting and behavioral learning. Uh so any process that turns off multiple processes and checks in multiple key vaults is not uh is not your friend. There is no valid security architecture wherein you're pulling secrets from Azure vault and AWS key manager. Uh so we also uh rather than look for that specific web hook which would not have helped us 6 weeks later when shyoo came back um we knew that no uh we understood from doing enough thread hunts what normal behavior of our product is and nothing in our product should be reaching out to web hook.site. So we just blocked that entire domain outbound um and alerted if something tried to do that. Um

and uh we also went for uh not sure why it says end. Uh one of the stories that my one of my mentors told me very early in my career and was very helpful for me and I hope to pass it on to you guys is uh apocryphily uh Loheed Martin was trying to sell the B2 Spirit Bomber to the US Air Force. Uh and there was a big briefing and all the top brass was there and they were going over all of the amazing engineering. It really is a miracle of engineering. And they get to the front part of the show where they talk about the radar signature of the B2, which even though it is a massive

metal airplane, it has the radar signature of a duck. And the brass were all really impressed and they all clapped and you know, but there was one guy in the back who was just completely unimpressed. Uh, and they all turned to him and they're like, "Well, come on, man. Like, this is really cool. How can you how can you not be impressed?" name shrugs and he goes, "Well, I'm just going to shoot down the duck that's going mock two." >> And so, a lot of security works like that. Ignore what it says. I know truffle hog says it's a security scan, but if the process from truffle hog is also checking KMS and Azure, like it's

not your friend. Shoot it down. Uh, we did some code review at scale during the the event. Um, it's incredibly useful. I think so we did this in Python because the malware is in no.js. So very much like switching from uh Linux to Windows or vice versa uh if you're searching for malware or or writing uh code that's going to interact with an adversary switch languages. So it's a little harder for their tooling to pick you up. Um I've also I have found that uh scripting during an incident is very common. Um, I have some reservations about using generative LLM. The last thing I need is a hallucination in the middle of a investigation. Um,

when we're trying to reestablish trust, a hallucination breaks it. Uh, however, when we've coded very small, very easily auditable bits of code, um, we have found that to be extremely helpful in in security events. Uh, and that's very helpful for retrieving um, evidence. uh you should not while you're reacting to an incident you should not make the assumption that your evidence is going to be there even in 30 minutes. So grab everything you can while you can. Uh this is the actual code that we used to hunt through our environment for shyoon. Uh it's very it's intentionally simple because it's very deterministic. We know we didn't miss anything. Uh and then the short-term solution is

version pending. Uh this is actually what the Node.js uh community suggested. Um it's a literal one character change. We changed the configuration with Node.js rather than to take the bleeding edge. We pinned it to the known safe package. Uh and we're good there. The reason that this is a short-term solution, of course, is that whichever version you're pinning may have a CDE found in it. And so now you're stuck between the bleeding edge that might have a supply chain attack and the tailing edge which might have a CVE. Uh I want to acknowledge uh this story should not make it sound like I did this on my own at all. Uh I had help both

from the security team and the development team. Um I'm not going to talk about people's names publicly but uh I work together with very smart people and it's it's really fun. uh common pitfall solutions. I'm actually going to skip these uh because we're running low on time and I want to get to long-term solutions. Um I I don't think a cool off period alone is enough. I don't think that vendoring your code is a good idea. Uh what I do suggest is that uh you sign your builds and check your signatures because if you sign your builds, it's very hard for somebody else to forge malware into your builds. And if they spearfish your credentials or if

they do a subdomain takeover, uh you still have a problem, but it's going to raise the minimum of a bar for compromising your packages. And then uh I strongly recommend caching your dependencies in a repository and scanning those prior to serving them out of your cache. Uh and so most of my def my defenses revolve around that. Uh since I've got less than five minutes left to go, um I will say uh really quickly uh I'm gonna use the example of JROG and uh X-ray and Artifactory. uh sonet nexus does this at uh enterprise scale and then if you don't have enterprise money to throw around at the problem uh which if you're at an

enterprise you absolutely should because I guarantee you guys are using third party software at scale and you need to be able to treat this problem it's going to cause enough damage the likelihood is there um but if you're at a smaller company uh there's the open- source tool pulp um and the difference there is you're going to have to com what you're saving in licensing you're going to have to pay for in sweat acting. Uh P doesn't have a scanner. You're going to have to build and automate that yourself probably in Python. Um so what this what happens is either your developer laptop is going to have this CI/CD runner. Um I'm using GitLab here because GitLab is something

we use at F5. Uh it could just as easily be uh GitHub or uh Gy or um Bitbucket, whatever your CI/CD runner is is here. And rather than having it go directly to the public internet, pull down and trust whatever you're going to get, which might have that supply chain attack, it's going to go to your cache. And if your cache has seen this before and has scanned it, knows it's safe, uh your cash is actually going to reply much faster and cheaper than the public internet would. um it hasn't seen that before. It's going to reach out to the public on your runner's behalf. It's going to pull it down. It's going to trigger a scan, which means the first

time you get a dependency, uh it'll be much slower than just getting it from the public internet. But every time thereafter, it's a cache. Every time thereafter, it's much faster. And the truth is, you guys don't update your dependencies more often than you update your code. And so in addition to providing additional security, this is actually a long-term cost-saving measure, especially if you're building in a cloud environment where you're paying for all this traffic. And then what you as a security team do is you uh deploy or leverage some dependency scanner that pulls in the the scanner and does whatever security assurance you want to do on it. Um, if you have to implement this yourself, uh,

open sourcemalware.com is a fantastic API that I want to plug that will check for known malware, uh, or suspicious behavior. Uh, and of course, virus total has an API that will just scan it with every virus scanner known to man. Um, and it does it pretty quick. So do that. Uh there's also out of the box solutions for uh build signing uh both for open source you have free options uh or enterprise grade use the one that's appropriate. Um there's also uh safe chain via Aikido and supply chain firewall. If you're a tiny team of like six people uh the downside with these tools is they live in your command line and you can always accidentally just go

back to the old panjs and load it. But if you can trust your team to use safe chain or supply chain firewall, um you don't have to host anything. You don't have to maintain anything. And um so for very small teams and I recommend that and with that I'm out of time.