BSidesSF 2026 - When the supply chain hits a sour note (Kennedy Toomey)

Welcome to this session. Good afternoon. This session is by Kennedy Tumi and she will be talking on when the supply chain hits a sour note. We'll be doing Q&A at the end of the session through Slido. So, if you haven't already gotten the QR code, you can also find uh find Slido through bidesf.org/q&a. With that, over to you, Kennedy. Thank you. Let's go ahead and get started. And today we're going to talk about when the supply chain hits a sour note. My name is Kennedy Tumi. I'm an application security researcher and advocate at Data Dog. But I think in this case, I have something else to add. I'm also a terrible singer. So, I'm pretty used to hitting sour notes,

especially when I'm singing, which I love karaoke. How can you not? And when I was creating this talk, I was kind of approaching it thinking about karaoke. So, let's go ahead and keep going. Who's afraid of little old me? This is the story of an npm package. Everyone blindly trusts it. It's configured to upgrade automatically. And guess what? Everything's working great. And now a new version's released. Awesome, right? What could go wrong? Doop d another one bites the dust. Get it? Um, but seriously, another day, another supply chain incident, right? We saw this specifically back at the end of August and September. We were seeing new supply chain attacks every few days, every week and a half or so, which was a

lot. I'll go through them briefly. The first was singularity. Now, this specifically was it um the attackers injected a malicious script into a vulnerable GitHub actions workflow and this was a credential harvesting script and it ended up resulting in thousands of leak secrets. Obviously, not a good thing. And our next uh instance of an attack was just a week and a half later and a npm maintainer was actually specifically targeted and they did this through a fishing email. Now this malicious code that was injected was a crypto stealer. The bad news is these malicious versions were downloaded more than 2.5 million times. The good news is the impact was minimal and only about $500 was actually

stolen from these. Now we have Shy Halude. I'm sure you've heard of this and if you haven't, I encourage you to look it up after this. Now, this attack was actually using unrotated tokens from the singularity attack and it was using a self-replicating worm. The bad news is they exfiltrated secrets and they exposed private repos. The other bad news is this wasn't the last time we saw this kind of attack. There was a second one. Again, a similar self-replicating worm that backdoored almost 800 unique npm packages and over 500 GitHub users were affected. So you know maybe these dependencies are a chain of fools. One vulnerable dependency might lead to more and more and more and more and more. Now this

could be due for different this could be due to many things. It could be transitive dependencies. One dependency is using another which is using another and guess what one's malicious. Or it might be chained attacks like we saw Singularity and then the Shy Holute attacks. The attack surface just keeps spreading and it's really hard to keep track of. And then we have a manic Monday because guess what? These attacks, they never happen at good times, right? It's 3:00 on a Friday or it's 300 a.m. on a Tuesday or even 7:00 a.m. on a Monday, which is the last thing you want to wake up to when you're starting work for the week. So, you're trying to scramble and

figure out what happened, right? You're having to do all the research, understand what's truly going on, and then you're also trying to figure out, are you affected? Do you have an SBOM? Is it up to date? If you don't have one, if you don't have something up to date, do you know how to figure out how you're affected? If you don't, keep that in mind later. So something that's key here is you want to stay organized. You need to find a way to track how you are investigating things. Now one way is a spreadsheet or through different tickets or through just a document. You also want to assign clear responsibilities. And this is something I learned very early on in my career.

Uh, previously I was working on something that wouldn't necessarily be a security incident, but it was kind of a hands-on deck situation and we created a spreadsheet. We told the team, "Here's what you need to do and just jump in, pick up things, start working on it." Well, guess what? Not everyone did. And that's because they weren't assigned what to do. And I was talking to my manager after, and he's like, "Well, you did everything except you didn't assign it." And I don't know if y'all have heard of the bystander effect. And now this is something you learn in first aid and CPR training. I don't know if anyone is first aid or CPR certified. I had to

do it a few times and this was specifically called out. When there's an emergency, people think that if someone calls for help or says, "Hey, I need a first aid kit." Everyone assumes someone else will do it. That's why during first aid training, they teach you to specifically point to someone and say, "Hey, you, you get me the first aid kit. Hey, you, you call this person." And that's the same thing for security. Now, obviously, it might not be a life ordeath situation, which is good. However, it is important to truly point and say, "Here's your responsibility to make sure that everyone's on the same page and to make sure everyone understands what they should be doing."

Now, everyone's under pressure in this situation, right? Leadership, they're breathing down their necks. They want answers. Customers, they want to know the impact, right? You're getting emails from customers and clients saying, "Hey, what's going on? Are we affected?" and you're trying to do this, answer the questions. Well, new information is still being released, right? These attacks, it's not normally oneoff. You're getting this information. You're having to stay up to date because new things are being found out. And so, you really need to come together as a team. You need your developers, your DevOps engineers, and your security team. Now, this can't just be a lastminute thing. Well, it can, but that's not when it's good. Ideally,

prior discussions of expectations need to happen, right? We don't want this to be something that no one expects, no one knows what to do in this case. We want to try to actually train our developers, our DevOps engineers to know what to do in case an attack happens so that everyone can jump in and know exactly what their roles and responsibilities are. Something else we need to do here, we need to communicate with leadership that if an attack happens and we need all hands on deck that the teams need to kind of stop what they're working on and hop on the instant response efforts. This has to be top priority and leadership needs to understand that.

Now, the good news is you can't always get what you want and this is specifically focused on the threat actors. The good news is hopefully we've implemented some security practices at some point. Maybe it's lease privilege and defense in-depth ideas. Ideally, we've reduced the attack surface. So even if something happens, guess what? Nothing really bad happened. Now one recent example of this we actually saw at data dog. There is a well there are many but specifically there was a data dog open source repo that was seeing malicious PRs opened against it. Now this is interesting because we actually were able to detect it when it happened and we had orgwide configurations in place that were able to prevent it from

being merged. Now I do include the whole write up of that at the end of the resources if you are interested to learn more. In addition, we have proactive defenses. In some cases, we're using fine grain access tokens instead of personal access tokens to make sure that we've narrowed down the scope of what these tokens can access. Also using MFA that will never stop being a trend. Something new that actually came out of the attacks in August and September is something called minimum package age. And I didn't hear this talked about much prior to it, but it's the idea of setting again the minimum package age that you want to use in your software. So if you set it for 7 days, the tool

will wait until the package has reached that certain age, 7 days, 4 days, whatever it might be, and then it will automatically upgrade. Now two tools specifically have rolled this out in the JavaScript ecosystem that are alternatives to npm which was targeted. It's yarn and pnpm as well as GitHub's depend tool um very similar called cool down which is the same thing. It gives you the ability to wait for a 7-day or a 3 or 4 day cool down because the thought is if you give it that time to cool down and to let researchers look at it or to see things blow up, it gives you some breathing room, right? So, I wish I could sing like Adele about

this out for you, but I cannot. So, you have to imagine it. But hello from the other side. At this point, you've made it through a majority of the response. You can see the light at the end of the tunnel. It's not over yet, but you're almost there, and the show must go on. Urgency has lowered for everyone. Now, for the security team, it's lowered slightly. You're still kind of prioritizing this. This is not over for you yet. But guess what? It's pretty much dropped off for other teams. Normal work has to resume at some point. You're still monitoring it, but not everyone is. However, I'm still standing and so are you. It's time to take a deep breath

because guess what? You did it. You were either you were able to either protect your systems, reduce impact, rotate a ton of tokens that were exposed, whatever it is, you did it. And the good news is what doesn't kill you makes you stronger. Now, this is a pretty common thing. We want to do a retro after any incident, a retrospective, a post-mortem, whatever you may call it. Think about what went well. What specifically went really well during the incident? Some things do go well during incidents, even if it might not seem like it. But also, what did not go well? What did you struggle with? Did you have an easy way to find the malicious versions, the

vulnerable versions? Did you face friction from other teams or from leadership? Did other people just not know what to do? Did you have a communications plan in place? Did you know exactly who needed to be notified? And then what we always do after we figure out what didn't go well and what went well, we need to fix it, right? We're asking all these questions, but it's kind of useless if we don't do anything about it. So the goal here is to figure out what didn't go well so that you can fix it for the next incident. So when I was working on this, there was a social media trend that was popular where this song specifically this part of the

song because if you've noticed this is the first time I did not use a song title. It's just a lyric. But why do you write like you're running out of time from Hamilton? And this is important because after an incident, you've got to document. You've got to write everything down, right? What did you do? What was affected? What was the result? Who worked on it? There is so much that you have to document as you're working on these incidents. And the next thing you have to do is you have to communicate with people. What happened? Again, the clients, they're emailing you. They're asking, "How are we affected? What do we need to do? Now, you have to figure out the best way to

communicate with all these people because guess what? Leadership wants something different than your engineers, than the customers. You really have to figure out what works best. And with that, create templates. At this point in the incident response efforts, you're exhausted. When I'm doing things and I'm this tired, my writing skills are the first thing to go, right? I want to make it as easy on myself and my teammates as possible. Have templates in place that you can just fill out different blanks so you can easily email clients or customers and still have it sound professional and you can send exactly what you need to send to leadership so that they understand it but they're not

getting too much information. Something else you need to do, write SOPs or standard operating procedures. Write down instructions on how to replicate this again. Because guess what? This is not the last time you're going to see an attack like this. And if you weren't sure how to investigate this time, make sure you write down the steps to investigate next time. So, at this point, we've all got friends in low places, right? These incidents can take weeks, they can take days. It all depends. Please, I beg of you, check in on your team. These events, they're stressful. They're high pressure. They're exhausting. I know personally when I'm working on incidents all day, I might not be eating

properly. I might just be snacking and not eating right. Also, think about people who might be missing dance recital and sporting events of their children. There's so much that goes on during the week and over weekends that you might be missing because you're working on an incident. Check in with your team. Make sure they're okay. The good news is it's 5:00 somewhere, right? Take time for yourself after an incident. Take time to breathe. This might be going for a walk outside because it's finally nice out. This might be going and grabbing an ice cream or getting French fries. Whatever it is, take a second to yourself, log off of your computer, of your phone, and just

decompress for a second because again, these events, they're stressful, right? We don't want to be a part of this, but we do it anyways because it's our job. The next thing, mandatory fun with your team. Now, this is focused a little more towards leadership. I'm also going to emphasize, do not do this the next day after an incident. No one will be happy with you if mandatory fun is literally the next day because at that point, that is not fun. But this mandatory fun, it can be a few different things. You know, maybe it's just taking a normal meeting and saying, "Hey, we're not going to do this today. Let's just socialize or let's play a

game on this meeting just so we can decompress and get to know each other personally. Now, if you're all in the office together, this might be something different. It might be going to all grab lunch together as a team and missing out on an afternoon meeting. It could also be telling your team, "Hey, you've worked hard this week. You've worked a ton of overtime. Now, if you can take comp hours, but also without that, maybe take Friday afternoon off. Now, everyone can't always take Friday afternoon off. Rotate which Fridays people can take off so that you can really tell and show your appreciation because guess what? 99% of people appreciate being appreciated. They appreciate being thanked because guess

what? This is hard work. It It's kind of fun sometimes. Like that's obviously why we're all in security. We like doing things like this, but it can get exhausting. So, thank your teammates. Thank them for doing great work. The next thing, 0% of people were will turn down free food delivery during these events. Maybe tell your team, "Hey, why don't you get food delivered and expense it or even send them a gift card to get that food delivered?" People will appreciate that because again, a lot of times you're not always eating properly. You're having whatever snacks are in your house. Show them that they're a valuable and important member of the team because people appreciate

food and it's one last thing they have to think about after the incident because no one wants to go have to go grocery shopping and then cook and do all the prep work and then have to clean up after. No one wants to do that. So, what comes next? Soon you'll see. There is always going to be another incident, right? It could be tomorrow. Hopefully not because I would like to watch some of the talks tomorrow. It could be in a few months. It could be a year. There could be three back to back to back or there could be none for a long time. We just don't know. Prepare now so next time will be a

little easier. repair now so that you can train your teammates to know what to do in these different situations. The goal here is to truly understand what the team needs, right? We can create templates, we can create SOPs, but we truly need to understand what does your specific team need. Are you a small security team of two or three? or are you a huge security team where there are too many people? You have to truly dive in and figure out what works best for your team and maybe this is by doing a mock incident. Maybe this is figuring out, you know, hey, we want to make sure we're prepared for next time. Let's do just a round table

to see how we would respond, who would be involved, how we get people involved. But again, this is not going to be the last time we see a supply chain attack. I have a feeling we're going to be seeing many, many more. So, with that, I want to say thank you. And I do have time for questions. I do want to point out that this QR code, this is going to take you to all the resources. So, this has all of the different um research that I included. It has a copy of the slides and the most fun part, it has a playlist for all of these songs. Now, I don't know if y'all did notice hopefully that each slide was

a different song. I have a pretty eclectic taste of music, so they were kind of random, kind of scattered. Um, but that playlist is included on these on this resource. So, again, we do have a good bit of time for questions. Thank you so much, Kennedy, for the great uh and insightful presentation. We'll do Q&A through Slido. So, as a reminder again, if you have questions, please post them on Slido. You can do that by going to bsidesf.org/q&a. And while we have questions rolling in, I'll make a fewformational announcements. Uh if you want you can take a break from the day's events with a stop at the bar and chill out space which is sponsored

by Run Zero. You have two complimentary drink tickets that we provided to you during registration. We've already paid for them so make sure you use them if you haven't already and you can use them for both alcoholic and non-alcoholic drinks. We also have prayer and mother's rooms. If you need to use one of these, please go find our info desk staff upstairs and they would be happy to guide you there. We also have head shot all day sponsored by Nebulock right outside of talk tracks by concessions. If you want, go get your new headsh shot perhaps for your LinkedIn profile. Uh please note that they will close at 5:00 p.m. today. And as a reminder, you can also support

charities, EFF, B the Diana Initiative, etc. by buying a t-shirt by code check on the fourth floor. T-shirt sales will run till 9:00 p.m. today. All right. Uh if if not through Slido, if somebody wants to raise their hand, we can do that as well.

Okay, I think then we can wrap up early. >> Awesome. Thank y'all. If you do have any questions, feel free to find me on LinkedIn and I will probably be hanging at the data dog booth tomorrow if you want to say hi. >> Awesome. Kennedy, actually, we have a couple of questions. >> Oh, awesome. Okay, I take it back. >> Okay. For minimum package age, have we seen any supply chain attacks adapt to be more similar to timebased malware that only activate at a certain time? I personally have not seen it yet. I think at this point the minimum package age is such a new idea that it's not being frequently used, especially um last I checked it was only in PNPM and

yarn for JavaScript. So it's not in npm which is the main package manager. Of course dependabot also supports this idea of the cool down but again I don't think it's as widely used as it probably should be. So I haven't seen anyone try to go around it yet because it hasn't been needed. >> Okay. >> Okay. Good to know. Thank you. Um we have another question. What's the fastest response you've seen in a company for a supply chain incident? Minutes, hours, days, or weeks to say, "We're confident this library is, you know." >> Sorry, can you ask one more time? >> What's been the fastest response you've seen in a company for a supply chain

incident in terms of minutes, hours, days, or weeks? >> Um, I'd say within minutes. I think it all depends on your team. I think if you have a global team where people are kind of up at all hours because some people are in Asia and then Europe and then the US and North America like we really can see news happening at all hours. So you're able to start the response efforts a little faster. I've worked at big companies. I've also worked at small companies where no one's seeing it at 3:00 a.m. Right? I'm personally not. I can tell you that. So it all depends. I think generally the more distributed a team and a company is the faster the

response time is just because you'll be able to see it earlier. However, there are paging systems. So if you are seeing an active attack depending on what kind of tool you're using that also helps if you are not a globally distributed team to see incidents happening as they occur. >> Awesome. So with that we can wrap up this session. Again, thanks uh to all of you for attending this presentation.