Trailblazing Lessons from Oregon Trail for the Secure Software Supply Chain - Kadi McKean

Yeah, we need some energy in the room.

All right, I'm going to read this uh this here from my from my notes here, but I want you to know and welcome Katie from the east coast. East coast. Welcome to New Mexico, Katie. Katie is passionate about DevOps and DevSec Ops community since her days of working with Cobalt and mainframe solutions. You heard that right, folks. Cobalt and mainframe solutions. At Reversing Labs, she collaborates with developers and security researchers to help entities prioritize their open-source risk, reduce technical debt, and meet compliance objectives. when she's not working with developer community. She loves running, traveling, hanging out with her dog, Milo. >> I love dogs, too. All right, take it away. Thank you. >> Can you hear me? Oh, good. Now we're

live. Okay. >> So, um, thank you so much for having me here. I'm really excited. This is actually my first solo speaking engagement, so this is a big deal for me personally, but I'm very excited. Sorry. I'm also excited to be out here um in New Mexico for my second time. So, thank you. Um before we dive into this though, I just want to get the temperature of the room. Who has heard of the video game or trail? We got one, two. Okay, we got more than I thought. If you don't know what it is, who has seen Yellowstone? You will fit right in. Okay, we'll be fine. So, um I'm gonna have a lot of bad

jokes in here today, so there with me. Hopefully you learned something. So, a little bit about me. I'm I'm Katie Keen as uh he mentioned. So, I'm actually a community manager at Reversing Labs. What does that mean? It means like I'm a little agent of chaos um who interacts with the open source community and trying to help improve security posture. Um other than that, I also run my own podcast. called 10x insights where we talk with people from development in application security uh to learn more. Thank you. It's okay. Um to make sure we're sharing insights so it's everything like it's something everyone should know. It shouldn't be kept as a hidden secret, right? So a little bit about reversing labs. Um

we're in the top 20% of Fortune 100 companies, but you guys can read off the slide. I'm not here to talk about this today. I I like this slide when it talks about just who we are because at a high level we have a 27 pabyte data leak. If that doesn't strike you as massive, it should. So in our DNA, we're actually a malware binary based analysis company. So we have this huge data lake of all these scary things so that we can run our own behavioral analysis and talk a little bit more about what we're talking today. So agenda, I hope you like by photos because that is what you will be seeing a lot of today. Um, but today we're

going to talk about what the trail is, the gating process, cutting corners, common deaths on the trail, sudden disasters, and then if we have time, time for questions. So, this is the Oregon Trail. I tried to find one that was uh within the the, you know, green screen uh vein, but they all looked a little weird. So, this is the best one that I saw where you can kind of see it from the 1800s. So, as you can tell, this journey, it's a 2,000mi journey, right, where people are leaving from Independence, Missouri, and they're going all the way out to May not uh valley out in Oregon. So, along that route, there is some crazy stuff that

can happen and a lot of different ways you can die. Hopefully, as many of you remember from the game, right? Um, just like Oregon Trail though, survival in modern software development really requires preparation, wise decisions, and the right tools. So, We're going to get into that and how we can really win the game against these modern challenges in in software development. So, what does our trail look like? This is the trail I'm going to be talking about today. And this is where you're going to really see there's different stages for build, develop, test, deploy, um, and then updating, right? because you're running that constant feedback loop. Because in today's world, it's not a matter of if something bad's going to

go wrong, it's a matter of when, right? And I wish I had a crystal ball so I could be better at predicting that, but we're not there yet. So, um the key here though is what I want you to pay attention to is the gating process. So, there's strong security checkpoints along the way and that's to try and help hinder things like malicious code getting injected or catastrophic failure. So, we do have these checkpoints in place. It's just a matter of how well they're followed, right? So, the other thing that I found interesting about the game, right, is there's actually three different players you can be. So, who chose to be a banker? Who chose to

be a carpenter? Anyone ch I always went for the farmer. I don't know why, but I just thought it was cool. But each of these roles, right, they have a different um resource amount. So maybe it's more sheep if you're a farmer or more seeds whereas if you're a carpenter you know you you have access to tools. So you have to really choose your role wisely and much like you know the farmer banker and um carpenter today's players are are similar right everybody's got a different skill set um they have a different resources available to them and they have a different perspective in mind when what they're you know for day-to-day actions. So their end goal is

a little bit different than someone else's. So for example developers really just care about shipping code fast. We want to get those features out the door quick, right? And then security. We want to make sure we're shipping secure code and hopefully preventing vulnerabilities. But we'll get to why it's just more than vulnerabilities in a second. And then operations. So operations wants to make sure things are running smoothly from each part of the different process moving through the gating procedure as well as then once it's in maintenance mode, they want to make sure nothing's going on. So everything should be running seamlessly. And again, different understanding, different goals they have, and they have a different little bit of a thought

process when it comes to this. So, as I was talking, we were talking about these different checkpoints. So, for those of you familiar with the game, you'll remember there's, you know, a blue river crossing, there could be a fort crossing, and all of these are different stages where you're like, "Yes, I've made it." Or like, "Okay, I'm a little bit safer." That's kind of how you feel when you're doing the different parts of the gating process in software development. So once you move to the next stage, you should be feeling a little bit more safer knowing that they have passed certain security checkpoints, right? So the feature that the developer made it made it into say like production, you know,

it should have had certain checkpoints to go along the way. Um what I think is interesting, right, is Q&A, right? So Q&A used to be something that not everyone do. But now we're seeing everyone worried about Q&A, especially from appsac um and more people are worrying about it. So we're starting to see like for example folks at OAS meetups, you never really saw like QA mentioned, but you are starting to see more of that now. Um the main point I wanted to talk about here is that if you skip these different checkpoints, you can risk losing everything downstream. Um and no security released to just focus on vulnerabilities, right? You you have to be aware of other things that are going

on here. So, where do we integrate some of this these testing points, right? So, I talked about the different um checkpoints and this is kind of what it looks like um in a perfect world or one idea of a perfect world where we're injecting those security tests. Um, and why why can we do this is because we can inspect large software binaries. Um, where we can be kind of that last line of defense before you push it into production. So, it's a final build inspection right? Um, I think that's all I wanted. Yeah, because you guys know this stuff, right? Okay. So, we've talked about the different gating points, but we haven't talked about shortcuts. So, who here has

looked for a shortcut they can take. Like, let's let's be honest, like I'm I'm right there, too. If I can take one, I probably will. The problem with that is in in regards to software development is that you're going to have some problems. Whether that's, you know, if you're taking a shortcut in compliance, you're probably going to have someone come talk to you from legal at a later date in time, which really, I don't know about you, but I personally never want to be in a lawyer's office unless it's like for something good and I'm inheriting like $3 million. That's not usually how this goes. Um, so you could end up in a lawyer's office. Um, you're

talking if you're missing some of these steps too for that final build inspection, you could be more vulnerable to malware injections, misconfigured pipelines, incomplete vulnerability tracking. There's a variety of different things that could go wrong here. So, few of the common themes I just wanted to point out real quick is pitfalls when it comes to cutting corners usually is compliance and especi pipelines is also something you want to make sure you're not taking a shortcut on and making sure you're choosing the right open source components. So, let's get into some real world examples because, you know, this little gift is only so fun. So, what's wrong with this picture here? Anybody have an idea? So, do you know why anybody would want

to make a package with an extra Y? So, this is what we call typos squatting. So that's where people are maliciously and intentionally using, you know, fat finger basically or like putting an extra Y on it because they think someone's just going to grab the package because, hey, I use cryptography all the time because it's written so similar and looks so similar. It's hard to know that there is malware injected in here. But there are tools out there where you can do a quick inspection like this and see that, hey, something is definitely not right here and it's not passing um you know, my snip for what I want to look at. So, what do we want to look about?

Here's another example, too. So, this one actually happened to me in real time. Um, I was at Pyon and someone wanted to look up the request package. It's a very common package used in in PPI. Um, and this is I almost had a heart attack and like it wasn't failing yesterday, so what happened? But hey, you put an extra U in there and now you got malware. So, let's talk about another another event. Anybody had plans to enjoy their Christmas, New Year vacation in like late 2021, early 2022 or anybody else like were they just ready for like a fire girl? >> Always ready. Yeah. Stop roll guys. It will help. I don't know if it will in

this situation. Let's talk about lock for shell though. Anybody remember this? I know it personally like did a lot of weird stuff to my holiday schedule which I didn't care for but you know I'm only on the community side. So I can I just remember looking at one of my colleagues being like dude have you slept in three days like do you know where this is? How are you going to fix it? So when we're talking about blog for shell it's a heavily open source uh heavily used open source library in the Java community. Um we kind of want to like I don't want to on it too much, but it kind of is a

little like mini celebrity vulnerability that became known from this. So, what happened? Something functionally was added in in 2013, right? And it was added to log for J by a guy named Ralph Goers. Um Ralph isn't, you know, he's not a nation state threat actor implementing a vulnerability with schemes to exploit it later. He was just an unassuming guy from the US. He's just, you know, I've got some job knowledge and I I want to help out the open source community. So he contributed well back in 2013 that commit turned out to be what became the unmitigated disaster eight years later and ruined a lot of people's holidays. So what this is it allowed for remote

and often unauthenticated users um to execute arbitrary code on their server. So that's about as bad as it gets. I mean anybody else think that's there's nothing worse than that? I mean that's that's pretty bad. Um, and to this day it was one of like few CDEs that got a a CDSS score of 10, which is again literally as bad as it gets. So what I thought was interesting and the reason why I like this example is that nobody noticed the vulnerability until late 2021, early 2022, which ultimately, you know, forced the public disclosure. Um, and it's just wild to me that it took eight years to find this, but again, people are injecting these little

ticking time bombs. So they're just waiting for it to get picked up or looked at one day. And again, you just don't know when. So all right, what about this one? Does anybody remember uh UA parsers? This also happened about 2021, right? November. So we're thinking of like again holiday season. I like Thanksgiving. This wasn't really fun. Um so UA um with this one what I thought was interesting is that this is where credentials got stolen. So the guy the hacker actually gained access to the maintainer's npm account right and published three malicious packages which contained like was right in the pre-install script. So it executed automatically um upon installation to involve like a crypto

miner um targeting Windows and Linux um and and ultimately a Windows password stealing Trojan. So again, another wild example of something that could go wrong if we're not putting best practices in place. Um I'm new here. So this Um, this is also one thing that I like to show too is just you can see how easy it is to detect these these packages and what's wrong. So you can get a quick visual and you're like, "Hey, red means no, yellow maybe." Right? It depends on what your level of risk tolerance is, how your environment is configured. Um, but it's a really good way for you to be able to decide, you know, what you're willing to take on um here.

And then we've talked about this, right? So we're starting to build in more and more of these testing procedures and making sure we're having more gating points in there. So that way, you know, if something is found, it's hopefully after it's in in production. So your your testing gets a little bit more noisy, right? So, this is where I'm trying to articulate, right, that there's a lot of different noise you're going to get be getting with these tools, whether it's like free open source tools, a tool you commercially buy. They're all going to have some type of noise. So, you have to figure out which signals are the ones you need to be listening to the most.

Maybe that's, you know, only CVSS10s. Um, some of us like to look at them, you know, for seven and and up. Um, but again, it just depends. So, this is also just showing where there's other types of checks you can be doing. So, it's not all vulnerabilities or like typical spawning type stuff. Um, we're also talking about compromising at CIC or tampering. So, that's where you're looking at different versions where like this version's okay, but that someone injected malware in that one. So, you want to try and figure out what the diff is between the two so you know, hey, this this is what happened. This is how we can adjust it. Um, and it's also good to see that

there's a history, right? A lot of these packages I personally like to know like are these ones getting tampered with more often than others because if it is getting tampered with you have a higher likelihood of being attacked um just based off of looking at its previous history. Right? So the moment you guys have all been waiting for, right? How am I getting back to working for this? So um we've talked through a bunch of these different options today and this is just where I came up with. So dysentery is malware, right? So you're going to have CVE, CVSS vulnerabilities, but to be honest, malware trumps vulnerability any day of the week. Um, so snake bite

you'll see is that vulnerability exploit because that's still really bad. You might have like a couple holes in your arm, maybe die poison. Um, starvation is that dependency confusion we were talking about. Hunting accident is always my favorite. Um because like you haven't lived unless you played the game and you actually were involved in a hunting accident. It was very sad but like it happened. Okay. Um and that's CICD misconfiguration. So cold exposure just typo squatting, right? So that's adding that extra Y adding that extra U. But I know what about the elephant in the room, right? What about AI? So AI also is providing its own unique form of of death for us, right? So we start to

see some of this happening where bones we're starting to see are more AI agent based. So if you think about things like um anybody here of the Disney attack that happened last year that was a good one. I don't know maybe it's because like I'm a big mouse head but I I found this one to be truly interesting. So um this guy who who you know unassumingly downloaded a malicious file disguised as an AI image generation tool. Let me tell Yeah, you're laughing because you know where this is going. Um, he granted basically an unauthorized access to their personal computer in in downloading that agent. So, this guy had couple terabytes of data from Disney's

like private servers, 44,000 messages off Slack, you know, he knew like the product road map, where you guys were looking to go, other documentation of like doomsday scenario that is not not publicly available and this guy did publish it. So, that guy lost his job. Um, unfortunately, I think he tried to fight it, but I, you know, I don't know how you come back from that. But, um, the point being though is it could look like it's a helpful little tool. You need to vet out these tools though before you're downloading them, right? Because we're starting to see more and more of uh these types of attacks coming in through developer tooling, whether that's like a VS Code extension or an AI

image generating tool. Um, you can also tell that malware in large language models, we have been seeing more and more of that. So, it's going to become more commonplace, right? Just as hard as we're working to keep these guys out, they have a whole team and they're working, if not harder to even get in. So, with more tools being available like AI or um LLMs, we're going to start seeing more and more of these attacks happening. They're not going to be novel like oneoff things. Um, there's also a learn bias, right? So, the hunting could just be like, "Hey, Katie's just not a great player, so that's why she's having this accent." But there's also a learn

bias in that, you know, we're teaching these large language models what we want to do without thinking of full um all different options of what could occur, right? So, it's only thinking from like my point of view or it's not including maybe three other people's point of view. So, you have to realize that it's probably not capturing everything you wanted to. And then, you know, there's always like the in real life Armageddon and that's basically Skynet and we're all just slaves robots. So, you know, that's something to look forward to, too. Um, the other thing I wanted to talk about is how many of you heard of Jim Manico? Okay, so I'm glad no one does because

this is very helpful for me to teach you guys a moment. Um, Jim Mano is a friend of um, one of my colleagues and you might be hearing about coding where AI is heavily used um in the software development space. Well, Jim Vgo is a premier educator in in the area of secure coding and secure application development. So, he's also a speaker at many of these application security conferences um in the OOS space, but lately he's been commenting and really shouting from the rooftop about how AI creates horribly insecure code, right? So, to address that, he's actually built some prompt engineering rules. um some scripts that can help with AI. So, if you're looking to use more AI but using

it more securely, like call this guy. Um he's really good at it and he's just a really great educator and well respected. Um we're cruising. I know I have like five minutes left, so I'm cruising. Um so, pickle files, right? This is also something. It's not the pickles you eat, but these are pickle files in Python. Um where can you find the malware in the model? Right? So, this is something that we are starting to see a lot more of. Um, and it's really just sticking with um the whole AI theme, right? It's a very popular data model within ML. Um, pickle file is really just like a serialized data set and when the file is

uploaded, it gets deserialized, right? So it turns out that this allows for thread xers to really embed attacks for pickle files quite easily. Um and those texts are executed in the des serialization process. So some of the the bad things can be done including like spotting processors um on the machine communicating with the network or even executing code which always is fun. Um so what you see here at the bottom are some of those checks that you know reversing labs does to help you when it comes to pickle files. So, just thought that was cool to highlight. Um, so we've talked through a bunch of these different, you know, options, right, where there's there's really like three

different types of ways we've seen some of these coming in. There's more, don't get me wrong, but like the three big ones I've been talking about with some different examples are are the state sponsored attacks. We've been seeing them for decades, right? Um, supply chain poisoning 2020. How many of you remember Solar Winds? I hope you were a customer, but actually were the ones who were able to figure out by breaking apart their binary and telling them we knew this would have happened. Here's how and here's what you can do to protect yourself moving forward. There's actually a whole case study on it. Um and then 2023 3CX and then dev tooling kind of what I talked about with that

Disney hack, right? Um the XC utils is also one that is more currently where you know some guy was having you know maybe some mental health problems had asked for help um for a very important package you know blogging app and you know Gan just raised his hand and was like hey yeah sure I'll do it. So he looked like a good committer for a little while until he wasn't and that's where he just injected the malicious code and ruined a lot of people's days quite quickly. So again, that's a little bit more on the social engineering side, but we're starting to see more of these types of attacks, especially when it comes to dev tooling, um, even into the

VS code extensions that I talked about. So this is my favorite slide. So what happens when they all kind of converge together? Um, and this is really, you know, because some of them can be multiple different things, right? It could be state sponsored, it could be poisoning, it could be dev tooling, but also state sponsored or poisoning, right? So the trick really becomes is how how do we avoid the stampede? Um automated security scans and CIBC CI/CD pipelines good final build exam great. Um you can't deliver or deploy without it. Like let's let's be honest. And then making sure we have reproducible builds. So some good counter measures. uh it's tough but it's challenging based off

like hashes or net has some of the native support but not everyone does so we do it based off behaviors and that's looking at the differential you know analysis that I was mentioning earlier or detecting build system tampering um or even improving where ways where you can harden your solution right um so I don't know if anybody else remembers this I I have no idea there was a cheat code in this game. Did anybody like has anyone used this knowledge of it? No. Okay. Well, at least I'm not the only one. This makes me feel better. Um, what I wanted to talk about though is like why is there a cheat code and how do we help ourselves, right? And what is

that cheat code when it comes to looking at malicious software or some of these things that we're seeing in the the software supply chain space? And that that cheat code, right, is what I've been showing a few times here. And that's software free tool, right? There's tons of them out there, but you know, I support one that has um six billion files in it. So, it's huge. We also are one of the largest contributors to um the open SSF for providing that threat repository. And it's just another tool you can use to really help bolster that security. Just a final inspection to make sure like, hey, is this the right type of extension I want to be using? Does it

pass test like What's the history of the malware on this? How often is it tampered with? So, this is just one free tool that you can use. Again, secure. Software super easy to remember. Um, and this has, you know, the six different categories. So, I've talked a lot today. Key takeaways that I just I hope you guys have fun with this talk. Key takeaways is every choice you make matters. So, if you're cutting corners, it will come to bite you probably at some point. You just don't know where. That could be in, you know, misconfiguration or legal, it's your choice. Um, collaboration is key. So, those three players I mentioned in the beginning, while they have different

goals, they're also working together and that's to help create like secure software that's just used by everybody. So, you need to make sure that you're collaborating with your peers whether that's in development, security, operations because at the end of the day, you do have one thing in common and we talked about that. Um, the other thing that I think is key is, you know, you can't eliminate risk. So, how are you going to manage it? Right? You you need to be able to understand that there is always a certain level of risk. It's just how are you going to manage it? What plans do you have in place for when something does go wrong? And how are you

going to rectify it? Um I think the last thing I just I always find this funny, but if you wouldn't rack blindfolded down the Columbia River, why would you deploy code you haven't fully vetted? And that's it. Thank you. Amazing.

Does anybody have any questions? We have a a minute or two for for questions. >> All right. Thank you very much for that presentation. >> Thank you. >> Round of applause for Katie.

>> All right, we're going to get things switched over here. Welcome to the newcomers in the back.