Developers, Developers, Developers: Your Software Supply Chain is Compromised!

Okay. So, this next one, no pressure here. This is um I'm I'm about to introduce my boss, so I got to get this one um right here. Uh my my job I'm I'm the chief technology officer for SAN's Internet Storm Center, right? And um I had the pleasure of picking up that position where we maintain over 3,000 honeypotss all over the world. Um and I was hired by a guy who created the internet storm center when it was dshield, right? Dshield was the original or was there a name before DShield? >> No, it was originally dshield.org. And what when was that? That was back in >> 2001. >> 2001 dshield um was created where we

have internet storm center handlers and all that stuff. Um, so it's my pleasure to introduce Dr. Olrich, Dr. Johannes Olrich. His official title is the dean of research at the SANS Technology Institute. He's a SANS faculty fellow, uh, the founder of the Internet Storm Center, dshield.org, where you get free uh, analysis. So, we have we have our honeypotss. You can download all of the data we collect from all of our honeypotss. Um, there is free threat intelligence. We also have internet storm center handlers who are analyzing that data and uh posting journal entries each day with things that are relevant. And of course then there's the internet storm center uh threat um threat I forget what we call the threat icon

which is either green blue lets you know what what's going on. Um I think that thing is only turned yellow or orange like 15 times since 2001 when it's created. So if that's if that's red it's going to be a bad day, right? because we've had some bad days with malware. It's only gone to yellow or orange. I think it was orange for like a half a second or something like that. Yeah. Dr. Witch is also the author of several SAS courses. Um he's uh an an instructor and an author for SEC 546 which is IPv6 essentials, co-author of 522 defending web apps and security essentials and f um and he teaches uh 503 network

monitoring and threat intection threat detection in depth. So with that join me in welcoming Dr. Johannes,

>> I think I think I need uh I need this one as well. Yeah. Okay, good. Good. All right. >> Okay. >> Have it all up there. Good. >> Yeah. First of all, thanks for having me here in August. Uh here I'm living actually in Jacksonville, Florida. So it was a reasonably easy uh drive up here and uh and also thanks for the introduction here by Mark. I think I'll keep you around for a little while longer. Uh but uh anyway uh so what I want to talk today is developers and some of the threats that developers are facing which is something that I think is a little bit underestimated somewhat. Uh over the last few months actually I

think it became sort of more a public thing. what really happens here with uh developers u don't really have to look don't really have to introduce myself because Mark already did a great job for me I originally uh grew up in Germany but um what um what surprised me a little bit is know why am I invited here for a keynote speech I guess Mark is it's Mark's fault somewhat but I've been in this industry really too long to give a speech like this, they should have one of the young kids here do it. You still look like you believe you can make a difference and such. So, um when you're doing this for 20 25

years, uh you get two kinds of speakers. Um you get the jaded speaker who pretends that you can make a difference and you can get the jaded speaker who well basically just jaded. I'll leave it up to you what I'll end up with doing here during this presentation. Uh hope a little bit still get a little bit to make a difference part across but um it it's a tough business. It's a tough business to uh really in particular from the education front. So I'm teaching as Mark said here for our graduate school in particular a lot for sans.edu edu and um it's kind of hard to see people making the same mistakes over and over

and in part you know that also is a little bit what that talk is about that things like no trust in tools um basic things then we had just earlier someone from the AI part here uh introduced himself and introduced the facility um anybody remembers buffer overflows anybody knows what the big problem was with buffer overflows

Failure to balance check is no that's not a problem. It's mixing control plane and data plane. It's mixing data and code. What's a prompt injection? Um it's it's mixing data and code. It's mixing control plane and data plane. But well, you'll learn it again. Eventually they'll fix buffer overflows and prompt injections. Uh but uh so anyway so that that's enough introduction. So developers, you probably all remember, you know, that uh that famous like know Steve Balmer. I'm I'm a bad Steve Balmer impersonator, so I won't really do the developers developers developers thing that he did at that famous conference when they Windows XP or whatever they introduced there uh when when they sort of had that big introduction. And uh but

the re the reason what I like about this is it really shows the importance of developers. Uh developers are those people that hide in their cubicles that uh don't really you know talk to other people. So they sometimes are a little bit forgotten and lost uh when it comes to security too. But in business in general uh where they're kind of ignored but uh at the same time developers are your supply chain. Everybody talks about supply chain risks, but what is the supply chain when it comes to software? It is your developer. Your developer puts the software together uh in the end. And uh that's you know here the big problem and we have day after day pretty

much the same problems in the news these days. I sort of picked a couple of recent headlines here. The one just from this week with like that big new get new. I'm not sure if you're familiar with this. There's like the .NET uh ecosystem. So like your PIP and npm for forn net, you know, that's that's sort of where that comes in. And of course that's what I referred to earlier when I said that finally sort of gets a little bit in the public um public uh conscious kind of uh this problem and that was like that big uh shy Huloot worm here that infected npm repositories at the lark. Anybody knows what module actually was infected there?

It was a it was actually a kind of a neat module. Um they call it um it's it's one where you if you display code as part of your software, it nicely marks up the code sort of know give it nice colors and such. But in particular when you talk about npm this module itself I think uh when I count it depends on like 256 other modules that you and um being a little bit an old-fashioned developer who isn't really used to quite some of these ecosystems. Did you know that one of those 200 something modules is just removing spaces from both sides of the strings that you're including? So um it's um this huge complex ecosystem. Anybody

ever played with Electron? Electron that framework you've done that you uh it's it's actually a neat idea. Electron Slack for example is a famous application written in Electron like the if you're using the native application there a couple other big ones but it allows you to use JavaScript and HTML to write native applications. a couple frameworks like that. Um, I played with it. I think the hello world application I wrote it something like 370 dependencies. If you're coding that you may know the exact number but um nobody's ever going to tell me that they verified all of those uh when they when included this really one of the big problems. uh developers are your supply chain and

at as part of your supply chain they have access to everything. Those people you never talk to or well some of you may actually hear the they're probably sitting in the back the developers uh but um anyway so um they have access to your source code which often you is your intellectual property. they have access to credentials in order to connect to to customer data. Um, yes, you know, we all as developers and I I'm kind of a developer myself, I admit. Uh, we're supposed to have special developer databases with data, you know, for developers to use, but uh ultimately you have to make sure that that data is actually correct. they have to have

access to real data uh to create that you know test data if you even use that and of course quite often we don't use test data we do use real data and um with that you know stuff like that gets leaked so all that confidential your development pipeline which is another important sort of business secret what's the next product we are going to develop how are we going to finally shoehorn AI into our product and make it uh a little bit more sexier or such. So, uh, all of that, you know, developers know about it. So, I I want to start with three threats here that I sort of see as important, and then I have sort of a fourth bonus

threat at the end, and I'll I'm going to cover them sort of inverse order. I'm starting with the what I consider the less severe ones, a little bit of the easier ones first and sort of uh work my way down the list here until we sort of get to really I think the big one here which is of dependencies. Uh but um anyway, what kind of really made me aware of what's going on here was this last pass compromise from a couple years ago. Not sure if you still remember the Last Pass. Nice software. They make this uh password manager. They were compromised and as part of the compromise customers encrypted password data was stolen

and actually then later it turned out because of well you know hashing encryption stuff like it's always a bit tricky to get right because they didn't quite get it right the way they probably had should have gotten it. some of that data was actually decrypted and then used by the attacker who stole it. So, how did that happen? We have our developer here. And uh in my talks, I always use dogs for the good guys and cats for the bad guys because anybody who ever had a cat knows um cats are bad. But um anyway, so um we have our developer and we have our customer secrets. How did attacker get from that developer in the lower right

hand corner here all the way up to the customer secrets? Well, um developers need workstations to work with and developers want to have fun. Anybody here when they're coding likes to listen to music or such? Yeah, it's very common. So, this developer had Plex installed on their coding machine. Um, okay. Watching videos while coding. I'm not sure. But, um, either way, that's sort of what they had here. Okay. So, what happened next? Turned out Blex here had a vulnerability, a relatively straightforward remote code execution vulnerability where you could connect to Blex and trick it into executing code. I think you just had to upload a file and get access. So it was a relatively easy to

exploit vulnerability. An attacker found this vulnerability and this was the it was an older vulnerability. It was a couple years old at the time it was exploited. But then again, you know, software like Plex or so that's not part of your corporate patching framework or vulnerability management system and such. Uh so it it just got lost. Who knows? Maybe he wasn't even using it actively. That also happens. I think I have a system at home where I have Blex installed just because I was playing with it and yeah, I got bored with it. Had real work to do and um it's now sitting there on the system. I hope I keep it updated. I should check when I

get home. But um anyway, so they exploited this vulnerability and with that they installed a keystroke logger. So standard, you know, kind of a malware uh that you commonly find in exploits like this. And with that keystroke logger, they were now gaining access to anything that developer typed, including credentials that gave them access to their customer secrets. So there was a specific encryption key here and some credentials to access cloud storage and such that this developer had access to. Once the developer used it, the attacker was now able to access that cloud storage directly download all the customers encrypted vaults and uh yeah rest is history as they say. and our customers most secret passwords are out there yes

in an encrypted form but and you know you can't hash them because you actually have to use them in the password manager password manager has to eventually know what these uh credentials are uh for a password wault and um that um was just a matter now of basically brute forcing whatever password the user used to secure that wault. So that was really the only thing that stood behind between the attacker and all of these secrets. Anyway, yeah. So, uh, one person essentially blew up the entire company. Now, LSP still exists. They survived this particular incident. Um, but the worst possible compromise for a company like this is losing customer secrets. And it's also a big part of the

business. I remember in the old day um like um I use one password a lot not much last pass but when they came first out you could synchronize uh your vaults directly like systems to system it just had to be connected to the right wireless system. Um I actually like that but the problem was you paid for the software only once this way. Uh now to make it a subscription they all make you use some kind of you know cloud service uh to synchronize uh your your passwords. So that means you know these passwords are now or these password vaults are now accessible uh to an attack like this. Could it be you could definitely be me? So this is definitely

one of those things that I think anybody here probably has some software on their system that they're using to develop code that um may not be strictly necessary to actually write code but is nice to have and whether that's various messengers that you're using some of them you know may be needed for sort of corporate communication but some of them you just may have like some discord or whatever uh to you know link up with other develop velopers ask questions and things like this that uh that you have here on your on your systems and then what you have noticed something like this happening and I'll talk later a little bit about some of the challenges when it comes to

protecting developers. So anyway, so all of these unrelated software that you have on your system, email clients, web browsers, which are a huge attack surface. Just uh last week, um OpenAI came out with their browser and as a developer, I haven't yet. I haven't gotten around to it yet, but as developer, I expect you to experiment with that to see, hey, what are they doing there? Is this anything that we can use? And of course not the day after they released it, people listed all the prompt injections and so that you have in that browser by just visiting the right website, you know, and attacker can take it over. And yeah, as a developer, you often deal with insecure

software. That's part of what you're doing. uh also um outofdate software and I don't do it that much these days anymore but I remember in some prior project I was working on where I always had to have some old version of Java sitting on my system because I had to test if my software works with those old outdated versions or old browsers and things like this you know that you uh that you need for testing uh and yeah that's again sort of a risk that you're accepting here as you're developing software. So how do we fix it? You know patching of course the obvious fix here is everybody's software patched all the time particular software that you may

not necessarily have sort of in some kind of corporate inventory or such. So um it's hard it's hard to get that right. It's particular hard if you have these unique requirements for developers like the need to sometimes have outofdate buggy vulnerable nonvulnerable software on your system because you need them for these tests and uh that makes particular difficult so sometimes you can have sort of these virtual machines with like the outdated stuff. I've done that plenty of times where you sort of keep an old Anybody ever had that problem with like the old um bias control systems like IPMI and old Dell DRA stuff and such that required a very specific Java version or such to run and um in those cases that

yeah virtual machine will do. So that's that's one way of doing it. Endpoint protection software. We'll talk later a little bit about the difficulties with endpoint protection software or network protection software and developers and some of the specific challenge you have there. One thing that would have helped in the last pass case is that idea of a privilege access workstation. Anybody ever used something like this or has experience? I see a hand going up back there. Uh back there not a lot of hands going up here. But the idea is well the developer workstation should never touch live credentials to collect to your uh to your cloud storage. And the idea of a bridge access

workstation you have one vers that is really locked down that you just use for these specific use cases. I find it works it sometimes works better in theory than in practice. It's uh it takes Burke to get it actually to work correctly and and set up correctly. And it often gets in the way too like you know if you have to move some larger code updates live and such where depending on again how you set up. The other problem that you deal with here is these de these things weren't created last year. Uh like my code base Mark mentioned earlier, you know, I started writing on that site, yeah, in 2000. There's still some code from back then.

And um, yes, I get embarrassed when I see it these days. Uh, but realistically, you always have this leg that you have to carry around with you. And that sometimes gets in the way of these a little bit more modern solutions. And nobody has the time to nobody has the time to rewrite all of their code all the time. So, next threat, developer tools. Developers need tools. Yeah, I still do a lot of coding in Emacs. That's my editor of choice. Um, some people prepare VI. I never really got used to that. Uh but um most time if you really want to develop well, if you want to develop fast, if you want to uh take advantage of a lot

of the modern stuff that you have like even vulnerability scanners and things like this, you probably use some kind of IDE, use Visual Studio Code, you use Jet Brains or uh you know all these different nicees that are out there that help you write better code. These idees are great, but there's always something else you could add to it. Dark themes. Developers love dark themes. Uh so, um the darker the better. Sort of dark gray font on a black background and and such. Um but and and you know, sometimes the tools don't come with those themes and such pre-installed or extensions that you have. So all of these tools have these extensive extension stores that you can

use to download these extensions, themes, and the like. What a lot of developers don't realize as soon as you load that extension or a theme, you give that code complete access to everything you're doing. And there's usually no real good way to sort of separate this. Even those different colors you need code to run. And um and this has become a huge issue. VS Code and uh Open VSX. Open VSX is an extension store for well VS Code version that aren't VS code. Uh Visual Studio Code um there are a couple of other IDEs that are essentially sort of wrappers around Visual Studio Code and they have their own extension store. And so that's uh sort of where where they come in. But

here an example. So you're a developer. You heard about this AI and wipe coding thing and um you want to start using copilot as part of your day-to-day development. So you need an extension to run copilot. On the left hand side here you do see the extensions that you s see at the top of the list. Let me say there's some obvious good ones like there's an official GitHub copilot extension here. That's probably what you want to do. But um then there are others that uh you know sound interesting too. And I'm not saying any of these these are I picked them like earlier this week. So I don't think any of them here are malicious.

Uh but how do you tell? How do you tell? Um, yeah, maybe you speak Chinese. So, um, cat GPT, the cat should give it away. Yeah, that, um, that this is bad, but um, other than that, uh, Blackbox AI, um, I have no idea. It seems to be popular. 4 million downloads. No idea what it is. uh Bad Boy 17G and I think these were sort of the second or third page of this these weren't like all the way down sort of on the list uh those extensions that you have here. So yeah uh it's hard it's hard to figure out and I have some examples here. So these are some malicious ones here. These are dark

modes um that you can have um tracular dark. Okay. Um, not sure about no 42,000 or 45,000 installs on this one. And that one actually turned out to be malicious. This one here is on the left hand side the good one on the right hand side. The malicious impersonator for a particular extension. Um, you can tell the difference. Uh, so here it is prettier. Make the code look prettier. That's kind of the goal here. Um, last commit. Yeah, same same number of pull requests, same number of open issues. The only thing that sort of gives away this one was released in 2017. This one here more recently. Uh, but not that recent where you would

say, okay, you know, it's probably something malicious. And then of course the other thing that was sort of with the npm issue a couple weeks ago where a good extension or in that case a library goes bad. That happens too where developers credentials get compromised and then someone is taking over a popular extension and and making it malicious. So that's that's but yeah there's no way that you could tell really here the difference between those two. So uh I mentioned already recent events. Uh this was the latest greatest one of these class warm and um the biggest extension being affected here was called code joy. Um code in Cllingon was another one that I kind of like. My wife is a a little

bit of trekky but uh she would probably tell you why you shouldn't code in Cllingon. Um, did you know there's a big effort out to actually have Cllingon finally added to the Unicode character set? So, uh, but, um, I'm not sure if they're using it. Maybe in there. But, but one real interesting trick they have here is they used Unicode characters to actually have some malicious code in transparent font. So actually they had encoded strings in essentially white spaces and then code that decoded those white spaces in the code to execute. So when you look at the code for this extension there sort of just a looks like a block a list of empty lines in in the code.

And um this of a little bit reminds me of um when cross-ite scripting came out first. Um this was like uh back in the late 90s when I first ran to cross-ite scripting. And initially I sort of thought, hey, what's the big deal? People are exploiting themselves because they're clicking on the link and then that code comes back to them. You classic reflective cross-ite scripting. Uh I think with the unic code stuff we're a little bit at the same level right now where um you know everybody focus on the URLs with like the lookalike characters and stuff like this which actually most browsers doesn't really work well these days because they don't really display those uh characters

uh but in developer tools in editors did you know that many language like JavaScript swift are uniode compliant Uh you can write code in other languages which again you know if you don't speak English that's a great thing. You can finally use the poop emoji as a variable name. Uh so um but with that come things like left to right and right to left. You can code in right to left instead of left to right. if you set the right unic code character to change the direction of um of the text. So there's a nice example where code looks benign when you're reading it left to right but when you're reading it right to left as the parser will

interpret it will actually be malicious. You can think about you know when you sort of change a crater less than or something like that you where uh where that sort of you know changes a comparison of like a password or uh something and opens up a back door and this is I think another nice example here uh where uni code is being used to basically obuscate code here in in new and interesting ways. Another thing I liked about this particular event is they um they used Google calendar as a command control service which I think actually better than that blockchain thing. They also like added URLs to the blockchain which yeah is neat but blockchains are really just for

scamming people. Anyway, so um the um Google calendar is one of those things like particular me of being a little bit of a network nerd and ids nerd um hard to detect that someone is using Google calendar and sort of built some indicators of compromise or such around that. Yeah. And then uh what was also interesting here is they were like all of these tools still developer credentials that sort of know basic uh table stakes kind of but they were particular focusing on credentials that developers use to publish their own extensions to these marketplaces. And then of course they can turn a good extension into something malicious. maybe something that you wrote yourself to share, you know, in your company or

something internal that they then add there. And that's sort of the warm part here of this particular uh malware that it actually turned good extensions bad by stealing developers uh credentials. Yeah. And then the other stuff is pretty normal fair for any kind of you know malware some kind of you know remote access trojan here uh hidden BNC and stuff like this. That's actually I think if if I would write that software and you know by the way you can tell that I didn't write that software I would remove that stuff because it just makes detection more likely actually if you have somewhere hidden BNC in there. Um but anyway, a little bit AI here.

We now not only have developers, we also have people called data scientists. They're like developers, but they know less about code and security. And um in 2003 we noticed in our honeypotss attacks for example against geio server. Uh geioerver is a tool to manage geographic data. Uh the other thing we saw was um what's the tool here? Again I draw a blank now. Nfix nfix. Uh it's a data management tool. It pulls in data from various sources and then you know outputs standard JSON or whatever you want to um to basically be used in your machine learning model or whatever. Uh this tool is particularly interesting. It gets you the this nice little uh web gooey web- based guey where you

can sort of arrange different modules that bases of your data pipeline you can build there. It's a Java based tool by default when you install it. Who needs credentials? Um it's it's just open to the world and you can just upload your own modules which are code. So it's it has remote code execution built in. It's not a bug, it's a feature as they say. Um and um yeah and that stuff attackers keep scanning for more and more. I say in 2000 so two years ago we sort of saw this uh increasing big times these scans earlier uh this month or last month SISA actually demonstr focused on a nice attack where this was actually exploited

um where an attacker accessed your geo server in this case here they actually used a vulnerability like I said oft you don't even need a vulnerability they're just exposed to the internet those systems and um in particular when you're looking at, you know, how these systems are often set up these days, uh, you're not setting them up um, like in your little lab at home because data scientists usually don't have labs at home. They do it all in the cloud. So, they just spun up some virtual server in the cloud and then uh, install the tool. It's now hidden by IP address kind of. So, that's the only thing between the attacker and remote code

execution. Yeah, in this case um that CISA documented they compromised geio server and uh then basically used it sort of for lateral movement because that geio server now has access to all kinds of other systems in in your data pipeline and uh with that you know you basically can just access whatever uh you want to. Okay. Oops. The monitor is acting up a little bit here. Hope uh it's visible in the remote. Let me see. Okay. Yeah. Um other part of the pipeline here uh DevOps Jenkins as I some have it seen referred to the word press of DevOps when it comes to vulnerabilities. Um it these are complex tools. These are complex tools that again you know

execute code that touch code that you write the touch credentials in order to build containers and and move them live and then that people expose them. I think the only thing actually keeps something like Jenkins particularly secure is not exposing it. But that's again hard if you know you it's all in the cloud and all remote managed and such. So not exposing it can be hard. similar uh atlation conflence every month there is one or two vulnerabilities uh in these tools that are being used and um a second going to make sure that it's still visible here. Yeah.

Okay. Yeah. And uh because exploits are so common in those tools, their exploit frameworks written just for it. So whenever there's a new vulnerability, it now becomes a lot easier to exploit those vulnerabilities. This is vulnerabilities are similar from each other. And then once you have that access to the tool, you here like ghost jobs. It will basically can add jobs to Jenkins that will not show up in the console. You can dump credentials. Also interesting, you can basically capture here with this Jenkins attack framework the output of the last job that ran the console output, which often is like error messages and stuff like because it's code that doesn't really work well yet. And of course, for

developing stuff like this, you're more verbose in what kind of error message you get. So there may even be a credential or something like this in there, you know, that uh that wasn't uh removed. Yeah. And then basically just manage Jenkins for you. I would love someone would manage Jenkins for me. I have to expose it much easier than doing it myself. But um anyway, yeah, famous compromise of this kind of thing Jet Brains. Um Jet Brains I mentioned before, you know, big sort of developer tool manufacturer. Um and very popular actually really nice tool. They have one tool they have is team city which allows basically people to sort of work together on projects. Yes, it also had

vulnerabilities. This particular vulnerability was exploited two days after the patch was released. Uh so who is patching all developer tools within two days? And in particular those complex tools like um Jenkins and such uh they're not easy to patch. It's not something that you auto patch because you customize the tools. You write your own code that interacts with them, your own scripts and such. So, you know, whenever you patch, you have to make sure that those scripts still work and that it didn't like change a feature that you ran. So, how do you protect all of this? I already mentioned, don't expose it to the world. That's probably kind of your biggest biggest thing that you can do

here. Uh, patching is fine, but like I said, it comes with a big caveat. You cannot patch as fast as they're being exploited. So, you really have to buy yourself time. And that's you do that by isolating the tool. Like the isolation I always say doesn't prevent the exploit in the end, but it buys your time. It makes it harder. Makes it less likely because the first thing they're going to hit is everything that can they can just sort of know scan for. Yeah. And then uh that's the other part is make sure the patch are applied correctly. I see this in particular with the Jenkins stuff where you know you may sort of have hit the update button but

the patch didn't apply because some files were open and in use and couldn't get overwritten and such. So uh always make sure these these are complex tools. Yeah. Yeah. Other defenses here that's sort of I'm not sure how many of you are sort of on the security side of this like security teams and such. A couple Yeah. And then then we have developers on the other side who sort of more developer versus security. Anybody here sees themselves as developer in the back? Of course you I see the developers as I mentioned earlier. Uh but uh there's often a little bit tension between security and developers because developers want to get work done. Obviously developers are nice

people. They some have a hard time thinking evil and figuring out what could possibly go wrong. So it's really important to sort of set up that partnership uh where the security team assists developers in getting their tools secured. It doesn't just tell them, hey, you can't use that tool or that library because developers sometimes also really creative in finding workarounds for that and then they use the tool without the help. Um, so develop some guidelines with developers, include them so those guidelines actually work and don't get in the way as they're trying to develop code. Yeah. Dependencies. That's like I said sort of the big big thing here. So developers last night at dinner talking about so

some of the early computing experiments. My first computer was actually a Commodora 64. Anybody Anybody old enough? Couple of you here uh played with it. Have Have you typed Have you typed in like hex code from some magazine? Yeah. Uh so that's what that's what he did back then. I remember like there was this one assembler uh that I typed in. It was like sort of know 30 pages of magazine. You type in 30 pages of hex code uh to to to make that work. Um we didn't have much dependencies back then. There wasn't really so much of issue, but we probably didn't really know what we're typing in when we're typing in that hex code.

Um, these days developers don't develop, don't code. They take libraries and then duct tape them together. That's how I code. So, I'm not I'm not blaming anybody here. That's just how stuff works. Um, and I mentioned earlier like know with Electron and such know this all the frameworks are the same. Yeah. um just to get started usually download like hundreds of dependencies. Some of those dependencies are like these minuscule pieces of code that you could probably have code in the time it took you to download and install it. Uh but um yeah, so now you have that dependency sitting here um on your system and um anyway so pip as an example. Now here in this talk I want to

specifically focus on attacks against developers. So this goes beyond including malicious code. What happens if you install a pit package? Well, you would think, hey, I'm just downloading stuff and then copying the right location. No, there are setup scripts that run or that can run and they can run any code as a developer. That's whoever is running it. So these setup code scripts basically run whatever code they want and you need them. Some of these packages you may need to compile something or such. Um, so you may need to adjust them to your system configuration and and that's what these kind of tools are for, what these scripts are meant to do. Now, there are two ways you can install a

patch in Python. They're these wheel packages. They're really more kind of what you think where it just copies stuff into the right directory. So, they're kind of safer in that respect. And then you have tar packages which basically just download the source code then compile it. So you would say hey developer just only use wheel packages don't use these tar.gzcz they're insecure only use the pre-ompiled packages. And here sort of just for those of you who think you can review the code I have sort of a little uh setup script here a malicious one. You probably know it's malicious because it's obuscated like this. Um yeah, this particular one extracts secrets from the browser, exfiltrates

them and then um steals Discord data. A lot of the tools target developers are after Discord data because that's what developers love to use these days. the kids to to communicate with other developers. So this is how an attacker then gets into the communication channels between developers and sees what they're working on and uh have you ever like you know in Slack, Discord or something like this uh send like a credential to another developer. Hey to connected to database just you know use this password. So uh that's what they're after here. gaming credentials because developers love to game and uh so they take them as well and I think that's really just to get sort of their game accounts for

other malicious stuff uh and screen capture. So you told your developers, hey developer, only use wheel packages. Well, here's an example. The cryptography package is very commonly used package. Pretty much anything that use like SSL or anything like this in Python needs that cryptography package. For ARM, it's not available as a wheel package. For x86, yep, you can do the wheel, but not for ARM last time I checked. Maybe they fix that now because ARM is becoming more popular. But I was working on our honeypot stuff where um you know we running them on Raspberry Pies and such. So we need the ARM version of the package and uh yep um not available. So we have to do the source

distribution. We have to compile it. Entry points. You may have seen this in uh Python. When you install a Python package, it often installs these uh little command line utilities that you can use to sort of get some of the functionality of the particular uh package sort of as a self-contained binary. Uh like if you install some image manipulation Python library, it may come with like a little tool that lets you run this. Well, these are called Python entry points. They're basically little scripts that are installed in a location that's in your path. in um on your system. So now when you type, you know, manipulate image, it calls this. What about an entry point called ls

or something like that? So um I can trick you into executing code this way just by installing a package again by installing these entry points on the system. Hey, so PIP, one of the big ones. NPM, you another big ecosystem. Um, again, Discord is a big target here. So, these particular packages here, they're actually meant to sort of, you know, help you write little bots and such for Discord are stealing Discord tokens. And this is particular tricky because now even if you would review the code, you say, "Oh, okay. this particular script needs access to discord credentials. Well, I want to write a discord bot with it. So that's actually kind of normal but later it also exfiltrates those

credentials to some other place. That's where it gets more tricky. Yeah. So in npm we have these in again sort of installed scripts package.json JSON like before you compress it, pre-ompress, postco compress, there of different scripts that you can execute as the package is installed on the developer systems. So pretty much the same thing and all of these ecosystems need something like this because they all have the same problem that just copying the files in the right location doesn't always work. You need to run some code, you know, to adapt uh what you just downloaded. So yeah, you could review the script section which again is uh okay. There's an ignore scripts option. So again, you're telling your developers

always run the ignore scripts option and um well and then a lot of the packages won't work. Yeah. Here's another example. So here where it s of runs this node index.js JS that's the malicious script here you know not going to necessarily it doesn't stick out as anything malicious then you have to look at that script and then maybe you see the obuscated code and even if you deoffuscate there's really a lot of benign stuff in there that doesn't necessarily immediately uh send alarm set alarm off alarm bells your sh key they're stealing secret SH keys that may be the one line here that's a little bit odd kind of Yeah, NuGet. Same thing when it comes to

um .NET I mentioned NuGet earlier. Um same basic problem. We have these installed scripts that can run arbitrary code on the developer systems as they're installing it. Anyway, I promised you a fourth one. So, uh AI. Everybody wants AI. So, yeah. Um, I got some AI for you here. What happens when you download an AI model? Many AI mods you download are pickle code Python. So, Python pickle code is what you're downloading. You're downloading a Python script. you're not downloading just data as you're installing this uh m this machine learning model. So you're going to hacking phase the website where you download these and um just like all the other ecosystems same problem anybody can upload stuff including malicious

scripts and then as you're downloading as you use the model you're actually basically running eval again you know combining code and data we have that model that looks like data but then we call xc or eval or something like that on it you that'll actually execute it as code. Yeah, in Python like Python, PyTorch, that particular machine learning library, it has sort of that um weights only equals true, which basically only use the data, not the code. Uh that actually didn't work right and uh you still executed code. There's a guy who who has a real good YouTube video about this. I I forgot his name. Um Mark Bagot. Yeah, Mark Bagot. Um he he has a YouTube video about some

of that. Yeah. And um so if if you search for him about some of the risks here actually basically you do serialization here which another sort of mixing or des serialization. It's another mixing code and data kind of problem. Yeah. Um and then of course we have AI coding assistance. Um actually just uh back to this here and it's often not clear which code is written by it and not written by it. Um again it's uh something that you can't really tell your developers not to use or not to do. You want to put some guard rails around how it's being used. And then the good old you know garbage in garbage out problem. We had one of our uh SANS edu students

recently write a research paper and one thing about these copilot and these these tools is they scan the code that you already have on the system uh sort of as a baseline uh to figure out like your coding style and various things that you're working on and such to better suggest code. The problem with this is if the code you already have is bad, they create more likely bad code. So this year what came out of this um probably used a little bit more data but um he basically you know had co-pilot create software one on a system where he sort of in pre-installed some bad insecure code and one where he had secure code

and you see how the insecure code base on developers system did affect the quality of the code that was created. So yeah, it makes worse developers worse or bad developers worse. So what about endpoint protection for developers? Anybody knows Vizou? I like to play with that. It's great tool. Yeah, I I don't want to diss any tools here. Uh but uh one problem I have with Vizou on my developer workstation is I use a lot of Docker containers and then you end up uh in docker is with all of these sort of files in slashdeaf on your Unix systems which yes are a legitimate sort of you know indicator of something bad happening not very docker

containers. So you have to tune them you have to tune these. This is something I ran into um with my cat. So, um, my cat has a little bit problem. Um, she's a little bit overweight. Uh, so my wife came up with that, um, that's sort of a cat feeder that sort of doses like how much food she gets. And, um, every hour or something like this, the cat feeder tried to talk to BU and um, some ids sort of went off on that. uh turned out it was just a random Python library that this uses that uses B to check internet connectivity. it's probably written in China and you know Google doesn't work there so they

um they used BO instead and that stuff again on developer systems when a developer use that library all of a sudden you have sort of beaconing it was classic beaconing behavior was not malicious as far as I can tell but um certificates how do you manage certificates for developers um I like to develop like TLS on my development websites too and uh you need various to connect database you need certificates uh so developers often manage their own certificates or you have some uh not that tightly managed authority for it because it has to be flexible. So you get a lot of like self-signed certificate alerts and such on it. I do get uh crowd strike which is tool I

use. It sometimes it sometimes flags when I crep on my bash history. Um any developer ever craps bash history kind of uh so uh because you get sick of doing the up key kind of thing. Uh so that um that's another thing where you have to tweak your detection rules know for developers because they work differently than your average uh office machine. This is a quick conclusion. Don't ignore your developers. Help them. don't become their enemy. Uh I mentioned so with extensions such but also with libraries make it easy to add new libraries to the allowed uh catalog. You rather want to know that they use it because if you don't let them use it let me show you a little

developer trick. Copy paste. Um so then you end up copy paste. So then you have the code but you don't know that you have it. And uh so so make it easy make that partnership work. make explain show demonstrate why it is a problem to do insecure things and uh do your best you can to scan for it you won't find all the problems. So the hybrid approach endpoint network you know scanning um is probably going to help you the best anyway um do we have a couple minutes for questions? Okay good. So, any questions?

Nobody. Okay. I do have stickers also. They're out at the sand pool with Oh, sorry. >> Oh, sorry. I didn't see that. >> Okay.

>> Just shout and I'll repeat it.

Uh what's the second part? >> Yeah. >> Yeah. So, uh how does the vulnerability work? Because the parser compiler for this language is uni-ode aware, it will then tokenize the other way around. So this only works in languages that are actually uni code aware. The question here was how does it work with the light right to left versus left to right and you know how does the tokenization work and the tokenization is not left to right in that case it's right to left. Does that answer the question?

Yeah. >> Yeah. Yeah. So, yeah. The the comment here was just, yeah, it's it's not what you're usually doing. Use Emacs as your editor and you won't have that problem.

Actually, Emacs is UTF8. So, Emacs may have that vulnerability. That's why >> that's why you use VI. ED ED may be safe. Uh I think VI does UTFA too. >> But uh Okay, good. Yeah. So if anybody has any more questions, I'll stick around for a while. My information is here on the slide and there are internet storm center stickers at the SANS booth uh if anybody's interested. Uh so thanks everybody.