Developers, the weakest link in the supply chain?

We have David Leadbea who's going to be talking to us about developers, the weakest link in the supply chain. Thank you David. >> Thank you. >> Yeah. So, hi, I'm David. Um, I work for G Research Open Source and I'm talking about the supply chain. So, what is the supply chain? Well, this was a bit of a massive problem in 2021 when um the Evergiven got a bit stuck in the Sewers Canal and obviously the impact on the physical supply chain made the news. Um more recently, some supply chain attacks on things like npm have made the news as well. Um but this one was possibly more impactful uh across the global economy. Um, so here's an iPod Nano from 2006.

And on the back, Apple put designed by Apple in California, assembled in China. Now, legally, they probably have to put where it's made, but they decide for branding reasons to say designed in California. And why do they do that? Well, you know, they've got a big office in California. They're a Californian company, but also this means that they're writing the software and making the hardware designs in California. So more more of that goes into that iPod is not just the hardware but the software inside it. And so that brings us to the software supply chain. So I'll distract you with some chocolate for a minute. Um why? Well, if you look at the back of it,

it's got some ingredients on it. And you know the ingredients say things like full cream milk, sugar, cocoa butter, and so on. But also just below that it says may contain wheat, gluten, peanuts apparently. Um so that's because they share the tools that they use the production line. They share it with other chocolate that they make and so there's the chance of crosscontamination. Now for bits and bites we don't quite have the problem of crosscontamination but maybe we nearly do like some of the things that can happen are sort of unexpect unexpected crossovers between um technologies. So that's what I'm talking about. So let's have a look at some software ecosystems. So if you're writing Go

code, you have a go.mod that specifies what your dependencies are. Um in Rust, cargo, um package.json for Node.js and some other JavaScript ecosystem stuff. Uh Python, you have pi project.toml, although if you're familiar with Python, you have various other options as well. And that's kind of a problem as well. It's not a standard way of specifying what your dependencies are. Um, which doesn't matter when you're building software, but when you're analyzing software for security problems, that begins to matter. So, something recently has been developed called software bills of materials or sbombs, and there are various formats for those. So, one of the formats is the SPDX format. Um, there's also cyclone DX. Um, and basically they they say what your

dependencies are. And so they're a standard way of extracting the dependencies from the ecosystem specific um files and other other places that that data is stored and turning it into something that can be analyzed by a system. Um but that doesn't capture things like tooling. So there's a lot of things there that are up to what the developer does and that's where we sort of come to this talk. I'm not really going to talk more about sbombs because there's a lot of things that sbombs can do and that could be a whole other talk, but they're an interesting area if you haven't looked into them. Um, particularly for sort of open source security and keeping track of your

dependencies. So, let's go on to some Go code. So, I've mentioned dependencies. Um, and when you import some packages in Go, this is the syntax you use. You import um strings. So, strings is part of the standard library. That's pretty normal. Um, okay. Also, import something off GitHub. So, how does that work? Well, you just put the path there and it magically imports it. Um, but you can also put something like git.example.com and then that will also work and go will go and check out the code from that remote system. Um so how that actually works underneath is if you start looking at this you when when go sees that URL it expands it into an HPS URL and then

in the actual source code of that URL there's a go import meta tag and that tag explains what that for the package on the left hand side there's a particular version control repository type and there's a particular location that that is to be found at and if you go and read the docs on um for modules in go it explains that if the URL doesn't have a scheme then it will fall back and find the attempt to find where the code is by trying various options um and so it also mentions that there's a module path um is matched with the go insecure environment variable then it will use other protocols that are not secure like https and git so that brings

us to the first vulnerability there was a bug where basically go just didn't do what was documented. So if um you had in this example the the last one where you've got fu.git, if that particular package path was used, it would attempt to check it out as it should do over HTTPS and then attempt to use git and SSH combined. But if those failed, it would fall back to HTTP. And so what's the attack there? Well, the pretty obvious one is some kind of man-in-the-middle attack. It's an attacker can't can't change the HTTPS certificate or get that, but they can via a downgrade attack force the client to not be able to connect over SSH or

HTTPS and then go fell back to HTTP. Um, so I found that in 2023. Um, and it's actually not a very serious vulnerability because Go has various mitigations here. um the when you add a package to a go u module it basically says what's the what's the check sum of this package I'll download that and then it saves that in a separate file called gossum which means if you have already checked out a particular package it can't change and it essentially is immutable um so it also need a particular environment variable set so go has this google service that it uses automatically called the go proxy which is a basically cache repository of modules source code. Um, obviously you

can't attack that and it's always over HTTPS. Um, and it also has something called the go sum database where the as well as the local file with go sum there's a there's a central repository of what all the expected check sums are for particular module versions. So overall it was a very low severity CVE but it does show that in very unexpected places there are ways of attacking the supply chain. And you know the amusing thing about this is imagine a developer at a conference or somewhere on open Wi-Fi. It would be possible to attack them through this kind of mechanism. It's a bit of a like esoteric case but there are there are potential exploits

there. Um so go may be a polarizing language. Um you know many opinions but it is well designed. There's there's defense in depth here. So although there was a vulnerability where it it attempted to use something that was insecure, it it had multiple check sums on top of that which meant the impact was quite limited. Um and just to sort of mention a bit further, so in a go.mod file you have something like this where it declares the go version it uses and go now will automatically download the correct version of the tool chain that has been used for a particular module. So the neat thing there going back to tooling is if there was a vulnerability you

essentially upgrade the tool chain in your go.mod and then it will automatically download a new version of go. Um the obviously the problem there is that you do actually have to keep that up to date and so it's not perfect but um it is a good way that it captures the versions of things that were used which like I said for sbombs that might not always be captured. um you know the buildtime stuff sometimes isn't captured in sbombs and there's some other um things in that space. So moving on now um so open SSH I mentioned already that um git and sh are often combined but we'll just look at SSH itself for now. So here if I SSH to example.com it asks

me for my username and password. Fine. Login example. Good. That's just how SSH works. Uh enter DNS. So worms if we do this. Well, that's silly. Why would that work? Name servers not known. Okay, let's try that on a different system. Oh, that's strange. Why? What? What? Um let's try that on yet another system. And so on this system um it actually says the authenticity of host can't be established. That's a bit strange. Okay. So what's happening here? So this host does resolve. So this isn't my test at home. Um there's a wild card DNS entry. So anything before wild works. So fine. Um what's actually happening here? So on the on the system where it said host key

verification failed um it says invalid host name. So there's some checking in SSH that checks whether the host name is valid. So the the thing here is that what's valid in DNS is a larger subset of characters than what is valid for a host name. So you can have entries in DNS that aren't a host name. And that's fine because if you've used things like SRV records where they sort of have an underscore, technically an underscore is not valid in host names. So this is kind of expected and used although there are lots of edge cases here as you can see. Um, so this was a bug in open sh that I found and this was back in 2023 again.

Um, and basically the there's a long description there, but basically if we go back to the the host name, it's got a comma in it. Now, if you're familiar with how SSH records known hosts, so it uses the time of first use check. When you first SSH to a system, it it asks you uh this thing where it says the authenticity can't be established. There's no certificate CA there. it just asks you to yourself verify that fingerprint. Um there are other things you can do like using DNS. Um there's more to be said there but I haven't got time for that in this talk. But basically when it when you then say yes, it writes that to a file. Um and so what

would happen is it would write the comma to the file along with a wild card before it which meant that what the user had just accepted became accepted for any remote host. And so if you somehow trick the user into accepting a weird thing with a comma in it, that's a bit strange. Maybe a user or developer won't do that. But I did find one one weird configuration where SSH used DNS to canonicalize host names and it would it would be a bit of a strange configuration. But there would be a case where someone could man in the middle DNS which obviously in most cases is not authenticated and could essentially trick the user into connecting to a

different host and it would bypass the time of first use check in SSH. So yeah, configuration dependent. Um this actually gets more fun though. So what happens if we put a asky escape character, sorry, an escape character in there and it's got a it turns it red. Well, it turns out at one point SSH actually outputed that to your your screen and so obviously this doesn't really matter. Um, I'm not going to accept that. But what about if we put the entire host name to be what the sh will print? So it says your authenticity of hostg.com can't be established. And if you look closely, this starts to look a bit fishy because it lowercased

everything. So, it's not a really great attack, but I've I've replaced the SSH time of first use banner with something else via injecting it through DNS. Um, it sort of could potentially confuse someone, but as I said, it's all lowercase. Um, it I didn't even fix like some of the weird uh cases there cuz there's also a length limit in DNS, so it's really hard to just about squeeze in the thing here. But, but anyway, it does sort of work. Um, but it's not, you know, it's it's it gets down to like with the npm attacks recently. Developers were fished by accidentally replying to an email or clicking a link in an email that came from npm.js.help

rather than.org. Like maybe this could fish someone. Um, but well, first of all, they're not going to be shing directly to a host name like that because well, probably I mean, if they're doing that, we've probably lost at that point. So um anyway there was an another fix for this that I realized later on that actually it previously wild cards were disallowed. The comma wasn't. So if you noticed in this variant um it's github.com comma and then a /b which is a backspace which hides the when we look at the output it hides the comma there. So if you accept this, it replaces an accepted SSH key for github.com with something that you have provide you have you have access to

because it's your host that it's actually trying to connect you through wildcard DNS. So um anyway, and you you could maybe come up with something a bit like this where it's like optalsample.com make that look believable sort of maybe. Um so how do we how do we fix this? So there's obviously the SSH side, but the other thing is like I said, what's valid in host names is different to what's valid in DNS. And this was a commit to OpenBSD. Um, OpenBSD restricted the characters that their resolver allowed resolving. Um, so that you can't put a comma there, which fixes this problem at another level and you know, good defense in depth. Um, so actually this can't

happen on GIC C, it can happen on Muzzle. So, a bunch of those demos I was just doing just then were using Alpine because it uses muzzle. Um, OpenBSD has now fixed this. Mac OS, I've reported several issues to Mac and Apple and they're not that interested in fixing things unless there's an obvious r unit. So, uh, Mac OS hasn't been locked down. Windows already does filter this and has for a very long time, although there's some different edge cases there I won't go into. Um, so let's move on to Git. So, I mentioned SSH combined with Git. So this was a vulnerability that someone else found back in 2023. Um, credit to Vinci. It's cut off his name at the top

of this, sorry. Um, so anyway, the this was using proxy command in SSH. So it's again, it's a very particular configuration and that particular configuration meant that um when you connected to a particular host, the proxy command would allow escape character injection and um that sorry shell meta character injection. Um and anyway, so there's a proof of concept here. So the the idea here is when you get clone a particular repository, um it with this vulnerable SSH config, um it uses subm modules. And so subm modules are where you have a git repository that refers to another git repository. And probably the best way to show that is with a output of GitHub. This is just a small screenshot

to a bit of GitHub. And you you'll see in this repository it's got CVE at some commit ID. And so one repository refers to another repository. So the the CV we're talking about here was basically having the SSH URL was Bactic's open calculator. Um and so as you would expect if you if you configure it like this and then you just clone his test repository. Um well that happens. Um so again this is very configuration dependent. The user would actually have to have this very particular sorry wrong bit uh this very particular SSH config which is uh you know possible but um so anyway g has interesting attack surface. So this is an example of

another bit of attack surface in git. So git has some other URL schemes. So I've mentioned SSH and https. It also has this Xed URL scheme which basically just means run this command. Fine. Okay. It doesn't let you do that off the command line. You actually by default it's turned off. You have to configure it. Um so how does get work? Um so Git has this nice feature where if you can't type very well because you know you're like me and whatever. Um anyway, so you can't type clone you you can set this um config option called help autocorrect and it helpfully auto corrects you. And so it says warning you called a command called clon um did you mean clone and

helpfully runs it for you. So if we get trace a clone we can see that underneath git runs a command called get remote https to do the clone. Okay fine. So what if we just put a dash there? Yeah it turns out that works. Um so it continues under the assumption that you meant remote x. No, you hadn't even typed remote-x. It just added that for you. Um, so fine. Um, so yeah, that does what you'd expect. But before you get excited, this is not actually a real problem. You can't use X from subm modules at all. It's another another level of things that blocks this particular attack. Um, so you'd have to somehow um engineer the user into

actually running that command, which is a bit strange. And you know, at the point you've got code execution on the user, then they could have done something else. So that's an example of what could be possible, but that's not actually a vulnerability in itself. Um,

what has happened here? Oh, that's exciting. We haven't full screen, right?

Oh, sorry about this.

Just Linux problems. Oh, there we go. Okay. Um, so I use this on my blog post about what's coming up and I I quite enjoy, you know, vintage technology. So, this on the side of it is a typewriter. On the side of it, it has a carriage return lever. Now carriage return is an interesting character that we have as a legacy of ASI and if you've transferred files between DOSs and Linux um you probably come across that you have a CR LF ending and then you depending what you're looking at you see a bunch of carrot m or something else appear that you don't expect to be there and you need to convert the file or whatever. Um

so this vulnerability used that very character. Um so how does it work? So get subm modules are as I've said sort of the vector for a bunch of these things. Um in this case I just try to add a subm module that um when you when you do add you can give an option optional name for it on the end. So I'm trying to call it / r which is carriage return in uh c style escaping. Um and then it fails. It says not a git repository. Um but notice that it's it seems to have stripped the last character there which is interesting. um but it hasn't in the middle because it's got this question mark here. So that's a

bit strange. So what happens here? Well, if I instead add the subm module and just call it foo directly and then I move it. So I I use git mv um in git modules then you'll notice that it says path equals fu and controlm. Um like I said that's what the code return looks like in depending how you output it. So what happens here? So when git goes to par that subm modules file because it sees a doss line ending it strips it and so when we when we see when we run git status after doing that we see fatal can't change directory to the thing ending in fu no such file or directory. Well that's

interesting. So we we've sort of confused git. It thinks the file is called one thing but it actually has another name. Um so what if we make foo exist? So in particular, what if we turn foo into a sim link to the.git directory, which is where git stores all its important stuff and then we create a hook in there called post checkout and um chmod plus exit because it's a it wants to check it's a shell script. Um anyway, so if we do that um we we clone a repository. So I'm using GitHub desktop to clone it here. So this this this works in various contexts. Um but definitely for a GitHub desktop. Um cloned it. Okay. Clones. Oh,

interesting. So yeah, basically this was an RCA against git itself. Um which worked both on client side things. So if you cloned on the command line, you include minus minus recursive, then you would get um a bad repository could do bad things. Um, also through GitHub Desktop. GitHub Desktop by default turns on the um the recursive clone option. Um, there's a URL at the bottom there. Sorry, that's not very big. Um, to the full write up, but I'll it's just on my website, so I'll have a URL to that at the end. Um, there are other places that this could be used though. So, GitHub actions, for example, um, has an ability to check out Git

repositories. That's what you'd expect it to do. It underneath just uses git to do this. So you you do that with something like this. And so if you did have with subm modules true, then potentially if someone were to open a PR with a bad subm module, it would run that code. But that's actually fine because you expect that code that is run through GitHub actions by default runs on that user's um for repository. So they hopefully don't have any extra permissions there. So that's not so bad. Um but there's this lovely feature that GitHub actions has called pull request target. And if you were to find something that uses pull request target to check out the subm module from

basically the other um branch. So it it runs it runs in the pull request target runs in the nonforked repository against if you specify it against the forked repository. It's generally a really bad idea but um it turns out people do use it a bunch. Um so here's like a you can code search for this quite easily. I haven't actually checked whether any of these are vulnerable to this and GitHub because I reported this through GitHub. Um, GitHub actually had a new version of Git installed on GitHub actions basically at the same time it was released. So, there wasn't really any risk here. Although, you know, the bad actor discovered this then yes, maybe this could have been used as yet another

supply chain attack against GitHub actions as well as a client side attack. Um, and you know, as we saw in the previous talk, people like North Korea are trying to get developers to do evil things. So um yeah, so GitHub actions is a bit scary. Um that that particular image was from um Yerion who I'm probably saying wrong, but he also has a a open source project called Zismo, which audits GitHub actions. And one of the things it does is it say you're using pull request target. Don't do that. It it doesn't actually have any it doesn't say why you're using pull request target. It just says no. Um, so yeah, that if you are using GitHub actions, that's that's

something worth looking out for that. Um, so did I get a juicy bug bounty from GitHub? No. So little rant here. Um, GitHub said, "Yeah, well this is in an open source dependency of ours, so we're not giving you a bug bounty for a GitHub desktop rce, even though I proved that, you know, it was usable from there." Um, which is fair, but you know, it's like a bit of a special dependency when your entire company is named after it, but fine. Fine. Um I you know I don't have time to fully rush about how open source and bug bounties intersect but there are many cases where some of these vulnerabilities I've I've reported to

companies and they're like yeah that's not very serious. Um and they're like well yeah but it's an open source project. Can I just patch it? And in one case some company just said actually well no because you've you've agreed to the bug bounty terms so now you can't re reveal this. So it's like yeah I could just write up here anyway. So there are there are multiple cases where I mostly don't bother to submit things to bug bounties. I naively thought GitHub might be more reasonable about it when it was good itself. But anyway, that I could rant more about that and I don't have time. Um, is this a new thing? So, I'm sort of in summary. This was the

cheering award lecture from Ken Thompson in 1984. That was where he backdoor a C compiler, right? the back developers have both been looking at security, breaking security and not understanding security for as long as developers have been developing software. Um I sort of identify in between a developer and a security engineer. So I I think there are many things that can be done better improving the tooling. Um you know it's a bit strange to fish someone through an SSH um link but it's possible. So, um, yeah. Anyway, um, in summary, I've I've only covered Git and SSH here. There's many other ways developers can be attacked. I I mentioned GitHub actions a bit as well. Um, editors are

interesting, particularly when you combine that editor with AI through an MCP or other things. Um yeah, I think XZ changed mindsets as well like you know how much do you trust the developers when a developer might turn up to an open source project and start doing scary things. Um so this is why you know a vulnerability in git like this is a bit scary because well actually if you just pull the um latest project and some bad developer has put something bad in there and attacking the other developer like there's all sorts of scary things. Um so yeah um thank you