BSidesSF 2026 - Practical (and impractical) git commit signing (Matthew Garrett)

So, Matthew Garrett's here to present on his topic of practical and impractical get commit signing and uh we will take questions on the Slido app. So, if you have the application, please log in and submit your questions and we'll field those down here and if we run out of time, we'll make sure that your questions get to Matthew and he will be able to email you back the responses. With that being said, I'm going to turn it over to you, Matthew. Have a good time. Okay, thank you. Um So, just a quick check. A couple of bits where I'm going to try to perform some very very faint audience participation. Can people just stick their hands up so

I can figure out whether I can see if people stick their hands up? Perfect. Thank you. And also, you're all awake. Brilliant. So, my name's Matthew. Uh I have to be clear, no formal qualifications in computers other than a certificate that says that I am competent in the use of a word processor, a spreadsheet, and a database. And I will say that I did pass that with distinction, but also I was 11 years old. My training is in fruit fly genetics. Um I'm an elaborate fraud as are so many of us. In the past, I've been a Linux kernel developer. I occasionally stole cos players one. I have poked at various embedded devices. A few years ago, I

spent a while breaking the security of um some rental electric scooters such that it was possible to just unlock them and then ride them around without paying anything. Um so, obviously I didn't do that cuz that'd be illegal. And any of you who have seen any of my recent presentations or read anything I've written in some amount of time are probably familiar that I am overly enthusiastic about the storage of cryptographic keys in hardware elements that are not directly accessible by the operating system. This is foreshadowing. So, let's start with I Okay, is anyone here not aware of what Git is? That saves me time. Excellent. There's a short version of why do we want to sign a Git commit? And that

short version is that is the same reason you want to sign anything you want to have evidence of possession of a private key. And you want to be able to demonstrate that this commit was at some point in the control of someone who has access to that private key. I That's the short version of signing. The longer version is you actually want that to mean something. You presumably want to be able to associate that key with some sort of identity. You want to be able to, as a consequence of seeing this cryptographic signature, say at some point this commit was under the control of someone or something that has access to that key. And I want to be

able to make that association so I can have some stronger indication of who wrote this Git commit. And this is relevant like I'm assuming that any of you who have wondered around have probably been If you've been talking to vendors here uh and certainly if you were talking to vendors at RSA next week, there are going to be several people trying to tell you that they have fixed uh supply chain security. I am not going to tell you that I have fixed supply chain security because fundamentally I don't think that's a thing that can be fixed, but there are certain things we can do to reduce the risk around that. We can make it harder

to inject commits into trees in ways that uh allow untrusted individuals to land source code. We can make it harder for a user's account to be compromised and still allow um that compromised account to pass itself off as the legitimate account. We can reduce some of those risks. Nothing I'm going to present here is intended as a doing this will mean that you have supply chain security. We are still going to be dealing with humans. Humans are still going to make mistakes. There are going to be bad outcomes. Nothing here is a guarantee of anything. Also, to be clear, um none of this is related to my employment. I am not selling a product. Any code that I reference here is free

software, freely available, and also no, you can't pay me for support for it. Git is a sort of quintessential piece of open source software in the sense that instead of doing one thing well, it does many things to varying degrees of competence. And one of the things that Git does is support signing commits. Um so, it also supports signing tags, and honestly, signing tags is probably a more frequent use case because that's a thing where instead of having to make assertions about the authorship of individual commits, whoever merges those commits can then at the end say, "I reviewed all of these and I approve this message." But, I think we can do better than that.

I think focusing on commit signing, making it easier, making it viable to uh sign every Git commit is a worthwhile step forward. Git supports performing these signatures with way too many tools. Um You can use GPG. You can use X.509 uh identity certificates, the same sort that you would use for S/MIME. And you can use SSH keys. Um Um If you have some sort of cryptographic key that doesn't fall into one of these categories, there are various ways you can probably still make it work anyway, and that's sort of something I'll be alluding to later on. GPG's first on the list. It's the oldest implementation in Git, and you can tell that it was there first because all the

configuration for the other signing formats is under GPG. It's in the GPG namespace even not using GPG. Because GPG is I really cannot overstate how much I hate GPG. Thank you. Nobody booed me there. Anyone wants to boo me? I promise it's not going to be any sort of code of conduct issue. No, awesome. Great. Um so, GPG is a problem for various reasons. GPG GPG has a trust model. GPG's trust model bears about as much resemblance to reality as I do to the Mona Lisa. It there's some sort of vague shape correlation, but very clearly no. GPG The focus of the GPG trust model in general is the idea of a web of trust.

You find someone who is already considered trustworthy. That trustworthy person indicates that they trust your identity and then signs your key with their key, and there's now a web of trust. The problem here is that this firstly, kind of went out of fashion in the early 2000s, and finding someone who's already passed the web of trust to sign your key now is not straightforward. And the second thing is, what that ends up telling you is someone's legal identity, and someone's legal identity does not necessarily correspond to their online identity. They do not necessarily want it to correspond to their online identity. And so having a trust model where you're saying, "This person's possible name is

this." is not a good way of providing any meaningful security or any meaningful assertions around the identity that many of us use. How many of you have ever come Sorry. No, this is a bad question. I'm not going to ask those of you who have committed code under a name other than your legal name to put your hand up because that kind of defeats the point. I see a couple of you put your hands up anyway. Well done. GPG is just fundamentally a poor way of asserting trust in arbitrary individuals. And I This is largely me just having an excuse to complain about GPG because if you're going to put me on stage and give me an

opportunity to complain about GPG, yes, I'm going to do that. But more realistically, if you're an organization, the GPG web of trust isn't super meaningful, right? Anyone can create a GPG key with any identity in it. And unless you're going to have some sort of root key and then you're going to sign all your employees' GPG keys, that's going to be painful. The process for an employee proving to you that uh they had access to a GPG key is going to be immensely tedious. There's also like you can't easily do short-lived GPG keys. And the reason for short-lived stuff is we'll get to that a bit later on. There's not a good way as a corporate

entity to say, "These are my employees' GPG keys and I trust these." and also be in the position to um handle people leaving, uh people changing role, people having access to different stuff. GPG's not great for having metadata associated with the key cuz you'd basically need to produce a new key to have that metadata in there. Bunch of nightmares. And also it's very difficult to with GPG make an assertion that this GPG key was generated on a specific piece of hardware. That's again something we're going to come back to a bit later on. Moving on from GPG, there's X.509. As if I hate GPG, boy um the RFCs around X.509 certificates as used for subject identity in stuff like

S/MIME have Okay, this is a audience participation point. How many of you have ever used S/MIME? How many of you have used S/MIME for a reason other than your employer forced you to? One person. What? I'm so sorry. Um I'm fascinated. I would love to talk to you later. We'll talk later. Don't worry about it. Okay. I didn't actually hear what you said there. I just said that's okay. But so things in principle are better than they were. The certificate authority in browsers forum did specify a new set of baseline parameters for what X.509 certificates used for this purpose should look like. And unfortunately, nobody really paid much attention to that. Because while in the browser

space, if you say CAs, you have to do this, then if you don't, then Chrome stops trusting your certificates, and that's a problem for you. Uh if browser peoples if the CA authorities If someone tries to write a spec that says if you don't do this, email clients are going to stop trusting you, you're going to point at them and say, "No, they're not." Because the probability of Outlook suddenly changing how it's dealing with S/MIME as opposed to preserving backward compatibility with whatever immensely up private PKI someone is using is basically zero. So doing this in the real world means you're having a bunch of huge complexity. But, also there is still the problem of with X.509

certificates, what is your trust model? Who are you trusting to issue these certificates? And the um like the good part of S/MIME certificates is that the ones you can obtain in a reasonable way, as in the one vendor that will give you one for free, only verifies your email address. The only thing the certificate says is this person has access to this email address. I think that's actually good. I think that is the right level of authority for like community-based projects. You're not talking about which individual do we trust, which legal identity do you trust. You're saying someone with this email address who I have corresponded with has this certificate. And I think that's an

appropriate model. Um but, if you're using GitHub or GitLab and the majority of you, if you are publishing Git content to the wide world, even if you are a corporate entity and you're dealing with Git, there's a high probability that you're using either a managed GitHub or GitLab, or potentially an internal instance, but whatever. Their model is that they will only trust X.509 certificates that are issued by a globally trusted certificate authority. And that means you can't issue those certs yourself. You are going to have to get a globally trusted CA authority to do that, certificate authority to do that. And that's also going to make it very difficult for you to do like um

short-lived certificates, because if you want short-lived certificates, every time one of those is going to expire, someone's going to have to prove possession of their email address again. And this is not fun. SSH keys. Um so, SSH keys do not make any pretense of identity. An SSH key is a key pair. It is a private key, it is a public key. You can give someone the public key, you can sign stuff with a private key, person with the public key can then verify that signature. It's just a key pair. It tells you nothing about who has those keys. Um I mean, in the general sense, uh there's actually, you know, some amount of fuzziness here,

uh which is Oh, wow, that slide's got messed up. Um That's supposed to be on one line, but anyway uh OpenSSH. So, when we we talk about SSH as a protocol, we have the RFC-defined SSH, and we have the commercial product SSH. And then we have what basically everyone uses, which is OpenSSH. And the When we talk about SSH certificates, we're talking as defined by OpenSSH, which could be implemented elsewhere. If you're not using OpenSSH, then stop. Just use OpenSSH. The nice thing about SSH certificates is that all the SSH tooling treats them pretty much as if they are just public keys. An SSH certificate is just I'm going to say just, this is a very low-bearing

just. An SSH certificate is just an SSH key type with an embedded public key and a bunch of signed metadata wrapping that signed public key. Um so, don't think too much about what an SSH certificate actually is. All you need to know is pretty much anywhere you can use an SSH key, you can use an SSH certificate instead. And that means that Git is absolutely fine signing things with SSH certificates because horrifyingly, the way uh the way that Git performs signing operations is to shell out to another command to do it. So, may say horrifyingly, "Okay, fine. Whatever." And if it's signing with an SSH key, it will just shell out to SSH-Keygen. Please do not ask why SSH-Keygen does

something other than generate keys. SSH certificates are unlike X.509 certificates in that you can't have intermediates. Certificates have to be directly signed by a CA. And that simplifies everything a great deal. Also, SSH certificates are not ASN.1. Hooray. I mean that Sorry, that sounded sarcastic. I genuinely am very, very happy about this. Uh so, you have an SSH certificate. It chains back to a single arbitrary CA. Who is that CA? It's whoever you want it to be. And so, let's think a little about who you're trying to convince when you're signing something. Uh Who you saying that this commit is from? Is this someone with a specific email address? And Git authorship is very tight sort of

associated with email addresses. The author field will typically have an email address in it. And so, fine. That works pretty well with Well, that fits very nicely into the X.509 model. That fits somewhat into the GPG model, although GPG, while having an email address in there, typically when people sign uh a GPG key, they may not actually verify possession of the email address in the process. So, you're not necessarily making a commitment about that email address being associated with this. GPG, just do not even consider saying yes. Uh When we're saying this, are we just saying this is signed by someone who has possession of a key? That in itself, without information about what that key

is, does not tell you much about the provenance of the signature. It doesn't tell you how trustworthy it is. It doesn't tell you who this person was. But also, who's doing that validation? And where are we doing all of this? And I'm going to sort of branch off from the theoretical now, and I'm going to go into the practical side of things. And um realistically, if you're doing stuff with a Git, you're probably using GitHub. Uh Anyone here who's from GitHub, I'm sorry, I'm not picking on you. Basically, everyone who isn't GitHub is actually worse at this than you are. But you are pretty bad at this. So, let's talk about how the different key formats um that we've talked about

fit into GitHub's model. So, have any of you actually ever enabled commit validation on GitHub? That's the thing that means that on your repositories, you get a little tag next to commits that says validated. Verified, rather. Okay, some of you have. What do you think that's telling you?

Right. So, you're basically asserting that someone at some point has access to a key. And the question there is, how is that key considered trustworthy by GitHub? And the answer for GPG keys is the user added the public GPG key to their account. So, if I have access to someone's account, and if I have access to their GPG key, I can post stuff, and that'll show up verified. Fine. X.509, um the certificate you not in the X.509 case associate a certificate with your account in any way. Uh the certificate is associated with the email address. You sign it with that certificate, and then GitHub verifies that the email address in that certificate matches the email address in

the author field, and then also verifies that the certificate chains back to a globally trusted CA, specifically uh a CA in Debian's CA certificate package, which is roughly the same as the set of trusted certificates that browsers pay attention to. Um so, that means you can't have an internal CA for this purpose. You need to be making use of globally trusted things. For SSH, basically the same as the GPG case, the user added their SSH pub key to their account, and now uh signed with a private key corresponding to that public key. This is kind of nice for the users because if they're using GitHub, if they're pushing to GitHub, they basically already have an SSH key

pair associated with GitHub. So, this means that the user doesn't need to add any additional stuff. The downside, though, is if someone's account is compromised, if someone's GitHub account is compromised, the attacker can just add a new GPG or SSH key. And there will be no The user will get an email notifying them that a new key was added. Uh the attacker will need to MFA in order to be able to do this, but if someone has been sufficiently compromised, they can just have an attacker-controlled key added to this. And the other aspect of this, also, how many of you have actually checked how many SSH keys you currently have registered with GitHub? I'm pretty sure that some of those with

their hands up are lying. But, okay. Possibly you are all much better people than I am. I I really do enjoy these heartwarming opportunities to discover that I am not the best person in the room. Uh but, how many of you How many of you would say you have absolute confidence that no private keys associated with an SSH key that is in your GitHub account has ever left your control, ever? Nobody. Okay. Brilliant. Um X.509, so sure. Uh that works. It is giving you an assertion about this email address, and then you're still going to be trapped in X.509 purgatory. What you probably actually want is to be able to say, and I'm going to point out

here, I'm sorry, I've been working in security for large organizations for some amount of time. My brain has been somewhat melted and I very much think about things in terms of how would this work in an enterprise. And in an enterprise, what you almost certainly want to be able to do is be in control of this key management. And that's difficult for X.509 if you're not able to be your own CA. And also, even if you are able to be your own CA, you probably don't want to deal with S/MIME. Nobody likes S/MIME. Um Does anyone here like S/MIME? I'm sorry, you put your hand up as someone who had you voluntarily used it. Hm? No, you

don't like it. Cool. Okay. Uh GPG um I have no good way to deal with that from an organizational perspective. I'm going to need to uh have some sort of keyring where I add people's GPG keys. And keeping that automated in a way where I actually prove possession of those keys, prove that that's is associated with someone. This is going to be a whole bunch of infrastructure I'm going to build and it's not going to get me anything else. Um X.509, uh if you're going to have to rely on a third party, that's also going to mean what happens when someone leaves the company? What's your revocation story? You're not going to be able to make this

work well with short-lived certificates, which is the easiest way to avoid revocation. You can't have It's not going to be straightforward for you to end up in a situation where a new X.509 certificate is issued to a user every 24 hours without that user having to really become unhappy in the process. Um SSH certificates, well, right now the sort of model is unfortunately in GitHub even using certs, it's still going to be based on what is the public key that someone added to your account. And this is especially infuriating with GitHub because if you're using GitHub Enterprise, you can limit your organization's access such that someone has to be using an SSH certificate signed with your CA key.

And that works. That prevents someone authenticating to GitHub via SSH. But there is no support in the product for then doing the verification of the commit signature with the same CA key. Anyone from GitHub here who is able to say like Firstly, please, I am not going to embarrass you by asking you to take responsibility for this. Um but if this is something that there is any hope of ever changing, that would be incredible. Now, the nice things about SSH certs compared to other things are not just there's lots of simplicity. You're not having to build a huge PKI around this. You're not dealing with intermediates that need to keep being rotated. You are just signing the thing and if you need

to reissue all those certificates, you can just roll over to a new CA and do that. That's going to be fine. Anyone who's ever used SSH agent forwarding where you do SSH {dash} capital A to a site knows that you can then use your local keys on the remote system as if they were on the remote system. And this works for signing as well. That means you can SSH from your local system into a remote system, do signing, and it does not require any special configuration. It just works out of the box. So, this is already a huge usability advantage. Uh you can easily issue short-lived certificates. GPG, not easy because the validity of a key is

associated with the key and you can do some stuff with subkeys. It's going to be painful and miserable. Don't do it. Um and you can add arbitrary messages to this. So, you're going to be able to add a list of group memberships. You're potentially if you want going to be able to indicate this certificate says that this user should only be able to commit to these paths in my mono repo or these repositories in my organization. That kind of thing. You can have very fine grained control. The other nice thing is well, this is technically true for all of these keys, but key types, but most of them would involve using PKCS11 and if I hate

OpenPGP and if I hate X.509, I really hate PKCS11. Uh you can generate the keys in a private key store. You can use a trusted platform module. You can use the Apple secure enclave and you can then if you are in a position to make use of device identity attestation, um which if that's not something you're in a position to do, come talk to me. Again, that's not a product I sell. Uh all support. Very enthusiastic about it, but no. Then you're going to be able to say this commit was signed on hardware that my organization owns. You can make sure that this commit was generated on a computer that has your management infrastructure, that has your endpoint

security software. You can have much more confidence about this. If you want more details about trusted platform modules, then 85 minutes ago Eric Chan gave a good talk about that. You should go do that. Like watch the video, basically. Uh now, making this work at the moment, unfortunately, as I said, GitHub's verified status application infrastructure doesn't really do this. Um so, you can however have a separate run as part of your CI. You can have a GitHub action that will look at a PR and will then go through every commit and will verify the signatures using whatever infrastructure you want. Um so, if you want one of these things, if this is a well, okay, that's all very

good that you are telling me that you can do this, how do I actually do it? Then this URL contains both a more detailed write-up of everything, which somehow will apparently only take you 7 minutes to read instead of having to listen to me for um 28 and 3 seconds. But the code is available at that URL. The write-up is there and there are links to public Git library pose that contain all the source code. It's all free software. Do whatever you want with it. Please if you have a reasonable feature request, do that. Please do not ask me to do a lot of hard work for you. Anyway. Um we've got maybe two or three minutes

for your questions.

Link yes. How do you suggest we do signing? How do you I suggest you do signing? Um I would would absolutely say use SSH certificates. Is there a hint there? If you are in a position to do so, I would say using hardware backed uh keys. The fact that you can easy if you have infrastructure to issue certificates, it doesn't matter how many keys someone has. You can just issue as many certificates as you want. You don't care. You can associate the user identity with this um private key in some way. You're then going to be able to just push out certificates to them. They're going to be able to use multiple machines. They're going to be able to

SSH into dev environments and do their signing there. And then like my blog post talks about the linked post talks about how you actually configure Git to do this. But once that's done, it's transparent. Uh you do Git commit um and if you have auto signing turned on, it will just be signed. If you do commit amend, if you do a rebase, it will resign everything in the process. And if this is just a hardware backed this thing, then um yeah, you just speak to an SSH agent that does that. There's an implementation of an SSH agent that does that links there. Just to make sure I understand, the assumption you're making is that nobody

is really basing public Uh if you are going to rebase something, then the signature is so if you're rebasing on a local system, the rebase commits will be signed with whatever key you have. And that's not going Because you said rebase, so I just wanted to make sure Yeah. Whereas, if you use GitHub to do the rebase, if you use the API, then that's going to end up signed by GitHub, which makes things differently complicated. Um, one more? Okay. Um, I'm going to be around for the rest of the weekend. If any of you have any questions about this, then hit me up. Thank you. Thank you, guys.