HSMs in Plain Envelopes: A Code Signing Story

hello welcome everybody to another b-sides Villages talk happy to be having this this year and looking forward to many more I'd like to introduce Alexis Hancock uh director of engineering at eff she's going to be talking to us today about hsms um and want to give a quick shout out and thanks to AFF for being always a wonderful partner here uh uh with b-sides and just a great Village to attend so please thank you very much Alexis okay all right I guess I have to talk a little loud it's all good um so as you said I'm Alexis Hancock director of engineering primarily over the web encryption projects at my job eff has several open source projects in

particular I manage dessert bot project and I kind of want to take you all a little journey on a little journey around um what it looks like to kind of like integrate security into an open source project and kind of follow best practices while coming across a lot of barriers to that and I will just get started so yes open source and security has had a rough road we all kind of know about the heart bleed bug that happened with openssl and the government has been highlighting certain things around open source and the prioritization of security um but a lot of Open Source product like projects have actually paved the way for security [Music] um

so there's you know kind of like a two-edged sword with that where you had a large collaboration of people come together and try to push the envelope when it came to security standards um and also I want to highlight the lack of investment in open source the lack of investment in the people behind open source There's real people coding behind like all these wonderful tools and burnout happens lack of funding all of that contributed to the lack of security in the open source ecosystem this kind of sucks giving a talk next to a lunch but it's okay um so I wanted to highlight that I think everybody's kind of like familiar with the meme around uh that one guy in

Nebraska taking care of a project that shouldn't be a thing going forward in my opinion it should be a more supported effort let's go forward and try to create projects that are well supported that have a lot of investment from the ecosystem that uses it right so I don't want to keep hearing about you know just one person that had like actually contributed to like one project because you're depending on not only a single source of failure but just a single person who can only do but so much um so and I also want to preface code signing does not prevent malware or malware installs but code signing does help an Integrity chain behind it

so today I'll take you on a particular journey of getting Trust on Windows through code signing obtaining a code signing certificate talking to etokien hsms and smart cards signing with an e-toking token and esthetication and code signing ecosystem at Large so my problem for our project was I needed Easy distribution among a small team I needed cost efficiency I needed low-cost uh barriers and I needed compatible with our modern release pipeline as a team so I was very naive to think I could just have that up front so I want to talk to you all about kind of like the challenges and barriers that I came across when trying to get Trust um obtaining a code signing certificate

some of you may be familiar with this process where the model that we were coming across was we are publicly distributed software and we need to be trusted on a major platform from several users right thousands of users millions of servers so coming with that model we needed to be publicly trusted through another entity and so in particular with Windows there's something called the user access control system that you have to bypass in order to be a trusted software you might see it pop up when you install you know some sort of like software or installer of some sort on Windows and it will pop up saying this is trusted by this publisher or not trusted do you

want to proceed so we wanted to get that trust on windows so we could you know follow certain guidelines available but there was two different issues on this particular platform there's EV code science certificates which are way more money than regular code signing certificates but the trade-off is EV certificates which are about I want to say at minimum 300 that may be a laughable number they're probably more than that but a couple hundred dollars and keep in mind this is an open source project not a private company so Evie is immediately trusted or extended validation certificates are immediately trusted on windows so you get immediate reputation with regular code signing certificates you get trust over time

um no one's really quite sure of the exact formula that goes into getting a code signing trust with a regular certificate on Windows but it's everyone's pretty sure that's dealt with the ecosystem and even Microsoft kind of halfway confirmed this that's like number of installs over time that's been successful and then you can gain reputation and Trust on Windows that way but the trust will not be immediate the reason why they have immediate Trust on extended validation certificates is because of the fact that they do like an extensive background check process for that certificate but then there's a little caveats because they do some background testing and checking for regular co-assigning certificates as well but you know I digress so

had to choose a Civic a certificate Authority a certificate Authority um if you may or may not know like there's a couple out there that actually do work with Windows or trusted by Windows as a issuer of code signing certificates they're very easy to find you have set to go digicert or digit trust and then you have several others as well so they vet you as a person as an organization first and you get phone calls automated checks things of that nature you have to submit business documents and then finally maybe like a week or two later you get your EB certificate so I got it cool right and then I got the HSM or the smart card with the

certificate on it and a plain white envelope in the mail with no documentation with no other instructions besides get a PIN code and good luck um I wasn't quite sure what I just received in the mail considering I wasn't quite familiar with this ecosystem of code signing and trust and what exactly all this meant I just looked like a USB token it just looked like a USB drive so for all I know I just got a regular USB drive to mail from set to go or digitrust or whichever CA you choose right so these USB devices are interchange the term e-token HSM smart cards even though there can be some variants there but all of them have to be fips 140-2

compliant what's that nobody um Phipps is a federal information processing standard why does that apply to this because they use this nist standard for these smart cards in particular so in June 2023 all code signing certificates asked by the cab Forum as voted by them last year they all co-signing certificates have to come on these devices so June's right around the corner so they all have to be generated and stored on 140-2 fips compliant devices so if you do get a code signing certificate in the future you will have to deal with these devices so the token that we got was really interesting because it said safenet on the token body but the actual um connector piece a gemolto and I was

really confused because the documentation said safenet and I was I was just like what the hell is going on so I had to find out that Talus was the vendor that issued the USB because once again I got this in the plain white envelope no other documentation didn't know what the hell was going on Talus acquired gemalto gemalto acquired safenet so if you see those three names they're all the same damn company so whose fault is this like with this confusion that initially occurred I might I mainly blame them I blame the the USB vendor and I blame the ca for out of date documentation on their sites it took some time to go on the ca site and find the proper

tools to talk to the Token because they provided me initially rules and instructions to go use the GUI for this token but as developers with different types of pipelines at play we don't necessarily use guise to do that we use the cleave we need clean tools for this it didn't readily provide that I had to dig for a long time solution modeled for a single developer deploying on the same machine that they develop on it's kind of like the approach they took when I have a team I need a team of people who are trusted to code sign how do I distribute this among them this is just treating it as if I'm the only

person working on this project with my singular machine which is probably how co-signing certificates get on the loose anyway and sign malware but you know and it's a mix of me and my own experience I didn't know how to talk to cryptographic tokens or smart cards in depth and this is why I'm sure sharing with you now in case you come across this in the future yourself and wish someone had told you uh talking to the Token uh you have open SC uh is a open source Suite of tools uh to talk to cryptographic tokens they Implement two different standards um there's PK cs11 and pkcs15 it's the data format pkcs 11 is the API you'll

also see pkcs 11 called crypto key uh crypto key meaning cryptographic token interface sorry for all the alphabet soup um but there's pkcs set of Standards they're like 1 through 15. I will only be talking primarily about pkcs11 and the API to talk to these tokens and when you have these kind of Standards floating around with these smart cards some of the smart cards may be pkcs 11 compatible but the module provided um to talk so it may not actually be compatible with this open SC even though it's open source so sorry if you come across that sometimes but it does happen pkcso 15 is a data format for the smart card that may not necessarily be

compatible either with opensc or even have pkcs11 proper support I'm sorry that's just kind of the state of things right now but if you know these standards and how to talk to them you can kind of work with the tools that you're given and eventually you'll be able to talk to the tools with either tools that the ca will give you like they gave me proprietary clean tools to talk to the Token that they issued or I could use opensc which was thankfully compatible with the token that I was issued so make sure you talk to the ca like what are you issuing here and make sure you're not like me where I get it in the

plain white envelope and then figure out what does it support and what's on it and what modules are able to talk to it signing with the token uh so we had to use our own infrastructure for this authentic code signatures are something that our window format Windows format code signing signature um The Challenge right now is the pipeline and tooling that we mainly develop on are mostly Nix based machines and you have to sign in particular for some reason on a Windows machine with sign tool which is a exe sign Tool uh so not no one at my job develops on Windows I don't know about y'all I mean you could strike up a cloud

container or two but then it's outside of your pipeline for your CI see where I'm going with this so there's open source tools to help play with this so you have Json um some angels out there made Json and Osso sign code um that are out of band tools that can Implement an authentic code signature when you want to talk to the token and they support different key stores so if you have a yubi key or you have a different type of key store that supports pkcs11 you can use these tools to talk to the Token to sign um remember that the time stamp that you do you utilize in a signature counts as a counter signature so if you mess up

the call to the timestamp server that the ca provides it can mess up the digital signature for the entire piece of software so I had to learn that the hard way when I had did like a bad call to a parameter for a 256 hash and it came back saying the digital signal was wrong and I was like do you mean and then I found out I had messed up the call to the timestamp server Cas do provide some cloud services like ssl.com provides like a e-signer service of some sort but I don't really like that direction because one once again you have to pay for it pay for it monthly or by sign I forget but it's

just these are things that can happen for free you know and these are things that can be utilized if you know the ecosystem was Stronger around that so I don't necessarily support the cea based tools but they do exist if you just don't want to do any of that with your own infrastructure and just want to utilize the e-signer that they have with their Cloud infrastructure uh what can make this all easier estustation so you'll have your own say a UB key and you can attest to the certificate originally with that UB key and a test that there's a private key that's been generated and stored on that device and they'll send you the intermediate and the cross signing

certificate to install on your UB key and you can distribute among your team with a tested UB Keys um the UB keys do have to be fips compliant thankfully UB key ubco provides Phipps compliant keys so that's a opportunity there to actually expand among the ecosystem um say you have three developers on your team you can have them all have fips keys attest to the certificate and you all can sign that way it's easier to revoke like a physical key from that person than it is to revoke people from A system that is a singular centralized system that everybody knows the PIN code and they leave the company and they may know the PIN code and that can get messy

so I find it better for auditing and management of cosigners um the code signing ecosystem I only talked about Windows here but there's other opportunities of code sign that occur and happen uh pgb keys and UB keys and smart cards um you have to you can like make an internal system where you upload a pub key to an internal system possibly um and have like internal pki management with that um digital signature slot on UB key I'm sorry I'm like really UB key heavy there are other smart cards like Nitro key and all that I just primarily worked with yubikey um slot 9C on UB key you can use that for a singular use or internal projects

for pki and uh we use actually on certified pgb keys to sign some of our packages for Debian so that's been something that's been really really helpful having open pgb keys on UB Keys distributed among our developers who are all remote um and then there's six door which is a new project I think out of the Linux Foundation um they support something called ephemeral keys and they do use oidc or open identity connect for identity purposes which may you know cause some issue here and there but some people depending on your threat model and how sound you find that solution but I do like their idea of ephemeral keys because cosigning certificates they last like a year it's really long term

certificates in the ecosystem CLS certificates normally get issued for like 90 days at a time now or at least that's what's at least that's what's being um proposed as a better TLS ecosystem for putting out TLS certificates and so that every 90-day spam versus the EV and regular code science certificates that are issued like once you know and they expire in a year or three that's really long term and an easy chance for something to slip and keys to get you know actually leaked in some sort of manner because of that long-term presence of that code signing certificate so I do like their idea of ephemeral keys and just two days ago six store now supports npm signing npm

packages and beta so check it out um and then I talked to the director of engineering over there um there is like little Whispers around trying to get a chain of trust with on the Windows not with Windows authentico but on the Windows ecosystem and Pi Pi packages um signing six door with that six store check it out there's a whole like different sets of like tools that they use they have their own CA internal monitoring and logs it's very nice stuff um signing mobile applications have their own ecosystem I didn't focus on noise as much but I do know like with iOS there's xcode and some of these things could be actually implemented to

be a little bit more open there's different tools so I didn't focus on it as much but there is code signing for you know apks and different like iOS packages in apps my requests and wishes uh of this ecosystem is better support from certificate authorities when it comes to um code signing there is the pki Consortium uh they actually have like a GitHub an open repo if you have like any issues that you've had over time to go Express that in their um discussions panel on GitHub and then there's the ca um browser form that also has done their due diligence and actually passed a standard that goes into effect in June around not only getting co-signed

certificates supported and stored on um 140-2 compliant keys but you can actually use cloud hsms now with this new made mandate you can attach to a cloud HSM you can um Unfortunately they don't have attestation readily available I think only Google Cloud right now has esthetication available the other two need to catch up but there is more talk around remote cessation in particular so you can use cloud hsms now Within new mandate it's a little bit more looser just as long as you have a 140-2 compliant key they'll allow you to do a test um so let's focus also on proprietary Cloud signing more distributed signing I support Association in that way because you can distribute among a team safely

and the important part is the private key never leaves the device you can't export the key and you can't put it anywhere else it stays on the device unless someone out there hack ubiky you know you're getting paid but you know what I'm saying like it's just kind of like in this case that we know of that the private key will stay on the device um and now that the cryptographic standards are the same for code signing um Regular certificates and Eevee certificates I honestly don't see the benefit of an Eevee certificate over a regular co-sign certificate now because the cryptographic standards are the same um especially as of June so I'm not quite sure outside of like background

checks how beneficial EV over regular can be in the future past June I would just probably tell most open source projects to just utilize regular co-signing certificates um and in my opinion accessible security is better security especially for open source projects like mine and I would much rather we go towards a stronger ecosystem rather than creating proprietary tools down the road that would just make code signing even harder so some plugs uh pivot is a cash app based open source project that I contributed to that actually you can manage some x509 certificates um on ubikey um I actually added additional slots with my contribution so go play with that if you want um it's not set in stone but uh it's a

project that I thought was pretty useful for talking to yubikeys and if you want to kind of see what it looks like to sign with a smart card I did create like a shell script to kind of see what that looks like particularly with EC elliptic curve ECC certificates and seeing how that can look it's a really short shell script of sorts so it's something that I played with like especially if you want to like build your own internal pki and want to see how to uh how the actual signing mechanisms work on a UB key and they also have their own docs on the site too on ubico they're pretty good Dev Docs thank you for listening

[Applause]

any questions yes

could you talk a little bit more about ephemeral code signing certs um I was under the impression that this signature would only be valid as long as the signing certificate was valid so the way six store set it up if I understand your question correctly is that they they have different mechanisms they have their own internal CA right that they have that issues these ephemeral keys that are mostly um I would say kind of like a key store ring situation where the base key gets rotated but then there's a chain that gets connected over time um which is why they have the monitoring monitoring and logs hopefully the director of engineering from six door doesn't get mad at me and I'm like

misinterpreting what he said but um that's what I understand as far as like being valid as long as the signature is valid um I think they have it where there's like a permanent log of that and they check against that versus having the co-signing certificate long-standing so making sure there's a there's a record of it and having that there and solidified into kind of like mimicking CT logs to an extent and kind of like having that chain of trust going back to the original co-signing certificate that did you know lose its um I would say permeates in the system hopefully that answered the question

okay thank you any more questions thank you Alexis yeah [Applause]