How to Use Code Signing to Protect your Critical Software

Oh

hi everyone can you see my slide now yeah get started like you're sounding great okay perfect hey it's great to be here thank you and I've got a lot that I want to cover in the next 25 minutes so I'm gonna just get started so a few years ago this company was in the news Maersk they were the world's largest shipping conglomerate as you can see here very very large and they're responsible for about a fifth of the world shipping capacity and they had a really bad day on June 27 2017 and not sure if any of you remember what happened then but this is what their employees experienced their computers started to display a

screen like this and what had happened is that not petia malware struck Maersk and it infected 4,000 of their servers 45,000 pcs basically destroying all the data that they had so it took them multiple weeks to Therese restart and restore from backups but this impacted their entire operation their their voice over IP phone stopped working their ships were out that around the ocean stopped moving because the navigations went down terminal gates stopped going up and down so so customers couldn't deliver their goods or get their goods from there their ports and it disrupted customers all around the globe it took them about 10 days to start to bring things back up they had to do it from

backups and their internal their accountants basically arrived at a 300 million dollar cost to them but most external experts believe that it was much much higher than 300 million for this much downtime for this company so huge situation for them but it didn't affect just marys get affected these other companies my brother works for merit Merck pharmaceuticals and at the same time I remember hearing that he couldn't go to work because their computer systems were down and impacting physical security systems and computers and and all of that so it impacted subsidiary of FedEx and the in Europe and some of these other well-known international companies all in all there's about ten billion dollars of estimated damages around the world so

huge situation and how did it happen it's a doctor investigation basically this problem got linked back to a small Ukrainian business as a family-owned business called link OS and they basically provide turbo tax like software but for Ukrainian for Ukrainian businesses so every business that every company of the does business in the Ukraine they needed to use this particular software package and we all know what's been going on politically between the Ukraine and Russia and what had happened is Russian hackers had infected link OS with not no Petya and that in turn infected all of these companies and it was a very virulence piece of malware and that's why it impacted so many companies and it

impacted them extremely quickly there was an account for for Maersk that they were literally trying to go around to every every office and unplug their computers and they still you know couldn't couldn't stop it so you have to ask how did this happen and you know if if link OS was using code signing if Maersk was vetting the software that was being installed on their servers they should have been using code signing and so obviously there's a failure of some sort and it had ten billion dollars worth of consequences around the world so we got to think about the risk and and I'm always one who likes to think about the risk and there there are basically two sides of the

same coin and you know as a group of InfoSec professionals it's you know which of these are more would be more urgent for your business you know is your business more like link OS and you're providing software to customers and so you're therefore concerned about the risk of what happened to link OS impacting your business and you infecting your customers I mean that that would be really bad especially if you're a tech software tech kind of company or are you more concerned about the risks that you might have that Maersk add where they had a disruption of their business operations again if that happened and your business was down for 10 days it's gonna be extremely

disruptive and extremely costly and frankly I'm concerned about those and I talked to customers and obviously the you know though depending on their business they might be more concerned about one or the other but it's a lot here for you to think about and I know code-signing doesn't always come to top of mind when you're thinking about the risk that you have to contend with on a daily basis but this is a great example where it's done some did some tremendous damage and this leads us to it to a simple fact today software supply chains are more vulnerable than ever and why is that and first is is that we're all experiencing digital transformation so

our businesses are relying on sophomore more than they ever have our employees are downloading software from the internet sometimes we know about it sometimes we don't and every time they download it we run a risk of of infecting our systems with malware our software infrastructure comes from many different suppliers I mean at the company that I work for it's a fairly small company but we probably have between a hundred and hundred and fifty different software suppliers in terms of the applications that we use for business operations as well as open source libraries that we use in our product and other libraries and software that we use in our product and then finally we get the issue where cyber

criminals are now more active and creative than they've ever been and that's really what the rest of this talk is about is is how are they doing it and what can we do to stop that and in Maersk was just one example I mean this has been going on for a while just last year the Taiwanese computer manufacturer Asus had a situation where hackers discovered some of their private code signing certificates they inserted malware into legitimate Asus software updates and then resized it with a sis's private code signing keys and this got pushed out and I'm going to talk about that particular incident in more detail in just a few minutes but you know we

think back a few years even before that there was issues around Stuxnet and what happened in Iran and their their nuclear processing equipment and if you look on on the dark web code signing certificates and keys are a hot commodity but especially if they come from a trusted brand name so you know if you're if you work for a company where the brand is highly recognizable and someone gets ahold of your code signing certificates basically anything it could be signed and cyber criminals know that and they sell them on the dark web McAfee did a research project a few years ago and at that point in time said I think this was 92018 they've discovered 22 and a half million pieces

of malware that that had been signed with either stolen or forged code science certificates and credentials that's a lot of malware out there that's being signed from using legitimate code signing credentials so I started this off with a story because I really want you guys to take notice that this is a big big concern for us all of us an InfoSec that we have to care about this and what we see frequently when we talk to our customers is it's not always a high priority issue so the rest of the time that I have I'm going to just give a really quick refresher of what code signing is what challenges we see and we hear about from our customers when they

try to do code signing and protect those those credentials and then I have some some tips for how you can do that moving forward and then finally I'll give give us time to for me to answer some questions as I can so what is code signing really simply it's just a digital certificate 'el signature that we use to sign computer executables that computer executable can be anything like an application a mobile app a disk image drivers firmware if it runs on a computing device it's considered code and that code can be signed and and it does two things one is it verifies or it's supposed to verify the author's identity so if I work for company XYZ

and I sign a piece of code with companies XYZ co-signing certificate everyone out in the world needs to be able to trust that really did come from from company XYZ and furthermore it ensures that that piece of code doesn't get modified in route between when we release it and when someone actually downloads it installs it so it's it's basically meant to protect from malware from being inserted into a legitimate piece of software so that's what code signing is in a nutshell and you can kind of think of it as a birth certificate for the software so you know we spend months and months developing software and at some point we want to release a bit it's like the birth of it

and then the code signing certificate is like the birth certificate saying yes you can trust it it comes from us and it hasn't been modified by anyone else and then we think about okay an organization who's responsible for code signing and and this varies widely some some companies they leave it up to their software developers because they're developing so much software so quickly they have really really aggressive schedules and it's just there's no visibility to the InfoSec team that code signing is going on and it the InfoSec team did it they may not be able to keep up if it's done by the development side of the house they're probably using either build engineers to run a code

signing script or they're actually doing it as part of their build infrastructure so I'm guessing a lot of you probably have companies that are embracing DevOps and and that's all driven through automation so it's those scripts that are actually doing doing the code signing but then there are some some companies who take a different approach and they basically want to centralize code signing because a understand the risk of those keys becoming getting out into the wild so they have either their PKI team or their InfoSec team be responsible for all code-signing operations and and life like in life everything is a balance between positives and cons for these ways of doing it in general if

developers are signing code they're gonna be doing it on either their their own personal workstation or laptop they're gonna be doing it on a web server a build server or maybe even in the cloud and then obviously if the PKI teams signing code it's done in a secure vault we have one customer that this secure lab isn't even connected to the their internal network it's completely isolated and they actually have to take physical media into this locked room to do the code signing operations and obviously if they do that it can't be very fast so when we looked at the pros and cons if it's done by the software developers it's going to be extremely

fast it's going to meet the demands of business but it's likely to be very insecure because most software developers do not appreciate the risk around code signing and especially if it's not protected so they'll do what's convenient for them though they'll choose to do things that makes their job go faster and for them to be able to put out software faster on the other hand if you you know have the InfoSec team doing it it's going to be highly secure they understand the risk and they want to protect it protect those credentials but it's gonna be extremely slow you know it's going to you know someone's gonna have to take a USB flash stick and run

from the development side of the the building to the InfoSec room so extremely slow and both of these have pretty serious and significant disadvantages and and that's really what as we think about the right solution we have to be able to balance balance both so let's talk about some of the challenges and the risks that organizations have I'm going to point out this particular piece of literature it's a paper from the SANS Institute and it is really enlightening it's a great great read it's very short but basically what it outlines is that yes code signing is so effective that now cyber criminals are attacking the code-signing infrastructure so they're looking for those private keys and once they have that they're able to

pivot and then actually attack the code-signing system itself so this is bad they can either steal those keys or they can compromise those code-signing servers and in this quote I found really powerful it's not an exaggeration to consider private code signing keys as the keys to the business's kingdom now think about that and let me explain why I think this is so profound when you sign a piece of code as long as it was signed with a ballot certificate at the time that it was signed and it points to a time stamp server that that has a valid time stamp that piece of code is going to be installable for all time so think about this if a hacker signs a

piece of malware with one of your ballot code signing certificates even if you revoke that it's still going to be installable and it's still gonna look like it comes from your organization and I'm like you know what we thought might fine with TLS or SSL Certificates and you know they expire after a certain period of time so it kind of minimizes the damage done and it's also only good for a particular IP address co-signing keys any piece of software can be signed with that if the hacker has the private code signing key so let's talk about the the specific challenges the biggest one is private key sprawling unprotected private keys and when I say that what I

mean is that it's that scenario where developers have access to the private keys and they decide to put them wherever it's convenient to them on their laptop on a sticky note and on their desk on a build server web server you name it if once you hand out a private code signing key it's out there for pretty much forever usable for pretty much forever so let's look at this in a little bit more detail obviously if keys are not protected they're going to be prone to theft term issues and really as I said it's their unbound risk around this is that you just don't know where they're store you don't know who has access to them who's

made copies whose reuse them and things like that and there's a great example and this is the example with Asus computers I wanted to walk through how this happened because I think it might be enlightening for you to understand the severity of what happens when you you do have private keys sprawl and the private keys end up in places where they're not supposed to so with with ASUS like with most computer manufacturers they produce drivers and they produce updates to drivers and they would push out automatically to all of their customers after all that's the security conscious thing to always plug security gaps with updates to software and they did that it was done automatically most people have

their systems configured to where as long as it has a valid code signing signature it's their operating systems going to install it but one of the things that asus did that was really unusual and again a great example of private key sprawl is that they had private code signing keys on this update server and that is extremely poor bad practice they should not be doing that but they did it at least two keys were found hackers were able to break into the system and it was not too hard because it is you know it's connected to the internet because it's the web update server they browsed around to see what they could find they found these two

private code signing keys and then what did they do they inserted malware into Asus update so it's a legitimate update they added their malware to it if they didn't have access to those private keys nothing would have happened their customers computers would have rejected it because it would have been an unsigned executable or it would have been a signed executable but with with a valid invalid signature but they found these keys so they were able to sign their malware and the the infected update and then that got pushed out and it infected Asus as customers and it was an estimate of about a million computers were impacted so again if you're in the business to where you're delivering

software to your customers you have to think about what would this mean to your business if you infected millions or a million customers with with malware and the scary thing is that a sistent know about this for months they didn't realize that this had happened and it was only after a malware company I think it was his purse key discovered this and then reported it to Asus but I bet you there were some serious consequences of that company for allowing private code signing keys to be on their update server so what does this mean from there from a business standpoint and I know a lot of us you know you aren't really thinking about things in terms of you know this is what

my company's revenue is but it impacted their stock price that impacted their revenue impacted their market share and it made their customers trust them less and perhaps there's some liability issues there where you know customers were asking for money in return of being being infected and depending on the the segments of the market that they work at there could have been regulatory fines as well so pretty serious consequences so that's private key sprawl another challenge that many companies have today with code signing is is lack of policy enforcement so you know I'm sure we all know what's the right thing to do in terms of policy well don't keep those don't let the private code signing keys

outside of the secure location require some level of approval before they get used things like that a certain encryption strength and I talk frequently with many customers and they do have those policies defined they have them written down in a great little manual sitting on their desk but if there's no way to enforce that then people are finding ways to bypass that policy so if you don't have the ability to enforce policy and when I say the ability to enforce it it's an automation around being able to enforce it people are gonna figure out ways to bypass what you have in place to cause this less convenient for them and then you'll end up with cosine signing

challenges there's also this issue of lack of glozell global visibility and so if you're on InfoSec team for even a large company with thousands of employees you probably your group is not that large maybe fifteen maybe fifty people at the most but if you think about how many people in your organization is actually writing code and then that codes getting signed it's happening all around the globe and you probably don't have visibility into everything that's going on but if there is a breach of some sort like with what happened with ASUS or north mirrors whose what's gonna happen they're gonna come to your team and say how did you let this happen so it's really

imperative that that we do get global visibility we need to understand things like what code signing certificates are being used what pieces of software use those code signing certificates when they got signed who did a code signing operation it's things like that so without that global visibility it's really hard for you to even gauge the magnitude of the risk that your organization has and then we also have to deal with rogue development teams I'm an old software developer that that's my background and really you know we software developers I have so much pressure to get out new features and get out new features faster than ever before and they're focused on testing and integrations and things like that

security usually Falls pretty low on their list of priorities and if as a InfoSec team we're trying to put process on them and say well you can't do things this way you've got to go through this set of manual steps development teams are going to find workarounds and we see this all the time with with our customers is that they find workarounds they go rogue and then they put the company at risk because they just need to be able to move fast with their software releases and they'll code sign in any way that they can so to finish up here I wanted to leave you guys with some tips that that we have on on how you can make this better for your

organization and I'll walk through each of these and just a little bit of a detail but before I do that I want to think about code signing a little bit differently than we think about other kinds of PKI information code signing is really a process it's a prawn and I say it's a process because it involves people so these are the people that are actually doing the co-signing operation it involves things like the signing certificate or the keys and it involves activities so it's actually near the exercise of actually running that code signing command that's a sign the to create the digital signature for that piece of code and if there is a failure in any of these areas so if there's if

this is insecure if we can't control who has access if we can't control the actual keys and we can't control what gets signed that means the entire thing is going to fail and there's going to be a vulnerability across the board so as we think about protecting code signing we have to think about how do we do that in terms of securing the entire process and that's what these best practices are centered around is is how can we secure that process so first what I want to talk about is really it should be obvious to all of us it's it's to secure those private keys and to secure those private keys we're either going to put

them into an HSM and keep them there or into some other kind of encrypted storage to where we limit the access of who can get to those and the thing is with secure storage is that if that key needs to come out of the secure storage in order to do a code signing operation it's immediately not secure anymore because someone can easily make a copy of it and then at that point you know that code signing key is not secured and we're back at square one so secured storage means they're in a safe location and that we they never leave that location once that key gets created for any reason so the next next best practice is to provide your teams

with enterprise availability so even if you have developed teams around the world you want to be sure that you know everything that's going on around code-signing because it's an important part of securing the business we need to know who assign what pieces of code when do they get signed what build servers or what servers were used to do the signing operations what code signing tools were used what time of day what days of the week all that is really important information for you to have because it helps you to build risk trends and and be able to identify patterns that might indicate that that a code signing credentials being misused next step is you need to

control the process so you need to be able to say okay only these select people can access the key and when I say that I don't mean they actually have the key they just can they can access it for a code signing operation you need to be able to specify who should approve the use of that because that is a very important control mechanism that if I'm a developer I shouldn't be able to use that key whenever I want to I you know there should be some approval process in order for me to use it because that's how critical these are so we need to be able to say you know this person this person this person has the authority to

approve the use of that particular key if there are requirements around encryption strength or which certificate authorities you're used or which HSM should be used that should all be defined and you should be able to have a way to automate that that control and then we need to think about an automation process that's convenient for developers again I go back to that whole notion of rogue development teams and they go rogue just because it's easier for them so it really becomes important for us to find a solution that is going to make their life easier and it's going to make them want to use the solution in place and that usually means don't make me learn something new and don't slow me

down from what I normally do and allow me to work with the tools that I already use so those all are really important to keep those development teams happy and then finally you need to be able to have intelligence around around what's going on within your organization on code signing you need to in some cases you need to show compliance that yes you these are being followed you do know you have visibility into every code-signing operation that occurred you know whose access what certificates and and be able to identify risky trends that said that might indicate a problem and then again you know this has to be developer friendly and I'm saying this twice because it's that important we have so

many customers that we work with that they have tried their best to create an infrastructure that secures code-signing but it's inconvenient for their developers and so their developers will Rumble for a while and then eventually they just stop using it and when that happens they're back to square one of having an insecure process and finally I just want to leave you with this white paper it's it's really really useful a lot of what I covered in the best practices comes directly from this it's a NIST paper that talks about what security considerations your organization should should put into place to protect code signing so with that I'd like to thank you for your time and wish you all a great

weekend and I'm happy to answer any questions

okay Eddie I think there's one question for you in the QA it's about revoking code signing keys and you could revoke the certificate but the problem is is that if that certificate has been used to sign a piece of code and I and a time stamp was used then even though you've revoked it it's still gonna be a valid piece of code that's gonna be have that it's gonna have a valid signature that's the whole reason why this is so scary for me at least that you don't take all the precautions and then your key gets out your private key gets out someone signs some malware with it there's really nothing you can do at that point