BSidesATL 2020 - Connect: Software Supply Chains and Code-to-Code Combat

hi everyone I just want to confirm can you see my slide it should be chilly now it's great to be here I'm calling in from the west coast so it's a little on the early side for me so that's why I'm not sharing video my name is Eddie Glen I work for a company called vinify and just a little bit about me before I get started I started off my career as a software developer developing avionics software so I spent many many years concerned about software that controls aircraft and making sure that it's safe and secure and over time my career just kind of led me towards being more security focused so I'm very happy to be

here today to talk about software supply chains before I get started I wanted to tell a story and there's this company in Europe called Maersk they're the world's largest shipping conglomerate and they're responsible for a fifth of the world shipping capacity it's a huge company lots of employees lots of different offices spread around the world and they had a really bad day on June 27 2000 17 and I don't know if that date rings a bell for any of you but uh something really bad happened at Maersk that that impacted a lot of people around the world and basically what happened is their employees started seeing screens that look like this on their computers and what had happened is

that they had been hit by the not petia malware and 4,000 of their servers got infected 45,000 pcs got infected it basically wiped out all the data on all their computers their voice over IP phones stop working their ships stopped moving that port terminal gates stopped moving up and down so their their customers couldn't get product through the port and this ended up affecting not just mayor split obviously all of the customers that depended on them for their worldwide shipping activities and it took them about 10 days to to get everything reboot it to where they were operational again and their internal accountants said that it had cost them 300 million dollars but some external experts thought that it was

probably maybe double or triple that for them based on the number of days that they had downtime so extremely serious incident for Maersk and I'm sure that their leadership will always remember that that particular day but it didn't affect just them it affected other global companies Merck pharmaceuticals was in the news in the US my brother works for fur Merck and he he wasn't able to work for about two weeks because they were experiencing similar things and and this happened with the subsidiary of FedEx in Europe and the list goes on with with other companies that work that were impacted by it and the estimate for the number of damage has done worldwide by this one piece of

malware was ten billion dollars so extremely significant amount of amount of money and impact on companies and companies workers so how did it happen and this is what I wanted really to focus on today is is the how it happened and how what steps we can take to keep this from happening to businesses like yours so it turns out that there is a small family-run Ukrainian software business called link OS and they produced a piece of software called me doc that's basically used by all businesses that do business with Ukraine so all these global companies did business with Ukraine so they had this piece of software and we all know what was going on and has been going on

between Russia and Ukraine for a number of years and hackers within the Russian military infected this piece of software with no Petya malware and that ended up infecting everyone that used it and this is really you know an important question that so far I've not been able to find the answer to but if if link OS was doing proper code signing practices that shouldn't have been possible but it was so obviously there was some problem with code signing and that's what I really want to talk to you guys about today to help educate you on the importance of how to do proper code signing within your organization so when we think about the risk that we have

from a security perspective there there are two risks that are almost different sides of the same coin first you have the company the Ukrainian company company linka linka and their risk was around do they protect the software that they deliver to their customers adequately and did they infect their customers which they did so that you know that's a huge risk for especially for those of you that work for businesses where software is your primary product but then there's also the risk that you have to think about from Maersk where if they got infected with with malware it's going to disrupt their business operations and you know when I talk to our customers I frequently say well which of these risk

is more palatable to your business which are you you're willing to to bear and most companies say well neither they're both both extremely important and things that I want to avoid so this kind of leads us to the question about software chain software supply chains they're more vulnerable than ever and there are a number of reasons for that first is that our businesses rely on software more than they've ever relied on software before you know if you go back to merican and think about you know they're their port gates going up and down I mean they stopped working because they were controlled by software their voice over IP phones stopped working because they're controlled by software

so businesses rely on software more than ever the other important aspect is that the software comes from many different suppliers so if I'm part of the InfoSec team I you know look at the inventory of software that my a my company uses and I would I would guess that maybe a hundred to two hundred maybe five hundred different vendors provide your company software and how do you know that all software's trustworthy and that it comes from the source that it says it comes from another aspect is that we're downloading software from the internet every day and your employees are downloading software from the internet every day so how do you trust that software that's being downloaded and

then finally but and it's not lie and certainly not least here cyber criminals are more active and creative than ever before and we'll see that later on in my talk here so there are other examples where malware has infected of very critical pieces of software based on poor code signing practices and policies just a year ago the Taiwanese computer manufacturer Asus had a very severe issue around hackers adi malware to their software update so i'm gonna dive into that in a little bit more detail just a few minutes and then if we go back a few years before that you had the issue of Stuxnet that was was causing quite a few problems and then here's the thing that if I were you

I would really be concerned about is that code signing is an effective way to stop this malware for being inserted into legitimate software hackers know that so what are they doing now they're focusing their attention on how do they steal code signing certificates from legitimate companies so if you work for company XYZ and you aren't protecting your code signing private keys sufficiently and a hacker stills those keys they're gonna be able to sign malware in your company's name in my case research study about two years ago and they'd found that 22 and a half million insistence of malware were signed with either stolen or forged code signing certificates so again think about this if there's a piece of malware floating

around the internet and it's signed with your company's code signing credentials what's that going to mean to your company which your CEO going to think about that so you know keep this in mind that that if your company is doing code signing that you really need to focus on how do you protect those keys so and the rest of the time that I have I just wanted to do a couple of couple of things I'm gonna go through a quick code signing refresher I do find many many of the people that I talked to they they are aware of code signing but they aren't really aware of the details of it I'm going to talk about the challenges

that likely your organization is having and then I'm going to finally give you some best practices on how you can improve things

so first what is code signing it's really just a technique that allows someone to add a digital signature to computer executables and when we talk about computer executables we mean anything from the software applications that that your employees install OS images disk images containers firmware drivers you name it if it's a piece of code and execute on something then that's a piece of code that potentially can be signed digitally signed and that signature is going to show you two things it's going to show your consumers two things one is it verifies your your identity your company's identity and two it verifies and ensures that no one has modified your software since you signed it so you know let's say you put out a

software update a hacker if it's been properly signed a hacker can't install malware or attach malware to it because then the digital signature will no longer match and the operating system will reject it so think about co-signing as a birth certificate for the software so just like with with our human birth certificates code signing certificate in key is extremely vital to protecting the identity of the software that your business consumes as well as creates so when we think about who signs code and this is part of the problem that the businesses have it's kind of across the board if you think about who's actually writing the code it's going to be the software developers and if you're like

most of the businesses that I talked to they they have hundreds or thousands of software developers spread around the world so you have lots of development going on where they're they're the ones responsible for signing signing their software you have build engineers you maybe people don't even do it it's done by automated scripts that's being driven by is something like Jenkins or in another DevOps kind of platform but code signing you know could be done by the developers or the people associated with the software development there are other companies that recognize the [Music] significance of the risk around code-signing and therefore they restrict code-signing to just the PKI team or the InfoSec team so I know one company where

they have a literally it's a locked room with one key that one or two people have access to and when they need to have signed a piece of code they take they unlock that room they take a USB in with the code on it that's where their private keys are and then they they perform the code signing operation and that's how it gets done and when we have to ask a question where does it get signed and again this is going to be part of the problem one of the challenges that that that you likely are experiencing is that if your software developers are doing it it's gonna be signed on their laptops or their build

servers or maybe it's a web service that they're using to build their software or it's done in the cloud and if the PKI teams doing it it's like likely to be in some secure lab or on a secured computer with very limited people access and so we've got to think about this from terms of pros and cons obviously the pro for when the developers do it is that it's extremely fast and I don't know if you hear this from your developers if you talk to them but that's what drives them they they're being pressured to get out new software updates more quickly than they've ever been done before you know used to be that we would measure

software updates by once per year and then it got down to once per quarter and then once per month and now it's not uncommon that software developers are being asked that push out releases once a day or several times a day so if you have to if they have to rely on a single point a PKI team it's going to be very slow for them and it's going to cause problems so that's why they want to do it themselves but the problem with this approach is that the keys aren't secure if you have keys that are being stored on a developer's laptop or a server or in the cloud they're gonna be very unsafe and accessible by potential

hackers then if you look on the other side obviously if you have a single point of signing it's going to be extremely but it's gonna be also very slow and that that often doesn't work for today's business so let's talk about some of the risks and challenges around code signing this is a great read so if you wanted to find out more about code signing in the risk around it I would suggest that you read this very short report from sands Institute and basically it's all about how hackers have now pivoted in their attack vectors to where they're now attacking the code signing system itself so if you have developers with code signing keys on their laptops hackers

are going to be searching around your network trying to find those private keys on laptops if there is a lack of process within your organization the hackers going to try to exploit that lack of process and one of the the great quotes from this article is that it's this one that just showed popped up it is not an exaggeration to consider private code signing keys as the keys to the business's kingdom so let's think about this for a minute if you have a TLS certificate that's going to have a limited lifetime and it's going to be assigned to a particular IP so it's if it's stolen and it can do limited damage but with a code

signing certificate in key if the key is released that can be used to sign any kind of software and it's going to look like it comes from your business and if your business is in the business of producing software that's going to be a direct hit on your reputation that now people are thinking that your business is producing malware the other thing about code signing keys is that when you sign a piece of software if it's time-stamped that piece of software is going to be valid the the code signing signature is going to be valid pretty much indefinitely as long as that timestamp is still valid so that means that even if you have a

breach and hackers have stolen your private code signing keys and you've tried to revoke those those certificates if they've already signed the software it's not going to matter because that software is out there on the internet forever effectively and it's going to have your company's name associated with it so very very from a security standpoint to not really guard those private codes hunting keys with everything that you have so particular talents first one is around private key sprawl and unprotected private keys and this is really you know the issue where error we have to think about where those keys actually bein stored you know is it on some developers laptop is it on a build server it's on a

web server and there's a great example of what bad can happen if you don't store these things properly and I'll go back to that Zeus example basically this is what happened to them they had a web update server that pushed out their driver updates to all their customers and was done automatically those updates were signed with a Zeus's code signing keys and all the operating systems you know assume that you know if they were being pushed out with a valid digitally signed update that that would just go ahead and get installed and we see this all the time with their computers and laptops what happened is that for whatever reason Azusa decided to keep private code signing keys on this update

server and it was probably for convenience you know that they were rushing to get some update out and and some developer just assumed that hey let's just put the key here and then I don't have to be moving software back and forth between my build environment and in the update server so they they stored at least two private code signing keys on this server and some hackers were had broken in because the server is connected to the Internet and they were looking around and they found those private code signing keys and they also found the updates that asus had put together and so what they did is they just added the malware to the Aces

update they rescind it with the valid legitimate key from aces and then what happens that got pushed out to their customers computers who thought that it was a valid update because it was signed with a valid key and it impacted about a million computers worldwide so again you can see that by not protecting these keys you really open up a extreme risk for hackers dealing with your software so the impact to Asus was lost revenue you can see it in their stock price when this came out they lost market share it was a hit on their reputation because now there their customers were saying can we really trust what Asus puts out and depending on you know who they serve

what what industries there might have been some regulatory fines as well so what are some of the other challenges around code signing the next one is lack of policy enforcement and I see this a lot you know that I'll be talking to an info SEC team and I'll ask them about their codes on your policies and they'll say oh we have a you know we we've got very strict policies and you know we don't women will have any concerns and they'll show me a 30 page document that describes that and then I'll ask the question well how do you enforce it and that's where things break down if they don't have a way to enforce that with

the development teams then they don't really have a policy they just have a written document then there's also the problem around global visibility you know that if you're on an fo SEC team that's responsible for the security of your company and you have developers spread around the world how do you know what they're actually doing you know are they following the policies that you put in place are they using the right kinds of code signing keys are they following the right kinds of code signing policies and practices you really need to have that visibility if you don't that's gonna mean that your company is at risk for an incident happening like it did was with the Ukrainian company or or

aces and then finally you have rogue software development teams and I put rogue here in in quotes but their focus is not on security and their focus is on getting updates out as quickly as possible security is just something they have to do and I know that you've probably have heard this before so if you have a policy in place that that slows them down or it's inconvenient or makes them do different things that they aren't normally used to doing they're gonna try to find ways to work around this and we see this happening all the time with with the companies that we talk to so what are the things that you can do so I you know this sounds kind of sounds

kind of bleak but but there are some some steps that you can take to make this better the first thing that you need to do is think about co-signing as a process and it's different from a TLS certificate or another kind of PKI in that it's a process that involves people so it's the people that use it it involves things so it's the certificate code signing certificates the code Society in private keys and it's the activities so it's it's the build process it's the signing process and if any one of these things has has a failure or weakness then your entire codes on your process is going to be weak so think about trying to solve

this from a process perspective and it needs to be a secure process so how can you secure that process first and foremost you need to have secure storage so you know it is not acceptable to have a private code signing key on someone's laptop desktop machine a build server a web server that's just going to cause problems so those keys have to be stored in a central location and that that sounds obvious but the reason that it doesn't happen is that when when we do move those in to a central location then it becomes inconvenient for developers and then they go rogue and and they find ways to get around it you next you need

to have the enterprise visibility so even if you have development teams around the world you really need to know exactly who's signing code what certificates were used where the private keys at who did the signing who approved the signing and that leads us to number three here process control and and it's really important especially for for code signing credentials that are used to sign very highly critical software that your company produces or even software that your company relies on is to know who actually has access to use that key under what circumstances can they use that key who needs to approve the use of that key if your process doesn't have a couple of approvers that are required to

the approve the usage of code signing key you got a broken process that's that's gonna eventually lead to some some pretty significant damage and then when we talk about process we have to think about automation has to be easy to do so if you require you know two approvers to use every code signing key that needs to be automated there needs to be a way that people can actually follow your policies through an automated fashion if you require that every piece of internal code that use gets code sign you have to provide a very automated way of making sure those kites code signing certificates to get issued and and eventually rotated and revoked and finally you have to have intelligence

around what's going on in your company you need to be able to show compliance if you do have these code signing policies that you know that everyone's following them you need to have this ability to be able to detect risky trends let's go back to you know a development team that uses an automated build server and that build server signs code once every day at 2 p.m. but if you start noticing that certificate being used at 2 a.m. that's a trend that then indicates some risk it's out of the ordinary and this all has to be done to where it's developer friendly I just go back to you know rogue software development teams there if there's a way

for them to work around what what you need to have done they will if it's not easy for them so and when we talk about it being easy for them and he needs to not slow them down it needs to work with their existing tools and processes and it doesn't require for them to have PKI experience because they they often do not have that and the last thing I just want to leave you with is if you wanted to read more about best practices there is this great piece from NIST that's a security consideration it's for code signing so if you wanted to look at how you can improve the the code signing situation at your company this this

article would definitely be helpful there and I just want to thank you all for your time this has been really nice to be able to participate in this and I'm happy to answer any questions hey so this is about um if you want to go into slack and go and connect there's been some questions popping up if you want to go and glance at those and maybe have some conversations I'm going to keep us on schedule or try to keep us on schedule I know I was I ran over a little bit and you got started a little bit late so I was trying to cut you off and let you have all the time that you needed for your

presentation but I know it's important to keep us on schedule so if you want to go into slack and ask any some questions and maybe give them some comments about his presentation we really appreciate you showing up today and getting up early I'm presenting at besides Atlanta it's very helpful and important to us and we appreciate your time ok great thank you you're right the first comment here from Franco Sonia is not a panacea particularly in the supply chain is predominantly open source but this is this is what you can use code signing for it and in that particular situation is that presumably your development organization working with your organization when they start to use open

source software they run it through whatever tests and security tests that that youth you deem is necessary and required and once you've deemed this particular piece of open source software and it's you know source code then sign it so that you know that no one on your team or in your organization can actually modify that if they aren't supposed to and so we have customers that are signing you know multiple aspects you know they're if they're getting source code in from a third party they sign that when they receive it after they run it through their tests if they're using libraries for third party after they run their tests they sign it so that they

know it at no point does that get get changed or modified by either a malicious employee or a third party that's broken into your network but yes I agree it's it's not a not a solve all but but definitely it can go a long way to helping prevent the problem that Maersk had and even a Zeus I mean a Zeus is a great example where they just literally did not protect their keys and can I get the name of the article that you recommend again it's I mentioned two one was a NIST article on security considerations then the other article was from the SANS Institute about code-signing security concerns but thank you very much I know I am over now so if

you have other questions feel free to reach out to me either through slack or Twitter or or my email address okay we appreciate it thank you