When Doing the Right Thing Goes Wrong - Impact of Certificates on Service Based Infrastructure

service exper something I left that word out sorry on service based infrastructure all right if you're uh if you're not going to listen to The Talk would you take the conversation someplace else it's really loud in

here red

shirts all right Robert Lucero is a physics major turned software developer he has nine years of software development experience working on various projects at Microsoft and uh more recently at OCTA an Enterprise identity and App Management Service in my phone oh good I got it uh judging by his untapped badges if you want to know more about him he'll probably tell you over a beer uh but he might make you run a marathon first or beat him in a video game marathon or maybe just you know pay uh so you know maybe try that first um but not during the conference because no alcohol uh instead please give a warm welcome to the guy who will be telling

me where to find all the best beer in town tonight Robert lero awesome thanks for that introduction uh had no idea that I would have been researched uh or most of that stuff being read uh cool hi everybody um so again this is when doing the right thing goes wrong and specifically I'm going to be talking about how uh certificates based uh infrastructure uh oh thank [Music] you we're learning we're learning how microphones work and I'm I'm pretty bad at this sort of things so um yeah so so people tend to deploy certific ific Ates and htvs uh internally in their own systems behind the scenes not just for uh front-facing web properties and that

can be fraught with problems and we're going to kind of go into that throughout the talk so again a little bit about me uh I'm a software engineering test I've been a developer I've been a program manager I've been a product manager uh and more specifically and kind of where my heart lies is developing software to test software and I had a really good opportunity to do that when I was working at Microsoft uh I came in at the tail end of the is7 development these demos will be using is8 yes I did patch it um and I got an I had an opportunity to kind of work in this space uh as my first job sort of

out of college um I worked with a really cool developer we got to learn a lot about it and again I've been working in software development for the last nine years and the last the thing that I've noticed is that certificates constantly keep coming up people want to use them but it is sort of a black magic it is sort of a a mystical art especially if you're deploying it uh for the first time within your your companies so this talk specifically we're going to go if if you've never ever had to work with server certificates um this is going to give you a quick introduction to it um we're going to talk about server certificates

we're going to talk about certificate authorities we're going to talk about trust chains um just as a quick show of hands are there people here who have never ever used these before have there been people here who have deployed their own Casa all right you guys are you guys know more than me probably um how many people have run into trouble after they've deployed their CA bam see same group of people um so we're g to kind of go in um I'm going to describe a simple U microservice architecture it's going to be an htps uh a uh API endpoint we're going to have a client talking to it it's all going to go uh be secured with htps a certificate

and then we're going to look at some of the bad practices and we're going to identify some sort of Trends in what makes these things vulnerable and that's both from a client side so the client connecting to your service as well as the service side what makes it vulnerable and not vulnerable in the security sense all the time but as well as being up and highly available and then we'll talk about some real life examples of bunch of links um I happened to be at Microsoft during two of these incidents which was cool uh because then you got to see people running around uh trying to fix certificate problems um finally we're going to kind of talk

about the pros and cons of implementing certificate authorities uh and specifically thinking about why you're doing it um just that last talk that we talked about uh if you were here they're like everything should be over htps I would argue in some cases that's not always the best thing um depending on what you're trying to secure and depending on what you're trying to do and the key thing is it's always finding the middle ground implementing what you need based on your requirements and identifying how you're securing your critical infrastructures so let's jump into it certificates um we're just going to take the really quick and easy way to do this this is the Wikipedia definition an

asymmetric cryptographic public key with a verification signature um it's a way share secrets but built into it which is really cool is a way to uh verify that who you're encrypting your data with is actually who they say they are um that's that's what really makes it key it's both an encryption mechanism and and an authentication and a trust mechanism um we use it most of the time in hsps you know connecting to a web endpoint it encrypts that communication you know that you encrypted it with one certificate it's going to go over the wire it's going to get decrypted um can be used as an authentication mechanism so people who use client certificates I

issue a certificate to you you authenticate to my service via your client certificate I know that it's you and you're only talking to me because I issue you that public CER um last but not least these certificates are only valid for a period of time they have a specific validity period and anybody who's deployed their own Casas know that certificates expired how many people have been caught with an expired certificate so certificate authorities these are trusted authorities that issue digital Sears these are the people who sign certificate requests and basically say I trust that this person is who they say they are they're kind of a a middle point there're somebody that you can verify against and I'll kind of go into

a an example here in a few minutes um it's used as that common point of reference to determine if you're a valid certificate um um they also published the list of revoc like a certificate revocation list a list of certificates which are explicitly not to be trusted um if for any reason a CA is compromised you don't want people um using those certificates maybe they were forged maybe people were making certificates in your name um so there there's a lot of reasons why you don't want to trust somebody even though it's been signed in the first place um other really interesting point is that root certificate authorities are installed everywhere everybody has them if you use

your phone if you use your iPad if you log into Linux Windows or Mac OSX there's a set of root certificates that are already installed on the box that's that's it's a very unique special thing that makes sort of the the web infrastructure work the way that it does um I as my own certificate Authority can't become one of these people who is installed on the box uh and we'll go into more details on that here in a minute as well so we talked a minute about revocation lists um these are again lists of certificates which are explicitly not trusted they're published by authorities so as you're going through a a trust check you want to know

hey did you did you revoke anything should I not trust anybody that I'm talking to uh and it's kind of paired with the the validation of a certificate so what makes this all work is being able to identify a trust chain so I have my own certificate I go to Vera sign and I ask them please sign my Certificate request and then they sign it and they give it back to me and I put it on my website then when somebody accesses me they come and they talk to my site say hey here's my certificate and they're like who signed you and I say ver sign did ver sign's like yep totally did Rock on uh in that case we're assign is the

root CA so how can we make this in more readable terms I have a passport I am now giving my passport to a bouncer that bouncer uh looks at my passport and he does some checks right he basically opens it up he looks at it determines if it's forged checks the checks my birth date to see if I should go into the bar um bouncers of z i are very picky about that they only going to take you know certain forms of identification hopefully my passport is you know valid but the but the important thing here what what makes this really interesting is the state department issued my passport and I have an implicit trust to the state

department and the bouncer at zge also has uh an implicit Trust of the state department that is kind of the root Authority Bas we both agree that whatever they give out is trusted now if we go through the same sort of example if I were flying into SFO I hand them my passport I hand it to a Border guard a border control officer he opens it up but what he does is he looks at it he determines if it's valid as well as scans it to determine if it's been suspended or revoked or if he should do anything else so he goes through a separate sort of trust chain to verify that my passport is who I say I am this

is the authentication side of it this isn't really the cryptography or cryptography side of it um and this is kind of what what ends up being more fraught with troubles and dangers uh not necessarily like are we using the right encryption algorithm but more of is the system going to work when I go and issue out certificates so a lot of companies do this when they go and they create their own certificate authorities this is a a basic sort of trust chain example you you create your company's root certificate Authority you create some intermediate authorities sometimes the root certificate issues there root Authority issues certificates the intermediate authorities issue certificates and if you want to think of

an example it's I have a VPN cert I have a VPN server and I want to have people authenticate with certificates so I create my VPN certificate Authority and then maybe I'm using mobile device management so I'm going to create my device certificate Authority but they're all anchored by my own internal company uh certificate Authority so we want to look at it one level further and kind of what we're going to highlight here for for some of the demos um up top we have kind of be like the web facing side of the world if I'm going to create a web property if I'm G to go put a website up I need a certificate that everybody trusts so I'm

going to go get a certificate signed from a top level Authority go to at a ver assign digit whatever um when somebody comes and connects to me they're going to verify that certificate via the root authorities that they have built into their mobile devices or on their OSS that doesn't work the same way if I've built my own internal certificate Authority I have to find a way of getting that root trusted root Authority onto all the boxes that are going to verify it so so here hopefully this example kind of makes sense we have a web server that's doing some work and then behind the scenes that web server is now talking to some sort of job

server again using the microservice architecture it has an htpn point but it's using an internal CA versus you know digit search sign CA cool all right um we're gonna kind of keep rolling with this so I'm G to give you a quick demo um I'm going to kind of go through a few examples on my box um and let me know if this doesn't make sense or if I should explain more I'm the last Talk of the day so I'm between you and whatever parties you might be going to um here this example we have a web service that's just an HP rest service we have a client application that can be a mobile client that can be

another web server that can be anything just talking HTTP to uh my rest service and then in this example specifically The Client app is sending credit card data over https to the service and the service is just storing it so we have some sensitive data that we want to protect um and the other key thing is the the microservice the the rest service has its certificate signed by uh an internal CA I created an internal CA to sign the certificate so demo time all right who wants to see a demo okay now I didn't practice it this way so we're going to do a little bit of we're going to kind of pivot and turn

hopefully not get in the place of the mic say that one more time that is it's it's all in one I wanted to tell you that that this year that's what the the Defcon badge is going to be um I actually just changed this background to be less nerdy it was like an ocean pick I just found something from last year that I figur people will get kicked out of okay so um real quick we're going to kind of just show off what we got running over here so right now um I'm going to use a couple of tools I'm going to use Postman to just make web requests to my service just an easier way to show it and then

I'm going to show you the client's application making requests to my service and then we're going to read some data back and forth so number one thing let's talk about um let's talk about the the service itself so let's go to my endpoint we'll notice one thing when it starts up here I have that lovely lock that everybody loves loves um Chrome trusted it which is awesome which means obviously I'm sharing information correctly uh it has one problem with it it's kind of complaining that I don't have audit records it let me log in so I really don't care and it says hey you know we trust this and if you want to look at

the certificate you know Windows it thinks that my certificate's okay you can go to the details you can go to the certificate certification path um certificate is okay you can't read that you can trust me um and then here the the root Authority it certificate is okay it's because I've installed it on this device now what's the big difference between like a web browser client and like another client like a mobile client that's connecting to my service so suppose that I go to maybe an evil Endo uh

okay so if if a web browser notices that there's something wrong with my my trust chain Chrome every browser now does this it basically says you're not I shouldn't go to this website I'm going to actually make the end User make a decision if they should continue uh sending data to this website specifically it says you know the certificate Authority is not valid and my first option is back to safety it's not okay keep going you know I have to click an advanced button and then I said it couldn't prove it and then I can I say that I can go on so we're g to we're going to stay away from that website um just for the example

perspective um wow this does not scale well uh we're making a get request to the endpoint there's some credit card data already on the server um we're g to post some extra data so Johnny Test we're going to send it along uh here it's getting a 2011 created and then if we go back and we we say Hey you know give give me that data we show that it's sted it that's basically the premise of the service and then we're going to go over here if I can drag this back over we're going to run this this client application which will spin up in just a moment and here we're just going to be every five seconds we're going to create

a new user we're going to store some sensitive data we're going to store some credit card information all right so any questions about this service the idea behind it real basic I got it um cool all right let's jump back into the presentation let me make sure I took notes to make sure that I wasn't skipping anything yeah it's pretty much it real simple service but it's the simple services that are the ones that we depend on so going back here crap this is where my S current slide cool all right so let's talk about how this service may be vulnerable so who is the certificate Authority you know is it online or an offline certificate

Authority how much data is being published online where do I get my intermediate trust uh certificates where do I get my crls simultaneously if you have a a client that's connecting to it how do you know that that's properly configured what is the level of trust in the certificates is it doing error handling correctly what protocols are being used you may have spent you know six months and a whole bunch of money and a whole bunch of time making uh you know a certificate system setting up https and find out that everybody is still talking HTP to you know your service what's the point uh again trusting too little or too much and then the error handling specifically

you know that that's a challenge like developers La I being a developer am lazy uh and when I'm testing these things setting up a trusted chain is is often difficult in my development environment I can't go to my operations team and say please give me a valid certificate they're going to be like why would I do that I have to revoke it anyway so let's let's talk about kind of the first vulnerability what we all think of a of as a vulnerability in in a security sense this is the man-in-the-middle example right I have a client's application has been tricked into going to the evil microservice which is then turning around and relaying the information back to the

good one it doesn't matter it's relaying it uh it's basically sitting in the middle right everyone's kind of familiar with this concept with this idea and there's a lot of different ways that you can trick a client application to kind of force you to go somewhere you don't want um for example the go- go thing where they were intercepting all traffic they were sitting in the middle um you know if I get somebody on a fishing attack can I get them to update their configuration on their phone to post to some other random endpoint you know these things are problems these this is this is why understanding how your client's application works in regards to

this system is important so the next one thing is let's let's demo this let's try this out so and just a quick question if you're up close you guys can read this right yeah cool all right so what we got going on here is okay drag everything over so let's let's stop the service let's take a quick look at the code so I'm going to basically fake um my config being compromised so we have a client we're posting data and and now I'm compromised that sucks um and if we take a look here we're going to notice something I'm G to post data oh crap uh it's still working uh why is that still working

that shouldn't be happening so let's go figure out what what's going on so I I changed the endpoint right it's still working still posting data my my client thinks everything's fine and and even from from my perspective on on the site side you know that's a lot of people uh Mindy Stone did Mindy Stone get created yeah Mindy Stone got created you know everything looks fine to me here remember that evil website that we were talking about a few minutes ago that I shouldn't have gone to when it goes oh man it's an evil API and they're posting all my secrets on this website um so we're getting man in the midd it's very straightforward what's different is

that my client application didn't care it was still posting data still throwing data up there didn't do any checks why did that happen as a as an end user through a browser I got blocked I got stopped and they were like you should not go here something is really really wrong uh let's go figure out whoops thank you Windows um let's go figure out what's wrong let's say take a look at one kind of area can anybody spot the bug I will buy you a beer if you spot the bug anybody three two one okay that's okay this is this is some crazy C this line right here um if you ever get bored go search

for how do I buypass certificate checks and you will see examples for Android iOS windows yeah super simple line uh we can talk about C and delegates but this is wow holy crap and and I have seen this I've actually seen people write this code in production systems because they they are setting up an HP service like this is this just makes it easier for me to test I I know that it works and I don't have time to set up a trust chain so this is a bad bug this is a dumb bug let's fix it I really should delete it put a comment say never ever do this but we're you know we're hacking right we're at

bides okay we're starting it up again now we get this now we're now we're broken that sucks well it's good right because we don't want to send data to a bad website this is this is one way to to protect our service we want to at least trust that we're SSL and we're we're we're sending it to trusted certificate implant so this is one aspect of the problem problem so I don't I don't want to keep sending errors let's go let's go back to base URL the the happy fun one before I got compromised maybe I wipe the device cool and then we'll let that keep running and then key thing here we have validation now it's a trusted end point

and the the the application continues to run so exactly what we want cool awesome things are rolling

now let's talk about the problems with strict certificate checking okay we we fix the bug everyone loves fixing bugs right we're more secure than ever you know that certificates aren't going to harm us anymore but now we need to know all of the various ways that our infrastructure can fail behind the scenes when does that certificate expire is that certificate revoked did the crl expire because that's part of your trust chain uh are there connectivity issues to the places where I'm trying to validate that the that the authorities are are valid uh do I have a fully trusted

chains how much information leaks it depends your crls are very public you those those things it really doesn't matter because it really just has the signature of the certific and says don't trust

this so um there an interesting question so so we I I'll I'll and I'll be better about an answering that responding so so the question was should I be worried about leaking data um it depends and for the most part crls even even crls for for root Casa they're HTTP it doesn't really matter because it's built into the certificate itself that that's the end point that you should validate and they want to make that very easy to get so optionally you know if you intercept that and you start inserting certificates you can cause a problem so that's that's a very unique way of attacking um so so here this is the fragile system that we have in place

right now my my certificate Authority signed certificate and if you notice that this is a lot of repeating of the same words when I was practicing this I realized that that was a big problem so my service has a valid C the client application has to go to the internal CA to determine if it's valid there's a thousand different ways that this can break because you're encrypting you're you're sh you're sharing encrypted data and things can go really really wrong if you sharing data with the wrong people as we saw earlier so one more demo um we're going to kind of go we're going to go into what happens when we break trust when we

revoke certificates and then if we you know just fail to even follow the protocol all right so while this is going on I want to make sure that this is still running and we're going to do some adjustments here so all this is going on and I'm going to bring up I I don't actually have I'm in airplane mode right now to make sure that this all works um the way that we're going to kind of do this is is I I like Windows for these demos because Windows has its own internal certificate management it makes it a little easier to experiment with you can do all of this stuff um via op SSL and Via uh Mac OSX or Linux or

anything like that where you have access to either the keychain or your own trust chains you can revoke certificates and you can do a lot of this and and setting up a service like this makes it makes it a lot of fun and kind of makes it more visual so the number one thing that we're going to do as I go back to my handy dandy notes um the key thing is is is we're we're trusting the the internal CA so if I go to my trusted root authorities here we're going to notice that secure arero test my root CA is installed to The Trusted authorities and if we go back here I want to verify the certificate

again my certific certification path is USS Endeavor and it was signed by secure our Lucero test and we're going to do is we're basically gonna going to remove that implicit trust and we're going to see what happens so if I uh come on Windows delete this Windows really doesn't like you deleting things out of the root CA obviously they got placed there for a reason unless you got compromised so that's a good reason yeah so okay other fun thing I noticed this while I was setting up these demos when I refreshed it it still trusted right it's holding on to that session it it's it's Chrome uh and uh uh Internet Explorer both did this so we actually

have to to do the the really ugly you know start up another one let's let's try this so I'll slide this over in just one second okay so this is this is iie IE is saying the exact same thing oops um click here to close this web page you know why why is it not trusted and you know it's not telling you much not Windows doesn't give you or uh Internet Explorer doesn't giving you that much information I'm actually there was a way to do it if you continue to the website site you know you get the big red bar again and if you view the certificates it says you know cannot be verified up to a trusted

Authority so so even though I created the certificate for my service and I trust it I don't have I don't have the the capability for clients to verify the root Authority because I deleted it it's not there and I don't have a way that you can get it over over the web and and if we were to look in the details sometimes you you have information about where where you can verify that root Authority so so that sucks we should go fix it I I like running a trusted site so I'm going to go back over to this side because that's where I have it installed um let's quickly fix this problem so we're going to import this uh

other other important point of note don't do this unless you know what you're doing um adding random certificates to your trusted r Authority is not recommended but this is just my machine so hey why not so yeah see it's telling me that I'm trying to to trust the root CA the import was successful and then if we go back to window IE come on uh SE API and then B we're back in business okay so that was that was one uh the next thing you want to do is let's talk about how how do things look when we revoke them if I explicitly don't trust them so suppose I bought my certificate from super cheap Hoster and Vera sign

revoked that certificate that would be bad um we want these things to work I like it when my certificates are trusted so suppose we do this suppose me drop the root CA into my un trusted one and we go through the same route again we should have just saved it like this bam click here to close the web page and if I click more information damn this is right because I did this all through Chrome we're kind of we're kind of doing it live on a couple of these demos let's go take a look at

chromes internet disconnected oh oops that's because I tried to

search huh okay that didn't work I think I revoked the wrong thing that's okay we can we can revoke a whole bunch of things uh use this endeavor

and again we're doing this live so let's close all window let's bring this back up bam there we go okay uh that was one thing I did want to point out I forgot which one did it I was practicing a little bit too late last night um Chrome is really good about kind of expressing like what's broken and if you notice here now that we broke the certificate Authority will notice that that my client while it was continuously running has started to freak out again like so this is this is good but it's bad in the sense that my certificates revoked this means revocation is working um we want to test these systems because if I do

spend money on a certificate or I do build this system I want to know how it behaves when these things go down this is a really big problem when people go off and create their own internal Casa and we'll talk about that in uh here in just a minute but just because you know we want to make sure we want to see things running we want to see things work let's go delete these they're no longer revoked and we'll see it again come on Zach green thanks Matthew usero cool we're back in business so what does this mean what can we learn from this big companies are affected by this as well every single every single one of

these these issues was related to a certificate and resulted in a major corporate outage uh I had I was lucky enough I I didn't work on these teams at Microsoft but I had the opp opportunity to talk with some of the people who worked on them and it's a very very interesting problem and again for everybody here who runs their own certificate Authority you know how much trouble this can be the first one had to deal with an expired certificate Microsoft was notified 180 days in advance that the certificate was going to expire weren't able to get it on the boxes due to a different bug um the service disruption the second one there February 29th it's subject to leape

problems right most certificates only have a lifespan of one year uh unless you're explicitly overwriting that and how you're issuing it if you're running your own CA nine times out of 10 it's one year Google uh their trusted root Authority was compromised they were off issuing certificates as if they were Google right that just happened in the middle of March and then I just as I was searching around for this uh smtp.gmail.com one of its intermediate authorities expired and Gmail was not sending mail for like two or three hours so these things are critical these these little certificates that we depend on so much like they're part of these our systems and we need to make sure that we

know how they work so yeah try not getting revoked all right so let's talk about the pros and cons of internal Casa and and we're going to kind of wrap this all up here because I think we're coming to the end um the pros of running your own internal CA you you have control you get to control everything that's one advantage of running your own thing anyway uh you get your own your secure encrypted data that you wanted uh if you did want to SSL pin things uh you know that it's really trusted you control both the certificate as well as the pinning on the other side uh and if you do have to reissue or you do have to

revoke it happens much faster assuming that you manage and monitor it effectively which is the number one con these things are very difficult to manage and monitor if they need to be highly available you have to build them that way uh bad implementations will will underrot systems so if I'm a crappy developer and I write in my certificate bypass um no point in having the service if um if I'm not implementing it correctly and I'm giving all my certificates 30-day expiration periods I'm going to have outages like we saw with the example of the the client application um if somebody were to get into your system and find your intermediate Authority they can now issue certificates as if they were you

that is really bad um that's a dangerous thing so so these are very protected systems where they to be depending on what they're being used for if it's for people to access your VPN now you have people random people who you don't know getting access to your VPN um and the other hard part about running your own internal CA is that you have to be able to distribute your root CA or make it available for people unless you manage the machines you have to go tell somebody hey go put this certificate on your box oh you know those warnings that Windows said and and probably Mac OSX said oh why are you installing a trusted rth Authority just

ignore those it's okay because it's my certificate or it's our company's certificate so how do you find the middle ground when you're building these Services if these are all way way behind the firewall maybe and you're you're storing data that isn't that sensitive maybe you don't need it um people will argue with me probably left and right about that but identify the risk areas build the system in a way that you protect your key areas um again design and implement it accordingly do you have to protect the whole world how many individual microservices do you have to protect how many end points do you have to protect uh and then weigh the costs if you only

have to protect like six end points maybe it's just more cost effective for you to go get six digits you know put a calendar reminder update them once a year drop the whatever 600 bucks it would take to to keep those up and running save you all the headache of of going in and creating your own internal CA um here are some tools that you can kind of use and think about it um windows I love windows again for this because you can just open up the certificates uh or you can use cert util and OSX and Linux op SSL does everything that we were just doing uh you just go from the command line you just run it it

dumps all the data it says if it's valid or not um when you get a certificate open it up review the the the expiration date go look at all the endpoints that it relays you out to does it have an internal Authority can you access that intermediate Authority and then access the crls make sure that they work make sure that your systems can access it uh I recently had an experience talking with a group of people where they couldn't access a crl because their firewalls were way way too strict if you're working on clients or people that are talking to you know microservices or service endpoints code reviews are critical have people who are familiar with these

Services you know understand them uh test with all the various expected scenarios if if you're going to build these test them right um it's hard and it's timec consuming and people will argue about them but uh argue with argue with you about them but it's it's really important and then if you're using SSL Penny test your Rey strategy if you have to revoke a certificate that it's SSL pinned you're going to hurt a lot of people if you can't re key it effectively and this is it um know what you're getting into before you set up your own CA test and monitor have practice drills be able to redeploy be able to revoke be able to reissue and then um I documents

and identifier your system just be proactive about the system that you're building that's pretty much it if you have questions you're welcome to email me or uh ping me on Twitter or LinkedIn or whatever uh I will put these slides up I made some last minute changes they'll be up on get Hub and again for anybody who has questions feel free to ask

now I've

been