PG - Check That Certificate - Jacob Jernigan & Andrew Sorensen

not

[Music]

also

[Music]

so

[Music]

so basically he was like that's

all

[Music]

so

all

there we go okay folks we're going to go ahead and get started I'm G to introduce Andrew and Jacob and their talk is check that certificate genten all right all right good afternoon oh sorry ni bad way to start the talk all right good afternoon my name is Andrew senson this is Jacob jigan and today we're going to be presenting check that certificate project to make it easier for developers to communicate securely so SSL or TLS is critical to the security of a large number of applications including desktop applications mobile applications web applications command line utilities database they all use TLS or SSL to communicate with another system and while a lot of research in SSL or TLS focuses on

theoretical or difficult to exploit attacks um such as and have resulted in vulnerability such as Beast or poodle um there's a whole entire class of vulnerabilities within TLS that are simple or easy to exploit and these uh stem from a feature in TLS which is often considered optional by developers however it is completely non-optional uh and must be performed by the developer so unfortunately certificate validation is an aspect of SSL or TLS is often seen as optional raise your hand if you've seen a post similar to one of these or working on a project or uh trting to get someone else's project to run okay and of those people how many of you have for example taken one of these

sample these sample Snippets and either used it to get something to work for example curl DK wg-- no check certificate uh x509 trust store right we've all used these things right but why do we use them well we're trying to turn off a security control so we can get something to work right we don't have a lot of time to get our project to work and we need to get it to to do what it's supposed to do and so what we do is Trop certificate validation and it works right it always works it does exactly what it's supposed to do but what we don't notice is that we're potentially communicating with anyone right we've

turned off a critical security feature of TLS and when we do that we subject ourselves to in the middle so unfortunately these type of posts highlight this problem within TLS that if you have a security feature that can easily turned off well someone's going to come in and do it they're going to turn the feature off and they're going to be vulnerable so if someone who does a lot of security or application Security reviews occasionally I'll come across software that uses TLS or ssls and transport to communicate with a remote server in fact frequently applications do this right more and more applications are communicating with a web server or an API and they're doing that to perform

some sort of sensitive operation on the back end they might send credentials to the server they might send data to process to a server Etc all these all these use cases for it now unfortunately it can be difficult to determine whether an application actually does certificate check uh certificate validation now it's easy to stand up a server and man the middle of connection but for an average developer the tooling and time it takes to actually set that up and actually carry out the attack can be significant and on the other hand it's easy for an attacker to set this stuff up because they're motivated and they have the potential outcome of being able to obtain all the sensitive information

sent on that transport so um and let's just imagine that you find an instance where certificate validation is missing well it's then difficult to tell a user or developer how to solve that issue because uh again you run into those posts right you say uh you do a Google search for how to do certificate validation in PHP python C dot dot dot and you come back to a post on how to turn it off inad of how to turn it on right so you struggle with this thing you don't know what to do with it and at the end of the day you end up turning it off which is not what you want and so

while you have people on the kind of the the the quote on the top really illustrates this right so people who don't understand why it's necessary to do certificate validation so the first quote came from a blog post uh that was providing details on how to turn off certificate validation and so this user comes along and says well thank you very much you solve the biggest problem in my project right it works now I've got my code to work it's communicating with you know Google Maps a whatever it might be right but they've completely turned off uh a critical feature of SSL and they're they're happy about it right they turned it off works great and then you got this

second class of people right that just don't understand what the impact of Navy turning off peer verification would be right so they they've actually looked the documentation here because they've got a specific option and argument that they're playing around with right this SSL verify Pier thing what is this right and so well they turn it off and then they post on this this discussion form and say well you know as I understand you can turn turn this thing off and uh curl will still communicate securely even though it doesn't verify the peer certificate right so there's something in there right there's something that they're not doing what is it that they're not doing well they're not

checking their certificate now what does that mean well if you don't check who you're communicating with you're basically using encryption right you're using encryption you're still using encryption you've always been using that but you're not checking who you're communicating with right so I can I'm establishing a secure communication with so and so in the audience right but I don't know that it's actually you it could be someone else anyone right just so as long as they happen to be at the right time in the right place I'm communicating with them and I've uh I'm communicating insecurely because I'm not verifying who I'm communicating with so when perer verification is skipped the connection is only secure if an attacker

can only intercept traffic and they cannot modify so they can only passively capture it they don't modify it then it's secure but as soon as they can make any modification to it it's done the attacker can stand up their own server you're you will freely communicate with it and you'll share all of your secrets with it defeats the purpose of SSL so the focus of our research uh as I mentioned earlier a lot of research has been performed on kind of three different categories of SSL or TLS design cryptography and implementation a lot of a lot of research in those particular areas reveals um kind of interesting vulnerabilities within SSL or TLS for example implementation heart

bleed right disclosure sensitive information that's in the memory of the SSL server right that's a big deal uh also the clients in certain cases cryptography right all the issues we hear about with rc4 to uh you know certain ciphers being bad export ciphers weak Keys all those sorts of things design considerations right like well you know you designed your your your protocol in such a way that it reveals information about whether the padding was incorrect or not well okay then you have a vulnerability but in particular we're most interested with this last category which is using TLS right so TLS is complicated you can screw it up and if you don't use it correctly in

particular you don't do certificate validation correctly uh you could be subject to a man in the middle and then all of your security guarantees all this stuff that you everyone invest all this time into developing becomes useless right doesn't provide any security at all so two major areas of previous work um that are that were notable for our research the first one is a uh research paper published by the uh Stanford crypto group in 2012 and in this paper titled the most dangerous code in the world validating SSL certificates in non-browser software they basically went through and evaluated a lot of different software packages to see how well it performed certificate validation and they came back with some pretty

devastating results as you might imagine for example the Chase Mobile Banking application for Android the PayPal SDK and Amazon payments sdks did not validate the identity of the remote server and when they don't validate the identity of the remote server right they're subject to man in the- middle which means that all that information that that those clients were communicating with the server is now all subject to man the middle which is bad right cuz we're trying to S sensitive information and now it's potentially disclosed um and while this paper was really focused on solving specific instances of the problem there's another project that kind of took it a step further so this project from ISC

Partners another Consulting Group uh basically what they tried to do is they put together examples of how to use SSL or TLS correctly in particular they focused on certificate validation right it's a problem at hand problem that we're most interested with now unfortunately they were really limited by the number of examples that they uh that they had in their codee base to to work with so they only had an example for C in op SSL which is a difficult one that people frequently mess up and then they also had one for iOS and uh Objective C which is also helpful but again it doesn't encapsulate all those other cases where people do certificate validation um in desktop or mobile

applications and would need an example of how to figure out how to do it so now I'm going to hand it over to Jacob to talk about some of the specific areas where uh people kind of confused use or misuse TLS okay so excuse me so Andrew talked a lot about um the developers and their knowledge um but what I'm really going to focus on is libraries because as I'll talk about in just a second uh libraries either have incorrect documentation or what we've kind of dubbed obscure implementation flaws where the library will do one thing but the documentation says it does something um not completely different but different enough that it could be a

problem so um TS is also checkbox on an application security feature list um right so I have TLS enabled and we're good except obviously we know that that's not necessarily true um because we need to make sure who we're talking with along with some of the other issues that are outside of the scope as far as ciphers and things like that um please excuse the pun but outside of the scope trust and Key Management are key failure points in cryptography as well had to include the terrible pun somewhere so understanding the complexity uh certificate Val ation is extremely difficult um especially because certain libraries effectively leave it up to the user WordPress is a good example of this

where you have PHP and WordPress had to write their own certificate validation um because the one in PHP is kind of hard to use and the problem with that leads into edge cases like um non-standard characters like puny code domains or wild cards and you have to account for those and all of the failure cases that could come with that what that leads to is larger projects can often like WordPress get it done but smaller projects either don't do it because they don't have the resources or they do it incorrectly or non feature Complete because they just don't have the resources so a good example of the the documentation not fully explaining the ramifications of an option would be WG

so the the WG uh option CA certificate uh in the man page you'll see without this option W looks for a CA certificate at a system specified location what that does does not tell you is that it also uses the system trust store so if you're not expecting to use a system trust store and only passes CI certificate that you have specified you're going to run into a problem and an example of this would be like if you have a mano laptop vulnerable to uh superfish and you don't want to use that certificate and you try to use this option it's not going to work so net net is is a really interesting bug um if you look at the

equals method for an x509 object which is your kind of default uh object for SSL or t certificates um you'll notice in the excuse me the documentation that says uh very specifically two X5 objects are considered equal if they have the same issue or in serial number that should make you very afraid unfortunately in the remarks there's nothing about uh the cryptographic secur security of that uh and you actually have to look at the windows API the underlying calls for that from the net to find that that's insecure so if you don't know that from reading this line you can use it and we've actually found a few projects where that had happened um unfortunately if you happen to know

that a project is using this to validate a certificate um you can easily create a third certificate that you can use to man in the middle so python the https connection class does not perform validation by default in certain versions most of the newer versions they do but anything before 2.7.9 I believe does not um the problem with this OB viously is that you'll never know because if you don't get an error in development you'll just ship the code and as a user you know if you expect it to be secure you have no idea until you read some documentation eventually hopefully and find out that it's not so open SSL uh this was an

interesting that we kind of tossed around whether to include or not but s client very specifically mentions that it's a test Tool uh but we found a couple of pieces of critical software that specifically a VPN client that actually use it to connect to the VPN server without validating the pier so obviously newer versions of asls client do include an option to validate your peer but these particular pieces of software we found did not use that so even when the library is correct working with TS is still hard as as we've kind of tried to talk about you know if you're working with internal service you have to try and roll your own pki and manage not only that with

the ramp of time but you have to try and securely distribute your ca files and manage all of that across your network obviously there are tools for that but that's a significant amount of ramp up time that a developer just might not have so local development or excuse me um let's see so the the other interesting case that I like is time so if you're not managing time on your server or on your client um you could possibly run into a failure case where a certificate is either valid or invalid because um you're within the window when it really shouldn't be so as we like to do insecurity let's go ahead and blame the developers

because it's just all their fault but is that really fair because as we've seen the documentation is either limited or incorrect the need to perform validation isn't clear to a developer if they don't know anything about TLS and to be honest if you consider when you're trying to ship some code and you're not aware of the security implementations and you just have to get it done because it's due at 5:00 on Thursday because you can't ship on Friday so it's got to be ready for Monday let's just disable verification because it just has to be done so I'm going to go ahead and pass it back to Andrew for the conclusion all right so in incl

conclusion right this certificate checking is a really difficult problem right and developers often run into issues with it and their first instinct is to turn it off because well you don't see an easy path out to do it correctly right and so your only option is to just turn it off and proceed along right well that shouldn't be the case right so let's try to see what we can do fix that so what we did was we took the approach similar to ISC Partners to write documentation on how to do things correctly and while we're still vetting uh kind of our examples what we're doing is we're going through and looking at all these different libraries and

Frameworks and trying to find code that I would actually do certificate validation so right as a consultant um you know frequently I review applications that need to do certificate checking and I always had this struggle with uh needing to be able to provide a recommendation that someone could take and we're act on right so they need to be able to have need to have something that a normal developer not a security person not someone who understands TLS completely can take and Implement certificate validation right so they shouldn't be dealing with things like host name verification writing that themselves peer verification how to deal with uh you know bundling certificate authorities with your system if you're

running you know like a sigwin compiled application on Windows right where you don't have access to the trust store all those sorts of things so we wanted to do is make it simple for developers to do certificate validation so we created these examples PHP python C and Ruby are some of the things that we're starting with and a couple of those Standalone tools that we described earlier like WG and curl and we're essentially making a website uh it's still still in the process at this point um but you can certainly look at it now um to give people a place to do the right thing so instead of uh you know in the future instead of those stack Overflow posts

that tell you how to turn it off and they're all always upvoted answer is always you know if you just do no check certificate you know you'll be you'll be fine and dandy and then you know someone will follow up and say well thanks it's great you know it works um instead there'll be some really clear uh directions that will say well you know the first thing you need to do is identify who you need to communicate with and then you get the certificate for that machine right and then dealing with all the formatting issues abtract that from the user and then this is what you do you take the certificate you bundle it with your application right

you do certificate pinning um and you can use those self- signed certificates right you don't have to go through your uh your admin your it admins that uh you know to do test test work on your own machine so instead of turning off certificate validation you can roll your own certificate locally and if you're shipping a mobile application you can do certificate pinning in your mobile application and instead of relying on the trust store right when you don't need to trust a whole bunch of extra parties to uh vouch for your certificate well since you already have it bundled with you well you just trust just the host of of the machine that you need to

communicate with right so we solve a lot of the issues with certificate authorities we solve some of the issues with having to deal with setting up your own pki and we get to the point where you can just look at an example and you can implement it in your application are there any [Music] questions validation PHP ask to go uh yeah so the questions about what about certificate validation and PHP so uh PHP is is an interesting one right so for 5.6 uh there's post name verification something that you have to implement by yourself um and a bunch of other details um some of those cases are a little more difficult for us to to work through in

since we're trying to work with a bunch of different languages actually the thing is I'm going to direct you to this next slide which says you know you follow up with those PHP Problems by you know if you want to help sure let's do it right um and certainly there's a lot of people that rely on PHP and they should be able to communicate securely that's totally goal um we don't want to Discount a language just because it's difficult we have a recommendation other questions uh yeah so you mentioned in your research you came across a lot of you know documentation like you know WG and C stuff like that that was either misleading or incorrect so as part of

you know the website work which is awesome uh are you actually going back to those projects and stuff and committing and saying hey we need to update the stock or here's a change list with these stocks and things like that right yeah so the question was more about like just you know you've got findings right so what what about reporting those back right so we been working with some of the parties for example on Microsoft we work with them to to identify kind of where the issue was within the documentation what kind of the best approach would be for fixing it um for the others sometimes there there are open bugs that been open for

many years like the s- client bug um there was a little screenshot back in there it said it actually been open since 2003 in bugs. debian.org right so you know 12 years later we finally got a fix actually it was fixed in 2014 so 11 years later we finally fixed this issue of getting certificate validation going python right it's it's been fixed right it's been fixing the language the problem is not so much um the language needs to update it's just that the supporting documentation around like examples of how to do it or sample code is incorrect does that answer your question yeah thank you and and also also curious you know you mentioned stack exchange and there's so many bad

examples like is part of the effort going to be to troll stack exchange when people ask these same stupid questions or you know not stupid but valid questions over and over like hey here's some good examples right because people are not going to not go to stack exchange because then nothing would ever get this so I'm actually good friends with several of this assignments for stack exchange and I think they would murder me if we did that so I think you know as time comes we'll try and um update some of those where we can uh and to be fair like we we didn't talk about it but on several of those posts it they there are

answers that say do not do this please do this instead unfortunately they're the hard answers and the easy answers often get uploaded because people oh yeah let's do this and it works right so we we should be fair and say you it's not all bad but most of it is thank you yeah and just to just to sum up on that note I think you know the other thing is that stack Overflow and these other posts are kind of collaborative nature right so there are things that can change over time I think as you know a project or any project that kind of attempts to do this right um provides more accurate and easy information right

some to some extent that will kind of trickle up right people get the right answer uh so depending on which language you're looking at sometimes it's a lot more or less work as you said to is right if you consider an addition to just providing examp providing libraries in these languages to like actually make it as simple as possible for people to do the right thing right so the question was uh essentially You Know Not only would you consider developing examples for particular languages but what about just in improving or wrapping those libraries um around with some other library to make it even easier I I see I definitely see a value to that right um

to some extent the question of who should write those libraries and whether they should be included or not in standard library is something that would be be considering right I don't really want to see people need to download you know yet another library to do cryptography right like the idea of need someone needing to download bouncy castle or some other you know alternative SSL library in python or PHP or any of those other things to me is not very valuable because there's still going to be people who have their own version of those things right but certainly yeah I can see definitely see a value to doing that um and you know nabe once we got the the rudimentary

issues solved right first level things the things you have to interact with in the library certainly wrapping those things up into you know a simple class or something like that that the developer can take and replace uh to override existing functionality within the language to make it secure by default definitely that's definitely valuable so we actually did something very similar to that in our net example um I think that one of our concerns was like to um we want to make sure that we're giving people secure code like we don't want to give you something that's now vulnerable so we want to be extremely careful about what we publish so um we're we're looking at that very

carefully

y experts yeah so that's definitely an option like you can go on there like we actually definitely want you to do that so you can go to the website and get to the GitHub and tell us we're doing it absolutely wrong and this is how we do it right please do that because that's what we're looking for we want to help people so yes definitely do that right the the whole goal here is to is to make this something that people who are experts in can get involved in right right now it's a lot of the same people posting who don't know about this topic that are posting back and forth about how to turn it off right so we want to

change the conversation than

question use CRS [Music] CP right so the the question there was uh more about crls or ocsp so um to us uh certificate revocation is kind of a less interesting case of certificate validation right because most of those systems to some extent work on the principle that you have to download a crl which is subject to denial service and then in that case the CR crl won't be checked um since we're kind of focused on certificant pinning in particular that also makes it a little bit less interesting but certainly that is a that is a highlight of our project and if you go to our our main page it list that as on on the kind of the

criteria for writing example code just to be able to check the crl sorry just a quick note on that um one of the big things that we're working with is this is to try and help folks that are trying to test their applications so they may not need to work with that yet I think our hope is that somewhere in the organization that we can there'll be somebody who will help them change these things for production right so I think there was one more question nope okay okay uh we're we're just about out of time here but if you want to have any other questions for us we'll be hanging out in the chill out

Lounge just a couple minutes after the talk so thanks for coming thanks guys