Survival of the Fittest: Password Edition!

thank you Scott so who likes changing passwords okay there's a couple of you this move maybe I'll have let's see what kind of people we are who likes forcing other people to change their passwords there's a few more of you okay okay well I hope I'm not trying to change your mind about that just a little bit so a little bit about me Scott gave a great intro I have done a few things in my life I started out sysadmin I worked as an attorney practice law for a couple years decided that wasn't for me and it came back to tech and security and now I run the security program at Pierce storage down in the Bay Area so

you know I'm not gonna go I'm not going to bore you with some really deep technical details I'm probably a lot of you have tried cracking passwords before I am however going to give you all of the kind of tools you need to do that because my goal here is to set you up with a methodology and some of the tools that you need to implement what we've done at pure storage at your own place of work so I'm gonna review our approach and and then I've got here the slot us but the slides there may be slightly different I made a couple of hits last night but the slides and some some scripts and some how to walkthroughs all

in the github and that will also be at the end so why so you might have heard that NIST released this guidance kind of surprising many that something we all kind of knew is that path changing passwords on a regular basis is actually bad practice so it results in people either using basically the same password every time by changing a character or or writing it down if they have to do that if it's long and they have to change it a lot so it results in all this kind of bad activity and then also I don't know how many of you were this but I've been a Blue team almost all my life so you

know we kind of are often in firefighting mode we're responding to things we're triaging events and whatnot and we don't get to do some of the fun cool things that everybody in the conference's talked about cuz conferences are mostly filled with red teamers talking about what they do so this allows the team to kind of get in there and do this kind of cool thing of cracking passwords and seeing what they get and also you get to learn some interesting thing about your coworkers so some people choose some very interesting passwords and also it's proven to be pretty effective so what you see here is the number of passwords we've we've been doing this for about

three years and you see here the number of passwords we cracked as well as the percentage of employees that was so you see that in the first couple quarters while people were still learning we were cracking more but we were a fairly rapidly growing company so quarter-over-quarter we actually see a steady decline and then a little bit more precipitous decline of the percentage of employees that we crack and I don't have this data on this slide but it is a little bit disproportionate to new hires so and this is kind of why so the common password policies that we're all accustomed to basically say that the first example here is not OK while the second one is perfectly

acceptable for you to put in your kind of put in your password to meet the policy that they want right so so what happens in reality and I think as most of us probably know is that those are just as easy to crack because these substitutions are so common and so regularly used by people that we just check in and so you know kind of information security twitter agrees so there's a lot of people that talk about this talked about the you know make fun of the password policies that they see when they're signing up for their bank like why are you making me do this why won't you let me do this but you let me

do this so this is fairly common so we wanted to take this approach but where the difficult part is is that we have customers who put security requirements upon us or or ask us to meet certain security provisions they will say here's our 80 page security policy before we can buy from you you need to meet that we're a business-to-business corporation so they can do these kinds of things or we've got auditors and you might be in a regulated company you might have a shock sarbanes-oxley that you've got to worry about you might be working for stock to certification ISO 27001 any of those and you'll have auditors that come in and they've been doing this for years and

they've got a checklist and this is you one of those easy things for them to check off right they're gonna just say what's your password policy when they say that they're what they really mean is what are your settings in Active Directory you have to do this fairly long amount of time to train them that is not a policy that your policy is something that is in our case is you must have a strong password and if it's strong you get to keep it and if you have a weak password we're gonna make you change it and we test that every quarter so you have to do this kind of retraining of your auditors and

compliance personnel to let them know that's what you mean and that those settings in Active Directory are just a mechanism to assist your users to do that so so how do we do this so I'm not gonna go super technical but I do want to you know kind of give you an overview of what we do and so the first thing you need is you need some sort of equipment right so you need either use of the cloud services such as AWS or Azure or build your own kind of password cracking rig which is also kind of a fun activity good good reason to spend a little bit of money and and then you need to come

up with some words and something that's really good for for coming up with dictionaries and word lists is really knowing your users so or knowing your target if you're kind of red teaming but in our case it's knowing our company and so knowing our culture knowing you know knowing the geographic locations and knowing the language that they speak and then knowing kind of this goes along with culture the jar again the internal jargon that we use and then you've got to choose some kind of tool and so we've actually landed on hash cap but hash can John the Ripper are both really good examples of tools that are both free to use and get totally get the job done

we started with hash cat it's historically a little bit more his relied upon GPUs which gives you a higher crack rate or hash rate but in both cases those work I don't know how long some of you've been around but back in the day when we were all in the same collision domain Cain and Abel was a real fun one to run because they would listen on the network pick the pick up the the hashes that are flying the LM hashes usually they're flying across the network and just started cracking away and then what we do is I mention we do this a recorder and we take about seven days to do it and and I'm actually

new information here we're in the middle of this process right now we there's four of us and so we each take turns at a quarter and we started this past Thursday and decided to use the newest set of GPUs in Amazon and our we usually took two days to do brute force and that's just an easy way to check those those low characters in case in the administrators kind of behind the scenes changed someone's password to a to a seven character password or whatnot which is below the minimum that we've set in Active Directory that used to take us two days and that was actually kind of nice because what we would do is take any research that's been released

over the past quarter any dumps that are like listed on Krebs on security comm or something like that take that take those and incorporate them into our word lists unfortunately he told me that it only took them nine hours so he didn't get as much of a time frame but now he can spend more time on building rules to go the word lists to do some mask processing and then on the seventh day we then all of the people that we've cracked we then start sending out the notifications give them two weeks and then they have to change it everybody else is cool so the brute force is straightforward right that just means crack it as you know try every single

combination for X number of characters so that's that's super straightforward and then we start doing the word lists and that is basically some of the stuff that I was talking about like if you know your culture you know the geographic region there are some tools out there where you can where you can essentially creates maybe geographically specific words so you will go out there and you specify the Bay Area for example and it will pull things like warriors and Stanford and street names and things like that things that people tend to use with their with their to make to construct their passwords and another really good source of information here is if you just mind your internal wiki

or your internal knowledgebase whatever you use to to support that because that'll get in any kind of especially any kind of code names that people we use all original company names if you're fairly young and you had a different name when you're a stealth startup those are interesting so then we move on to kind of the rules and this is how you take those word lists which which are going to have an okay but fairly low hit rate and turned them into that example at the beginning where you substitute the a for an @ symbol the S is for dollar signs are fives and things of that nature and it's it's got a number of rules that

are that are built in or you can develop your own that then apply to the word list that you've fed into the tool and then lastly one that I actually find a lot of benefit from or a lot of really high hit rate is using masks and masks are basically a targeted brute force so in the example I've got up here the this is this is hash cat kind of language I guess and so the you know you do a question mark a and it means any character and L is the lowercase ones the use uppercase and once you learn your your user behavior or just people behavior in general you can just pull down a word list just take a glance at

how people construct their passwords now a very common format is to do uppercase a bunch of lowercase and then tack on a couple numbers to meet that that password policy that they're they're trying to meet right which they don't really want to do so you do that and you're now able to check a lot more of likely key space whereas if you're trying to brute-force 10 characters it's gonna take you a few years you take just check the 10 characters for upper those lowers a couple digits and it takes you like 30 minutes so that allows you to really get done and so in here I've put a couple of examples of exactly the kind

of the the command line that you would use but that's boring to look at right so I thought I'd show a couple so I'm gonna have to don't have this on my screen so what we've got here is is straight up just hash cat running with the parameters mean a three the attack mode is a brute-force mode it's a it's an NT database or extracted from an NT Active Directory database it's got the user names in there I'm specifying where the where the results go and then I'm just trying everything and I think that's actually 9 characters and it doesn't look like I've got any but we'll check the status and I'm just running this on a Mac Book here this is not what

we do but you'll see that it's gonna take a really long time right so let's maybe kill that one and check one where I'm talking about where I've gone a little bit more specific with my mask and you'll see I start to get hits immediately even though it's on a pretty low powered system and what I've done is I've just done uppercase or any character and then some lowercase and then a bunch of any any any this is actually our database from back in 2015 so all these have been changed but they're they're pretty common and then and again so let's see didn't say yes so that was going to take 115 days and like

I said it's gonna take like 20 minutes on the stuff that we run but then we can do something a little bit more interesting such as using a specific word list that I've downloaded there's a place on github that's got probable word list so it's kind of a constructed likely set of passwords I think this one's a few hundred thousand and we're doing that and then we're also doing what's called a hybrid mask attack so we're taking those words and then at the end of it we're tacking on any four character so we're gonna brute force just the four characters tacked on to the word so if password was in the probable word list then we would try

password 1 1 1 1 1 1 1 2 3 4 ABCD and so on and so forth so we run that and we'll see that we very quickly get a lot of passwords even even on a MacBook Air so so you you run through them very quickly and it works pretty well in questions about those ok I'm gonna move on so now now that you've kind of seen how we do this and again I just want to say that you know that just kind of giving you the the surface of the actual technical piece of this and that all of its very fairly well documented on github with the ability to extract the data from

Active Directory which which is easy but it takes a while then you forget how you do it every quarter and you have to do it again so eventually we documented it and then what we also do kind of for the auditors although I've never been asked for this but for the honors we actually store all of that in our internal github and a private repository just for my team so we'll we'll upload the original hashes the results all of the rules that we use all the masks that we chose to use that quarter so then it's all a fairly reproducible and we can we can very clearly explain to the otters now they have asked they

said do you document and do you have this evidence and I say yes would you like to see it I've never been taken up on that offer I think I think that's that's where they stop so what we found you know I've just got some kind of interesting results and and stuff that we found over the course of doing this for three years and this is a little bit not exactly how we do it cuz I told you that we brute force upfront but if I were to rearrange the ways that I I do the testing and just like I did with the testing that last one was much more efficacious than the first one right so

if I put them in that order what we find is that really in the first four to eight hours we're gonna get the bulk of the passwords were going to get and then it's it's a real low tale afterwards but because we kind of do it that way to give us some time to set up this this is not what our graph looks like unless we choose to do it that way but that's what you can find so if you're really either crunched for time or crunched for money because renting this Eneida as or as yours pretty expensive then you can rearrange them like that and and just and just kind of maybe trust that

everybody's meeting that a character or nine character or ten character minimum that you've set in your password settings and then the composition can be interesting so unsurprisingly we see the bulk of nine because that's the bare minimum that people are allowed to have we did we have caught some with the sevens and the eights like I said but then we also catch a fair number in the in the 12s and the 14s and the 16s and even higher I think the thing that was most interesting to me is that the most that the majority of people that we found of course this is all just what we found out the people that we don't find

is that they actually were using all four characters I was really expecting the bulk of people did he use more of a minimum and just use three characters and so then we also found you know you've aggregated it and you see what's common over time and some of these could be service accounts but what's most interesting for us at a company called pure storage is that pure is the base word and four of the top ten passwords and this link should be in there so you can do some of this analysis once you've got the the raw lists of passwords that you've cracked there's a couple projects I think the one I use is called PI pal

so pipl it's in github and the link should be in the notes you'll just run through it and make these kind of cool ascii graphs but also give you some just summarize the data and tell you where you are so that's that's how I've done these then I turn them into Google sheets and to make a pretty graph so you know we were doing it for a couple years and what we found is that you know just by kind of eyeballing the results and what now we started to notice certain ones that would come over and over again as one person in particular he had Starcraft 1 then Starcraft 2 and Starcraft 1 - and

so we decided to take I mean seriously it's it but so what we decided to do is we started to give them a separate set of notifications right so we wrote there's a just took a Python module that takes simulator you have one string to another we decided on 70% so if if if you were caught both times in a row and you're a current one is 70% similar to your previous one what we do is you get a special email that says hey we noticed that your password this time is X percent similar it's just a mail merge of your last week last month's password or less quarters password of this and we put that in the email and I think most

people or many of the people don't even realize that's not the current one it's the previous one because it looks almost the same and so we'll get some interesting reactions there people will say how can you send me my password in email it's totally unsecured and I'll say well like I mean there's some interesting research out there that shows that given given knowledge of your current password they have a 20 percent chance of guessing your net 20 percent chance of guessing your next password within five tries and so what's interesting about that is that probably gets under the radar of any of our password lockout policies because I think ours the six no ours is five but

that would be get within five you're gonna miss it right so that's a fairly scary bit of research given that knowledge and given how many passwords are out there so this is just kind of an example of one of those repeat offender tries we're in August of 2017 we found 805 people followed by only 490 the following quarter which is pretty great but of those 493 and 334 were of the same what we're failing both times and of thirty-eight failed bow times and they got this special email so kind of wrapping up just just some lessons learned so I I'm in the piece on get lab and I think that's really important because our hardest challenge our

biggest roadblock was the compliance and honors piece it was not the technical aspect it was not convincing people we got a lot of complaints like we still get people that think that we store their passwords in plain text and then we've got to explain to them no they're all fully hash to the one-way hash but it's so easy to guess that we got yours so so we the get Habs interesting the github or get lab is interesting that was a combination of git live lab and github the the fact that we rotate it through the team is pretty great it gives a lot of people interesting things to do when their day job is maybe for

the rest of the years kind of triaging alerts from the Sam or something like that and so the other thing that's been interesting is that because this doesn't follow the standard change procedure at most companies where everybody's changing their password every 90 days but it's completely dispersed across the company right not everybody's doing it at once our helpdesk continues to get really nervous about this even though at this point we're only sending about 230 or 350 notifications and they get really nervous like they act like it's the the first time we've done it every time we've done it for three years but the reality is that most people are gonna even though they've got the two weeks

they're gonna wait to the last couple days to do it most of them are gonna do it and a few people are gonna be locked out but it's not really gonna be any more than lock themselves out during the course of business women' we're not forcing the change passwords so that's pretty much it I'm open for questions if you guys got questions okay

I guess I'm not familiar so so it actually gives you a password kinda like a password manager it gives you I am not so so the big thing here is and it's what we try to teach people we've we've got a companion document that helps them craft strong passwords because people say people are so accustomed to putting in meeting those criteria and it being called a strong password and so we've put this thing and that really tries to drive home and this is a hard concept for people to understand but tries to drive home the entropy is the important part and so it really depends upon the Safari settings and whether or not it generates high entropic work strings for

your password because if so it's very hard for us because remember one things that we do is we actually we actually test for lower entropy by using those masks right instead of testing everything through the key space which is computationally too expensive well test for those those patterns that we see people using time and time again so if the settings are good is probably a good tool

right so luckily I'm not I don't have to answer the DoD so I it is certainly helpful that I came into a company that was a start-up that had absolutely no password policy when I came into it and that this is the direction we went and I didn't have to steer a ship into a different direction so still the conversations with the auditors and again customers because they give us a little bit of a hard time too it's you just end up showing them the results it's kind of what we've done so I think I was lucky because I came into you like a new company that was kind of fresh and so I got to do it from off the bat but I

think you would have to do is do this alongside your regular change regime and then and then show the data and hopefully it works because what I'm able to show now is that we we were getting not only kind of better security but better security awareness because of the downward trend of the people's choosing weaker passwords if they straight up say cracking is is verboten or something along those lines perhaps you can come up with a way where you can really isolate that I mean we we do use it in a double yes but there's no reason that you couldn't extract the full like NTDs did out of Active Directory put that on on something and put it on an isolated

system that doesn't connect to the internet right none of this is really once you have the software really requires being connected the internet so I think that that might be might yeah might be a way to hopefully address those concerns

into their master so it so we've kind of thought about that and we've thought about a couple ways so it's in it we don't have a plan for it but I think it's it's interesting we do encourage it right we do encourage the use of it unfortunately there's no say group Paula tyno of group policy or Jam for configuration where I can choose their settings for them like as we were talking about the Safari ones because that would be really cool like if I could just set the settings that that chooses their random password would be great well what we are looking at though is we've done it in the cloud because that's been easier we haven't haven't

had since we only do it for a week it's been about cost effective for us also it gets completely buried in the ad aw spend from like the service right so nobody notices but you know we've been looking for a reason to build something on our own and all I'm looking for is the reason to be running it kind of all the time and if we were grabbing that because I was also thinking about just the the key chain manager and Mac we're about 85 percent Mac so that might be interesting as well because it's also it stores those passwords so I haven't thought about yet but it's it's kind of on the horizon I think that'd be a good

approach

yet su mo okay yeah so most of the methods that I talked about cuz cuz because at the point where you're talking out I get to hand it off to this assignments no this stuff that I talked about is mostly manual and we've written it there's some of the analysis once we've got the data we've script script if I'd and then I mentioned that the we mail merge out the the kind of repeat offenders the sis admins have done a great job of basically writing that in PowerShell and so they they do it they said oh no - at least we send out the notifications that's that's actually just a cron job on a Linux server and it's it's really

ugly but it just uses mail X and in the reason for doing that it was I did all this date math and bash because I kept getting the dates wrong I'm not good at dates so I'll just do it in bash but they they've done all that and so they just they just at the at the end of the time they just clicked the the you must change password at next log and flag using PowerShell and they just kind of do it so it's pretty low touch for them so it's pretty good and then

yet so what we do is we we do this on a 90 day basis and we force people to change their password if it's week so so we haven't correct which is always a moving target so that's why in the policy we define it as strong and strong means we can't crack it and wheat means we can and I think you could totally do that you could totally put the age of one year and then do this on a quarterly basis making others change it at 90 days and I think that would be fair and we've talked about it at the time we were fairly young and nobody had long you know now we've got people that might not

have changed it for five years so maybe we should start thinking about that it won't be no I think I mean it's it's just it's just hard to swallow right so we have a fairly heavily engineering culture and so I mean 16 might not work but 10 might work because because I'll get a lot of arguments about when I talked about entropy with with you I'll get arguments about like a high highly entropic 10 character password would be still very difficult for you to the code and I'm sorry I'm I'm out of time but I'll go outside and be happy to talk to anyone thank you you

you