BSidesSF 2015 - Your Users Passwords Are Already Stolen (Lucas Zaichkowsky)

are I'm so I'm excited so uh just to kind of take you through a basic uh list of what the agenda is on this talk you know Bas who I am um why all passwords are crap all of them and then I'm going to give because it's like a very diverse audience I want to give like a bit of a a relevant primer on applied cryptography because even people that you talk to us say yeah I know crypto when you go to talks about certain aspects it turns out they start to get lost so I just kind of want to go over the things that are most important there and then uh discuss data breaches and

how they're not disclosed properly and the real meat of it is I'm going to get to a case study where there was a big breach last year over a million user accounts stolen that was leaked online and I had my way with it and so those results will be interesting so who am I uh Lucas owski been in the IT industry since the mid90s uh CIS admin net admin telecommunications uh gave that up because I got tired of fixing broken and and ended up working at a credit card processor where suddenly PCI became a big deal and you know that was a lot better than it so I got to work and it was it was an interesting

experience got to work with a lot of point of sale developers who were still coding at VB6 and they thought encrypting things meant doing like rot 13 and Opus skating or having their own little secret algorithm uh went on from there to mandiant where I got to see some really scary crap that kept me up at night for about six or seven months and it really changed the way I think about everything including attacks and breaches and threats um and you'll see where I'm going with that so all passwords are crap why is that well first off go back to the premise of uh what are threats threats are people I mean they really are

there's no such thing anymore as a self-replicating virus no one writes a virus that just goes around self-replicating malware as it is is a tool that's used to achieve something and whether it's crap or being used to steal passwords it's a means to an in so you know going with the um intelligence-driven defense model kill chain analysis you got to look at it as from a threat actor perspective and know who your adversaries are now if you look at the nuisance junk that's out there your day-to-day spam and uh zbot type crap you know not such a big deal there but you start going into activism with the script kitties and work your way up

to Insider threats um cyber criminals state sponsored Espionage it starts to get more and more sophisticated uh how many of you familiar with things like uh dll load order hijacking show of hands okay how many of you think that it's just really an academic world thing show of hands okay it's changed a lot yeah they've been doing that for a long time now both state sponsored as well as financially motivated so um why is this relevant to the password thing because attackers want to be inside you they really do and looking over the anatomy of an attack and this is just kind of one example of an attack lifecycle there's always that initial point of

entry so first they start doing some vulnerability scanning and then once they find something maybe they start doing some Google hacking and find web pages that they know are outdated that have SQL injection flaws and now they're on that web server or you know heck they could always fish your users and get straight in that way there's a billion ways to get that victim zero which uh I think we all know there's no way to stop it there's always going to be a victim zero but the interesting thing is they start popping web shells and doing lateral movements so they'll break out of that DMZ going from one host to the next then they start doing RDP sessions

and this is where it gets really interesting because this is the point where they've got legitimate user credentials and they're using RDP just like a user would to get from system to system they start doing SMB sessions SQL sessions I mean there's no malware necessarily at this point except maybe the back doors they leave behind it's a lot of user activity and so this is why those passwords are so important um and then they just pivot pivot pivot get to what they're after set up a pipeline that basically gets from a to zed I'm not Canadian but I just like saying Zed and then they just siphon everything out and there goes your goodies and uh I've

seen a lot of attacks across the board and it's the same thing over and over again um all the cool things that you read about in the news they're cool and all but they're they're not paid for cool Factor they're paid to produce results and they haven't changed much except maybe getting stealthier and stealthier on an as needed basis so how many of you are familiar with like the AP1 report show of hands okay so consider that like the uh tier one hackers for the pla where yeah it works against 90% of the companies out there they're not extremely sophisticated but it's enough to get the job job done and they're uh uh operationally sophisticated they have Sops escalation

procedures and so on but you start getting to groups like um I'm not going to name them but there's other groups where they'll uh create different back doors um password dumping tools anything they need new variants every time they land on a different system they never use C2 infrastructure and every time they're done with the system they'll delete their tools and defrag on their way out just to piss off investigators so if it's interesting now you look at these incidents postmortem or even just looking at our environments today and I think we can all agree password reuse is a huge problem um users reuse their passwords on multiple sites um maybe they have little clever mechanisms

that's different from site to site for a robot that might work but for a human if your password is a certain set of characters and the first three digits of the domain name the human's going to figure that out pretty quick and then you have things like service accounts etc etc and um this is a really difficult problem to solve because it's a human problem password management isn't accessible to most people and so until that's changed it's going to continue to be a problem and even if it is addressed like I myself use a password management tool generate unique uh uh random character passwords for every account but if a site gets hacked and they

haven't done things properly at passwords toast right that's kind of where we get to is you've seen in the news a lot of um websites getting hacked for their user databases and yeah there's fishing schemes that can go on from that or um intelligence analysis that can be done by Foreign agencies uh but another big problem is those passwords are a huge liability and so let's go ahead and jump into why that can become a liability so why hash uh so you know I apologize if you know crypto inside now but for those of you that don't this is very important to know um you don't want to encrypt passwords because if you encrypt it you

can decrypt it it's a lot better to just do like a fingerprint of the password when the user creates their password and it can't be you can't create a person from a fingerprint right but when someone goes and puts their finger against it again you can still compare the fingerprint so it's the same idea with the hash so um for the password purpose there's that and then from a file uh perspective or larger binaries it's being able to verify Integrity so if you download something you can compare the hash against the calculated value make sure it wasn't tampered with um esime for digital identification on email so on and so on now a little Public Service

Announcement I keep hearing md5 is broken and in a sense it is but it's only from the sense of collisions so if you have large binaries there's weaknesses that allow you to go searching for other variants of another binary that would produce that same hash that's basically the weakness but if you have a password you'd have to have a really really really long password to start running into Collision scenarios at which point you might as well just Brute Force the password anyway or whatever so for password uses md5 is still good now key space is essentially taking a look at all the available characters for that password along with minimum maximum characters allowed and then

being able to compute how many possible combinations are there to this lock and of course bigger is better larger key space the more time it's going to take from a a brute forcing perspective now when you're looking at calculations for like median time to Brute Force crack passwords you always want to go off of uh the 50% so you look at how long to get through the entire key space and then just basically chop that in half and that's a baseline to go by now salt uh salt is essentially how many of you have heard of rainbow tables okay everyone's heard of rainbow tables basic idea there is you can pre-compute hashes for your passwords or your

dictionary files and then just do a lookup like it's a SQL database instead of root Force cracking things so the idea behind salt is for every time a a password gets created and put into your user database you want to Generate random data to append to that that way it's Unique on every instance if 15 people in your user database have the same password the hash is going to be different and why is that important you can't pre-compute tables you have to basically go through the CPU or GPU or Apu Cycles to do the brute forcing at that point it forces them to go down the more um timec consuming and expensive path now rounds this is an important

point that often gets missed um even if you're salting you can still Breeze through those things pretty fast but you start doing things like rehashing the hash a couple thousand times that's going to slow down brute forcing tremendously um you know typical scenario is 2,000 rounds so that slows us down by 2,000 so uh lot slower now it pisses off the attackers the good news is it doesn't really slow down your production system impact so you know uh no really bad side effects from that now getting to the implementation failures now that you've had the primer this is where you really start to see programmers f up all the time um passwords they stored in plain text

obvious skated their ra 13 scheme um you know even if they're storing it encrypted the p uh the key is usually really easy to get you can just look at their code or wherever it's pulling from pull it out of memory there's a billion ways to go after keys so encryption bad so in PL text bad or assault isn't used so again you can start doing the rainbow tables and it makes it a lot faster if you have a large database of users say a million user accounts if you have uh same passwords that that are very common you don't have to crack them multiple times so to speak and then password complexity being hindered how many of

you use password management tools to create random passwords okay fair amount how pissed do you get when you go to uh put a password at a site I'm already seeing the hands right it's like this character is not allowed or it allows it and then you can't log in because the programmer didn't use the same library or something like what tick like the tick character right they're trying to do input validation one spot you're like well looks like they're not doing input validation everywhere which is interesting um password length is known that obviously decreases the key space a lot so if for some reason it's showing the number of characters in a password that just

significantly reduces the key space because you know it's exactly that number it's not going to be six seven characters nine 10 characters it's going to be exactly eight um or not rate limiting authentication even though it's not um a user database problem it's more of a implementation of validating the users so remember the iCloud accounts getting hacked and all the celebrity nudes appearing how many of you remember celebrity nudes everyone yay okay so that was basically one of the API methods available for validating the iTunes accounts they weren't rate limiting whereas the rest of them they were you see that happen all the time um and then of course there's just stupid logic flaws uh insecure coding to just

bypass authentication alog together so all the time now looking at user database breaches and how they've been happening and how they've been disclosed it's a complete joke and it's a liability I'm surprised companies aren't getting sued the way they're disclosing or lack of disclosing so you commonly see phrases as no payment card data personally identifiable information or personal health care information was stolen they're basically saying what wasn't stolen that they're legal legally obliged to divulge that's all they're doing the lawyers are telling them what to say or passwords were protected but not stored in the clear it's like so um what exactly does that mean protected because you know again depending on whether it's salted how many rounds was

it encrypted was it hashed there's a lot of important questions to identify the risk that's imposed by that uh database theft and then also no one ever seems to bother to ask these companies do you know if the attackers modified your code to steal passwords as users are authenticating I never see that get asked by reporters and then last um they'll say all users are being forced to create new passwords as a as a precaution so like

getting into Enterprise environments for more Insidious means or more reasons so yeah looking at zapo here's some direct quotes the database that stores our customers critical cards and other payment data was not affected our access and that was literally in bold like they want to make sure that you knew that right up front and then they're letting you know that uh information was accessed yada yada and you're cryptographically scrambled password but not your actual password um and I actually hit up the CTO and several other key individuals asking hey you know I'm in the security space I just want to know what kind of risks there are so as I'm talking to clients and whatnot I could advise them on the

impact zero response which probably isn't good if you think about it now cracking user databases this is where it starts to become a lot of fun so it's a lot of it's about efficiency because it takes time and we're very impatient and we want to get it done quickly so so if there's no salt obviously start just downloading rainbow tables generate your own there's websites where you can just type in or upload things and they already have the pre-computed tables hit that up first and so if you have a database with let's say a million user accounts you can immediately start decreasing it by a certain amount and then from there you go to your word list how many of you are

familiar with word list dictionary files okay a fair amount so you start hitting that up and look for the one to one matches you decrease it fur further and it's basically going by what what's going to give you the most Roi and then you go down from there identify pattern and length distribution so a good strategy is um I've already gotten the things that were directly on the word list I analyze it and I see the largest distribution is on nine character passwords and then seven and then 11 or whatever and so you can start going for those specific lengths as a way to do it efficiently and um that's where you really start to get into Brute Force

territory and rules how many of you familiar with rules okay less hands so you got your word list with let's say it's just a dictionary file of the English word dictionary or a foreign language if it's some other uh country's website so the idea is they have these rules that do things like add digits to the end special characters at the beginning make incremental changes and there's even contests that are done ocl hashcat anyone so they I'm not sure if it was them that did it or someone else but there was a contest for the best 64 rules and so it's literally called best 64 and it is baller I love it you can get really good results where you do

your word list then you run the best 64 and it just gives you these huge huge additional uh cracked passwords so when I first started approaching this project of cracking this database that I'm going to get to um it was very computationally expensive and so I went back to the drawing board on wordless and I thought what would be the best word list possible I don't think a dictionary file is really the one how about a list of already cracked passwords because people reuse passwords and so if you go to this website that's up there they have it's like this group effort of people cracking these databases and you can download already cracked passwords it doesn't have the

associated user accounts with them and that's fine I just want passwords that are actually real passwords that have been used so I downloaded all of these merged them all D duped them and then ran them through the uh tool split Lane which is also available through o hashcat which separates it into separate files for like one character passwords surprisingly there were a few of those two three four five six and so on and the reason for that is when you're using gpus to crack it's very efficient to uh use the same length for your word list so use all eight character passwords and pipe it through then do all nine pipe it through 10 and so on

and so then I set about building the rig and I got a lot of funny looks when I showed this on Twitter I think and a few other places are you Bitcoin mining it's like no although that may be a good way to recover some of the cost of this rig so I got a used gigabyte motherboard for $84 uh semon CPU off eBay for 22 I mean you really don't need much I I grabbed like an ancient 2 and 1 half inch hard drive which in retrospect just wasted a lot of my time installing operating system I should have spent the money and got something faster um the real bulk of the cost was the six uh GTX 750 Ti cards

how many of you familiar with the housw chipet and how that helps so with Nvidia they added a new instruction set that makes it incredibly faster to crack things and it also is really low energy consumption so when I'm running these six cards I'm using a $144 a day of electricity it's not generating much heat it's freaking beautiful and uh fun story I left this at home in Colorado went to Upstate New York for Christmas and one day it stopped responding and I went oh man I bet that tape fell apart uh and yeah two of the cards fell on the motherboard fried it I had to do some rmas and long story short I just sat on

it for six months and finally did that so looking at the performance so those of you wondering what do you get out of six uh cards about a $1,000 investment there if you're going after ntlm passwords 45 billion hashes a second md5 22 billion I mean really this is going you know in order of the fastest to slowest in terms of going through keyspace um the one that I'm going to be focusing on in a bit here is the pH pass which was doing six million hashes a second because that has a well I'll show you um OSX pretty impressive they got to be doing some crazy stuff to make a comput ation Al expensive this is the database I went

after how many of you remember this database leak okay now what I really wanted to do again was uh I said when I set out on this project I wanted to think of it from a corporate perspective and what it means to organizations so I thought let's go to the uh DHS list of critical infrastructure segments and a few other interesting things and go after that so um you know looking at how the the user database was structured pH pass it generates salt then it hashes the salt with password and then it repeats it iteratively 8,192 times so remember 6 million hashes a second in performance but that's with a really impressive cracking rate and

you sure you could do this out in the cloud on Amazon or whatever but I thought it was way more fun to put together mil crates it was literally just I wanted to have fun now after two to three days this is the part that even surprised me after two to three days just using the word list that I merg together here's what I was able to accomplish chemical industry 34 cracked accounts 83 uncracked Communications 135 cracked 316 not cracked defense industry and this is by the way top 10 companies in each of these verticals major organizations publicly traded um so defense 54 cracked 140 not cracked energy 29 and 69 Finance I mean you see it's I I'm not going to

read off the whole freaking slide but every vertical was affected and there wasn't really that major of a a a variance from one industry to the next I mean it I guess fared the best but even that's pretty sad there's people in it that you know whatever now a lot of people would think and correctly think well yeah there's probably a lot of trash passwords in there throwaway passwords and there is but don't think I didn't go through that list just looking for funny passwords and whatever else I was really shocked there were really long complex pass phrases that wouldn't have been hit by base 64 I mean obviously was hit by this word list that

there's no chance it was a collision of one user choosing the same pass phrase or random characters as another user it was some of one of those databases that was cracked that they're still using that same password not knowing that password is compromised now I also went after doov um hope that doesn't piss off too many people uh 234 cracked passwords and 373 that was using the word lless and the best 64 Rule now the reason I went best 64 first is I found an RS Technica article frankly that showed the like someone else's results using these different rule sets so after that you'd probably want to go on to uh the Dead one rule and so on and so on but the

real important thing to remember here is if you wanted to get into an organization you really just need one user account that works and I have a lot here to go off of and even if they change it one character here or there it's pretty close you could probably get into an account or two or more and once you're in as that user just think any user in your environment if you can log in as them and be a complete you could do a lot of damage and that's a scary thing so using the rules on the data set again what I did was I just looked at the distribution of the password length and on this particular set eight

character passwords were dominant and that's what I did first now going after that bigger the top 10 verticals I mean I was looking at I think 4,000 accounts total and after doing the straight word list it was down to uh what I say 2,24 that's going to take some time I'm still letting it run right now in the background um but at 6% completion on just the eight characters I've got a 39 additional now now I go to do the seven characters the nine characters the 10 characters or I build a couple more of these rigs or I go ahead and stand up a Amazon AWS instance or something you could imagine I'd be able to get a lot

further how am I looking on time I might actually be running really early here yeah so I'm not going to use up the full hour sorry takeaways and resources uh how many of you remember this quote by that former director of uh at the NSA it's basically no such thing as secure anymore we got to I'm not going to read the whole thing idea is we got to operate on the assumption that our networks are already breached that there are thread actors getting in our environment and act appropriately um a couple days ago I was in Denver and I was working I was doing a panel talk with a bunch of cesos and they saying if

if you want these guys to take away one thing what would it be and I I brought up the question if you really want to do damage logging in as yourself and just trash everything or give give away intellectual property how bad could it be and everyone raised their hand and said it could be really bad and I go okay that's how hackers work they are going to get your account and they can do what you can do only better and scarier and so what we really need to do is um focus on and this is a paper that I published recently the rapid detection response model you can download it at resolution onse security.com it's

completely vendor neutral um I wanted it to have integrity so it walks through identifying your current processes on threat detection response from that whole intelligence-driven defense model um preparing so basically knowing where your weaknesses are and prioritizing improving your defenses on both visibility technology and uh procedures for when you do detect things and then it gives you a lot of primer information on threat detection response U being able to take the outcome of that and pipe it back in so I highly recommend and if uh you have feedback I'd love to get it I still haven't gotten it any except the FSI sack they ended up sending it out to the entire list of people so somebody liked it I

guess so you know what do I recommend doing is um definitely following that model same thing think of it in terms of a thread actor and how you would approach that thread actor so for yourself keep using password management tools that way when zapo get gets hit nothing else is affected but know that your users are not going to do that and that they are compromised right now their passwords are stolen and so you need to be thinking of it in terms of almost an Insider threat perspective so enforce two Factor authentication everywhere first and foremost just try to block off entrance ways and I've how many of you uh have two Factor everywhere there's nowhere you can get

into your environment with a single Factor one two hands how positive are you not very positive 99.5 so I start bringing up things like web-based email access mobile phones accessing um third parties that have access to your environment um there's a social media accounts that aren't even under your control I mean hell who's sweet for example so you got your marketing department that's using hoot suite and they support two-factor authentication but uh I actually pulled the slide out because they fixed it up until recently you could use the mobile app and bypass it they just didn't have two-factor authentication implemented on the mobile app that changed though so don't always assume it and then use a jump box where

you can't Implement two-factor authentication so stand up a box where you have to do two-factor authentication to get to it and then from there you access these other resources that you can't protect and then regardless start looking making sure you're auditing user account not just authentication failure but success and all the actions they're taking and also think in terms of that attack or remember the attack life cycle where they're pivoting from environment to environment you need to be able to identify all of that reconstruct it and understand all the back doors they planted all the passwords they've stolen along the way so that you can properly remediate um looking at past breaches remember Heartland when they were

breached yeah so how many of you remember them talking about having a sequel breach a couple months prior to the card data breach no okay so they did talk about that and I always suspected that they were related but it wasn't until I want to say about a year ago they were interviewing Bob Carr on a p podcast and he actually said on the podcast yeah when that sequel breach happened we thought we had remediated turns out we failed at remediation so the attacker still had other points of Entry they made available went silent and waited um there was one Financial organization that I worked with that lost millions of dollars out of their ATM machines spend

upwards of a million on remediation efforts first remediation filled turns out they had an entire segment of their Network that was undocumented and the attacker still had back doors there so it's very vital that you are able to reconstruct everything that happened um so monitor user account activity please and that's it if you want to download the presentation and scare the crap out of your bosses J.M PW nightmare um and I'll open it up questions

so the question is what would the price comparison be between building your own cracking rig versus a cloud type of solution I don't know not afraid to say it any other questions feel free to shout it or come on up yeah

I'm just curious in terms of the password that you've cracked whether you've been able to apply that to doing doing any WPA2 pre-shared key cracking whether you can apply those same or whether you've even tried you know if someone gives you a pcap and you're like okay we want to calculate what the pre-shared key may be can you take these databases and apply them so yeah when it comes to cracking WPA you know do a pcap and get the shared secret and go after all the possible combinations absolutely same concept could be applied there um what I find more effective though is find the access points that still have the pin mode enabled and then you can crack them

through that pretty quickly that's a much better way to get around it but yeah if that's disabled you can still uh Brute Force the keys uh for WPA2 pre-shared key any other questions for the speeds how how many have you gotten for bcrypt sorry for theed yeah um so for the speeds the question was uh bcrypt now when you use ocl hashcat there's a command which just does Benchmark testing and I did see it stream by but I didn't want to have a slide with all 100 different types of things that you could crack so I don't know off the top of my head um sh me an email and I'll just do a or we can talk

afterwards I'll just remote control my system and run the command any other

questions H the least privilege mode so the concept of least least privilege access definitely apply that everywhere because good example is Morgan Stanley how did that guy get so many freaking account uh oh no supposedly get so many account information or was AT&T one of their venders uh ended up getting a lot of the information that was needed to unlock phones and how is he able to C that many things so it's just again assuming you have an Insider threat so limit the amount of access that they have to what's needed it's just going back to the hierarchy of needs of security centrals segmenting your networks and so on any other questions

so concrete suggestions about using two Factor so definitely like the key dongles those are good um if you're doing it on a budget getting it as an SMS text message another possible solution like how many of you familiar with logman so they don't allow they have a partnership hello there yeah they have a partnership with phone factor for that and you got to pay extra instead what you do is you can have the second Factor go to an email address so you can put in your phone number at vex.com or whatever the email to phone number format is and yeah someone could technically get into like your Verizon account and get your text messages through that but the

barrier to entry is a lot higher that way at least um I I really wish there was cheap and accessible two Factor authentication to be honest uh Google Authenticator I wish everyone used that I mean Microsoft does it come on why can't other people anything else all right well thank you all for your time uh free up you got some minutes for a smoke break or whatever and if you want to chat with me after I'll just be right [Applause] here