BSidesATL 2020 - Protect: Serverless Password Cracking or How I Learned to Stop Worrying & Love AWS

I'm gonna do my sponsor read thing real quick because these sponsors have made this whole thing possible and we are super appreciative for them so our diamond level sponsor is Warner Media Gold level we've got Kennesaw State University Coles College their Department of Information Systems also Bishop Fox coal fire genuine parks company and NCR also at the Gold level at Crystal we've got crisp critical paths and synopsis silver we've got Aaron's binary defense Black Hills core light and guide point security bronze level we've got NC C group and our in-kind sponsors EC Council for their online training and secure code warrior for the virtual CTF we'd also like to thank crosshair Information Technology Joe gray

offensive security and pentester Labs for their contributions to the raffle prizes which are really great so make sure to hop into the raffle giveaways channel go chat with our sponsors in the sponsors Channel and yeah just hang out so next up for talks we have a stephan becker and ryan Baisden and they will be talking about Cerberus password cracking so I will hand it over to them awesome thanks Patrick let me all right so let me get this presentation slide real quick yeah I'd probably help

all right is that fullscreen we're good okay great all right so introductions first I'll start with myself and then I'll hand over to my project partners to fajn I'm Ryan Baisden I worked for a company called risk 360 we're a firm out of Roswell Georgia that specializes in cyber risk IT audit penetration testing compliance privacy the whole gamut my primary role there is managing and executing penetration test engagements and red team engagements I've been doing this for about two years now on top of a an IT security career from internal perspective organizationally you can find me at my relatively new Twitter account at SAS Oh sick and I'll let Stephane introduce himself yeah I work for epi slabs we have

offices here in Atlanta and all over the world we help our customers create powerful and secure sa P landscapes with our software value-added solutions and managed services I lead up our AWS team which just means that I help our teams from a technical perspective as well as managing our relationship with AWS globally cool so without further ado I'll jump into our talk now so this is serverless password cracking or how I learned to stop worrying and love AWS asterisk almost I hope there are some dr. Strangelove fans watching if if so you'll you'll especially appreciate this if not after this presentation go and watch dr. Strangelove it's a fantastic movie so today we're gonna be talking

about password cracking as you might have guessed by now and we're gonna be doing it all through AWS so this could probably be done through whatever platform you like I know AWS is not the only cloud infrastructure platform in the world there's Google cloud there's Adger if you're in Europe there's some other options but it just so happens that my friend and local cloud cloud expert Stephane specializes in AWS so first we're gonna do a demo primarily because one of my favorite talks ever at 25 was about backdoors built into x86 processors and he did the demo first and I really appreciated that so what I'm gonna do is I'm gonna cut the screen share over to staff on briefly who has

the demo setup and what we're gonna do is we're gonna use AWS to to crack a an md5 hash all from the command line awesome this is my favorite part live demos let's go cool can you see the right screen yep cool all right so we're gonna do we're gonna execute a Python script Ryan and I wrote called Ares py let me go ahead and give it a hash so I'm just giving it an md5 hash we don't know what password this corresponds to yet we're gonna find out here pretty soon that's exactly what didn't want to happen well it's again great so now what this script will do is first it will send our hash onto a messaging queue in

this case we're using Amazon's sqs service it'll then go out and create a p3 16 Excel which is just a bunch of letters which roughly equates to a server that has about 96 V CPUs and 8 Nvidia Tesla cards but then it waits and on the server side on boot the server will execute a script that pulls our hash off of our message queue puts that hash into a command and executes hash cat once hash cat is finished it'll upload the correct password to an s3 bucket that our handy little script here is actually waiting for and our handy little script will download that cracked password upon completion the server will kill itself meanwhile I have I've built

in a couple of feedback items on the server side it's kind of a black box so our server will also upon creation send us an email it's not the best but it will push a notification to Amazon simple notification service that'll tell us once the instance is launched and then once the instance has executed hash cat successfully we will also get a notification from S&S as soon as our cracked password is an s3 so I'm gonna hand it back to Ryan cuz this is probably gonna take about five or so minutes yeah this will take a minute so so what I'll do is I'll I'll start going through some background on this project and I'll let's define interject as soon

as as soon as we've gotten our cracked password back so let me swap screen shares here alright so a quick word on server list we we intended for this to be truly service in the sense that we could crack passwords with without ever spinning up an ec2 instance if there are any other AWS experts engineers architects watching this you'll know that this is not technically serverless when we initially came up with the idea we thought how great would it be to be able to do this without ever spinning up a server at all as we move through we realize that wasn't necessarily possible in the form we were doing it and we'll get into that a little bit later when we

go through the progress of actually building this thing so it's still technically used as an ec2 instance right now there's no way that we know of elegantly to do hash cracking with lambda but I will say if there's anyone watching this who knows how to port hash cat to lambda or use OpenCL without a kernel I will personally meet up with you buy you a beer and include you on this project what matters here is that relatively to the way people traditionally crack passwords which is maybe using a big rack of GPUs or even just spinning up in an AWS server for themselves and logging in this is relatively serverless so why did we build this first reason is that we

wanted to crack passwords more specifically I wanted to crack passwords because it's part of my job and I wanted Stefan to want to help me do it in AWS because he's a guru with AWS and I'm a noob with AWS he didn't want to help me obviously which is good otherwise this may not have happened in the form you see it today the second reason is that racks of GPUs are insanely expensive back when I was trying to figure out how to do password cracking for RISC 360 we had the unfortunate experience of pricing GPU racks based on all of the people who were trying to mind Bitcoin during that bubble that we all observed and lost a

bunch of money and we're trying to get their money back so in other words it was a non-starter what else is expensive is AWS if you forget to turn it off these P 3 instances because they range anywhere from 20 to 25 dollars per hour without spot pricing can become pretty expensive and if you go and crack a hash and forget to turn it off then you're looking at a pretty big bill so we weigh 20 grand a month by the way yeah it's a lot we wanted a way to be able to do this without incurring any cost risk so what we were doing before was one cracking passwords with laptop hardware which is not so bad if you have

a pretty modern laptop that has the hardware that can handle it but in a lot of cases it's pretty slow and not everybody has access to that kind of hardware so one of the problems we wanted to solve was how to let people crack passwords in an accessible way that they wouldn't have to spend tons of money fifteen to two thousand dollars on a laptop or even worse 15 to 20 K on a rack of GPUs and we were able to accomplish that by passing all of the computing power off to AWS the second thing we started doing is we started spinning up very small p3 instances less expensive p3 instances in AWS we would

crack the password over SSH after logging in and then we would spin it back down which again all worked but it required a lot of manual time and effort which was another thing that we felt like we could avoid by doing some some AWS maneuvering we correct it Ryan all right let me hand this back over to Stephane real quick so we can see that final end of the demo all right so here you can see my notifications that are received we have our nice little aries instance launched successfully message hash got executed successfully message as well as our message that said the item is an s3 so I could show you here that let's go do a

quick results oh that's embarrassing here's our cracked hash apparently this person is a big fan of Panic at the Disco which I can't necessarily fault them for yeah so that's so that's the end that's the end result we're able to do this entirely from the command-line now there's a lot of setup in AWS that goes into this but being able to send a hash on from a command line argument and get the results back in the command line was was our golden goal we wanted to get to that for a lot of reasons when we were thinking about building this I was taking into account a lot of the pen testing tools that I use on a regular

basis and trying to figure out what the best components of them were and I wanted to be able to implement that into what we were building so the first thing that a lot of these tools do really well is use AP eyes and when I say AP eyes I definitely mean actual ap is but I also mean that they get information that already exists or they use functionality that someone already built to do something new the second thing they do is they don't really require giant installations they don't require proprietary dependencies lots of system requirements they can be run with minimal setup so they can be used quickly get the results quickly in a usable format and you're done

sidenote port your Python to to go Ling this is kind of a joke because if you use pen test tools you know that using Python tools ends up in a storm of dependency handling and sometimes your virtual environments can break and it's just a nightmare going kind of takes care of a lot of that stuff by being a compiled language so this is kind of a joke on Python I still love Python but going is great the third thing is that they can be run from the command line so there this is not a hard and fast rule just like any of these are not hard and fast rules there are great tools like burp suite for web hacking zap w3 af all

of these great utilities that are used for pen testing that don't run from the command line but being able to run from the command line affords the person running the tool a lot of a lot of capability and a lot of customization that is something I've come to appreciate with the tools that I use so I wanted to implement that and ours as well and what that came down to was Ares dot pi we built this this tool and we named it after the Greek god Ares the God of War mostly for fun at risk 360 we do follow kind of a theme with Greek mythology so it seemed only fitting to to name this Ares and of course because

we're so thankful for Jeff Bezos and his AWS invention we put his head on on our Ares painting here I mean he might as well be a modern-day Greek god of war let's be honest so I'm gonna I'm gonna let Stefan talk about some of the cost savings because again we we can talk all day about how easy this is to use and how much how low the overhead is but when it comes down to it one of the first problems we presented was how much it was costing us or potentially costing us to do this very simple job of password cracking and we theoretically proposed that we'd be able to save a lot of money we weren't exactly sure how

much until we finished and all let's define go into that yeah so the real there the magic behind this tool is twofold one this saves you time and provisioning a server manually the script does all that for you you know it keeps it even saves you time on like SSA Qing into the server it's just a black box that you send things into the second thing is we are using what's called a bilious spot pricing that's just a way for the customer to bid on AWS is excess capacity in practically speaking what that means is instead of paying twenty four dollars an hour for this server we're paying seven dollars and forty four cents for this

server I did a quick calculation in our demo the server was only up and running for three minutes and effectively that test run only cost us about 37 cents in our testing time we used about 10 hours of server time and had equated to about what we affectionately like to refer to as 44 Bezos bucks and just for a quick currency conversion you know basis books is like a one-to-one ratio with a dollar but it could have costed us about two hundred and forty dollars I don't have two hundred forty dollars ryan is enough two hundred forty dollars it's it's just insane the kind of cost savings that's about a 70% cost saving over what's

called on-demand pricing so that's that's my slant on cost savings yeah what's what's especially important to mention here about this this picture we have on the right is that these V CPU hours aren't just cracking in fact they aren't even as far as half of the time actual password cracking this was the entire time we spent doing password cracking tests as well as just setting up the development environments configuring the server to work the way we wanted it to doing troubleshooting on our scripts this was the length of our project as far as building this so 70% savings on the entire project is not bad the fact that it costs us sixty six dollars and ten cents over the course

the entire project is still amazing not even considering how much savings can you can you can have once you actually run this tool when you're only cracking passwords like Stephane said it's it's cents on the dollar when it comes to cracking individual hash the server goes up and when it comes back down incurring no extra cost at all so is this the best / fastest / most efficient way of doing this probably not for us this was a classic hacker in a virtual garage solution to a problem but in my perspective that's the true nature of hacking having a problem and solving it with the resources at your disposal those of you who have contributed to the

open source community will sympathize with this we have the same goal in mind building this tool as we did when contributing to open source which is just do it the chances are that somebody will likely come along after you and they'll do it better they'll improve on what you did they might even make you look like an amateur but what matters is that you did it and you did it your way first so the fulfillment of this project for us was finding this problem that we had and solving it and sometimes a quirky way sometimes a disappointing way as far as things we had to cut out that maybe fit with our original vision but

weren't going to be functional so people probably are currently doing this better and people will probably do this better after but we solved our problem and that's what matters to us and so we learned a couple of pretty important lessons over the course of building this tool sometimes the coolest option isn't the best option in our initial iteration you can call our alpha version we had this ridiculous serverless orchestration layer where like a lambda function was created that once triggered would launch rp3 excel instance and then we'd have another one that would have to wait and go find the instance and kill it it just wasn't elegant and we actually found out there were just a lot of points of

failure and so we just settled on scripting all of this out in Python using a very powerful library the AWS provides called voto 3 tool creation is a process we learned so many of valuable lessons and ultimately you can build this ridiculous thing and find out that python is just powerful to handle the brunt of it in this case we had we realized that infrastructure of code and scripting is just better and faster for me as an AWS architect I found that it fit the client requirements over what I wanted it fit the client requirements better to just create a script that does all this for us I mean Ryan would you say that this suits your needs better

ridiculousness that we had planned before oh sure yeah I mean like I mentioned the last slide there were elements that we had to cut out that we were we were sort of married to but in the end it became choosing functionality over choosing what we wanted it to be and to meet our ultimate goal of an accessible way to do password cracking we had to make some sacrifices but in the end the moment where we we got it to run all the way through and we got a cracked password back was it was all the reward that I think we needed mm-hmm so what's next first thing more hash options if you were if you're familiar with with

hashing algorithms you probably noticed that the one that we passed through during the demo was an md5 hash which is notoriously common and also weak as far as hashing algorithms go during pen tests and red team engagements we come across too much stronger types of hashes that are separated mostly into two families the Windows family so ntlm v2 Kerberos tickets that sort of thing and then if you win on any UNIX based system you'll probably run across a shot to 50 56 or sha-512 hash that you have to crack which computationally are much more complicated than md5 so bringing this to a point where there's even a more practical use is one of the things

we want to do in the future the second thing we want to do is enable brute forcing this demo that we did runs through a word list albeit a very big word list in fact I combined the two biggest word lists I know of into one for this demo but word lists aren't always going to cut it you'll come across a password hash at some point that is overly complicated and has it been breached before so it's not in a word list and the only way to do that is to brute-force it we are doing some some internal rnd on how to brute force more effectively I'll keep that mostly secret for now because I definitely want to do

a talk on it later the third thing because I'm not a total hypocrite as a golang port I bashed Python earlier not because I don't like Python but because I actually love Python and I like to give it a hard time sometimes but I would love one day to port this to go Ling so that it's even again more accessible easier to use and can come in the form of something like a compiled binary I know we have about a minute left so I guess we could open this up for questions oh slack for this so I saw a question here that asked if the p3 is launched with an ami or via user data so we did

we did end up creating an ami for this we just found out that it works much better there were a couple of iterations where we did pass in user data but we ended up just you know landing on having a pre-configured ami with the installed Nvidia drivers and then including our server-side script in that ami also the other question that I see immediately from from Christians so can people use this today are we selling it as a proprietary open source so we we have no intention of selling this in fact in line with our accessibility goal I mean we we don't have it in a public repo now but our goal is to eventually have this

completely open source for anyone to use they can implement it into their own AWS environment they can use our tool just like we did in the demo and we don't have control over it we don't want control over it I have a soft spot for the open source community so this will be entirely open source there was a follow-up to the previous question that I answered is the wordless baked-in or sourced from s3 you know I just realized it would actually probably be a great idea to have it in s3 so that you know from a engagement to engagement you can use a different word list and I can speak on that more because he actually does this in the

field but just to answer your question simply it is baked into the AMI yeah we we put it into the ami just for convenience for us it should probably be stored in s3 the only only thing I can see being problematic and obviously you can write code to get around this but having it stored in s3 and not in the ami doesn't immediately give hash cat something to latch on to and use as a command-line argument so getting it to fetch from s3 is totally possible and then running hash cat but just putting it on the disk so it's already there was was easier for us and it's as far as the word lists again like I said it's it's a

combination of the two biggest word lists I know and they're both public one is the RockYou word list and one is the crack station word lists you can google them both I combined them and cleverly called it rock station dot txt you can do the exact same thing if you want to yes the famous Rock you somebody asked just what was the total elapsed time we already mentioned this but from the time we executed the script to password crack was five minutes with about only three minutes of billable server time and fun fact AWS actually does per second billing for Linux instances I don't know if they have that up and running yet for windows-based instances but definitely I

do for Linux

do you see a couple people typing so now thank you all right if there are no more questions Patrick we can we can hand this oh wait Alex asks is this concept also possible with Azure and what made you go with AWS I can answer the second part of that question what made us go with AWS is one that we know that it's extensively documented and a lot of people use it the other reason is that again my my very good friend Stefan is an AWS expert so it just it just made sense yeah we don't write at this stage our tool isn't portable between clouds this is a doubly a specific how do we keep up with your

work so so again what I'll do actually let me let me jump on to my first slide here since I know yes anymore so we do have a we do have a private github repo where we're working on this together there are a couple of things that we would need to do before we can make a public and just some nice to haves like we this instance doesn't use any programmatic access keys on the instance itself it's a big security no-no so what I what we you end up using is AWS roles and so as part of our github repo before we go public with this we'd want to make sure that like I'm working on a

CloudFormation template right now that'll go out and create that iam role for you stuff like that also you do need to make sure that you have the AWS CLI configure it on your machine and we just want to create documentation about how to do that securely or just find good links on how to do that securely so yeah so definitely follow both of us on Twitter I know I will be posting with updates this project I I wish that the the repo was public today but I promise that it will be at some point anything bite you with the surrealist flow dying without results absolutely so I know it weekly we've had so xq s is

is a fickle being I know that we we consistently had we had to clear the queue recheck the queue reread check the queue yeah I wouldn't say that's sqs is fault sorry please tell me I'm wrong you I don't know for some reason in our testing we did have some weird issues where when we were we had a server up and we immediately pinged sqs if the message just got there the instance would struggle to get it and so there doesn't need to be some exception handling where the instance like it's the script on the instant instance does like try and grab a message and if it can't it tries again we also had an

issue where because the script because I executed on boot when we tried to go in and fix our a.m. eyes the server would wake up and then die immediately because that's just part of the script is for the instance to terminate itself so funny things like that happened it's funny how like programming languages do exactly what you want them to or do exactly what you tell them to yeah so David says you could take this one step further collaborative cluster for cracking with a credit and cue system definitely we more than once had to turn down the temptation of spinning up multiple instances and and simultaneously sending perhaps so spin up five instances with five hashes that

we wanted to crack and send one to each I think if we had done that we would have fallen into a black hole that wouldn't have allowed us to actually have a working demo today but it's definitely something we thought about we have we have a pretty big vision for this moving forward yeah and you know the CLI command could easily be modified to launch five P three or four according to your AWS accounts off limits

cool well some we've we've posted our our links to Twitter in the track protect channel for you to follow any updates on this also you know stephane spawn posts regularly about the work he does I am going to begin posting regularly about the kind of work that I do not just on this but on other things again like I said I also do pen tests so there's a wealth of information for me to share yeah follow us there and thanks thanks everybody for attending we've enjoyed this and I hope you have - feel free to reach out to us on Twitter and even slack while the conference is going on right and I would love to answer all

of your questions if any more or two arise thanks guys