I Find Your Lack of Serverless Disturbing! Leveraging the Cloud to Automate the Mun

test test hey guys okay so yeah I had a great panel I'm gonna talk to you guys a little bit about a serverless platform that uh that our company has been working on that we're open sourcing right so the the kicker to this is the regional speaker for this just got sick and went home about an hour ago so I was not supposed to give this presentation this is his background but I am Bryce right and I will be giving the presentation on his behalf yeah so yeah so you're still getting the content I'll be it yeah so Bryce this is me so I used to work at homeland and I run the unclassified sock they're

instant response and focus operations for a while so a lot of a lot of endpoints and a lot of you know nation-state type tracking down and getting my other networks right and then I work to NSA for a while and I did offensive stuff there and then I came to Adobe and built up a red team in the digital experience business unit down there on the Lehigh Building and now I am I am leading stage two security and we test stuff like pen testing and we teach some courses at blackhat or privately and then we're kind of spending up on the hunt front over the next year so really the genesis for this utility was kind of trying to get most

of the mundane stuff we do off our plate and using some of the latest serverless cloud technologies that are available so and I just want to stay Michael Butler who's our VP wrote the tool and did the research so all credit goes to him for for this so but yeah you know when we're doing red teaming or we're trying to assess our infrastructure of a forward assessment there's a lot of like Rita recon and enumeration type stuff that we spend time on and you know that's boring we want to get to the fun stuff the hacking stuff right so so you know back in the stone Age's like back when I was like in you know I don't know

middle school or whatever you know you might you might have like a laptop and you would use netcat telnet or scripts to kind of enumerate a server and kind of the pros to this is it'd be really easy right you just grab your laptop you could walk into a client's I plug it into their network and start to you know do a pen test or assessment but it's really no scaling so as your pen testing team grows or red team starts to grow that comes problematic and it's really not automated and so everyone's processes are a little bit different and you may start missing stuff so then you know much much better right is a you

know nmap came along and really map kind of unified a lot of the pen testers and in a sense that you had a standardized tool and it also had the scripting engine so you could build more capabilities on top of it so you can walk into a client site with a laptop you could pop up an end map and it would go through and scan the network and say like okay this server has port 80 open that means it's running a web server and the server has 25 open so that's a mail server and we ran this necess script against it so we know that the mail server is running an out-of-date version and is vulnerable to maybe like this

type of CVE or something like that so you know that was that's much better but still you know really limited it doesn't really scale past one person very well and then you know these tools are really really cool so they're they're really progressive layer and Faraday both work a little bit differently in the design but basically they enabled a single pen tester or multiple pen testers to kind of have additional servers that they could task to do scanning tasks like essentially wraparound and map and then collect that data in some type of centralized database or note right so this this is cool because you get a little bit more scaling but you know some of the things that were

bad about this were like the cons is it was like hard to set up like you had to spin up servers or get servers and you had to install the software and configure a and then you had to make sure they're able to talk to each other and you know your workers you might have like static IPS on the internet so and you know once maybe a blue team discovers what your IPS are you'd have to stand up new workers and there's really I mean you could wrap this around like using a tool like terraform or something like that so you try and rotate these worker nodes but that's not really built into these tools they don't

enable you to do that but layer and Faraday they're good tools but there's still the biggest issue is like just takes time and effort to get them set up right and the kind of static so so we really wanted to build something that was more modern write something that's leveraging cloud technologies right the cutting edge today and and so we we built this tool that uses serverless technologies so in AWS you can essentially which is amazon's cloud you can essentially write scripts that will accept inputs and run and feed data back right so and the great part about this it you know it's very easy to spin up and spin down so if you want to scan a

target really quickly and let's say there's like a thousand scripts you or checks you want to run against the target that can thread out to a thousand different lambda functions which spreads across Amazon's Cloud they can collect the data and then bring that back to you in a centralized repository so right so I mean there are some cons to this way to but so this is the approach that we took with this tool so so serverless services you know really the future right as far as cloud technologies go I mean I think lambda is starting to enable developers to to have a few kind of common workflows right you know one is kind of using as glue so they can

stitch together data there in other posit orys and another is to kind of scale out by like a restful api so and when you're talking about cloud you know though when we're talking about the evolution of cloud you know back in the day a company might have a data center and then they decided oh it's really costly to run our own data's and iris let's let's use a Colo and what it what it really was a Colo is just somebody else's data center which you're a data center that you're sharing with multiple companies and then cloud providers came along and said look that's still pretty difficult to get servers in you get in racked and up and running why don't we

just have some type of self-service portal where users can come in on demand and spin up VMs whenever they want that's kind of where Isis or infrastructure or service came from and that enabled kind of this revolution of cloud to start to take off now developers could just click a few buttons and have a bunch of servers and they could have their different stats they need the right CPU the right memory all that and they don't have to go through a centralized you know data center operations team or wait for stuff to be racked at a Colo but then you know taking it a step further you know what if infrastructure was serviced and these

cloud providers the VMS really abstract away the hardware so you don't a rack Hardware anymore but cloud providers said we can do this even better right we could abstract away the OS so you can just write your develop usually use write your app and we have like a standard platform you can just develop that app on our platform and then servantless really is functions as a service right it's actually no way the OS it's also abstracting the way the runtime so you just write a Python script and you make sure that the inputs come in correctly and go out correctly and then it takes care of all the scaling spins up spins down and extracts

everything else away and that the most common for this is AWS lambda asher has functions and GCP also has a server list solution so all the major cloud providers Amazon Microsoft and Google have solutions here right and so really as you need more of your code to spin up EWS takes care of that and scales up your application and and then it runs on top of their shared infrastructure services so so you don't need to maintain that and really you're only paying for the time that your functions are executing in AWS so you're going to reduce your cost a lot rather than having a bunch of servers or VM sitting out there waiting to scan something this will only charge

you when you actually go to collect data against a target right so serverless way more scalable way more cost efficient one of the side benefits of kind of doing it this way is when a server list function executes inside of AWS lambda it's gonna and it makes an outbound requests to maybe a server on the internet it's it's gonna do so out of AWS as pool of IP addresses and it's gonna be automatically randomized so there's really no static IP that you have to worry about and these net blocks are very large so that it'd be very difficult for an like a network defense team to block those entire network ranges and you know very powerful right

all right but what what could we really like as a red team or a pen tester what could we really do with all that power right so we can do faster port scanning we do more subdomain brute forcing you know we can extend out our scripts and weak screenshots of web servers and more right so I'm just gonna walk through some of the server some of the AWS is serverless technologies that we leveraged when building out this utility so right so these are the services that it uses and then we'll talk about each individual one right lambda which is what I've been talking about is functions as a service it's kind of similar to like a micro service or like

scripts for AWS so this is where your actual code is going to run and generally it's gonna be you're just gonna be focused on the code you want to execute and super cool hacker tags yeah okay so how does lambda actually get executed though so typically you're gonna need some type of trigger event right maybe someone call an API like a restful api and that would send the inputs into lambda and then lambda will execute the script right or or maybe there's a queuing system sqs is a queuing system we'll talk about so maybe a message gets pushed to the queue and then lambda functions or Landro scripts process each one of those and then take those messages and you know

maybe there's a queue for when you upload an image website queue and then as the image needs to be converted to a certain resolution or size for the website lambda functions come and execute Python code which format it and then those get stored and maybe like an s3 bucket for storage so that's kind of there's got to be a trigger there generally speaking and then usually when you're done processing you're seeing the data through lambda you want to store it somewhere so generally you're gonna store data in a object storage system like s3 which is a simple storage solution by Amazon and it kind of think about it as like an unlimited FTP server if you've ever used an FTP server and

use like uploaded a bunch of files but then it ran out of space right like this is this is trying to solve that problem for you it's trying to let you programmatically store data in Amazon's cloud without you know ever running out of space and without you having you as the programmer ever having to worry about you know data availability type concerns but sometimes you know you just don't want to store files you actually want to store the data in a format that is it's going to be queryable right you want to format the date and you want to kind of you know do SQL statements again it cut SQL statements to pull the data

and join it together so AWS recently announced Aurora serverless so this is basically like an auto scaling Maxwell fully managed service so you can just as you would for a normal web application and store the data in a my SQL like database and then and then retrieve the data you could also use a solution than a table us-like DynamoDB but it would cost a little bit more money so Aurora's serverless Edition is pretty attractive these days and then you really if you're gonna write something and it's gonna enable you to go scan other servers you probably just don't want like anybody just to like browse to your web app and be able to use your

lambda functions to scan servers right so so we want to have some type of single sign-on system or or some type of authentication to ensure you are the creek you are who you say you are right and Cognito that's what it's used for so this is a like an AWS service and it enables us to authenticate you and frequently this is used in combination with you know like to go to a website and it's like hey do you want to create a count or do you just want to use like GC or Facebook or your other accounts kognito can handle that type of authentication as well on the backend and enable developers to kind of a

struct away a lot of that code so sqs is AWS is queuing service right so think of it kind of like as rabbitmq if you've ever used that or basically when you have a task and you want them to get processed but they may take some time to get processed you might want them to get queued up so you would send a message to ask to sqs and then maybe as lambda functions come they would pick off a message one of the time process them and and then format data and store it in Aurora or s3 so you know big things is like when you sign up for a default AWS account right you just think you get

really excited about lambda so sitting on my computer one night thinking like oh man I can write anything and I can just scale it as far as I want so so I was just thinking okay well I'm just gonna you know talk to the talk directly to sqs using the API and then I'm gonna have lambda pick off messages off the queue and then I I did some speed testing and I realize it's like this is not going nearly as fast as I thought it was gonna go right so so the thing is with a default AWS account they kind of put guard rails in place to make sure that you're not gonna you know shoot

yourself in the foot by charging up huge bill so so when you buy your default when you sign for an AWS account you can really only kick off 300 concurrent lambda functions at a time so you can put a support ticket in to raise that limit but but that's what I discovered when I was trying to uh you know play around with this a bit so so so but you know sqs they're still going to be things they're going to take awhile and maybe you just don't have enough you've hitting your lemon L lambda functions so this is a good place to process the move the data from point to point process it and that's an SMS is more

like real-time notifications right so you know if you want to push out a alert to someone's mobile phone right like maybe as part of some two-factor authentication type service you could use this to put by email or SMS to a user's phone so and also you can use this just to stitch together between services but you just have to realize the difference between this and the queuing service is in the queuing service you won't lose messages they will queue up and then be processed 100% an SNS if the server's are trying to push data to is not ready it will just drop the request and so you'll you will not get that processed right so if

you're hitting your max number of lanta functions like I'm re running 300 then that those requests are just going to get dropped and then API gateways are really the restful api portion and this is how you're going to use lamda to stitch together technologies right so to get you know either your other information systems or even internally talking back and forth and this basically creates a RESTful API where you can query and have some type of action taken in the lambda function all right and then tying that all together right like this would take you a lot of time if you want it go through and set up every service so that you run this

tool but luckily AWS has a service called CloudFormation right and in cloud formation you can have a template and load this template up in cloud formation it will configure all these arrest these services so they talk together in the way that you designed kind of as you're architecting or building the system and that way you can just hand this single JSON file to anybody and as long as they have an AWS account they can spin up the structure in virtually the same format so really what this tool is which is called hindsight it cloud foregrip that you would download from our github you would just load it up and then it would take care of configuring

and all there's just those services so that you don't have to worry about any of that right and just kind of putting this together cloud formation spins up all these other services that's that first box and then when requests come in they come into the restful api but we don't want just anybody to be able to execute our lambda functions so we go to kognito which is that little credit card icon and to make sure that you are authenticated with single sign-on system and then we can use either sqs or SNS to push the messages messages out to services and then if we want to process data we use lamda to process it if we

want to store it in a way that's easy to look up like a relational database we push it to Aurora serverless or if we want to store the data for long term like images or things like that we push it to s3 and we can have authentication on all these back-end services so there's no way to get to the data except for going through kognito right and we call this total hindsight right so mostly cuz in hindsight I really wish we had this tool five years ago so but you know what is it it's a hundred percent server this platform you deploy it by a cloud formation template and it currently just doesn't numeration capabilities but

we're kind of adding on features as we go it's been a Star Wars theme to this presentation if you haven't noticed right and it's really easy for you to deploy by the cloud formation template really just you got to go to the cloud formation service and upload the JSON file but it actually is rather complex on the backend we're using step functions inside of AWS to kind of walk down a logic tree of which enumerations to kick off based off of what information we're getting back from the target right so we're kind of using the information like is port 80 open okay if so is ATP so just running on that port okay if so

okay check for these other things run these other land of functions and then we're kind of doing that for multiple services and we've built out a UI for this so it's easy to use because if it's not easy to use it's I don't more just project right so and but it also has the API portion all the services kind of talk back and forth through the restful api okay great so we're working on making it more extensible as we go and and that really just means adding more functions to do an additional enumeration to automate common pen testing or red teaming tasks so the cost for running this really depends on what you tell it to do right so if you deploy

the cloud formation template your costs are going to be very minimal right but then when you actually go to scan targets then you're going to be leveraging all these services and storing data and so cost will increase with time right so but still should be pretty reasonable okay great so without further ado I'm just gonna show you the tool if it's gonna let me write and this is what the tool kind of looks like so after you deploy the cloud formation template you can browse over to this you can buy the API gateway browse over this lambda function when you first come here it's gonna ask you to login your username and password are gonna get

authenticated against the kognito service so you need to manually go into the Cognito service and add a user right and password which is pretty easy once you log into the AWS console we just search for kognito and then there's a button on the side that says like users and then you add a user right and then once you're logged in you can create a project so for example in this project I can just name it a test 0:03 right I'm gonna flip over this and securely okay so so let's say there's assessment and we see a default Ubuntu page right so usually when we see these things there's a reason that someone has stood up this server and the reason is

usually because there may be some type of web application in a subdirectory right so we we might try and browse like slash admin we see like oh it wasn't found right but we can see we're getting a not found error so it should be pretty easy to detect what is available what is on this server and what's not so instead of like manually trying to pen test this right we could just come over here and under test3 click new scan and we can say like hey like check for all the ports on this target and then if any of them have a running HTTP or HPS then go ahead a numerator blech try to guess for

common files and directories and then you can put in a list of IP addresses or in this case we'll just put the single IP address so that it yeah it's just as a simple example right and like literally when we click Submit here and this this is gonna go spin up a bunch of lambda functions they're gonna go and pull data from this IP address right here like port scan it using pure Python and lambda functions and enumerate which ones have web services on them guess four files take screenshots and then return the results back here and you'll see how long it takes for the results to appear right so we click Submit we click

over the results and it will take a second for those lambda functions to move and store the data back in Aurora serverless but generally speaking and there we go right and a pretty rapid time frame we can see okay port 80 and 443 are open here's the banner that we got back from port 80 so we can tell like hey this is running Ubuntu because it's in the banner and Apache guys probably can you guys not see my monitor so so and yeah so you can kind of see that data there right so so you know nothing super groundbreaking but really helps you automate the test right and then here it goes down and says look we

tried slash test on the web server automatically for you using an AWS lambda and it returned to 200 HTTP code meaning that the resource was available and so then we tried to start guessing files underneath that and we saw there's an index.html here in addition it took a screenshot of the default web server so in case the web server has gone offline since you scanned it maybe you kick off a big scan of a net block and then you come back later and review the data and you could click here to see what that is and really all that does is downloads a ping file from from an s3 bucket using a signed URL right so

when you so in a secure fashion and really the website just says test so that that's all the ping shows us is it says test and ok great and we can do some type of like validation and make sure that the IP address that you put in isn't crazy right so it checks to make sure like the IP address you put in is actually rattleball across the internet and is not like a known DoD IP address or something like that so not that that would ever happened but just in case probably don't want to do that so that's kind of the tool flip them back over right so and I just wanted to talk for a

minute about how we dev the tool so that in case you guys want to do additional modules or whatnot you'll be able to do that so so the we're big fans of cloud 9 I was using cloud 9 for quite a few years and recently AWS purchased them and integrated them into the AWS platform so so what it is is it's like an IDE so if you've ever used software like pycharm for programming it's like that but it's all in the browser so anytime you want to add more code or whatnot you can just go back to the cloud 9 service which is now inside AWS and then that makes it a lot easier to

Dev the code right you know there are frameworks out there to help automate serverless application deployments they're popular but yeah so yeah and then the tool is gonna be posted up to her github I was hoping to have it up today but he got sick so so hopefully be up tomorrow but that is that is the tool and the presentation right so so I we do do trainings as a company and a DBS and Azure exploitation and then we're kind of integrating this tool into some of that training content so the show any questions about the tool Yakko

cloud9 I mean I like it but I feel like for the things that we're building they're really discrete and functionality right it's like you're creating this Python script and is just checking for like a TCP port are you creating this type of Python script is just pulling a banner right and now I mean I've used it to write more full feature stuff over the past couple years but but like I'm not like I'm more of a like a hacker red team or by trade you know I'm not like a full-time developer so if I was a full-time developer I don't know if it's like as feature-rich as you know like a pycharm type solution but for I don't know for things that I

do I like it because the code is always there it's in a state that I know works I can use it from any laptop I don't have to have the endpoint device configured the way I like and and if I only have five minutes to code I go in there make make some changes right and then I can get back to what I need to be doing and then come back and forth right or multitask in business meetings right so so that's that's why I like it but you know pycharm is great too right if you're doing Python yep what's up yeah so so the question is does it support internal VP C's so it probably should be

a slide on that right it right now the way it does run is it deploys inside a VP C so the cloud formations script does build its own V PC and deploys inside of it so I don't see any reason why you couldn't run this against an internal V PC the only thing that might prevent you is the sanity checking on the IP blue box so you just have to go in there and like remove the ten dot like 10.0.0.0 slash eight from that list and then you should be able to run it on the internal right so ya should work for you so let us know if it doesn't yeah yep API gateways it's still kind of pricey is

the question this great question it doesn't feel like it to me sorry I don't I don't really know coughs so probably not the most responsible spender in AWS is there anybody who has a comment on that

[Laughter]

I mean I haven't run into any issues like that with crazy AWS bills when bills win building this tool right and testing it so because I definitely pay those bills so the so I mean for my from my use cases but I I can't speak if you were building like an entire platform and using a gateway to talk back and forth I mean I could imagine the cost could get higher right so yeah Wes you have a comment yes they haven't yelled at me yet right good question okay so Bryce how do you prevent your amazon from yelling at you right yeah so I've written a lot of offensive stuff in that's and ran from lambda right and to

date I haven't received any complaints from AWS so the only complaint ever got was hypothetically speaking if you scan a DOD IP address which never happened scott

yeah I just know I yeah I use it for all this kind of stuff and we don't we don't ever get any complaints so all right

hmm okay yeah all right let's talk more later so let's see on the subject of this you know there is there is like an AWS pentesting authorization form so if you are pen testing targets that are that are hosted in AWS you should be completing that form that they did modify the process so that like the route count has to complete the request for the Ford which becomes problematic in certain environments when where no one has access to the root account anymore so but anyways so you know there is a process there that's documented I AWS is website for pen testing other AWS systems right so yeah and then on Azure they're the rules are a lot more relaxed

like a sure I think they did away with even having you to you don't even need to ask if you're testing your own VMs on Azure anymore you only need to ask if your pen testing the fabric of a sure right so which they have a like a bug bounty program for so so I don't know what the rules are in GCP yet so other questions comments all right I do apologize oh yeah back here

yeah I mean you okay the question was what about from the blue team side right I mean you could definitely use this exact tool for blue team purposes and you could even modify it to automate like like let's say you had like you know a spunky ass instance and it was kicking off no like tracking noddle Bowl so you had some other sim and it has alerts right you could even say like hey win hindsight you see this certain information talk to this other API and kick off a notable or alert yeah that's the whole hundred percent possible and in fact may even be more useful on the defensive side than the offensive side right i far to say if if you're looking

to get more visibility into your own infrastructure in an automated fashion alright so I think that's a good call out right thank you yep

Oh guys so there's a great question so I know it's Python to seven right is the runtime for most of the functions we are using Amazon lambda step functions as part of the design right I apologize I can't I don't really know how it works under the hood other than that if you email Michael his emails right up here I'm sure he'd be you know happy to chat with you or answer questions or he's in P side slack too so so and happy to get you more details on that but honestly yeah I didn't expect to present today so I didn't even really I didn't look under the hood right so a ton so yeah no

problem so but happy to get you answers so any other questions yeah Scott wait they create you up to a thousand right but I don't know I will I wish it'd be unlimited the how long did how high to AWS crank up our concurrent lambda limitations so so I don't know I mean we haven't really run into any worst-case scenarios with this like tool like infinite loops or whatever so so and it seems pretty safe from our testing but obviously you should test off yourself before you use it right if you're using it an account that has high high limits right so all right well thank you guys for coming out I'll be up here I do

apologize Michael isn't able to make it he had to go I'm sick and and yeah thanks appreciate it