BSides Toronto 2019 Jamie Baxter

but now we have application pen testing in 2019 API is in the cloud please welcome okay Jaime Baxter sorry

hi everyone so I'm Jaime and thanks for coming out and I apologize I won't keep people from lunch too long so I'm well aware of my responsibilities as the last line of defense between you guys and lunch so with great power comes great responsibility so today's talk we're gonna talk about clouds and we're going to take it we're going to take a journey into the cloud and specifically looking at the deployment of api's or application programming interface specifically REST API is in the cloud and we're gonna see all the different cloud services that can help you do this and also some of the pitfalls some of the security concerns and how some things may not

work exactly like you would think it was and then when you're doing both as a developer and as a pen tester the results that you might see as your clients and as your targets have migrated their their backends into cloud-based infrastructures just for limits of time we're gonna primarily focus on AWS cloud and infrastructure and services not to pick on them it's just that we can only talk about so much so first slide is Who am I so I said I'm Jamie I'm an independent information security consultant I'm probably in that 15 20 year plus Club that they were just mentioning I focused on doing a security assessment so application cloud mobile infrastructure and previous to that I worked in the

aerospace also government and Finance sectors I'm primarily focused on application security so everything to apps writing apps coding apps make your apps are secure that's really that's really my passion that's really what I do and also red teamer occasionally and also a CTF one that when there's time so what we're gonna look at today so an initial quick background iran rest api is and and how those fit into the cloud cloud service trouble so this will be a first cut at and how some of the cloud services may not behave as you think they will we'll take a look at the and then we'll start diving into individual ones so you know how information to

closers error messages may be a lot more severe in the cloud the different flavors of server side request forgeries how how role switching and role escalation becomes the new lateral movement or privilege escalation in cloud-based services and then also taking a look at easy to and if there's some time we might look at some more complex server-side forgery requests and and then some funding with container registries and DevOps infrastructures so first of all let's get into some of our background so wrist epi in sixty Seconds now a wrist epi is defined as a contract between the client and server so typically we have a client this can be them this can be a browser this can be a mobile application this

can be another application or other back-end service and it will issue requests to our server that the request will look like any other normal HTTP request that will have a header it will have its method and will have a URL and in a body or payload now the interesting thing about like rest is that one when people are like hey you know how do we how do we design this API we're like well we already have all these HTTP methods let's use them as part of our API so informally a get method will get information a patched method will we'll edit information delete with the lead information on the server and this allows you to sort of like organize your

API is roughly around the HTTP specification and it's these different things always fascinate me because when the original like HTTP RFC was written you know I'm sure the implementers never really saw how it would later be used to connect these like dynamic mobile apps making tons of requests leveraging HT methods in in in the interests like way so so we have our server and then it makes it set its responses so you and again they reuse the HTTP specification so like 200 is okay 403 would be forbidden and a whole other suit the whole other series of like 500s for server error and so they basically wrote on this to provide this well-defined contract for

applications to talk to servers and servers to respond to the applications now where we start to get into trouble doll is and always gonna trouble is when when you start talking about authentications and sessions because there has to be a way of ensuring that the request that's made is made by someone who's authorized and that person has the authority to do that request and you don't allow the wrong people to do them and you don't allow someone to pretend to be somebody else so you typically have anything unauthenticated request which is your normal h2 request but when you start to get out to authenticated ones then you have to have a way of tracking who is who is doing it

and there's been there's tons of different ways of doing it like people will try to use tokens but people try to use like the original one is like basic off in the header there's cookies there's tokens and specifically and we'll talk about this in a sec is getting into how different apps will use bearer tokens or specifically JWT tokens so meant over all our requests will be lumped into these these sort of buckets unattended kated and authenticated and unauthenticated it can be any authenticated user bound to a particular user or administrative level axis and you'll see why these subtle distinctions are important because they map to a lot of flaws that keep coming up over and

over and over again specifically when servers check the requests so often what they'll do is they'll just check to see if the if there's a valid token but they won't necessarily check that the token is tied to the specific user that is requesting that piece of data or you will see a case where they'll just say or to allow any user to edit something that doesn't belong to that particular user and we'll see some examples so the more things the more things changed so back in the old days we dealt with ma like large monolithic applications the server's had constant current feeding they were difficult to scale and you know difficult to provide mobile versions and it was a largely homegrown

codebase now as we moved into using cloud-based infrastructure we use a lot more building blocks we have these single page applications that try to completely just decompose the front-end and then we try to completely pose a back-end into a series of you know we've moved from resti like individual servers like running on like ec2 or VPS and then we moved now to lambda with server like service infrastructure and we'll see more examples any servers are ephemeral meaning that they get stood up they get stood down the lifespan can be measured in in hours or minutes as opposed to months and years as in the past and it's it's designed to scale from from the core of it so let's take a look at these

building blocks that we might assemble here so what I've done here so this is just a fictitious a fictitious AWS deployment for a back-end of an application so what we do here is we have so here would be the cloud front the cloud front entry where the requests would come in we have our static content that's stored in an s3 bucket so instead of having a server storynest we're leveraging s3 here we're using something called API gate billion eight way a gateway again is another idea service that will allows you to basically handles the part of receiving requests parsing them valid it can it can handle some of the validations of the tokens and then it handles the routing them to

the correct processor or the correct weather the correct service that's going to handle that request and then also we have something called Cognito Na Davis again that's a service that keeps showing up again again and I can handle identity pools and also user pool so this can take care of you no longer have to sort of create a a user database with the sign up and manage users and user and deleting users you can it you can leverage this service to take care of all that functionality for you and then as we move back one more level in the infrastructure we have our our lambdas or I mean choose the buzzword of the day whether you want to

call it serverless or Computing's and an azure it would be compute instances and so typically these are smaller functions that are designed to handle one or two very specific things there's different sort of architectures that you see different sort of factoring approaches but but in general that you want to keep them limited to a few a certain subset of functions and so basically this is our like saying it let's say our REST API and then we've gotten it so that you have we've we've decomposed it into three sort of primary lambdas and was so register or so registration functions got associated as one profiles got associated with another and messages get associated with another now I've seen

architectures I've seen clients where they'll do like one server list one sort of lambda function for each rest function or I've seen ones that will have just one lambda function for all of them or sort of in the middle this this is an example of in the middle approach where you're you're clumping a few like highly correlated functions to the same compute instance alright so so why do we do why are we doing this well like I said these these cloud services will sort of offload a lot of the work that we do in a repeatable way so here for instance this is a this is how so instead of trying to stand up a server

configure it you know configure your front-end configure your back-end like do all the Apache configuration do all the PHP configuration all that good stuff instead here what we can do is say ok this is our API gateway API gateway use user canino pools for a user no user pools for user registration deletions sign up all that so basically what this does is this takes a huge amount of work off your front of your development because now you have these little LEGO pieces that you can place in so this is great and if it works as expected then what's the problem right I mean in theory you should never have a pen test where there's issues with user

registration and user pools unfortunately as we'll see that's not quite always the case and the other neat thing about this is that instead of your configurations becoming you know tons of different files and like you know configuration parameters if the script out to configure essentially your infrastructure can be represented entirely by JSON so and that moves you towards the you know where people say software as infrastructure and things like that it's really no more complicated than ya know files and JSON files but this just kind of gives you a view of what it looks like in actuality so let's say we're doing the API programming well here for instance we're showing the programming the profile get

current user so we're telling it that it consumes the JSON it produces a text HTML and it's going to respond with a 200 and and the content type of a string so we're still relying on the development team or the programmers to provide this functionality but at least at the front-end on the gateway the whole interface is already specified and in theory should behave in a deterministic way so instead of having to write out all these function handlings and URL handlers and parameter handlers you can just supply this JSON file to the API gateway and off you go now the problem is is we still are left with how do we handle authentication and again authentication remains rain comes

back over and over again and ensuring the right people or making them are quite right requests to the right things all the Amazon infrastructure relies on I mean you can you can force it to use different ones the one you really want to use is an oauth2 mechanism it's primarily based on identity providers we can use the the user pool as identity provider or you can use a third-party identity provider like Google so for instance when you see sign in with Facebook what it's doing is that it's doing the oauth2 flow or it reaches out you know checks your credentials against it then it creates a token and sends it back and says hey yeah this is the person that they say

they are and this is a token with whatever claims I agree with so the idea is that you have this validate you have this validation of the ID that you have access and also there's something called a refresh token to allow you to reinitiate a session without running through the entire authentication flow so for instance with a refresh token like you know if you have an app and you haven't used it in a couple days and then you go in and it seems to take a little longer to start up but it still works you don't have to put your username and password and again that's leveraging the refresh token and if it's if it's using an OAuth flow and the

Refresh token basically says hey you know you know hey I'm back I know the other tokens have expired but refresh was usually good for 30 days so you know Jimmy's give me a new ID token and an access token the entire the entire way these tools work like API gateway and and in the lambda authorizers all revolve around the use of tokens and that's the same with any auth to flow you is the develop ability of the ID tokens that they're from a trusted identity provider and that they and the validity of the access token which says hey you have the access to to perform that request where things start to get a little murky is its developers and even

Amazon and other providers become slightly inconsistent with the way they handle ID tokens access tokens and refresh tokens sometimes the ID token is is invalid but the access token is valid so they still allow it sometimes you'll have a you'll invalidate the ID token but the refresh token will still be accepted and we'll see some of these cases very shortly and then finally just the representation of all this so it's all represented as JSON web tokens they're secured in transit they must be secured in strands in storage they must be signed you can't encrypt them but typically it doesn't offer a huge value as long as you're not putting sensitive information there claims as a little aside when

you're when you're writing your own infrastructure and assigning your own JWT tokens don't put sensitive information in the claims and that could include for instance like personal data identifiers you know the plaintext passwords themselves and in general you want to avoid that and so our typical loss to flow we've got our canino back-end the authentication is performed against the user pool role again rolling your own I am should be the last option you receive the claim back and then the API gateway will control access to your api's now what is the token look like well that's sort of your typical token if you that you can kind of see but you typically have a header a payload and

then the signature the key and all this really is the signature because in the past there's been different attacks against token where if you set your algorithm to none and then don't sign it and it will still be accepted any modern library should prevent that the one that we saw most recently was that if you had sent the algorithm instead of two from instead of an H Mac if you had sent it to if you set it to an RSA validation it would actually accept the public it would actually accept tokens that were signed by the public key of the server so that that's obviously a no-no to write and ideally especially with Amazon you don't you're not gonna see any these

issues so what do we look at rusty P is cloud services user pools I am ia gateway authentication we're good to go so like life is good right I mean all these challenges of sign of Aladin tokens and forcing API access token expiry it's all taken care of from our help with the clout the problem is is the future isn't quite as rosy as we as we might like so we can rely on the cloud services to helping to help us manage sign up and log in they can certainly validate the token for the tokens for us the problem is is when we get into enforcing API access do the services link link the tokens to valid

user sessions so do they permit BA will de Lobo to make a request to Alice's data well unless the programmers have specifically addressed that that lateral privilege escalation or lateral data access it won't and this is this is something developers still have to take care of and this is a common finding over and over again and so commonly and they gave it a name ID or and the the other thing too is is token revocation or sign out so sign so whereas in the previous case it's well understood that hey it's up to developers to make sure that you're managing your your permission your your permission model the token revocation is a service provided by like AWS cloud services for

instance global sign-on API allow a p.i allows individual users to sign themself out from all devices hey that's great well and then if you dig down through the documentation well actually global signup revokes all tokens but the ID token and the ID token is a bearer token that contains that is generally used with services outside of user pools so that's a direct quote from the documentation but that's kind of a little ominous right because the ID token is what the developers are going to use to validate if the request can be made to that particular users data and and then if you dig down even farther you see API gateway does not reject it even after it has been revoked so if

you're using a devious gateway and you you are have a connected to canítö user pool and you log out that logout session is still valid for 60 minutes and the API gateway won't even even check the ID token so they only invalidate the access token and the Refresh token but you can happily continued to do requests and this is a finding that keeps coming up again where development team will be like no we're using we're we're following Olaf's guidelines we we're doing system management properly we have sign out we have a sign-up function and enabled our sessions are being expired properly and then as a tester you're like well no on my test I'm still able

to use these tokens long after what's going on and as you dig in you find out that no it's because it's because that it's just simply not invalidating the the tokens and you know straight from the issue tracker from February 17 is is somebody asking what is the purpose of a sign-up function if it does not invalidate the session token I don't know that was that was an answer so two years late to years later we're still waiting for the answer and and just as an aside halfway there's a little break I just want to call out that so little Blade Runner reference there a question lost in the issue tracker like tears and rain just so you know that the original

Blade Runner took place in November 2019 so we are event we were basically at the time the original Blade Runner took place so how's that for you eighties kids I know it freaked me out okay so another thing with cloud services again things don't quite always work the way you might expect them so if you've been at blackhat or all involved in the application security space there is a big a big sort of like a research released at blackhat this year by by James kettle from Fort swagger of I know wax probably one of the most amazing I've SEC researchers out there and he be we took an old attack that was first sort of talked about in theory in 2005

and basically you know weaponized it and really showed the implications and and now there's a whole bunch of like research looking at you know content length and transfer type of encoding and I think that we talked about later so I don't going to go into the details of the attack I just want to show you how the messaging has varied in the last couple months from from AWS so so obviously once this once is H this request smuggling desync attack came out people were asking hey I'm in the cloud am I protected well so if you use AWS cloud front you have partial protection they claim you had full protection but then after they dug into it they found

out that it was it was primarily a blacklist state implemented so we all know black lists are just an invitation to find evasions so although it allows specific header combinations it's not a complete combination on Twitter just yesterday albino max was saying that it only mitigates about 50% of the cases AWS load balancer provides no protection a devious application load balancer provides no protection these states in the roadmap but it'll be a user configurable setting AWS application gateway which we've done the API gateway that we've talked about quite a bit it's unknown I couldn't find an or get a statement of whether it protects against this some preliminary testing seems to indicate that it doesn't unless you add

some very specific header processing messages in your JSON but it certainly didn't appear to be an out of the box feature and then finally as their application gateway they do have in their laugh module they do have specific rules for preventing it for preventing smuggling header mismatch as they call it but you have to enable it and it's still unknown whether that will be a complete mitigation so just because you have protection doesn't necessarily mean you're protected now moving right along - so we've got 10 minutes left so moving right along to some other interesting things so the classic finding in an API pen test on a normal search on a on an on pram or non

cloud-based architecture you know is an information disclosure and typically an information disclosure might take the form of a of a verbose error message that sort of describes the that query that failed or or gives you information about how long a request took or any of that good stuff and typically we dutifully put them in findings and you know there's a chance if you chain enough information disclosures together it might give you some you know it might lead to something more more impactful but the problem is is that with the default configuration of say lambda functions is that the default behavior it was an error message is to dump the process environment variables the problem is the process environment

variables contain a WS key AWS secret key and session tokens so now there's ways to prevent that if you enable certain configurations but the problem is is that that's all you really need to you sort of essentials to take over the account and what's interesting or at least take over that role I should say that's more correct what's really interesting about about cloud infrastructure is as you start using more and more cloud services and cloud infrastructure it now all becomes about the roles and permissions that they that the entity making the request has and everything has a permission so whether it's the individual lambda servers whether it's the API gateway whether it's the ec2 compute and in the

background you know it's all tied to two roles and once you get one you can often privilege escalate laterally and privilege escalate or laterally move ie switching roles it's a new privilege escalation and to achieve and to get to an account that you the that might provide more impactful more criticality in fact the good folks at Ryan on security have written something called the PACU framework which provides at last count I think it was 25 different privilege escalations techniques within an AWS infrastructure and that's why you you really should be considering also having cloud security assessments where you give someone the lowest privileged role possible within the infrastructure and then and then from there see if they can escalate to

console control or or something critical like s/3 s/3 exfiltration so server side request forgeries again it's another mechanism so basically a normal infrastructure just means you can make the server request something so you can request local resources or internal resources the problem is is that in the cloud you can also hit the metadata instance and this is a sort of a classical attack is that as soon as you have an SS RF vulnerability within a cloud base something on ec2 for instance you can now start requesting the metadata and AWS key secret key session token' and then you can start doing that same sort of role shuffling and role juggling and role escalation that we

just saw and there's so many different ways to get these keys and there's so many different metadata endpoints to to hit so these are all different sort of if you do find an S or s Boehner ability in a cloud-based service these are all ones you want to start looking for and the interesting thing is that now people are trying to come up with mitigations and like how do we stop these SRS and so some people are trying to put in proxies unfortunately others are trying to put in wifeless and so these are just some of the common evasion so just keep popping up over and over again is you know is that you you can shorten IPS and

also you can rebind the NS entries and there's some there's a really good talk lino to recon it that really gets into some of the DNS entry binding so takeaways there's a lot of nuances of moving your applications to the cloud you know even from moving a simple VPS to just starting to leverage these cloud services a lot of building blocks I can really help they can really solve a lot of problems but they can also they can also create new ones if you're not aware of exactly what's going on behind the scenes the Devils in the details all these high-level platitudes of zero trusts and dev second offs aren't gonna make these ass more secure instead what

it really comes down to is rigorous and diligent work by technically proficient practitioners with the with the business support and you did and ultimately you know these same problems will keep coming back unless unless we address this so we have I think about probably about two minutes left so Nina application pen test need a cloud penetration test feel free to reach out or if you just want to talk about clouds or if you want to talk about how the Raptors are going to repeat this year thank you thanks great talk as far as documenting api's I've seen some people use swagger to auto document their api's is that helpful or is it much better if people don't just rely on

an automated robot to do that and document it themselves so so you always need that stage of manual check but swagger like having a swagger output is is actually extremely helpful because you know when I'm doing an assessment I'm talking to a client that's based around an EPS estimate one of the first questions they ask is thank as do you do you have a swagger or yeah I'll file for your API because then what you can do is you can take that and it can be fed into like the open a parser and burp and now you can have a repeatable like at least at least a standard baseline that you can say I have been through every single API call

and they've all been tested with these parameters it doesn't it doesn't mean that that's where you stopped it just means that now you can have a repeatable deterministic test of the API so very useful so to answer your question very useful I always love to get them on assessments but you still need that second piece where you still check manually for common API methods and ideally if you have the source code validate that the the routes that are actually in the code match the swagger file that you have because one of the common things that happens is you get a swagger file and then like a test starts two weeks later and then between that time period the

devs just fit in one or two more functions you know oh yeah and we change this one parameter oh yeah and we did that and we changed this header so it can be a bit of a moving target but no but times your question they're great to have hey so I'm not really an application developer so a little bit of your talk when I went over my head but um was very interesting and I'm curious about some of these issues that you highlighted seem to be more based around these services being offered by AWS such as when you log out it doesn't necessarily kill an ID is there something you can do as an application

developer to mitigate against that great question so the way you can do it is you need to sort of you need to keep track of every so when someone logs out what would have the app will house have to keep a revocation list so we would have to keep track of every token that had logged out and then when it receives a new request coming in it would need to go to that revocation list and say hey this request that just came in are you in the revocation I have have you been previously logged out so the app would have to take on some more work than they might initially think but most app developers don't do that even without

cloud services so but the ideal situation would be that you check against this revocation list because the alternate approach which I've seen a few times but really ends up consuming way too many resources is checking every request to see if it's valid well now you're just duplicating API gateways job and now you're doing all kinds of extra processing on the token you don't have to do so it's easier just to compare it against a map of invalids that you expire after an hour anyway so if the maximum expiry time is a window as an hour after an hour you drop it from the revocation list anyways

so okay so I'll do the second question first so that would have so you're at the end of the day you're still trapped by the idea that you have you have to you have to submit your credentials to somewhere right so in this case in like an oauth2 flow you're doing against the identity provider so in that case it's it had a it needs to be over HTTP but once you're in an HTTPS tunnel and assuming you're using like the unknown vulnerable algorithm and a strong key and you're not up against time AI or a few people get that Joe you're not up against like some you know up against like some crazy like ability to decrypt something that's

how you have to do it is a posed parameter within HPS a tunnel and and to it but to your second question for the storage of the token that that's a great question and that is a challenge so you have to the token has to be stored somewhere right so yes if the endpoint device is compromised then yes the token is is is now compromised and there's really no there's as far as I know there's no way around that particular challenge that you need to take care of your tokens and and care for them and keep them safe but the end of the day if someone if someone steals the phone and then and or someone steals the hard

drive where the cache file the browser has it then then they have your token and and to be honest one of the things you you look for an internal assessment when you're when you're if you have someone's account like your your honor you're on a box it is to see what sort of credentials are stored in local storage and browser storage and a JWT token is always an interesting find and we have one last question up here sure yes you mentioned some security researcher James something James Carroll James kettle albino axe oh and he's the lead security researcher at ports figure and his content is always amazing oh and also the company that made the article

that 25 ways to hack AWS so that's a so that's a so that wasn't quite the name of the article so it was a framework and it's a framework that's oh look it's a company called Rhino security and they create a framework called the PACU framework that specializes in AWS security there's a Derby con talk on it from last year fine either this year last year and they have a huge number of blog posts that really delve into individual privilege escalation techniques and they've automated into their their framework and they're working towards having a platform that will have these different AWS specific lateral role lateral movement and roll privilege escalation techniques baked in alright thank you very much thank you

token of our appreciation okay