Hacking Serverless Applications: A Treasure Map For Uncharted Waters

thank you thank you for having me Rob and pronouncing my my name correctly um so we're going to talk about today about hacking 7s application I hope you enjoyed your lunch and you have filled with energy to listen to this session

[Music] so uh my name is hen if you're hitting the slide you probably read it as Chen and that's totally fine if you call the chain I will answer I'm used to it but the right way to pronounce my name is um for the last 10 years I've been helping companies um delivering and developing their software projects and achieve their business goals and in the last two years I've been working at the Consular security as a software development manager leading our Cloud native application security testing product you probably noticed those little hearts there and well during those years I've had five kids with my beautiful wife and definitely that's my biggest project so far hmm

so what are you going to talk about today what is serverless application right what are the challenges in serverless application what are the security risks and challenges that we have in serverless application and we show some demos about how to hack serverless applications and how to protect serverless certification most of the things we will see are AWS oriented but everything is applied to all major Cloud providers [Music] so um how many of you are using serverless application please raise your hand oh quite a few nice so over half of organization in each cloud provider have adopted servers and that's a huge shift and we're seeing um as you can see in AWS gcp and Azure

all of them um and organizations are adopting serverless and we are seeing this Rising Trend among our customers as well that they're using more and more serverless application and deploying more serverless components um and well that's um here that serverless is here to stay right um I believe that it will it will become one of the dominant Technologies in years to come and that's why we are talking about it but serverless is not it's not just a technology shift uh it's more than that if you're looking at the architecture we're not talking about um any more about big monoliths with one flow one big logical flow we're talking about architecture that is event driven by Nature we are talking about

components that are even smaller than microservices we're talking some will call it Nano services and each serverless function is developed with a single purpose right with a very very simple single purpose and that's part of the architecture we're also talking about the Cycles right the development life cycle the dev Ops or if you want the devsec up Cycles um developers are deploying multiple times a day while using serverless you see it I see it in my team it's not anymore um deploying a dead on the cycle or even at the end of the day Developers Code test build and deploy um and the process right the process is no more manual you want to automate

everything if it's not automated it just won't work um everything is automated we're using infrastructures code and that's something that it's a must right we cannot do manual processes in any of the things many more pipelines a lot more automations and by that combining all of those things um the decision coming bottom up right the developers are getting more and more responsibility to their hands they're choosing their services they're choosing their tools um they're creating all the pipelines and that means they have bigger responsibility and the decision is no more top down but bottom up so let's dive the dive deep to serverless architecture and it's not of course not only about the code it's

about the architecture itself so if it was zoom in a bit about what a serverless component if you like looks like so we have this um this Lambda function at the middle and we have the triggers so another function can be triggered from multiple ways multiple triggers we have for example the API Gateway an S3 bucket an event from dynamodb a cloud watch log and much much more and then we have that Lambda function that hosts our code and the services that usually interacts with and again the Lambda function hosts R code which means it also holds our mistakes in which we will zoom out a bit look at a bigger serverless architecture it's not it's not that big but it's a fairly

typical serverless architecture we see that we have a lot of serverless function across um the entire flow and we see that um it comes from an end point from one like the user interacts with the application in one endpoint but basically the Lambda function is trigger triggered in multiple ways multiple triggers Happening Here the red circles you can see that we have an API Gateway we have dynamodb we have the S3 bucket and basically each trigger has its own inputs right so if something happened and triggers the first thing what's happening behind the scenes it's important because we see that the invent bridge will trigger some State machine and so on and each function housed its own code on

configuration and on policies and that's one of the challenges we will discuss about it more this is an application one of our customers looks overwhelming right but it's a fairly small application and we see again a lot of Lambda function around 100 Lambda functions and we see API gateways and so on and what I want to show you here um is the overall view of how a serverless application looks like it's not like one process going synchronously a lot of event driven you see those gray lines that are not connected um physically they're connected by configuration by their policies and and with that architecture it's pretty hard to secure each serverless function has its own code on configuration and on

policies and we need to find a way how to secure uh secure those functions so let's talk about application security how are you are you securing your serverless application less a lot less hence than before and that's fine um a Google trend from five or six years we see again that Rising trend of searching um serverless Computing it's rising and Rising a little drop because of the covid but we see it going above and above but when looking at the serverless security we see a flat line and that's something we see right security is somehow Left Behind until it's adopted like we have a technology that we adopt and security is trying to fill the Gap and that happens slowly and

what we want to do is close this Gap the sooner is possible and that's something that we will talk about so let's talk about the traditional application all right so we have um this little architecture user interacts with the application there is a load balancer and so on so here we have one entry point one and we will put all our security measures security components security rules and so on on this entry point and well basically we are covered in a way it's not the case for um serverless applications we have multiple ways to interact with that with the application uh we have the API Gateway and we can put all the some of the security

measures there but we also have the S3 bucket that might trigger another serverless function and we have maybe an sqs and we have a lot of again triggers that can trigger our code and our Lambda function and that makes it uh harder to secure let's talk about a bit about the serverless risks so I named a few here but they're much more we will go over go over them quickly so we have the over privileged functions right so each function has its own policy that basically let the function interact with the cloud environment and usually what we see is that the function has an over permissive policy hard to pick the right permission for the right policy for the right action

and so on and we will take a look at an example right after this slide we have the insecure secret management how we secure our secrets do we put them in environment variable do we use SSM or something like that the sensitive data exposure the denial of wallet right we are talking about serverless you can scale and scale and scale and don't have to worry about any um server or clusters or whatever but with that comes the risk so if I'm attacker and I will invoke my function again and again and again and again and again and I don't have the right configuration in place so you don't have the right concurrency limit um in configuration I'll probably see

that at the end of the month in my build right um we have the event injection multiple ways to injection as you all know the vulnerable dependencies so we host our code but we usually use multiple third-party packages and that may introduce vulnerabilities the broken authentication open resources open S3 buckets that we all know and more resources that are open and malwares earlier earlier this year there was introduced to denonia the malware that is targeted to AWS Lambda function and much more you can read about it and I will show you the resources at the end so let's dive deep a bit about overprivileged function which we see is a bigger challenge for our customers

[Music] so what you see here on the left side is a pretty straightforward Lambda function the Lambda gets an event extracts some data takes the data and put it in a dynamodb table with uh the put item action on the right side we see we see the execution role of that Lambda function the execution ball basically is a wall that includes a policy that basically allows the Lambda to interact with the cloud environment and told it what action it can do and what are the boundaries what are the resources that the Lambda can talk with so here we see the put item action that's the only thing that the Lambda does in this specific use case

um in the cloud environment but we see that the permission is dynamodb star which means this Lambda basically can do anything any action on dynamodb within the cloud environment dynamodb has like 20 or 25 permissions here it's put and can delete and read and so on besides that we see that we also gave a star on the resources which means this Lambda basically can do anything on dynamodb on any table we have in the cloud environment so how we can fix that right so first of all that's a problem if an attacker can compromise this Lambda and get its keys he can do anything not just put item on that specific table you can delete read

everything on our Cloud environment in any table we have and this is hard right I see it in my team it's hard to maintain these policies it's hard to know what action do what permission do I need for that action there are thousands of actions above 300 Services specifically in AWS it's hard for this example um what we can do so we know that we use boot item let's change the permission to only put item by that we are minimizing the risk dramatically and if an attacker will compromise the alumni the only thing you can do is putting an item in the database the second thing is of course removing the Stars the style from the resources

so we will change that to the and table we are talking with by that we minimize the risk again dramatically of course you cannot do it manually it's hard to do it manually and as we said before if it's manual it won't work so we need a way to automate it for this process and we will talk about it in a second so um now we're going to talk about a few demos we'll show you a few demos we will watch a few demos about how to hack serverless application um basically attacking without being noticed Lambda function holds its keys right it spin up when it needs when it triggers when it invokes and and then we have the

keys to interact with the cloud environment and if an attacker can get those keys and use them basically he proxy the Lambda without being noticed you can use those keys from his local machine or from whatever and just interacts with the Lambda with the permission the Lambda can do right with the actions the Lambda is permitted to do so um the first thing we will see is a simple application and the attack Vector is the API Gateway a user is interacting with a slack Channel with a bot sending messages this those messages are going through an API Gateway that invokes a Lambda function is Lambda function stores the messages in dynamodb and respond back to on slack API into the

channel what we'll see is um in this specific Lambda there is a vulnerable dependency with a known cve um that we are compromising basically we took the payload from Google search the cve took the payload from Google and you just interact with that slackbot and found a the exploit

okay so we're interacting with the slack bot we have an ngrok server open on the bottom left and now we are pasting that payload uh we got from Google and trying to see if we can do anything with it we are executing um the command there is a command injection in that cve and we saw that we got a request to our local server okay now what we can do we are changing a bit the command and we are trying to do LS and again we're succeeding we get the file name and now we will cut the file content and see what we have uh in that Lambda so we are cutting the index.js file

encoding it to day64 and then we can we will be able to see the code see what's happening inside the Lambda and trying to get more deeper so you got the code

now we will encode it sorry decode it

and we can see the code right in front of our eyes exactly what's happening what it's doing interacting with slack API which tables what are the environment variables and so on now what we're trying to do is using not a put item but using a scan command on the table to get everything from that table well if the Lambda have the right permissions in place we won't be able to do that but in this case it's not the case we see that we are constructing the command scanning the table and basically sending back everything from the table to the select Channel we're closing the inbox server we don't need it anymore boom we just have

everything inside a table in the slack Channel and that's basically it right so we found a vulnerable London right that that happens okay it's our code we have vulnerable dependencies and so on but the the attacker would be able to escalate it and get all the information from the table if the Lambda had the right permission in place that won't happen so let's see another demo in this one we will attack an S3 bucket um the demo we will see is again the code itself includes a command injection that execute our malicious command and what we'll see here is the user interacts with the feedback form um the attacker can see like what's happening inside the network and what is

happening is that when you click on the attach file there is a pre-signed URL to upload the file directly to an SG bucket a present URL is a URL a temporary URL with a specific expiration time this URL is responding back to the client and then the client is using that present URL to upload the file to the bucket and to the Lambda foreign um this application we're seeing by the way this is a dvsa application it's a damn vulnerable serverless application that we've been part of the development it's under oasp and basically it's an application an e-commerce application built totally on serverless on AWS that you can deploy with three click just to show you how

um a serverless application can be well then vulnerable so let's see that demo [Music] so again we have that anger server we are logging into the application as you can see it's an e-commerce application with games you can purchase you're going to the feedback form sending the feedback everything's went good now we will open the network and see what's happening so we are attaching a file see the payload attachment name and we see that pre-signed URL that we Got Back oh everything uploaded everything fine now what we will try to do is change the name of that file with a command injection and well we will see what happens

we're constructing and what we're trying to do is basically getting the environment variables of the Lambda function

touching this file and looking to see the respond back to our Android server right so we've got results we're going to the server and we have the environment variables encoded now we are decoding it and see what we can get from there oh so we have everything including the AWS keys that um the Lambda has to interact with the cloud environment basically we're taking those keys exporting them to our local terminal local machine and trying to see what we can do with it

first of all we're listing all the files all the buckets inside and and we've got everything right not only one bucket not only we didn't get access denied anything we got everything um on that cloud environment all the buckets now we're trying to go deeper and deeper and deeper and see what we can get looking for something interesting that we can see just going over and over until we find the file that we're looking for and now we're executing a get object command right to get the file

we get it we get a specific order right it's an e-commerce application we get a specific order and now we will change that order a bit change the file the the name change the address and upload that file back again to the same bucket to the cloud and again what we see here is command injection right we have the command injection inside the Lambda function but because we have uh over permissive policy we can do a lot more than what this Lambda basically actually needs just uploading a file we can list we can get and we can upload and that's exactly what we're seeing here excuse me

so again the same thing again we are exploit some Lambda function that had the vulnerability and because of over permissive policy we'll be able to go deeper and deeper and deeper and that can escalate to many many services in the cloud to any place to getting an encrypted keys to getting uh basically anything right anything the Lambda can do we can do within our local machine without being noticed again it's hard to there is no access logs there is no such stuff in a serverless environment so it's hard to track all this stuff and we need a way to secure it

okay in this example obviously all aware of the log4j that was introduced last year uh it was a lot of noise around that um so just a quick demo about how it can be we can exploit that cve in a Lambda function we will use the console directly uh invoking with that malicious payload that vulnerable payload and again getting the keys [Music] opening a server and then just invoking our malicious payload chatting the keys and there we are now we can do anything the Lambda can do um so again um the same vulnerability again not the sing vulnerability but another vulnerability that we can use to get the keys and interact with the cloud environment and do whatever we want

okay so we saw like how we can hack things in a serverless environment and what are the the risks there so how can I protect my serverless apps right so I think that awareness is the first step aware of the cloud environment be aware of the challenges that the cloud environment and the serverless applications bring with is an important thing if I'll be aware of the permissions you can be aware of the vulnerabilities that I have I will be able to protect my applications better so first first things first about the least privileged principle you're all aware of it basically give the Lambda as minimum permission as it needs keep the permissions to the minimum

and find a way to Monitor and track the permissions regularly right on a daily basis it's hard as we saw serverless architecture is big but there are ways um to do that and again try to automate the process as much as possible if it's not automated trust me it won't work the second thing is you need to know our own code right it's our own code be aware of the different triggers how can I protect um and validate the inputs from our multiple triggers as we saw API files events cues and so on so we need to be aware of the different triggers and the code and again use the tools and processes to monitor the code whether

it's the process to review the code whether it's the using the tools like Saft or whatever just use the tools and processes to monitor the code and again automate that process or it won't work and the third thing is about the dependencies right we use a lot of dependencies we use a lot of Open Source projects that can introduce vulnerable um dependent vulnerabilities and we need a way to keep our dependencies up to date monitoring tracker vulnerable dependencies whether we are extracting s-bomb from H Lambda function to understand what's going on um anything and there are a lot of SCA tools to use and we need to to do that monitor and track it and again automate

the process so today I want to introduce you with the contrast code Tech contrast code Tech is a free CLI tool we've developed at contrast to help you doing all this stuff that we uh discussed today it's a tool that you can scare and secure your code in minutes we have currently three um commands there one is the Lambda that we will I will show you in a second how it works we have the audit command which is can extract ACA s-bomb and find your vulnerable dependencies and contrast scan which is a static analysis tool to find the vulnerable static vulnerabilities in your code so let's see a quick demo about contrast Lambda you can download the the CLI tool via

npm or Brew to work on any any machine Mac Windows Linux so it's already installed I'm not sure if you can see something We Run The Help command just to see what's going on and what we will do is to run the Lambda command basically what we do is using your credentials your AWS credentials in your local machine to get the configuration and the code we're not uploading the credentials to our backend all right so now I will list all the functions that are available in the environment available to scan

by that we can see which functions we can scan so you see we have a lot of functions they're divided by the runtime so we will pick one function specifically this one is in node.js and with one we will run a scan on it basically within seconds we will get results and we will cover go over them all right we are fetching the configuration and policies for this Lambda function on your local machine right we're sending the the payload to our backend to scan basically we are starting the scan

a few more seconds and we will get the results Gathering the results and now what we will see is basically all the vulnerabilities that we have inside this specific Lambda function

great so we've got all the results within 40 seconds and the first vulnerability we see is uh least privileged violation a medium list privilege regulation that tells us exactly what we need to remove from our policy um to make it again more secure and minimize the risk so we have a wild card for an S3 we have unused dynamodb permission and so on so what we do behind the scenes in analyzing the code and understanding what are the services and actions that Lambda actually use uh in the cloud environment and in this specific use case we saw that this Lambda only needs at least all my buckets permission so we know how to map between action to

affirmation and you can just take this new policy based in the Lambda and minimize the risk the second thing we see is uh a known cve a vulnerable open source package node serialize and and again you can just upgrade to the latest version or to the minimum version to um to remediate this farmability

so a bit about resources and open open source projects so serverless our top 10 some of them we've discussed but as you can look up and see there is a lot of information there are a lot of information about which um each vulnerability and so you can look it up and and find a lot more details there [Music] we have the hours dvsa that then vulnerable serverless application we've discussed uh of course don't use it in production it's damn vulnerable uh so deploy to some test account or something like that and and it's nice to see and interact with this application and find all those vulnerabilities [Music] um we have the contrast appsec Observer serverless this is our blog about all

things serverless and Cloud native you can find there um recent Trends recent threats and a lot of a lot more information about the serverless and serverless security that might be interesting for you and of course contrast scotsak what we just saw you can scan it download it use it for free it can help a lot in terms of security and also productivity right the developer can use it to find which services and action and which permission it actually needs that might be very helpful so I hope you enjoyed the talk and found it beneficial if you have any questions feel free to ask thank you

do we have any questions

hi hi it's going to be a quick one actually is your tool open source uh yes would anyone be able to contribute yes it is open source on GitHub so you can look it up and see like the code and everything good stuff okay thank you

oh can we see your barcode again please the barcode of course thank you thank you you can also find it in the contrast booth there is a sticker with the barcode so you can get it from there

hi I'm my question is I often would like to use stuff like this or like Dependable on GitHub work but I often can't install anything third party that I have to give credits to at work just because of like internal work policies so if that makes sense I wonder if you have any ideas for how to still run these sort of scans or checks in an environment where you can't credentialize third-party apps that easily so you're asking about depend about no sorry so I mean to frame that in relation to contrast I so the bit where you put in their AWS credentials like we use gcp in production and I wouldn't be allowed to put in any credentials into

your application do you mean the contrast contact yeah but I but I wonder if there's any other ways of still being able to use some of the functionality Yeah so basically we're not uploading your secret keys for our backend right so we use you're using your um we're using Code Tech in your local machine we use it in your local which we just upload the configuration and code you can temporarily to analyze it and bring you the results so we're not uploading the keys just the configuration so I understand that but I'm I still wouldn't be allowed to do it at work so I'm just wondering if there's like an intelligent way around this because it

yeah I'll be happy to answer that uh you know in the both I will bring you get you more information about how you can uh benefit from that have any further questions for yeah thanks so much Ken and obviously thanks for mentioning all of us but serverless top 10. uh the question I have obviously everything in your talk including the demo was very AWS and Lambda functions oriented is there anything specific in with Azure functions which you think makes them either more vulnerable or less vulnerable can you give us some examples please so we have Azure it's currently in uh data phase but we see the same thing applied to to Azure as well same vulnerabilities in a different

way of course but the same thing you can have vulnerable dependencies you can have um you have the same list privilege there so it's pretty much the same and our tools will uh we'll we'll work on Azure as well so any further questions so thank you very much thank you very much