Serverless Security Testing: Challenges and Solutions

[Music] [Applause] [Music] thanks a lot for joining me for this talk about serverless application and application security testing we will see what are the challenges that we face when having a building a security testing tool for serverless application so a bit about myself if you don't know me yet i am a developer software engineer i started coding as a young kid 30 years ago done my phd in critical software safety basically helping that code that is a nuclear plant and so on will not explode and kill people and then 15 years ago i moved from software safety to software security i worked a bit on interesting soft on making a ssl library more safe actually giving some mathematical proof

that you can trust some ssl libraries and then continues working on application security tools static static analysis diaste sca and so on and now i am the technical leader of serverless engines at contrast security and this talk is about serverless so i will talk more about that

okay cloud native application is gaining in popularity we see more and more customers that are moving to the cloud and not only moving to the cloud but moving to serverless architecture some of them are already full in the serverless world and some of them have some hybrid some existing application and some new application with serverless but we see always that now everyone is moving to the cloud and particular serverless this is a slide from aws this is the scale we are talking about and now aws has hundreds of thousands of customers that are using serverless applications uh and if you if you take a look at every month lambda execution you have trillions of them so

it is already a major player in the software architecture and it is only the beginning and now we can see more and more attacks in serverless applications some of them are specifically tailored for serverless applications we have seen this denonya a few months ago where some malware was introduced in serverless applications and this is the first time that it is at this scale but again this is only the beginning and also all the well-known vulnerabilities like log fortune is one one example uh it is not specific to serverless but it is also relevant for serverless application serverless is not just a development pattern it is more than that we have seen a huge transformation in the software industry

the way we are developing software starting from the decision making now we don't have managers that are deciding everything but we we developers have a lot more power developers have their say choosing the right architecture choosing the right technology for some patterns and this goes well with with serverless the process are different the life cycles of software is different we want everything to be automated we want to deliver value very fast to the customer and so we have the devops the devsecops trends and so on so this is uh the world of serverless and last but not least the architecture is really different if we are talking about serverless we are talking not only microservices but nanoservices

we we have our application is no more a big monolith but it is splitted in many many a very small components very small services that are communicating to one another so this is an example of a serverless application you can see here the architecture this is a very small application a few dozens of lambda lambda functions here and other services you have api gateway and so on this is a very small application and you you need to understand that customers have hundreds of lambda functions thousands of lambda function in a single application so this is important to remember when we we will talk about security and how we can effectively give a security solution for

others so if you if we zoom in in the lambda function what is the lambda function how it works you have a the lambda function you have your code in the middle the lambda function this is your code but your code is not running in a server you you don't have as in a traditional web application a server that is running listening for a request and so on you have your code somewhere in the cloud and when an event is coming in the cloud provider is creating a temporary ephemeral container that will run your code and this is really new this is a in the event driven architecture the events that are coming through can come from various places it can come

from not only a rest api but also uploading a file to a three bucket changing some field in a database and so on and so forth iot and we have a lot a lot of possible events each event can trigger your code it will create a container and run your code and the container will die a few moments later when your code is running it will interact with many services that are running in the cloud and in order to interact with those uh services it requires some permissions and we will see in more details how the permissions are defined and what are the security implications with this kind of application we are losing the perimeter we we no more have

one big monolith application with a single with a single place where you can put security at the entrance and listen everything and so on you need to secure each and every function in your application because the events from attackers can come from anywhere so you cannot rely on a service that is an internal service uh talking only with your code to be safe you need to secure everything also the inside okay so a few more details about lambda functions yolanda function as i said the cloud provider is creating a temporary container that will run your code this temporary container doesn't really contain interesting things for an actor for an attacker that wants to steal your

data and so on almost all the data is temporary it will be removed very soon all the environment is read-only except for one folder for temporary data you cannot connect through ssh to the container so this is a very particular uh way and for security that's implication for instance if you have some very critical vulnerability like command injection code injection the impact on your serverless application will not be the same as for a traditional web application in a traditional way application you have a code injection this is game over the attacker can do everything on the server but here the attacker will be limited and we will be how but two things are interesting here

in the lambda function in the container you have your code your code is running there and the source code is in the in the lambda function in the temporary container and not only that and also very more interesting thing is that you have the keys the aws keys we have said that the small container is discussing with other services in order to discuss it requires permissions and the permissions are given through the keys basically the lambda function is telling to the cloud provider i have the keys you can check that i have permission i want to talk with the database i want to talk with the s3 bucket and so on so as an attacker i want to see your

card and i also want to get your keys and with your keys i will be able to move in your application to access your database to access your s3 bucket and so on to to to take everyone everything i want so let's take a look at the demo how an attack look like in this kind of application here we will see that we will upload the file and through gonna tie

okay so this is a web application that is a a based on serverless architecture we log in we can see the application the web interface we have some form here we want to send a message to the to the website and okay so it works okay but what we can do here you can see the attach file we can also send the file okay so let's try to to send an image yeah working we can see the connection the request and so on and the the name of the of the file that is sent to the to the server it is just a post request an upload to s3 bucket but it triggers many things in the

in the serverless application so now what we can do here we just we will just rename the file and in the file in the file name we will add some a curl request that will extract the data to our own server so that we will be able to to steal all the environment variables and as i said before in the environment variables you have the keys that give you access to other to other services in the cloud so we send the file and we will see that in the backend something is running and in where nbroc server we get the data we have received our data we just decode it it is a base64 decoding the data

and we can see all the environment variables we'll clean up the data in order to define a profile so that we will be able to use aws cli we can see the token the aws access key the secret access key we have a profile now that is already defined so now that we have the keys we can do anything we want with a standard aws cli we can do ls on the s3 bucket we can see that we have all the s3 buckets we can see the folders inside the three buckets we are using the profile yeah the profile with the stolen aws keys so now we can see all the the files that are in it

and we will be able to download the file also that we are this is the downloading of the file

yeah just a second we'll see name of the file downloading it putting the file in our local thump folder we can take a look of the file and we will see that not only that we can see and access the file but we can also modify it now we are modify it modifying the file locally okay so small edit and now that we have modified the file locally we can also upload a new file in the s3 buckets so we can basically edit anything we want in the street bucket of the customers

okay this is done we can now ls again and we can see the files that we have just uploaded so this is to show you how we can perform an attack on a serverless application okay so now let's take a look at the small example we have here a small lambda function this is the chord small example just taking an event handling this event and you can see inside it is doing some access to a dynamodb database just patching an item putting a new item in the database you can see here in the red box we have the pet item in order to to make it work the developer will write the code try to run it will not work

because the number function doesn't have access to the database in order to have access you need to change the configuration and give the lambda function access to the database now most developers are not security experts they are just looking stack overflow google it my function is not working i have no access to the database how can i make it work and so you will find easily in stack overflow many people many good people that will tell you to put this permission and great it will work but there is a big problem this is a big mistake here the big mistake is the star you are a wild card here and basically this function only need to put a new

item in the database but you are you are giving permission full permission to the database and not only full permission to do any action to delete all the database to read the database and so on but also to all the tables in that database it means that this function only needs a very specific access to this database for one particular table but you get you give access to everything so yes it is working but as an attacker i am very happy if i get your keys now i can do anything i want in the s3 bucket so what you need to do in order to er to make more secure and to block the attacker to limit

the blast radius of the of the attacker you need to give a very specific uh permission to your lambda function only the prototype action and only the relevant table so that even if some attacker get access to your keys to your lambda function you will not be able to do many things just add a new item in the database will not be able to delete will not be able to read sensitive information for the database and so on so now we can see how we can add more security in in the lambda function but remember we don't have only one lambda function we have a lot dozens hundreds thousands of lambda functions in one application

we have seen customers with million uh more than a million of lambda functions so how can you do this process at scale how can you check for each and every lambda function that it has the correct condition it is not over permissioned so as we said we have a scale issue here we have a lot of services and not only that the code is constantly evolving we are very frequent deployment as we said at the beginning with a fast ci cd the devsecops and so on we have everything automated we have a lot of new deployments every every day and so anytime we change the code you need to check that it corresponds to the

permission so it is a lot of work it cannot be done manually if it is not automated it will not happen so we need some automatic way to do that it's difficult to understand what is connected to what in those architecture also we have many developers who cannot really rely on the developers the developers are not security experts and it's hard for them uh to to ensure security for a large large scale application um so let's move on uh so now that we understand a bit more uh what are the challenges uh related to security of serverless application can we just use the existing security tools that we have used until now yes we have a lot

of security testing tools uh how can we leverage those tools in order to get more security so let's take a small example this is a case study from aws a very very small serverless application you have three lambda functions here and a few other services and it is a irobot that is basically sending a registering request to the server not to the server to the serverless application to an api that will trigger a lot of events inside and okay so i want to secure this function uh i can use sca everybody knows sca sca will able to to give you security warnings to to tell you that you are using an outdated vulnerable uh dependency

your third party code may be vulnerable unknown cvs and so on but if you do that that's it's great you need to do that right but it it will only address a very small part of the problem you you will only cover the security vulnerabilities that you are importing with your dependencies and recent research i've shown that it covers only about 10 percent of your code that is actually running even if you don't you import a lot of third-party a lot of open source libraries most of the library are not actually running with your application and if you take a look only as a running code it is only about 10 of the applications so

this is great but this is not enough obviously what about iac infrastructure has code you have tools you have even free tools that are available on the market that are scanning your configuration your terraform your kubernetes and so on able to give security warnings but this will be able to tell you oh you are not using encryption on this particular service but it will not address the whole picture it will not be able to address for instance the list privilege violations that we have seen before and it has zero code coverage it is not scanning your code it is only scanning the configuration the configuration file of your iic so what about sas fast is static analysis it is actually

looking at your code looking at your qr code looking for injection vulnerabilities and so on a lot of possible vulnerabilities in your code but there is a big problem when you are if you have already used some sas tools you probably already know that you have a lot of false positives a lot of false alarms and this is a big problem and even more in the serverless architecture because we have seen that not everything is in the code you don't have one big monolith with everything inside all the connections between the services are inside the configuration not in the code and the sas tool is completely blind to that this the software is only looking

at your as the card of the virtual lambda function it doesn't know the connection between the the services so it will give even more false positive in this case so this is a big problem and also it doesn't address the least privilege violations that you can have so what about ios i asked interactive security testing we hear that it is much more accurate and reliable we have less false positive and so on but if you want to deploy some ios you need to instrument your server right and we are serverless here with a traditional application you instrument your server when you are making your server up once and after that it is instrumenting listening to the traffic and so on

but with a serverless application when you create a new container every time you cannot really do efficiently as an instrumentation for each and every small container that is created every time containers are leaving for only for a very short period of time and time is critical instrumentation takes time this is the problem and there is no actual solution in the market for now doing some miast and serverless so what about dust dynamic security testing dynamics you can do some dynamic testing here with your application basically if you want to use dust you need to find an end point of your application send a request and the dust will craft the payload of the payload to your api

try to inject some malicious data inside some malicious load and and see what is the behavior of your application but there is a big problem here this is a coverage problem because if you take a look at this kind of application the dust will send a request and most likely you will only get a response 200 okay i have received the request i acknowledge that i have resized the request but most of the logic of the application is inside in other services because it is an even driven architecture so the dust will only only able to discover what is really very close to to your rest api and everything that is inside the dust will be completely blind to

to what is occurring in the background of your application so this is only superficial and it will not give you satisfying coverage so we need something else right so let's try to imagine what we can get as a perfect security solution for serverless okay so we have here our traditional cicg for a modern application it's a serverless application and you have a developer you have an application and upset people here what kind of solution do you want to ensure security of your application you need something that will cover everything all the stages of your cicd pipeline you want your codes to be scanned you want to configuration to be scanned you want it to be accurate

you want it to be friendly to the developers very easy to use not yet a lot of false positive and so on so this is the dream that the dream security testing tool that you that you can think of uh going into more details let's say that we want something that will install automatically maybe one line of code maybe one three clicks only in the in the ui something that will go inside your cloud and be able to do everything so what is everything the first thing is to discover what are the assets that you have in the cloud what are your resources what are your lambda functions your databases your s3 bucket all the

services that are deployed in your cloud if you are talking to customers or people that are using serverless in a more general way cloud application it is very hard to get visibility on what you really have in your cloud many many times we just run this discovery and the customers are surprised to see oh i don't know this function from where it is it is coming maybe some developers that has done some experiments and a lambda function forget about it and now it is in the cloud any vulnerability here you can get access to an attacker that will be able to enter your application and so on so this visibility this discovery stage is very very important

the second thing is that once uh you already have a map of everything that is inside your cloud you want it to be scanned and analyzed you want to discover all the vulnerabilities that are inside your code also the vulnerabilities that you are importing in the third-party libraries you want also to unders the tool you want the tool to understand the flow in this in this serverless application because remember you have a lot of micro services talking to one another so even if you know what is in your cloud it is not easy to understand what are the possible flow and who is talking to to who and what is connected to what so this is something that you want to be

analyzed and then you will get some kind of list of vulnerabilities that you need to address and when we have talked about sas for example when you get a lot of pulse positive here you want to to have some more confidence that the results that you get from the tool are really real results and you don't want to lose your time investigating what is the problem with a result that is only a mistake of the tomb so one additional step that you would like to have here is some kind of simulation if you have something that is inside your cloud security solution inside your cloud would be able to when we have discovered some possible

vulnerability we have discovered what are the possible payload that can trigger this visibility we want to test it we want to simulate it we want to send this payload we want to track what is the behavior of your application and to check if it is really vulnerable or not so because the solution is in the in the cloud inside your environment it is able to simulate and also to track the behavior of your application for instance if it is sending some payload in the lambda function and you expect a new file to be created on the s3 bucket after it is running you can check in the s3 bucket if the file exists and so you can check that uh the exploit

of your vulnerability really works and so this reduced dramatically the number of false positives and then after that you need obviously some reporting in your uh in the environment that you are using as a developer you want to get a slack message you want to get a pr request in your in your github and so on maybe a mail maybe something else whatever you are using you want it to be as actionable as possible as easy to use so this is a let's say the dream solution and this is exactly what we are building at contrast security this is what my team are working on so that is to to get a some example of the kind of result that

you can get from this tool for instance here we have said privacy that we have a problem giving too much permission to a lambda function so the kind of results we get from this tool is exactly a an updated policy that could you can just copy paste into your into your cloud that will replace the more the two general permission to a very specific one that will help you to reduce the severity of any attack in your application the second kind of things that you can get so let's say here you have a common injection detecting in your code this is not like a sas result here there is this result says that you have

a common injection we give you the exact payload that will trigger this common injection we are verified that this vulnerability this exploit is really working and if you want to reproduce it by yourself you can just take copy paste the payload send an event by yourself and you will see that the exploit is really working so obviously this is not a false positive so if you are interested and want to know more about serverless security we have published os top 10 similar to the general purpose of us top 10 but there is a no serverless top 10 that is more focused on serverless security risk so you are invited to to take a look and if you are using

serverless application and you have some data that you want to share you can send anonymous data so that the next versions of the top 10 will be even more accurate i will not enter into all the details here but this is the top 10 that you can check out online and as you can see this is not only about other privilege function but we have a lot of vulnerabilities that are existing in the serverless application and we need to address everything another open source that you may want to check out is a dvsa this is a this is a vulnerable application that we have developed intentionally vulnerable in order to to learn in order to test the security tools

that are really discovering the vulnerabilities that are inside so you can take a look you can learn from it please don't put this in production this is vulnerable it will only give you risk in your environment and so that's it any questions first of all thanks this is very aws specific can you say how it's different in gcp or azure

okay yeah sure a very good question yes around all this talk i'll talk about aws lambda functions this is the most popular serverless vendor for now but also you have azure you are gcp and you have other cloud provider and basically the same is applies also to other vendor and also in our solution we are starting to add the um support now we are adding support for azure and it will come also for other security cloud provider vandal any more questions raise a van no yeah okay hello if i understand correctly all of most of the exploits are based on rce with the other privileges it depends on rce that allows you to execute code or

retrieve the environment variables to get their credentials and do you know of exploits where without rc you can access other resources for example sql injection stuff that but something that is common in lambda and a serverless environment yeah thank you for the question so yes as the demo i have seen i have shown here was mostly about some kind of common injection stealing the aws keys but obviously you can do many things without the keys the keys give you the abilities to to move from service to service in your serverless application but if you if your number function is talking with an sql database and you have an secure injection this is also relevant for serverless applications this is not

specific to serverless application but yes also relevant thank you any more questions may you thank you very much [Music] [Applause]