BG - Serverless Infections: Malware Just Found a New Home - Erez Yalon

so hi everyone thank you for staying so late in this day some of you may be expecting mati seaman to give this talk it was kind of a last-minute change but it's the same talk so you're in the right room although it's a different speaker so today we're going to talk about serverless summer lessons infections I'm the head of absently search in check marks I've been there for the last four years these are probably the best ways to contact me if needed and if you have questions or anything else that you would like I'll be happy to answer also this is usually the place where people need to justify what gives them the right to stand here

and give you the talk so my justification is that I see a lot of elbows being developed in the industry by developers and now when serverless is getting more and more let's say out there the mistakes also we see them more often than we used to the problem is that a lot of people think that serverless gives them let's say the right to right vulnerable code and it makes it immune to two usual attacks so this is kind of trying to show differently so let's start the agenda here is to give a quick intro to serverless architecture I'm hoping some of you already know what it is we'll talk specifically about Amazon's AWS lambda we'll discuss a few demos that we

made pocs that actually managed to show that from a simple code injection we managed to create a viral infection we'll talk more about the security aspects of the lambdas and hopefully get to the key takeaways in the so let's start with some legal stuff I understand that well in the last couple of months or here it is illegal to talk about cloud and about serverless technologies without showing this picture so here it is it's very important that everyone acknowledged that they saw it now we can put it behind us and continue so what is this serverless thing let's talk in terms of evolution not that long ago this is at least oh I'm telling myself we had hard

well as a unit if we wanted to scale up we had to bring more hardware more servers put them in actual rooms we call them data centers this was back then in the past and we had to to make sure everything is taken care of hosting environment etc after that came the infrastructure as a service when we didn't really had to buy Hardware anymore a good example is Amazon AWS for example the operating system was the unit of scale if I wanted something just bring another OS so it obstructed the hardware the the next phase in the evolution was the platform as a service well the application was the unit of scale if you want an example salesforce

have iroko I don't know if people know it but it's a very good example of that also you don't need to know what the OS is you don't even kill you just deploy your application and you run and now it's the edge of several lists several lists means that we're doing function as a service the function is the unit you don't care at all about the of course the hardware the OS you sometimes don't even care about the language it runs on so it's all about the functions and a nicer way to show it Oh a different direction is talking about time and time lines so while we had data centers it took months to

deploy new servers and the application lives lived years on them it slowly minimized during the virtualized and cloud revolution and then the container deployment it got down to to deployment by the second sometimes application ran for a couple of seconds sometimes hours but not needed more than that and now in the last stage of the evolution with server list we just deploy in milliseconds we have not applications but actual functions that leave for between milliseconds and seconds and this is all we need the magic looks like this all you need to do is to develop a function you don't care about anything else you push it or deploy it to the cloud waiting for a trigger of some sort we

will talk about these triggers in a second when the trigger comes the function is doing what it's supposed to do and then it vanishes it cleans it dies and that's it no persistency nothing stays after that so the benefits other than being able to say that you're using server list which is a great buzz word you can say your scalable that's also a buzz word so you have to but it's really easy scalable I mean you have your functions on the cloud and then from that second you don't even care if you need to run and trigger the function let's say once a day or thousands of times a second it's the same it's transparent for you you

just run it the scalability is obviously a very good plus so that no ops you don't care anymore as we said before how do l memory hard drives patches updates all these operations these really lowers of the headaches of the operations and low cost I don't really have the numbers to back that up but I hear it from all around the industry people are saying that the model of paying for only what you're using and being able to throttle your use for down times instead of really busy times they really managed to lower the costs some of them significantly so this is definitely a benefit why why then people don't all of them why not

all companies move to server less obviously there are downsides one of the big downsides is the it's it's a new paradigm it starts from developers that need to actually learn how to code differently it goes through the designers and architects that need to actually think about new ways new structures of using their code and organizations that sometimes are a bit let's say have problems adopting new and new ways also it is a bit slower instead of running something local as we know you will need to deploy copy run it on a on a container somewhere what the clouds usually do is if there is a function that is used constantly it is being cached but still there always is some

sort of an overhead for this there is problem monitoring when you have everything running on one server it's really easy to monitor everything but now when you have a function as a service you don't even know what it runs I'm not even talking about servers you have absolutely no idea companies are trying to start thinking about how to cope with it but it's supposed to be a big downsize especially for management and the last thing and people say that makes them not go serverless is the platform dependent there is no unity so if you actually start deploying on Amazon you'll have really hard time moving to other platforms by Google or Microsoft or anything else so if you man if we

mentioned the companies probably most of you know all the commercial solutions there is the the Google cloud platform and Amazon AWS lambda Microsoft L they have their own and they also IBM there are also a bunch of open source solutions everyone has their own pros and cons but this technology of function as a service is probably here to stay for a while because a lot of companies really put a lot of effort to make it work seamlessly ok so we wanted to start and do some hacking and check how safe are these platforms we made the small comparison and decided to go with AWS lambda it seems the most common and mature they had an interesting wipe the

language coverage and this is where we put most of our effort in the demo so let's discuss that in a nutshell so the AWS lambda functions service it was introduced in 2014 they support a lot of languages nodejs Python Java and we got very excited because they added go in the beginning of the year which is nice donate call everything runs on Linux containers and everything is fun so if you want to trigger a function on the AWS lambda service there are two ways to do that one way is the explicit one just to use the REST API to call a lambda function and the other one is use any other trigger from the plethora of what Amazon

gives us we can see all the services here but you can use it as a trigger that s the update or pushing an IOT button CLI tasks a lot of people used the code pipeline for DevOps so it's really it's really nice to have all these triggers wired into the into the functions the common uses that we see for the for the service architecture are mainly mobile functionalities back-end also IOT back-end we see a lot of boats and chat BOTS using that which makes a lot of sense especially in the scalability part scheduled tasks everything that is high CPU utilization on-demand final imaging processing everything that is distributed computing fuzzers machine learning if you need to crack a password

this is probably the right way and alexis skills that use this as back-end so cellulous is great right what else could you ask for so about security for example as I told you a lot of developers and we see it now also in people who are doing auditing we show them the problems in the code and they say this is not a problem because this is server list this is a function that goes to the cloud and when we ask why why is it safe this is the explanations we get and it's true in a way so the lambda security adventure the advantages are mainly that it is running in its own environment okay when running

in the environment you have the potential damage is very limited what we like to call sometimes a sandboxed so here in a way as well it is sandboxed the second thing is that the environment is disposed after run this means that there is no stored no persistent damage even if there is some sort of micro damage it will be vanished after 60 100 200 milliseconds a lot of people think that combining these two together make every vulnerability obsolete so challenge accepted we decided to check this this way of using known vulnerabilities just because it's serverless and we decided to try and exploit a simple code injection in lambda function that everything everyone thinks that it will it cannot do

anything serious because as I said it's sandboxed and it's it's going to live for 60 milliseconds we decided to create something that will be called persistent we cannot really run something outside of code but we wanted to change the code to have the code itself all the vulnerability and make it even viral if possible the link below by rich Jones and you should read it probably it has a lot of other challenges up for grabs in the service architecture so let me explain our playground before we start the demos it's really easy it obviously runs on AWS lambda you have a very small app with three functions one function is the main that is parsing

input from the URL and calling either cars or users if any of you is a UI person I'm sorry so this is how it looks like I said I'm sorry you can see that in the URL part we call evil cows or users this is the part of the main function this is the part of the cars or users and the number here is just the number of a user this is the simple yet here's example if users which is user number 1 and this is how it looks like so this is the playground let's see how it works ok changing user to user - it is user - changing cars okay so let's discuss code injection what is

code injection usually I'm taking definitions from OS because they're doing really good jobs in definitions so quality injections code injection is an injection of code that is then interpreted or executed by the application as simple as that and this is what we wanted to put in our code so we made sure that our code is vulnerable to code injection anyone can see it no so this is the code the code of main just parses the URL takes the resource and the number concatenates it together and then uses eval most of you should cringe right now and it makes perfect sense most of you also know that this is extremely common out this is something we see every day this is this is what

what's happening outside as we speak evals everywhere so it's okay you cringe but this is this is life so this is our code ejection it's on purpose now let's start playing so we have a very very short-lived instance when we run it we wanted to see if there's anything interesting we can get out of this extremely short and again extremely short means milliseconds extremely short instance so we went to documentation and documentation said yes there is something interesting there is a parameter called lambda test route it tells you in the container what is the path the exact path of where your function seed the actual source and that's nice I mean I guess it's useful

so we wanted to know what path is it and we use the code injection you can see on top is the decoded URL and in the bottom the encoded one so we used it as a payload to get the process and to get out of it the lumber lambda passport so let's see what we got

okay so that was a simple we managed to get the path actually it wasn't that surprising because this is the default path when you're deploying new new functions but still it's it's good to validate that this actually works so this got us thinking we can get the path what if we want to get the actual code the code may be interesting we want to get the entire code of all the functions that are in this path so this is actually exfiltration of the code although as I said it's very short-lived when it runs also you get you're exposed only to one specific function usually maybe we can use it to get all the code out of this path so we have our

container we know that under the path of our task we have our three three functions what if we could use our own container or a hacker container to somehow copy it for future reference let's see and actually it worked pretty well so we managed to Tao the entire thing and send it to our so this is the container of the hacker you saw it's empty we're running the payload at the moment successfully now we're flashing the hacker container and here it is mayn't are everything just all the code you can get the index and this is the actual source code that you can get you can see the the first iea that we managed to get

out of it because it exposes the actual API key of the AWS you can lower your cameras we already changed it I saw that good thinking we already changed it but yeah that was kind of too easy but that's nice so you can do that what else can be done this was fun but it's not enough so we went back to documentation and we saw that AWS lamda really wants to help you so including the functions that you created and deployed you have the full spectrum of AWS SDK and AWS CLI which is excellent this is really excellent you have the ability to list functions and to deploy new ones to update existing and in everything can be invoked by code

this is really really powerful so this I'm sure I'm hearing your voice in your head now saying I'm sure it needs authorization authentication anything Asian no it doesn't actually it kind of needs authentication but while you run your own function you already got it so running your own function vulnerable function you can basically run anything you want so this is fun so we decided to change some source code after we got it let's update all the lambda functions that we can find in a specific path so iterating between all the functions well in our example who have only only the three functions in real life it can be thousands it can be even functions of

other applications that are being used so you can change a lot of things in this way so iterating and using update function code that is on our malicious container just bring it in bringing it in and updating code let's see it happen so this is our clean house painting it successfully refreshing and now after the refreshing both cars and main have this code although we only attack cars so obviously users is also painted at the moment so what we actually did here we can see the code we can see where you saw it in the demo and this is just what we added just the string extremely smart Hecht by string so this is pretty great

because we actually managed to get persistent infection although the function will run and then it will die if I run the same function later tomorrow in a month it will still be infected but let's say it's not enough because it's only a one-time infection if someone is going to clean the function or update it or something it will stay clean so not good enough here we can see that we added a reset function the reset function cleans cows restored and now we can see that cows is clean while mane is still tainted so we decided to push push even further the persistency and try to do some course contamination between them so actually this this jump is kind of logic and and

simple it's in fact in function makes sure that all other functions are still infected if it's not in fact the again this is actually an actual viral infection so we create cross-contamination to make the idea simpler if we attack users and run it excuse us attack mainland cards if we clean meinen users house stays infected and when it runs it will infect main end users and the demo again

so infected users successfully now we can see that nothing happened to car and name but now if we call users Karl is infected if you call cars then we see that now both cars and main is infected resetting only name

and users

and leaving cars infected is only good until someone calls cars when cars is cold

we have again cross-contamination you can see it so it was kind of great because we actually managed to to create this persistent code contamination we wanted in an environment that is by definition non persistent and this was really nice actually did some more plays and managed to do some heavy-duty remote code execution but obviously you can understand that if this was possible probably we can escalate it to things that are much more interesting than cars and users so a few more words about AWS lambda security um I want to talk about two important terms in it there is the execution roll and vbc execution roll is actually what the function is allowed to do while V PC virtual private cloud it

means which virtual private cloud it belongs to which means who where can it send the information from from inside this vbc so let's talk about execution roll first we can look about we can look at a Claudia Jas I actually did not know it well before but they had a chance to play with it it's really nice it helps to deploy in ojs projects to 2 lambda it easily and if you go to the community to the documentation you can see that Claudia asks you for full access you can see the AWS lambda full access is required for Claudia deployment it happens all over the industry in a lot of places including Amazon own examples

in github so this is from Amazon AWS server lists crud sample this is the boilerplate for many many many projects all these projects use the same code you can see that it's actually a prerequisite create a am role called lambda role and assign it AWS lambda full access so by default and by recommendation of Amazon please create your function as vulnerable as possible actually please let's talk about the VP CVP C's virtual private cloud by default it starts with no DPC at all the thing is that it opens outbound traffic just to remind you the first demo we had was exfiltrating the source code this would not be possible if VPN was separately so the default is no VPC we talked to

amazon about it they said it's fine we showed them the demos we explained everything they still believe it's fine and the only thing that they thought they should change is the example in github which they didn't do yet I'm actually checking every now and then it's been months now since we talked to them it did not change it so this is Amazon let's talk about wow this is really bad colors ok so I'll walk you through it so we're comparing lambda to Google cloud into Azure and talking about access code via code injection it's the same with all three of them downloading close to a hacker server AWS a lambda we manage because there was no

VPC in Google there is some sort of firewall block but it's just using ports it doesn't really care about IP which we find a bit weird but that's probably a reason for that and Microsoft again there is that the default is is really open but it can be blocked by network rules modify the function code so we saw that we managed to do it in AWS lambda Google they are not so keen to give you all the SDK by default so no in Microsoft it's it's complicated it depends on settings we didn't go very deep modification of code and we talked about permissions so absolutely no problem with AWS lambda not possible in Google yet and again setting stuff in

Microsoft and then our viral injection we already managed to do it in AWS lambda walking on the rest so maybe soon I'll keep you posted okay so approaching the end key takeaways and actions yes it is serverless but there are at least two servers that can be attacked in a server less architecture one of them is hosting the container the second is hosting the source code container should not contain unnecessary sensitive data I know it sounds really obvious it should not DPC must be defined if you take the default DPC off it's a problem so and there's no reason for it to be off so there's just no actual use case that that it's needed to

be off and the container is already authenticated as we said so you need to know that if the hacker has access to some sort of owner ability in your function he owns everything he has all the authentication needs he needs nothing else and obviously the AWS SDK is really a nice gift for that hacker so again useless pure bleed roll this is right for everything not only server less the AWS LOM the basic execution roll needs to be fine-grained and not be all-inclusive with the top permissions and it's always good to check the quality of the code eventually especially in service computing we started by people who are not checking the code because it's serverless now we can tell them

especially because it's serverless you need to make sure you have no vulnerabilities and that's it any question just a reminder if you have any questions if you can raise your hand really high so I can bring the mic really high so you can see because it's hard to say over people also if someone does not like to ask questions in a big room full of people you can contact me I promise to answer ok so I have a two parter if you'll allow it the first question is obviously the SDK if you get access that through an eval that's crazy you can potentially do anything but do you need an eval is that pretty much

prerequisite for any type of viral infection or what we what we try to show here is the reason you need to actually treat vulnerabilities in code of course we took the most simple one just as a POC it's really important to understand that if it can be used outside of the scope of functions it can be used in a function on a surveillance as well this was the point that we tried to pull so the evil was planted on purpose if you have any other vulnerability in the code we probably find a way to use it so the evil was an example oh and so the other the second part here is if you extend

the lifetime of you know if you add code to an eval or something right you got something that's processing if you're increasing the length of time of the average you know lambda operation is that going to be cut by lambda naturally is it gonna just say oh I'm alive for too long I'm going to terminate okay so this returns me to the reason why some organizations are not moving to server list and this is monitoring it's really hard at the moment to monitor all your activity it happens in many places at the same time different containers sometimes different places on the globe it's really hard to monitor I know Amazon is working on it so I would like

to say that there is a possibility to monitor that but I'm not not 100 cent you also also I think you have a bit of a problem to know in advance how long you want this specific function to work for you unless you have some sort of clue in general terms make sense okay does anyone else have a question

so the the the SDK provides a way to access the source code or what is that is that like intended part of it the SDK is something that Amazon gives you it's a bunch it's like a toolbox it's a toolbox of functions that enables you to work with you with all your functions sometimes it's only to list them sometimes it's to update them to deploy them change the code even delete so this is something that Amazon lets you do and hoping that you're doing it with the owners hat obviously

yeah just real quick question to check my understanding so the permission issue where it's they're being granted too many permissions for the for the function would that if you had the lower permissions the correct permissions that would restrict being able to modify other functions but it wouldn't restrict the exultation right exactly actually there is a very nice fine grain level that you can get to you can restrict what you can change if you can touch other functions when you can do it under which terms specific methods or specific functions in these terms that can do specific things obviously the easiest way is to say okay everyone can do everything and no debugging you need is from now on right

great thanks hi and your example you showed that the you know the function has access to an environmental variable that shared with the source code was being read from I was hoping you could elaborate on maybe some of the other variables that you might have access to in the environment the other variables are not something that we went for but other variables for example on which continent the container is what time was the last time that the specific function was deployed more like metadata let's say I think that the past one was the most interesting in our our eyes anyone else have a question one last call all right thank you so much [Applause]