Zero-Trusting Serverless Applications

[Music] good afternoon everybody uh my name is tripty and today we will be talking about serverless applications and how we can secure them using microservices secure design what are the things that we cannot uh secure using these design patterns and for that uh what do we need to do and before i begin and tell you guys about myself i would like to express gratitude to the entire crew of seattle b-sides volunteers and organizers for giving this awesome opportunity in the middle of the pandemic so thank you very much guys now little bit about me uh as i said my name is tripty currently i'm working as a principal application security engineer at elimio it's been 13 years but i started my

career as a mobile game developer and after doing game development for a couple of years i realized my passion is a security when i'm not doing security i like to travel and paint and about four or five years back i also got my yoga teacher's training certificate so at illimeo i not only uh do security i also teach weekly meditation classes and breathing exercises after this talk if you want to get in touch with me this is my contact information so this is today's agenda in next 1 hour i will be covering what serverless applications are what what do you mean by serverless in the context of microservices how they are evolved and then how we can analyze some

of the microservices design pattern from security perspective and towards the end i will be talking about some of the best practices for securing serverless applications and microservices all right so let's begin let's start with evolution and then try to understand uh what are the security pain points and how the overall role of security has changed with the evolution of microservices serverless awesome so as you can see on the left hand side there is a monolith application if you remember around 15 to 20 years back the world was full of monolith giant applications which contain a typical ui layer then a big fat business logic layer which was tightly coupled with the back end data layer and then

around 10-12 years back we saw the industry is moving more towards loosely coupled service oriented architecture but that was not sufficient and that's exactly when we started seeing rise in microservices so if you look at microservices these are nothing but more loosely coupled service oriented architecture and lately in last five six years we are seeing rise in serverless applications some of the serverless applications are aws lambda google functions and we will be talking a lot about them today but before we jump there i just want to put things into perspective um some of the uh applications which were built monolith it takes a lot of time to break them down into a micro service oriented architecture

for example netflix original application was a monolith application and it took netflix two years to break it down split into a nice micro service oriented architecture and then uh it took them seven years to complete that transition from the giant monolith to microservices so as you can imagine this is not something easy what this transition really meant to netflix was they were a traditional development company with 100 engineers producing a monolithic dvd rental application and when they uh changed their architecture to micro service oriented they became multi they divided themselves among multiple small teams responsible for end-to-end developments of hundreds of microservices that work together to stream the digital entertainment uh to millions of netflix customers

that we are enjoying pretty much every day in this pandemic so when it comes to serverless serverless is actually a deployment type of microservices only now the big question is are serverless application going to replace service oriented architecture microservice architecture well the answer is serverless applications have some serious limitation so they definitely cannot replace what we will see in the industry is serverless and microservices will exist together now let's dive deep into what serverless applications really are as the name suggests in serverless we don't have to manage the servers management of servers patches and whatnot and other operational duties are performed by the cloud provider serverless applications are like these extracted service which are even driven they are

short-lived and stateless the beauty of serverless application is they can scale up on demand or scale down up on demand and from business perspective the best benefit is you pay as you use and that's exactly what serverless application that's what exactly serverless applications are now let's take a a deeper look into different attributes from modularity perspective we have seen that monoliths are very tightly coupled whereas microservices are loosely coupled just like serverless application when it comes to inter process communication monolith applications uses heavyweight protocols like so but microservices use lightweight protocol like rest or grpc serverless we often see the user rest api when it comes to handling data we have seen with monolith applications

we uh most of these applications use global data model or shared databases whereas microservices they use data models smaller data models or database per services and in case of serverless we all know the data is f armor and we are going to talk a little bit more about it after a few slides what are the threads associated with the data handled by serverless now let's talk about deployment model imagine what it takes to deploy a giant monolith application lots and lots of time therefore the overall speed of development uh sorry deployment is slow whereas microservices these are small independent services so the speed of deployment is quite fast and when it comes to serverless because

majority of the operational tasks are taken care by cloud service provider the deployment speed is even more faster now let's take a look at some of the micro services pain point and i'm going to play a short video here still not understanding this why is it so hard to display the birthday date on the settings page why can't we get this done this quarter look i'm sorry we've been over this it's the design of our back end first people call the bingo service see bingo knows everyone's name though so we get the user's id out of there and for bingo we can call papaya and mbs to get that user id and turn it into a user session

token we can validate those with old knob and then once we have that we can finally pull the user's info down from raccoon yeah but couldn't the raccoon team basically just no raccoon isn't guaranteed to have that info before we do this we have to go to a query to see if the user is willing to take it to the next level or they're just playing the field now wingman is cool but he doesn't store any user info himself he has to reach out to other user info provider services like rgs barbie doll ringo 2 pll but how does it know what all the user provider services are well for that it has to go to go active

all knowing user service provider activator galactus has a mission knowledge of all current user input providers it doesn't have future site or knowledge of past user input providers so it expects a time limit to get all the current user input providers we need to pass a time range of the current time and a time representing the end of the universe we get from eks our entry chaos service vks is being deprecated at the end of the month for ominous megastar still doesn't support iso timestamps like they said they would a month ago so until omega star gets their together we're blocked we can't get signed up for our use case we can't use eks there's nothing we can

do so galactus won't be able to find our new birthday boy provider which means wingman won't know how to talk to anybody which means i will be able to find true love all right so that was a conversation between a developer who builds microservices for living and a product manager who wants him to do certain changes and we can totally see how common will not understand it how complicated it was uh for him to explain the dependency layers and how to introduce a small change and i hope the product managers got it this particular video is uh provided by shazam and it's available on youtube and i highly recommend go check it out now just the way the developer has these

pain points around complexity security people face exact same pain points micro service oriented architecture which is a mix of different micro services and serverless application it's highly complex if you imagine a monolith application because of tight coupling of different components and sub components there was this implicit trust and we humans are habituated we try to uh you know repeat the patterns in case of microservice oriented architecture this implicit trust is very dangerous it gets replaced by explicit trust and every time checking these micro services before they talk to each other for explicit trust can be quite complex next thing is in case of monolith there is a static order of api and if you look at this picture on

the right hand side now that has changed into dynamic order of api which is also quite tedious when we have typical monolith application we know the attack surface what is getting exposed what is not getting exposed but in case of serverless where the cloud provider have lot of leverage sometimes the attack surface is quite unknown we have to pay a special attention to define that attack surface most importantly if you look at the traditional application security uh tooling or application security review mechanisms that tooling is really not adequate for something that gets deployed like this with a level of complexity therefore the traditional application security always doesn't work uh and most importantly the speed of

deployment with the help of serverless application infrastructure the cloud service providers empowers developer with so much speed and easiness to deploy their innovation that if we go a traditional route it becomes difficult for absent person to do his job and as a result traditional application security reviews are not possible so those were some of the pain points now let's look at uh some of the implementation specific pain points the funny thing is oas vulnerabilities which were published 20 years back are still applicable to micro service oriented applications and serverless applications these are pretty much exact same vulnerabilities what this means is when the cloud service provider offers you the ability to write code fast and deploy fast

they don't necessarily offer you the ways to take care of these vulnerabilities by default so as a developer you have to pay attention towards all these vulnerabilities and we are going to talk some of uh we are going to talk some of these vulnerabilities uh shortly in detail let's take a look at uh aws lambda so those who know aws lambda is a serverless function basically you write the code you zip it and hand it over to the lambda we call that as a function and then we configure it to integrate with number of events for example let's say a record change in dynamodb then do this now when you're dealing with such serverless functions

all the debugging must be done through logging there are not additional interfaces available to troubleshoot uh this type of functionality and because of that the testing is very limited how can you use traditional tooling to test something that is so short-lived if we are not careful about wiping the data securely after the function is uh executed then there is a possibility of temporary data leakage and most important thing is aws lambda security is enforced using im policies and these im policies are not available to be secure by default so if somebody makes a mistake lets say use wildcard then there is a chance that the overall security of aws lambda can be compromised so we have to be extra careful just like

any software even aws lambda had security vulnerabilities it was introduced in 2014 and so far three vulnerabilities have published their most recent vulnerability was around code injection in their cli tool called aws lambda uh where a simple uh bound check was missing and as a result attacker can execute arbitrary os command so the point is we can't rely on the security provided by cloud service provider we have to make sure every aspect of the serverless application or function that we are using is secure now let's ask a question why is that and what's our role in this security as a developer or appsec engineer so if you carefully look at this um shared security model

uh provided by any cloud service provider you will understand how the role of security has changed even in case of serverless functionality the cloud service provider is responsible for taking care of operating system virtual machines storage database and security of the infrastructure part they are responsible for deploying the operating system patches but the complete responsibility of the core that we write the data that uh the application handles either at rest or in transit uh the identity and access management plus various cloud service configurations needed for your application to work properly everything is your responsibility and this is a big eye opener this takes away the myth from us just because something is hosted by aws

or google cloud provider or any other cloud service provider that means that is secure and waited no you know as a developer we need to take full responsibility of security now let's see how we can do that in this section i'm going to talk about uh some of the micro service design patterns so design patterns are basically uh reusable autonomous service that we can use it again and again so that nothing becomes hindrance to our innovation these design patterns uh basically allows us to accelerate application releases so i have often seen my developers coming to us with these templates and security team reviews those templates and make sure these templates are nicely wetted and in this way they don't need a

security team's review every single time uh they deploy these micro services so that's the beauty of using design patterns as such there are a lot of design patterns out there just on the screen you can see 26 design pattern obviously i don't have time to go through each one of them so i have selected few that i find are very interesting from security point of view you will see in these design patterns some of the security features are very well thought through they come as out of box and that makes it very easy for developer to configure them all right let's take a look at the first design pattern api gateway this is my the most favorite design pattern

now imagine your aws lambda service or google function is part of service a or service b in order to protect them just take a look at how many security functionalities api gateway provides out of box when we use api gateway we can uh leverage it for authentication api authorization it also provides capabilities to prevent a denial of service attacks it performs routing of requests uh it has mechanisms to prevent layer 7 or dos attacks by providing us capabilities to define throttling limits api rate limits basically it does the overall load balancing for us it makes sure all the end points are available on https which is a big big big plus point moreover uh you can

do security and resiliency monitoring using your api gateway so what i really like about this is there are tons and tons of good security functionality are inbuilt and if we enable them our applications are secure up to an extent but again this is not a silver bullet just because it provides all this nice wonderful security functionality that doesn't mean it is not vulnerable to attack so there are some attacks uh that api gateway cannot provide so this is something uh the developer of serverless applications has to keep in mind some of those vulnerable attacks are let's say layer 7 ddos attack with counterfeit request or ddos with cascading requests now cascading requests are basically simple requests

which when reached to the back end they generate many requests and then it becomes difficult for back-end microservices to manage it and as a result the service experiences denial of service next one uh is you know if there are any implementation specific login or identity attack let's say related to jwf token then that is not something api gateway can take care it's a design pattern so it obviously has some security features but there are definitely some attacks it cannot handle for example cross-site scripting and sql injection so it is the responsibility of the developer to ensure that these implementation specific attacks are taken care of level so let's take a look at one of the serverless

function here let's say we have a serverless function which is responsible for performing login service it accepts username password and returns a jwp token awesome right uh you will see these type of serverless functions are used for you know authentication of apis or performing authorization for each request which is basically needed between service to service communication now if uh we get certain things right for example uh in a typical jwt web token which consists of a header a cipher specification where we specify the crypto algorithm then a payload and then signature of the payload if we get the cipher uh specification right if i set it to none then basically all the cryptographic security which is

provided by the jwd token can be bypassed then the jwt specification provides a field called jpi if this field is disabled by default then our jwt implementation and the resulting serverless function is subject to every player tag and when you're not protecting your jwt token properly using right crypto or replay uh attack prevention then information leakage can possible so these are some small things which are extremely important and the developer have to keep these things in mind for secure implementation of your serverless application now let's move on to a next design pattern uh a circuit breaker design pattern basically the software design pattern which is used in many micro service based software development it works the same

way as how an electric circuit breaker works when multiple service communicate with each other to serve the number of requests there are many possibilities that the services are unavailable and are not responding for example this is a health check app and it is connected to redis cluster and dynamodb and some external apis let's say the redis cluster is down now what will happen is this particular microservice will inform a circuit breaker after a certain threshold is crossed and as a result the circuit will break the circuit breaker will communicate with the external applications which relies on smooth connectivity and communication with these internal services and as a result a failure is very gracefully communicated with all the intended parties

and this whole thing makes uh your application a design very resilient when the service is back and it reaches certain uh threshold circuit breaker checks it again every few uh you know seconds and once it's working fine uh the whole thing works again an appropriate message is communicated with the external application so this particular um pattern basically introduces resiliency in your architecture and it is very important because you know failures do happen and it's the ability of the service to bounce back gracefully from failure which is very important here now uh usually circuit breakers are fine in api gateways now they are not bulletproof what if somebody attacks the circuit breaker itself right you need to keep in mind how can

uh we prevent attack on circuit breaker itself from resiliency perspective so these are some interesting point that the security team members and developers need to talk out when they're designing serverless applications with microservices now let's move on to the next uh slide uh service mesh service mesh is actually uh infrastructure deployment of micro services in a typical service mesh uh proxies these blue color boxes you see yeah these proxies are injected into each service which is deployment deployed as a side car now instead of calling services directly over the network services then call their local site cars and these local sidecar proxies manage entire communication the reason i like this particular microservice design because um it uses

mutual tls to ensure the inter-service communication is completely encrypted it also provides mechanisms to perform authentication and authorization of all the services it provides access control policies to even enforce fine grain access control so it provides lot of security features out of box now again this is also not bulletproof because of such complex infrastructure and design this particular design pattern is prone to lot of specific security misconfiguration so we have to watch out for those all right let's move on to log aggregator and design patterns we all know that logs are extremely important from auditing and uh investigation perspective anytime there is uh you know failure or security incident uh we need the logs to do the

investigation so whenever you are building microservices serverless function always make sure you have this particular design pattern included in as part of your core applications this particular design pattern allows collection of logs which enables us to do real-time monitoring of enormous patterns and events you can also leverage this design pattern to perform automated login now just like other design patterns that we discuss this one is also not a bullet proof there are some limitations it is the responsibility of the architect to make sure correlation of logs happens correctly uh then as a developer we want to make sure that we do not log any sensitive information in our logs such as station tokens or authentication tokens

or secrets because many times the speed at which we deploy these uh serverless applications we don't have time to do double check so we have to be mindful of uh in advance that we are not logging sensitive information third thing if you are logging sensitive information or not but if these logs get into the hands wrong hands then game over because somebody can construct a lot of information uh from this lock so it is very important to make sure the logs do not have any uh unauthorized access all right so that was about you know our design patterns and what uh security functionality they provide out of box and we discuss many implementation specific attacks and misconfigurations and

settings that if we are not careful then uh maybe the designs are not as effective as we think in this section let's talk about some of the best practices and understand what kind of scanning and tooling capabilities are out there that we can leverage to meet the speed of innovation and security assessment so some of the best practices are first of all zero trust everything what i mean by that in monolith world we were having this implicit trust because things were tightly coupled in microservices world zero trust your code that means verify your code verify uh applications that interact with each other even verify the inter process communication what i mean by verify is make sure the authenticity and

integrity is validated before you talk to them uh in cloud we have seen there are many uh encryption at rest services available and this could be confusing to developers you know encryption at rest encryption in transit using server-side libraries or using client-side encryption you can bring your own keys or you can leverage uh keys provided by cloud service provider and this could be really confusing i would say for encryption or any cryptography related design implementation decision get in touch with your security team understand what option is right for you what option is economic for you because it varies from case to case in some cases where you're implementing end-to-end encryption or message level encryption it makes total sense to use client-side

encryption with your own keys but let's say your data is not so sensitive but you care about unauthorized modification to that data maybe it's okay to leverage the server side encryption provided by the cloud site provider so it it varies there is really no one-size-fits-all solution so definitely get in touch with your security team uh involve them early and find out what is the right encryption for you next best practices whether you're writing monolith application microservices or serverless function regardless you need to pay special attention to a secure coding convention whatever programming language you are using there are always a security best best coding practices available go read them and ensure you incorporate them as you write for

then we already discussed the importance of logs not only generate logs but analyze those log uh monitor those log for any anomalous activity and it would be nice if you can use some sim uh stuff get in touch with your infrastructure security team for that next one is all these uh serverless function that you write from scratch we all know there is heavy reliance on third party libraries and these third-party libraries always always have cvs so stay on top of those cvs use a good uh static uh source code composition scanner to find out the known cves and ensure that your code is secure and a small cve in the underlying library doesn't become the weakest link

in your application and i have said this numerous time always always always involve your security team early you will be amazed the kind of impact they can uh create uh just because you have involved them early because finding security issues right before uh production or in production no fun but when you involve them early you can prevent so many vulnerabilities proactively now let's take a look at some of the security best practices um more carefully so secure communication is extremely extremely important in microservices world if you remember in monolith architecture where there was a thick giant uh server-side application communicating with uh the client side we used to have tls connection these tls connections were not

necessarily mutual because again there was some sort of implicit trust and we used to establish this tls connection to prevent man in the middle attack but when it comes to microservice architecture or architecture which consists of native uh you know serverless functions or applications in a cloud environment we can't trust anybody so simple tls connection doesn't work we have to leverage mutual tls where both the parties are authenticated and validated and the reason this is important just take a look at on your right hand side how many attacks were published in last 10 years related to pls protocol implementation and what does that tell you just like any other software even security software is buggy and that's why we have to stay

on top of it do not use older tls protocols because they have one of these vulnerabilities in fact i would say make your code write your code in such a way that you introduce crypto agility you know we are on the verge of quantum cryptography uh post quantum cryptography attack so the cryptography that you use in your application has to be a post quantum cryptography grade so make sure you do not hardcore any cyphers algorithm key size but make your code agile enough so that if you have to upgrade certain tls protocol or ciphers you can do that uh tls 1.3 is coming january 2020 uh two so ensure that uh you know it's very

easy in your crypto code to upgrade to tls 1.3 when it comes uh last thing on this slide pay attention to the key sizes and cipher uh algorithm types as well as their modes and again crypto is not something i would say anybody to do it get in touch with your security team and they will guide you let's move to the next one um the next best practice is about input and output validation in microservice oriented architecture with serverless application i don't see those many cross-site scripting as i see blind cross-site scripting what blind cross-site scripting really is uh let me explain suppose uh you know your application deals with four more applications and out of four applications three

applications have uh input validation and let's say output when coding but there is a fourth application which is a legacy application and let's say we did not get funding to change the monolith legacy application into a more modern microservice oriented architecture and as a result it has vulnerability so there is some data which is not validated and when your application gets in touch with that uh data store it gets affected by cross-site scripting and when the cross-site scripting is not visible on the front end but it exists it is called blind cross site scripting so because of this complex a complexity arise due to a mix hybrid mix of you know monolith micro services which is a combination of

legacy and new applications it is quite possible that old code is making your new code vulnerable so therefore very important if something is in your application scope make sure all the user entered input is thoroughly validated let's go to the next one this one is about secure handling and storing of secrets usually i have seen in a monolith applications secrets are stored uh in plain text files environmental variables or runtime configuration files assuming that attacker can't get there and doesn't have access but in microservices world that's not true everything is out there and because of the sheer complexity just see how many different types of secrets we deal with secrets related to api service each api

service secrets related to artifactory secrets related to your ci cd environment secrets related to testing or code management all secrets must go in a weighted secret store cloud service provider provides tons and tons of options here always use the secret store provided by the cloud service provider and weighted by your security team super important now let's take a look at how we can scale security we have seen how difficult the architecture becomes how complex it becomes and therefore the speed at which developer wants to deploy the code is not the same speed at which security team can do all these reviews that's exactly where tooling comes into the picture use your static code analysis tool

integrated uh in your ide itself so that as soon as you write code you can find out about those silly you know security defects we make unknowingly even before the comment then use you know your application specifics uh secure configuration checks at the time of build and what this means is you can use some homegrown scanners as well which are specific to your applications in case you don't find commercial tools in testing phase make sure you're performing your dynamic application security testing or lately interactive application security testing is becoming very popular which is agent-based testing and it has access to both source code repository and dynamic execution of your application so it uses both the sources to come up

with more intelligent uh true positives and less false positives so make sure all these tools are nicely integrated in your ci cd pipeline if you are using serverless applications as container ensure that your templates go through security reviews and there is lots and lots of uh free open source uh container security tools available based on your need leverage those make your basic templates secure and hand it over to your development teams all right so that was about security scanning in cicd now as you all know these tools are in their first or second generation these tools do not have contacts and that's why they generate false positive or they do not provide adequate uh coverage to your micro services

in that case what to do i would say that write security unit test cases and automate them as part of your build pipeline do it once and run it every time leverage your security team to do threat modeling of the new feature or function you're building they will help you to come up with abuse cases and then write unit test cases to cover those abuse cases last if you see a pattern of application security vulnerability either related to the code uh or related to you know business logic in that case probably commercial tool cannot help so what really helps is uh build your own homegrown scanners to detect those vulnerabilities all right i know i have covered a lot of

content and now we are moving towards a conclusion so let me summarize um today we discuss different uh architectures how monolith architectures are different from microservice oriented architecture and then the rise of serverless function and why it is important to zero trust them by default each time you leverage serverless application it is extremely important to validate the authenticity of the serverless application to earn trust next is uh if you use weighted micro service design patterns provided by security team and of course it's a you know collaborative work between security team devops and uh the development team these weighted design patterns really help you to scale and foster security then we have seen that although the

design patterns take care of majority of security functionalities but they do not take care of some of the classic application security attacks so know that these type of attacks are still possible and think about how you can tackle them you can either leverage you know uh commercial security tooling which is available out there along with the open source tooling or you can build your own homegrown security scanners if you implement those scanners uh with your build pipeline uh then the shift left of security transformation becomes super easy and as i said earlier you know tooling is not everything it cannot guarantee you a hundred percent security nobody can but what really helps uh when you are in

design and requirement gathering phase is threat modeling sit down with your development team security team and do threat modeling in my experience i have seen is threat modeling returns the highest number of vulnerabilities it makes you think about abuse cases and all possible attacks and gives developer enough time to uh incorporate those in the basic design templates that you're building and last do not just focus on serverless applications or microservice but we'll take a look at a complete uh stack and ensure there is end-to-end security implemented now uh these are some of the great references that i used uh to uh build this presentation and uh yeah these are great um please take a look at these and i will

take questions now thank you everybody that was great trippy thanks we got one question so far in discord so let me take a quick look for you there and it's wouldn't sidecar proxies add overhead to the microservices of course they are that's why they are not one size fit all solution only when you need that type of architecture one should leverage service mesh with sidecar proxies anything that you design must map to the business requirement as well so yes they are overhead and that overhead actually introduces complexity