Harboring No Illusions: Navigating Risks in a FaaS World

All right, good morning everyone. Uh it's a beautiful day in the charm city and today Nimish and I will be talking about cloud security. So just few disclaimers uh we are not representing our employers today and whatever we are kind of talking here are ideas from our research which we did last year and uh apart from that uh if there are any questions feel free to ask throughout the session. Uh what we will be discussing today is something around cloud security. So a couple of pulse check questions. How many of us are working on cloud environments or running our workloads on cloud? All right, almost all of you. Okay, so with that being said, uh let's quickly

check the agenda for today. So the talk is titled harboring no illusions and navigating risks in the fast world. So as we all understand fast stands for the serverless kind of workloads what we see which includes the lambda functions ephemeral compute and whatnot right so to begin with the agenda looks very straightforward we will talk about uh the evolution of infrastructure abstraction which we have seen over the past few decades and then we'll move on to kind of examining how the attack surface has changed from the traditional operating models and then we'll talk about some economic impact of the serverless incidents what we have seen in the past and we will do a case study

of how we can secure AI workflows which are built on serverless environments and finally we'll have the key takeaways and some ship to Monday insights which we'll share with all of you. So with that being said let's dive into it. Uh so the first slide for today talks about how the infrastructure has evolved over past couple of years and to begin with uh what I would like to call the first phase is the bulockart era where the physical servers were running there was one operating system one workload and that's the bare metal era right so what would happen at that time you owned the entire cart there was just one cart one destination you fed the

bull, you fixed the cart, you cleaned the mess, you did everything basically, right? So that's the bare metal. It was predictable yet it was painfully slow to scale. And guess what? Some of the critical routes still use bulakas today, which means we still have the on-prem servers around. And I think it's for the good because there are a couple of industries or there is some information which we want to still you know kind of use the on-prem systems for and that's why they still exist. And now coming to the second phase probably it's the virtualization era where uh we saw there was one physical server and there were multiple kind of virtual machines and we

also call this era as the bus or the carpool era. Right? So the road is the same, engine is the same but multiple passengers and different destinations. So what happened here? There was better efficiency, faster recovery if one bride dropped off and fewer empty seats. But now the problem was we need to now have a schedule, right? So we need to have tickets in place. We need to have rules or otherwise chaos would follow. So welcome to the early cloud where we had the infra where the infra became elastic and scalable. So it unlocked the better resource usage per se, faster recovery, flexibility and here is where the tools like VMware or hypervisors, EWS, EC2

instances, they kind of changed how we thought about compute and as the applications grew teams wanted something lightweight, more portable environments. That's where the containers arrived. And with Docker and Kubernetes coming around 2012 to 2013, developers now could package their code dependencies, not now worry about the environment where they're running it, whether it is local in some other environments. So they could no longer have that excuse that it's running in my system but not in yours. Right? And this was the era of DevOps automation and microservices bringing that agility and also some orchestration complexity. And then finally in 2014 is when we saw AWS launched Lambda marking the start of the serverless era also called as

teleportation because now you don't book a ride you don't see a car you just appear at your destination which means you upload the code trigger the event and done. So it accelerated innovation for sure but it also blurred some responsibilities because developers lost the insight into what was kind of under their control once and that's the agility which serless brought but it also brought in some new risks and that's the risks which we'll try to uncover in today's talk. Now as we already spoke about each layer of abstraction which made our life easier also kind of added some distance from the underlying system and that's where we are calling it as the evolution where we don't have servers anymore to manage

but then what are the other things we need to still take care of. So that being said, the next slide up here talks about the shared responsibility model in the serverless environment. And here we see the markers in yellow, they represent the cloud customer. And the markers in purple, they represent the cloud provider. So from top to bottom, it talks about how the responsibilities have shifted. Now if we go from left to right in the on-prem era to the fast world how the responsibility on the cloud customer is kind of reducing and while the cloud provider is having more onus on them either if it is you know data class in terms of client and endpoint protection

or in terms of application level controls or network controls and other details. Now the key here is definitely how we apply these dev sec op principles like validating the inputs, restricting the IM roles, audit configurations, monitoring some function behavior. So that still stays with us and with that let's now talk about some serverless attack surfaces. So to begin with uh we have the first kind of uh kind of you know classification which is the trigger and the event source risks. Now why we talk about this here is because this is the front door for a lot of our risks coming in. It could be coming in through some SQS which is a Qbased system or it could

be a messaging coming from an SNS. So or the API gateway per se right. So what are we talking about here is how do we validate the inputs which we are receiving at the edge right and on the similar lines there are other class of vulnerabilities which is the functional level vulnerabilities now when we talk about these what reminds me of that is how we deploy our core packages to let's say lambda functions right so we do have packages involved there third party packages involved there and how risks from that can make make your entire application more vulnerable. Then the third class of the risks here is the platform misconfiguration. And in here there are certain uh

pointers which you can see on the screen as well which talk about how disabled logging can basically lead to some kind of loss in the forensics in general and how we can have the guardrails in place to ensure whatever misconfigurations we are having in the cloud resources which are getting spun up in our environment those we get alerted on time to time. Now the remaining two classes are a part of the platform misconfigurations but they still need a special mention right like the one which says CI/CD risks those are the risks which can come in through our pipelines and how we want to ensure that we have more safer packages being used in general let's say we can

use dependency versioning or pinning in general or kind of manage it through sbombs which is the software of materials and then talking about the monitor ing gaps, how we can have the right telemetry in place, which also means we need to have the right logs and we need the ability to query those logs time to time whenever we feel that there is a need to do that and also those help in understanding the threat activity in general. All right, with that being said, uh I'll hand it over to Nimish from here and he'll talk about the threat mapping to security frameworks. Thanks Sham. Um so a basic tenant of this session is that serless doesn't

remove responsibility it sort of relocates it. Now after speaking with different you know GRC analysts uh working within the space be it cloud or specifically you know subserverless or fast you know function as a service being a subset of cloud cloud environments. Um so you know the GRC analysts working within governance and compliance for uh monitoring and assessment for uh function based servers services and applications. We found that in addition to security engineering constraints associated with the infrastructure abstraction within the serverless architecture risk assessments and building accurate racy metrices uh is another major sort of uh challenge from a governance perspective. So this table sort of um maps three high impact serverless risks to two popular

frameworks. uh the OASP or the open web application uh security project top 10 and CWE uh which is the common weakness enumeration top 25 and sort of pair each u with concise mitigation steps that you can ship sort of come Monday morning and again this is like AWS standard but uh they can be translatable to other cloud providers as well. So first one is uh platform misconfigurations. Now in serverless platform is your runtime. So the OASP equivalent for this uh is security logging and monitoring failure and the CWE insufficient logging. U with the misconfigurations especially missing or weak logging. Essentially they turn incidents into sort of silent failures which are gradual and they're not very

you know um obvious to be detected or you know that's something which is um which which is which can be flagged within the dete detection engineering part um as if you can see can't see it I mean you can't defend it. So for mitigation strategy um capturing the right logs by default is fairly crucial. We can do that by enabling um again this is AWS specific but uh cloud trail or u lambda and API gateway logs um S3 data events and VPC flow logs u you know that is for specifically for egress paths and now enabling logs um is just the first although an important step but we also need to make the logs useful. So for

that um we need to sort of structure them say based on principal role request ID any resource ARNs and consolidate and centralize them into a seam solution. It could be open source, you know, proprietary. There's so many other options out there. Um, but that needs to be integrated with detection rules for anomalies such as, you know, sudden spikes in the invoke function or get object from new rules, uh, new roles or cross region copies. If you don't want to yeah like a you know there's uh seam solution um AWS provides their uh rendition of seam as well but there are so many like there's different other options as well. Second uh is the uh runtime monitoring

that's part of the mitigation strategy. This can be achieved by configuring cloud watch metrics and alerts for uh sort of error ratios throttle spikes and cold start anomalies. Moving on, the second risk is event injection or just injection within OASP and O wasp and um improper control of code generation within the CWE. Now, serverless apps are essentially event fabrics that are knit knitted together uh using a plethora of API requests, S3 object pools, you know, SQS messages, scheduulers and many many other different aspects that come together to essentially build uh and uh implement a serverless application. Now each event in this case uh is an ingress point and event injection captures the risk of unvalidated inputs forcing unintended

behavior code paths or even straight up denial of service. So the mitigation strategy primarily revolves around configuring boundaries to enforce at the edge. Now that can include schema and content validation at the API gateway. Um the API gateway security can reject any malformed JSON oversized payloads any rogue headers or or any unexpected uh you know uh MIME which is the multi-purpose internet uh mail extensions all the types before the function runs. Uh second strategy is strong identity on events that could be using the JWD authorizers um configuring MTLS if it's feasible or even enabling signed uh URL requests if you say want to grant temporary access to files or objects. And in addition to the secure

authorization, we can even use like S3 event filters and u you know that refer to the least privilege uh notification topics to utilize other you know attribute based access control. Now another sort of cool technique can be configuring fail safety. Now that can be done by using DLQs or you know dead data cues as async uh triggers and IM potency keys to prevent replay and rate limiter uh and VAF rules for you know obviously for Percy probes and other u you know custom ACL's. Now the core objective essentially is that if an attacker injects uh into the stream these boundaries sort of stop the lateral movement and keep um restrict the blast radius. Moving to the last one that's the

vulnerable dependencies or the vulnerable and outdated components in OASP and use of unmaintained third party components within the CWE. Now in serverless dependencies are part of the supply chain uh that are embedded within the lambda layers or container images and transitive packages. In this case for um mitigation strategy we can utilize steps like producing an espol like Maven um and verifying signatures for images or layers and blocking any unsigned artifacts within the CI pipeline. A good practice is to scan early and scan always. Uh and it's also a good strategy for runtime protection. Now sea scans uh software component analysis scans could be used to analyze open source components. Uh these static or dynamic application security testing,

SAS or DA scans, they can be used for analyzing sort of the code functions uh both before and after execution and image scans can be done on the pull requests. So you can configure something like um you know fail builds for critical and known uh CVEes and schedule rescans on occurrence you know confirming with your sprints. Uh third one can be constrain egress and a sort of third party access you know that is also that includes different steps like you know PPC integration uh with the egress controls. Now essentially the result is fewer moving paths within the serverless you know which has which is dynamic in nature and abstracted uh fewer surprises and less attack attacker leverage through your

build chain. Before I move on to the economic implications part I would pause uh to see if anybody has any questions. I think that's a good crowd. Well, so moving on. This section often sort of gets management's attention. So anybody who's a CISO or uh who is a decision maker within uh their company, within their organization. So it's typically you know u good to have these stats handy. Um we've established that obviously serless isn't a silver bullet and the responsibility shifts. So you know what's the sort of economic implications of that? Now in this chart I mean it illustrates the uh you know the rap the exponential growth of the serverless computing market and we are talking you know about

a market that was almost like 19 billion in 2024 and is expected to be around 50 billion uh by 2030. um you know it's not a which kind of says that it's not a trend uh it's a sort of like a shift that is driven by like actual genuine benefits like reduced uh deployment cycles and you know faster shipping um and obviously you know we use fast and serverless interchangeably that's also because statistically 61% of the serverless computing market is driven by the function uh function as a service uh and I would Okay, a major chunk of that is contributed by the lambda functions. So now the growth however also magnifies the risk which is why we have the second

graph uh which sort of relates to the projected losses due to serverless related breaches. You know this was created by forecasting on certain like certain known breaches that were directly or directly related to serverless. Now it's just an it the damage isn't just direct uh it also incorporates the cascading secondary losses. So obviously as the market grows uh so will the losses associated and um you know the base high and high plus is sort of um like three different scenarios that have 30% increments. By increments it essentially means that we are um that's the liberty that we are taking in terms of the forecasting and across all of them the loss runs into like you know

hundreds of millions. Now in the next chart that is for the in cyber insurance market obviously you can you know that's also increasing uh the share of the ser of the insurance market and serverless isn't just a isn't the only contributor but a major one um and I would say now with the uh agentic AI and the other emerging threats uh as a consequence and as part of it uh it's I I feel like this uh the numbers are you know not accurate anymore to be honest. So the growth is not just from the companies buying the policies it also reflects the underwriting uh you know reacting to these losses essentially. So that being said uh we will shift back

to our technical part of the session and have we have a case study uh essentially we this is regarding a open-source cloud utility setup. We've named it as links lab and we have um sort of a introduction of that architecture diagram and a quick you know demo of how to create the stack for that. Now for it's a sentiment analysis model uh but we are leveraging AWS's bedrock. It has um pre-trained several pre-trained models. It can be swapped by other classification you know machine learning models as well. That's the most practical one that we thought would be uh good to demonstrate but um you know we can definitely have um an LLM model in there as well. Obviously that changes

the you know diagram slightly. So coming to that um the workload itself is sort of intentionally simple um it's a sentiment analysis flow uh which essentially you know you pass on a review or a text and uh based on natural language uh processing algorithms it the output comes back as whether that text uh which could be a feedback or a review uh whether it sentiment is positive, negative or neutral. Now the simplicity is deliberate because it keeps the focus on the actual security controls uh trust boundaries and failure sort of modes which are which is what we are interested in I'm sure you are rather than on the application or the model complexity itself. So tracking the life cycle of

the process I'll start from the left um in which the user obviously sends a request through the API gateway. Now this is the public entry point and in a realistic serverless system this is where controls such as authentication, authorization throttling request inspection and you know input validations are typically sort of enforced and behind the API we have two separate lambda functions with obviously separate responsibilities. Um we have the authorizer lambda that is sort of like the bouncer uh you know it handles the access control and determines whether whether the request is allowed to pro uh to proceed. The worker lambda uh handles the business logic and orchestrates uh the call to the actual model which in

this case is the bedrock model um or the sentiment analysis model. Now the authorizer is as you can see also integrates with the secrets manager. Um this is important now because we do not want any credentials, tokens or any sensitive configurations hardcoded in the code uh or deployment settings. Now the authorizer retrieves what it needs securely. It validates the caller context uh and determines whether to allow or block the request based on the rules. We also have an S3 bucket in the workflow. Now that gives us a place to record requests and response artifacts uh which is sort of useful both from an operational as well as from a security perspective because it creates an

evidence trail for triaging and any you know post incident DFIR you know foreign analysis. Now the right is the uh model layer which is from the Amazon bedrock uh AWS bedrock. Now in this example uh particular example we have uh the llama 38B which is a very common sentiment analysis pre-trained model um but we do have as of this year I think it's been retired uh so if you search for it it will not come up we do have an alternative it's called the nova micro v1 you know in case anybody's interested. Um so yeah uh the worker lambda which invokes the bedrock model that receives the sentiment result and returns the response back through the

same API gateway essentially uh to the user. Now the diagram is just that it's simple um or at least it looks simple um but I would add a footnote and that is based on what happens you know we we had a question that how do we um so bedrock now even has the agent core I think so which basically it has like um different AI agents now that have some um actions that can be utilized like preconfigured actions So we had a question that can that be swapped one-on-one uh with this this bedrock model. Um so for that um the agentic version will not be completely different but will not be the same as well. So what it does is that

it's essentially adds another layer of orchestration on top of it. Now in the basic flow the worker lambda mainly acts as a request handler that sort of sends the prompt uh to the bedrock model and returns the response. Now in an agentic um orchestration the middle layer becomes sort of much more dynamic. Instead of doing just what one inference call it can plan the task um decide whether it needs retrieval call any other tools that are downstream or other lambda functions as well and pull or store the artifacts within uh the S3 bucket. It can also loop uh back to the model until it has enough context uh to send back uh the output uh to the user.

The key shift is mostly from a singlestep generation to a multi-step decisioning and action execution. Now the API gateway, authorizer, secrets manager and the storage pattern that can essentially remain the same but the worker layer sort of that's the one that evolves into an orchestrator and is uh you know can be replaced by uh a bedrock agent or a step functions based orchestration flow and that's what makes the architecture more agentic um in that sense that you know we just have one additional layer of orchestration in the middle which is much more dynamic uh than the than the current one. So this is we do have a demo for oh yeah so uh this is for the essentially the

stack setup. Uh the main idea here is to that we treat the infrastructure as disposable. That's the whole idea of all serless as well. Now we create the environment uh validate that it works end to end and we use it for testing and we also tear it down because it's reproducible so it can always be it can again be configured whenever you need to stack. So now that matters from both a security uh as well as engineering perspective because it makes the experimentation sort of safer. It reduces the drift and also lets us rerun the exact same scenario without rebuilding everything uh manually each time. And the video this is like a CLI uhdriven workflow. Um the process is uh

you know pretty much straightforward. We installed uh the required Python dependencies. We configure the AWS CLI credentials uh and the region. Um we set the environment variable used for API authorization and then we run the driver with a create action. The create action then provisions the entire stack which is what it's doing right now. Um and which is also what we saw uh in the previous slide. Now this is we are moving switching over to the AWS console. Essentially the purpose of the next step is the verification. We want to confirm that the resources are actually appearing the way we expect them to appear and that the architecture we described is you know it has been really deployed

essentially. So practically that really means that checking the API uh gateway exists and the authorizer lambda is attached to the API route. um the lambda worker is present with the correct permissions um and the supporting services you know obviously such as the S3 bucket and the sequence manager they're also in place so it's a verification of the um the essential requirements uh for the application

I think in this case it has the verification part is uh I think succeeded yes uh I think now We have the essentially the worker lambda is executing. So the worker is the function that will handle the business logic um as we discussed it reads the request parameters prepares the call to the bedrock model and sends the input for inference receives the result and then returns the result back through the um API response path. Now in links lab the worker also writes request and the response artifacts into S3 and we can also have it connected with and so Dynamo DB uh if you want to store the responses uh as uh in the form of a database as well. Now

this part uh the S3 storage is obviously useful um you know for not just for the logging effective logging but also for the reproducibility because we can compare the runs over time um and we can you know map the whether there are any discrepancies um we kept the demo you know recorded intentionally because um you know demos don't uh rarely fail if we you know in in addition to as opposed to you know doing it live. So that's good. Um yeah, I think that's about it. Yeah. So we have the stack ready and I think the last part is just the check um the verification uh of the responses and um the destruction of the

stack. With that, I will pass the buck back to Shbam and he will now discuss about the attack and defense scenarios specifically to our serless application. [clears throat] >> Well, thank you so much. Um, couple of things here. Um, I guess one thing was the services what we have used here are all part of the free stack. So anyone who is interested in checking this out can go in and there's a QR code right there which links to the repo repo as well and everything is free in the AWS tier and that's why we use those services and those were part of the serless stack as well and that helped us kind of you know frame the idea around

the session which we were planning today also we had a training yesterday where the same uh links lab was deployed by a couple of folks who joined us And we were happy to see that almost everybody was able to deploy that on their respective AWS environments. And uh I mean there were a lot of good questions which came out of that session as well. But yeah, please feel free to check this out and see how you guys can also contribute or if you have any plans around making it cloud agnostic or anything for that matter, feel free to reach out to us. Now with that being said, I think uh the primary uh focus for our conversation

today was around how we can have the attacks and defenses in place for the serverless environments. And in this case we have taken an example of the first one which we see here as the trigger risks part of the malicious inputs. So the one highlighted there in yellow uh towards the end where the user is also added shows how the inputs can be fed into the system which is via the API gateway. So let's say if you have a malicious payload um how would the system respond to that right I think that's the quick defense which we want to implement is validate the inputs at the edge have the schema validations done at the edge and also [snorts] which

ensures that let's say if there is some constraint around the content length which you want to pass in because finally this interacts with the breadrock model as well that helps you trim the content and only pass in the required information. Moving on, the second category of risks which we discussed earlier were the platform misconfigurations. And here for example when we talking about the authorizer lambda already nimush walked us through why it is used. So I think as we all know we want to know what is going on who is invoking that lambda function when is it being invoked. So let's say if we don't have the proper telemetry around those actions right what happens so I think

that's where uh the platform misconfigurations come in place because we will have those guardrails which will kind of alert us on the specific misconfigurations around it and that will help us alter our platform or the environment in the way it will become more um compliant. Then moving on to the third category which is the functional level vulnerabilities. And here as we see the let's say our lambda function is using a kind of not so safe package and uh then it tries to kind of you know take the input from the secrets manager like kind of access the secrets manager and get the secrets out from there. I think that's what we're trying to say that

that could lead to a leaky secrets attack and how can we ensure that the packages what we are using which is the third party packages are safe and [snorts] uh we don't have the vulnerabilities in the deployments what we are having for the lambda functions and for that matter we can have the quick defense as dependency pinning where we ensure that the right versions are being used in our environments and also the other uh kind of defense strategies include the maintenance of sbombs. So that covers the attack and defense scenarios. We also plan to have simulations of the same in links lab going forward and we are planning to kind of make it in an agentic way. Uh

but we'll see I mean how we can have these three use cases primarily kind of showcased via links lab tech stack. And off to the final key takeaways. All right. Um well uh the serverless uh as we know it removes the servers and the need to understand uh what is actually happening. Um now if you cannot trace the full path and permission boundaries you sort of end up doing these mental gymnastics uh during the incident triaging process uh instead of having like clear clarity on the controls and visibility up front. >> Yeah. Uh so you still own the identity data dependency and configurations with the abstraction and hyperscalable function triggers uh you know shipping gets easy. Uh so you definitely want to

move faster to prod uh and also get that non-technical PM off your back. Uh but if you're not gating your deployments with automated security checks and lease privilege reviews, you would end up throwing during the throwing up during the sprint, >> right? And that's where the uh structured velocity comes in place. We want to ensure the production pipelines are kind of enhanced via structured velocity even for the serverless environments. >> I think we had a question yesterday about structured velocity. That just essentially means following the playbook. uh you know we the the dynamic nature of deployment can definitely get uh you know enticing but we definitely need to have a similar playbooks uh for the serverless level deployments as

well. The third one um isolation is a design choice not just a wipe >> of course because it's a blast radius management and you need to isolate by function by the account by VPC and by the tenant. Next, uh functiondriven highly abstracted workloads can definitely feel like uh the compound we for production because it uh it enables faster shipping uh autoscaling and obviously fewer servers to babysit. Now, but the same sort of trust me bro abstraction uh can catch you off guard during the incident response if you do not have enforce trust boundaries, validate inputs and isolate event permissions. All right. So for third time in the talk today, you need to validate schemas at the edge and

keep your handlers small and at important. This is what a serverless architecture essentially uh feels like uh without having comprehensive telemetry everything sort of feels fine until it doesn't right and you need the traces to follow an event across the services and logs which you can query at the right time. And the last one u serless changes the shape of the risk and it's I think our job as security practitioners to design for failure and remember you lose the server not the responsibility. >> Yeah that's about it. We have a grown-up version of the banner which is muted. >> Muted right. >> It's just thanking everyone and wishing everybody a good rest of their conference. We realized that there was

no words and there were all right that's our talk today guys. Thank you so much. [applause] We can wait for any questions.

>> I have one question. >> Sure.

insurance. [snorts] That's a question I've asked so many times whenever I'm filling out those forms that we get from the cyber insurance companies. And it's like you ask for so many information and they just have like two people you know who are like oh this is all all what we cover they make good money and we're the ones who are filling out the form. >> I think I don't know if people are following but a lot of cyber insurance companies espec like especially the insurance companies that deal with cyber insurance they're also acquiring a lot of security companies as well. I think specifically I think PC uh it's a UK based I think uh this they're a big

cyber insurance company but they have made certain acquisitions and now they're offering the like an entire security stack and what they essentially say oh if you take uh like you know take these services your premium is going to go down so that's [snorts] a great uh strategy also I could reate for the uh links lab we are also you know definitely open for any contributions if you um have already scanned the QR code if you have the link uh for the repo definitely check it out and uh yeah please make >> all right thank Thank you.