Follow the Trace: How Traditional AppSec Tools Have Failed Us

Good afternoon everyone. Thank you so much for attending. Like to present Kennedy Tumi and her talk follow the trace how traditional apps tools have failed us. For slido questions, please feel free to either visit the QR code or the URL previously mentioned for theater 9. Awesome. Thank you. I hope everyone is having a good besides SF so far. As mentioned, today we're going to follow the trace and learn how traditional apps sec tools have failed us. I'm Kennedy Tumi. I'm an application security researcher and advocate at data dog. I did get my start in software engineering and then very quickly tradition or transitioned to application security where eventually I was a senior application security engineer at an

application development consulting company. So I have a lot of very interesting stories from that and that's that's why I'm here. I really enjoy appsac and today I want you to meet trace and trace is going to be your guide today to learn all about runtime security. And with that there are often a lot of developers, many vulnerabilities and a pretty small appseack team. This leads to burnout on both sides and there's usually frustration surrounding security. There are always new tools emerging. However, do they actually work and do they actually reduce the burden of security? The most successful tools provide accurate and actionable results. Today, we're going to learn how runtime tools have an added advantage

over traditional application security tools. So, let's go ahead and let the journey begin. And with that, we have to start at the very beginning with what is a trace? because I know that not everyone got their start in software engineering. Not everyone was a developer. So a trace is a detailed record of an application's actions and events. And when we talk about tracing, we usually refer to tracing as a single transaction. And then we use distributed tracing when we're talking about tracking requests as they move throughout services. And now with traces they help us understand how the data is flowing through the application. And with this that data flow can help us identify performance issues and it helps us pinpoint any

errors that are happening. And with that traces have become an integral part of application development. So why can't it be a part of application security as well? So today we're going to be going through a purposely vulnerable application. I will include a link at the end to this repository. However, it's a basic Flask app. So written in Python and it has a login and registration page and then once you authenticate there is a place to see blog posts that you've published and then there's an option to add blog posts. So, it's not very uh complex of a website. However, it serves a great purpose for today. And with that, we have our first request. It is a basic

post request to the login page. And the user input is username and password. And with that, we have the flame graph for the login page. And if you've never seen a flame graph before, that's okay. It helps us visualize a distributed request trace. And with this, it represents each service and each service call that occurred. And it does this with a timed color-coded horizontal bar. And in this, it includes error and latency data in order to help uh developers identify bottlenecks. So with that we can see at the very beginning the login and as we flow through we can see that the flask uh app actually dispatched the request and then if you can see on the very right hand

side there's a tiny little line that is actually our call to the SQLite database and if you look at the execution time it only took about 2%. So again, very small sliver right there. And then at the end, we can see a small flow is what I will call it. And that's actually where all the cleanup happened. So this is probably one of the most basic flame graphs you will ever see. And again, that's because a very basic website. However, once you get into some really interesting services and websites, these can get very complex and show you a lot of really interesting data. And with that, why are we capturing this runtime behavior? Why are

we doing this? With runtime behavior, it helps us identify what our application is doing right now, but it also helps us understand what it was doing previously. And with that, it does help us track function calls and we can start to see what code paths are actually being used. We also are able to identify any latency, throughput, and error patterns in order to help us if anything starts to go wrong in the application. And with that, we also want to see how things interact across our components. specifically how data is flowing through our components, which means we can see how different services are interacting with each other. And with that, we can also see what the

inbound and the outbound traffic is. And this helps us just understand and see all of the traces together instead of an individual trace. And with that, what even gets captured? So, we are capturing timestamps, spans, and traces. and then some metadata that are included with those requests. And then lastly, why does this matter? Why do we need runtime? Why do we need traces? It is really helpful for performance, for troubleshooting. And then it gives us a baseline of the application behavior in case something starts to go wrong. And with all of this again, why can't we use all of this information that we already have in security? Why can't application security people actually take advantage of this

data? So with that, let's start going into a bit about application security tools and the basics. So I'm going to start out with our SAS tools, which is static application security testing. And these tools are scanning the code, the static non-running code. And specifically, a lot of times it is doing pattern matching. So if we have vulnerabilities that we have identified this piece of the code is vulnerable, we're able to then take that and look against a database and we're able to ma match that pattern and go from there. So with SAS there, these scans are really good at catching vulnerabilities early in the SDLC or the software development life cycle. And this is because it doesn't need any

running code. It just needs code. However, this does usually result in a pretty high false positive rate because it doesn't know what's happening in the application. It doesn't have the context to be able to say, you know, this is potentially malicious or potentially vulnerable, but we can't prove it either way, so we're going to alert it. We then have dynamic application security testing scans and this is scanning the running application but it's doing this by simulating malicious attacks on the application. And when I say malicious they might not actually be malicious. They might not be coming from an attacker but they are acting and looking like they could be malicious. And with these scans, they're really helpful to see how an

application is handling real attacks, real world attacks, and it does this with an automated scan. So, this means we don't need to spend money or spend the time on a pen test or any manual scans. However, these scans do tend to have some limited visibility and they can miss logic flaws because it's not configured specifically for the application. it has just a pre-existing set of attacks it's trying to run. Lastly, we have our interactive application security testing scans and this is actually running the applications. It's analyzing the running applications behavior and it's doing this by using legitimate application traffic. So with that essentially it's following the data flow. So it's monitoring your code's interactions with

other components of your stack. So think of libraries, infrastructure, and it helps provide an upto-date view of your attack surface. And with this, there are four main steps that happen within these scans. First, it needs to track the data. Where is data flowing in? Where are these sources coming from? Are there any URL bodies? Where is this data coming in? Next, it has to analyze that data flow. It needs to understand if and how the original input reaches sensitive parts of the code. It then needs to identify vulnerable points. So, detecting different locations in the code where user controlled inputs are used in potentially insecure ways. And then lastly, these types of scans are able to

confirm the vulnerability.

Okay, we're changing it up. Um, so these types of scans are able to confirm the vulnerability because it's able to confirm that suspicious input actually reached a vulnerable point in the code. So with that is scans are really good at un using the context to understand the data flow to reduce false positives. However, they do require the application to be running which means we can't use it early on in the software development life cycle. It's more useful later in the SDLC. And with that we now have our application layer protections. The first thing I want to talk about is a WAFT or a web application firewall. And this lives right outside of the application. And I'm going to talk about a few things

that the WFT is great, good, and not so good at. So, first, the WFT is really good at understanding the threat landscape visibility. The goal is to find suspicious payloads, and the WF is pretty good at doing that. It's also pretty good at DOS protection or distributed denial of service and this is because it has visibility into inbound traffic at a granular level and so it helps reject these attacks. However, WAFTs are not great at exploit prevention and this is because they have no special insight into potential vulnerabilities in the application or whether the route being called even exists. The best they can do is block based on the rules that they're configured to enforce. And so with that, we have our

RASP or our runtime application self-p protection. And now this lives within the application. So the WFT was right outside and this lives right inside. And with that, I'm actually going to start at the bottom of this list. The RASP is not good at DOS protection. And this is because the request has already been through the application server before it gets to the RASP. So everything has been through all the hundreds of thousands of requests have already been inside the server. However, the a RASP is really good and actually great at the threat landscape visibility and it's actually improved over a traditional WFT because it can see the already interpreted requests which just improves accuracy a little

bit. And then exploit prevention is where the RASP really thrives and this is because we're able to look at the trace and identify whether an attack is attempting to exploit a specific vulnerability. So why runtime? Why am I here today? Why do we even care? Why don't our previous types of scans work? And they do work, but that doesn't mean other options can't be better or can't help you. So first, added context allows for more precise results. The more context you have, the more information you have, the less false positives you tend to have. And with the context, you're able to prove that the vulnerable code is reachable and you're able to actually confirm vulnerabilities. And then we have also

found that only 18% of critical vulnerabilities are worth prioritizing. This was actually just published in data dog state of devop devacc ops report which came out last week. And if you're interested in learning more about how we found that number exactly, I recommend you checking that out. But 18% we only need to worry about 18% of our critical vulnerabilities. And this is because runtime tools have the context. They're able to combine that CVE info with the runtime information to adjust the CVSS score, which is the prioritization score. And so it can use factors such as is the application running in production? If it's not, well then let's lower the score. Or is the service being actively targeted? Well,

if it's not, maybe we can lower that score, too. So we like to have it start the highest and then decrease. However, it depends on what tool you're using. However, runtime context allows you to better prioritize vulnerabilities. So now let's go on our step-by-step adventure and actually look at a vulnerable application. So we have our add a new blog site and there's an option to link a picture to add to your blog post. So again, nothing that y'all aren't able to see here. There's a text box, there's another text box, and there are a few buttons. If y'all want to start thinking about this and what could go wrong, feel free to because oh no, we've encountered a

threat. And if anyone has any ideas of what threat we have encountered, feel free to be thinking on that. I'll give you like 3 seconds. 3 2 1 We are going to see a server side request forgery attack or an SSRF attack. And with this, these attacks allow the attacker to cause the serverside application to make requests to an unintended location. And this unintended location could be internal only or it could be arbitrary external services. It just depends on where the attack is running. And with this, a successful SSRF attack can often result in unauthorized actions or access to data within the organization. Now this can be within the vulnerable application or on other back-end systems

that the application can communicate with. And in some situations these vulnerabilities can be used to even perform arbitrary command execution. And with that the common entry points often include sources that do not check for uh common user input. And with this it's not limited to HTTP protocol. We can also use different schemes like file etc. And with that let's take a look at the attack that I will be running and this is the script version of the attack and I will also show the manual version in just a minute. So here the vulnerable website is being run in docker and so we can see our base URL is the very first URL. We're then using basic

authentication and then with that we have the URL that we are trying to inject into our test picture URL endpoint. And with that let's take a look at the result. And again this is also how I did it manually where I took that URL and just went ahead and put it in that text box and clicked the button. And here we can see that we're able to reach the picture, which is good news for me showing this to you. However, not great news in the realm of things. And with that, let's take a look at the flame graph from this attack. And as I mentioned earlier, the flame graph helps us visualize the execution path of a

request. So we start out at the very beginning at the top span and this is the parent span. This represents the initial request from the browser. And as we can see as we move down, there was a Flask dispatch request just like how we had earlier in the login. But then we see get flag and that's actually in orange, not green. Well, that means we're going to a different service. And that means we've left our initial website, which means the SSRF attack was actually successful. And as we can see there, it goes from the flag to make sure it can log in and we can log in. And it does. And then everything flows back

up. And with that, we have our RASP to the rescue. And I do want to talk a bit about the RASP process before we get too deep into it. And first the RASP has to be embedded into the application. And this is normally done at runtime um via agents or libraries. And so it needs to have that runtime context, but it should only have to be set up once. The rest of the step should be continuous after it's set up. So the tool is going to monitor behavior. So it's going to actually observe the code execution. You look at the user inputs and understand the application context. It's then going to go and analyze all

this in real time. So it's going to start to intercept calls that are going to sensitive functions, anything like file access, database queries, things like that. It's then going to start detecting attacks. So it's going to flag things like SQL injection, remote code execution, and guess what? SSRF attacks. And then lastly, RASPs. RASPs can block attacks and then they can also alert. And with that, we need to talk about the configuration of a RASP. First, we need to choose the deployment mode. Do we want to monitor all the attacks or do we want to block attacks? And while we block attacks, do we want to block it based on the type of vulnerability, block it based on all

vulnerabilities, or do we want to block it based on the attacker? If we are seeing potentially one user or two users where all the malicious input is coming from, well, maybe we should block them or their IP address. So, how do we want to do this? Or do we not want to block anything at all? Because that's also okay. If we just want to identify the vulnerabilities that's okay. We then need to actually decide what detections do we want. Do we just want out of the box detections or do we want to create our own? And a lot of times out of the box detections are great. However, you know your application best. If there's something that you

think you need, well, why not create the detection yourself? We also want to make sure that we're not blocking anything that we actually want to get through. If we have any DAS scans, if we have any pen tests that are going on and we don't want the RASP set up for that, we need to make sure it's on an allow list. And then lastly, we want to set up alerts because we need to make sure that we're getting all the information that we know when attacks are happening. And if we're continuously blocking things, well, maybe we need to do some more research. So, what I went ahead and did on the same application, the same deployment, I went ahead and

blocked all SSRF attacks in my RASP. I used everything out of the box, but specifically was blocking for SSRF attacks. And as we can see here, I have tried the exact same URL to insert, and an error occurred. Please try again. Which meant before we saw that it was success successful. So we can see here that it was blocked by the rasp. But if you don't believe me because I get it who you know who am I to tell you that it didn't work. We can see the flame graph and if you remember before it looked a little more complex. We could see the orange spans but now it's only green. And again, the exact

same URL was inserted, but we just had the RASP in place. And we can see that it blocked it before it actually got to execute that vulnerability and execute that attack. So with that, let's discuss some security insights that we have learned. And context really makes all the difference. It puts less stress on developers, which is huge. There are less false positives for developers to investigate and they don't have to spend time fixing vulnerabilities if they don't have to. And with that, context also empowers security teams. It provides a lot of the info that's needed without having to write rely on developers. And previously as I worked for an app develop an an application development consulting

company I didn't always know what these applications were doing. I didn't always know what was going on and I didn't always have access to the code and honestly that makes things pretty hard to do from a an application security standpoint. So, I was constantly relying on the developers to help me out to say, "Oh, here's what we're worried about or," oh, here's why we're doing something a specific way. However, if I had that context, that would then in turn put less stress on the developers. So, it's a never-ending cycle. And with that, there are always challenges and opportunities. And I don't think anyone would tell you that there aren't. First off, many of the tools

require an agent to be installed. And not everyone likes to install an agent. Not everyone's comfortable with that. And you know that's okay. The setup is often more complex. Especially RASPs need continuous tweaking because you want to make sure that you're blocking what should be blocked, but you're also not blocking legitimate traffic. And with that, other tools are still needed for early detection. These tools are not a replacement for early detection tools. They're just found in different places in the SDLC. However, the runtime security landscape is evolving. These tools are becoming more and more available. We are seeing more runtime tools come on the market, which is great because that means they're more accessible to people. you're hearing

more about them and you're learning more about the benefits of these runtime tools. And with that, agentless tools are also emerging. So the companies that don't want to put an agent um in their application, they don't necessarily have to. And it all depends on what tool you're using. So let's go ahead and recap our journey because we've gone a long way. We've talked a lot about traces and security and application security. I personally think runtime security is changing the game in AppSAC. This is something that I'm very excited about. I really enjoy using these tools and seeing how they can catch things and identify things. IAS, sorry, I scans have the advantage of runtime context. However,

they're not going to replace our SAS scans. And then lastly, RASPs have added benefits over WATS. However, they're not great at preventing DOS attacks. And that might be something that's important to you. And so with that, I have a few QR codes. If you want a copy of these slides, if you want to check out the vulnerable website repo, Data Dog also has a community newsletter that is all practitioner focused. It has nothing to do with our product. It is written by our security research team. Um, so if you're interested in joining another newsletter, feel free to scan that. And then also, if you're interested in connecting with me, I've got that up there, too. And we do have a few minutes

for questions. And if there are more afterwards, I'll find a space, but I will also be at RSA hanging in the data dog booth if you have any specific questions revolving the product. So with that, here are some of the QR codes for feedback and for

Slido. And so thanks so much. A question for you. What's your approach to tuning uh WAFT rules to minimize false positives without weakening protection? WAF or RASP? It was for WAF. There is another one for RASP as well. Okay, so with WFT, I think it's different than RASP. Obviously, it's outside the application and so it depends what you're trying to get from it. And I think understanding what sort of data the W is currently catching and it differs for every application and every company. So it would it has to be a little more um tailored honestly. But I think it depends. You have to understand kind of what you're actively blocking. What attacks are you seeing the most and

understanding why are you seeing those sorts of attacks? Does that mean you need a rasp on top? Does that mean you need to go back in the code and see if there's a place to make the code a little more secure? But I think starting to really pinpoint what types of attacks you're seeing. Thank you. And our next question, thoughts on build versus buy RASP tools. I think it's whatever works for you. I think building your own tool probably takes a lot of time, a lot of energy if you're able to and if that's something that your company and if you don't want to add an external agent and you're only comfortable with adding one

that you've written, I think that's great. However, I also think if you buy it, that's what companies do. That's what they focus on. So it might save you, it might cost more money now, but it might save you time in the long run of having to actually develop it. But again, I think it's whatever fits your business the best. And for for the most recent adoption of rafts, how are folks ensuring it isn't introducing new vulnerabilities or stability issues? So, I personally recommend starting with just monitoring vulnerabilities with a RASP, seeing what is being identified and going from there. If we're starting to see a lot of different types of attacks, then I would start to tailor

the RASP tools and the RASP rules to actually then start to say, well, if I'm seeing X amount of SSRF attacks, well, maybe I'm going to start with just blocking those. I wouldn't immediately turn on all blocking for every existing RASP rule. And so I think again seeing what your application is doing and just knowing what's working best for you because unfortunately every application is different. Every company's different. You know a financial company with a financial website probably has different attackers than a you know food service website. So again it all depends. However, I definitely would start with just monitoring before blocking anything. Got it. Thank you so much, Kennedy. That's all we have time for.

Thank you so much for the presentation.