Pentesting on steroids using performance monitoring

foreign [Applause]

[Music] testing and how we can Bridge it with an observability and performance monitoring which is not like an object but we'll see how we can do fun things with that just to worry about me so I'm working as a abstract engineer at data dog I will CTO and co-founder at screen so we built application security from within applications we go to Korean by data log in 2021st previously I worked at the at Apple in the in the red team and even before I was I was a pen tester you can find me on Twitter via email and really open to questions but we'll have some time for that in the end and we have a good data Booth right there so

feel free to step by and and say hi to me on the or to the watching so first question what is performance monitoring sounds like a silly question right pretty pretty obvious but actually is the right question no no the real question is what is observability right so performance monitoring is one part of observability that's the art or science to understand when you are running a production when you have servers um when you have Services talking together is the art to understand how the system is behaving is it doing well is it feeling unwell do you have bugs do we have good performance are the customers happy if we have a bug how can we troubleshoot it that's what

performance monitoring and observability is about so it has little to do with security right you might say availability is a part of security you would be right so yes a little bit but beside that traditionally not so much but things are changing we as a as an industry as researchers as practitioners are starting to understand that there is a bridge to be done between observability and security and so um this bridge is starting to become a reality in a lot of defensive applications and now I'm gonna talk about how we can apply it to offensive applications so why do engineering teams care about observability as I said it's very important to understand the health of your system and

the behavior of your systems so let's assume you are develops in a company you are running maybe operating maybe 30 50 applications or microservices or maybe serverless functions you'll rise maybe a year ago so you know a big part of it but you don't know all of it obviously you didn't write all the code you don't know all the databases or third-party tools that that those things are using right so when you are a system engineer the complexity of the thing you're dealing with is insane and it's very hard to grasp because you didn't build it and you have a team of Engineers who are changing it every so often right like they deploy every hour they are using to

reform to change stuff on the cloud you don't even know my God it's a mess but uh how do you you are still responsible for the things to behave well right so you need tools to help you visualize understand how things are talking to each other how to debug it to monitor the deviations of the of the system and so on the right you have a small picture I didn't find the attribution of that one so I'm sorry for the author uh but yes load balancers Ingress to proxy service mesh sidecar and in the end application those are a lot of layers right that's a lot of opportunities for something to go wrong as well

and so when you attack a complex systems because that's why I'm here today to talk about offensive stuff when you attack complex systems well a lot of the needs that you have are similar to the one that Engineers develops as a rehab right you want to understand the system to accelerate your we can you would like to have facts instead of hypothesis why am I getting a 500 when I'm putting a code here or an internal IP well those are a lot of questions that it's hard to understand it's hard to get a clear answer to obviously and that's the job of of pen testers so you would love to be able to observe the response of internal systems

when you run stimuli on them right maybe discover hidden components um who in the room he's doing pen testing okay a fraction of this good who already dealt with pen testers who already received the pen test report all right all right all right so you will know what I'm what I'm talking about so how could we do this right and get answers to all the questions that are burning uh when we are trying to attack the system so the question is not how do we do this as security practitioners the question I want to answer first is how do product Engineers do this product engineer meaning the team that is making the system alive making

your production alive make ensuring that your services and applications and apis are behaving well so the the focus of the talk is around modern observability products right so um usually we distinguish two things but another way to put it is that they give you full stack visibility so full stack is is a big one right it's a bit almost gross to say it you have that olive oil LinkedIn Twitter any recruiter is looking for a full stack developer so it's it's not what I mean here what I mean full stack is from the kernel level up to the to the API Behavior right everything that is happening within um your infrastructure from the cloud provider to the application itself and

those observability tools they can give you a vision around all of that and that's very important because when you want to make sure that the system uh is alive you want to know about the container orchestrator how it's configured and how is it performant the host and container applications the data stores and what we call the cloud control plane how is your Cloud configured with users with security settings with system configuration so you have many path to get there all right so you have many products whether they are open source or commercial and here I want to do a small disclaimer I work at datadog but I will use data log as an example a lot

because that's the one I know best but you have all of those products and platforms that allow you to do something very similar so whenever you hear data dog you can replace it in your mind with any of those other things and that works in the same way all right who already used one of the products listed here okay quite a bit of you who already used one of those products in a security context okay good quite a bit who already used any of this product in an offensive security context two here three here is that it four higher higher if I didn't count you yet okay please come and we'll have a chat after okay not right now

um okay amazing so among that you have some that are open source that you can set up yourself I think we might be missing elastic I think which is uh uh open source as well but well that's that's the idea I'm not saying this is exhaustive so let's take a quick look at what actually is an observability product right so this is um this is data disclaimer you can do similar things with a lot of other products right um so here we have all of our hosts sorted by availability Zone we can see that some are misbehaving right we can see OST we can see a information about the networks and the on each of these hosts

uh when we have matching results which is the demo effect of course um you can get information about the processes that are running within that host and and so that's kind of things you need when you want to investigate why something is good is going wrong with your with your production so that's really interesting if you are an UPS or a necessary or devops and to get this level of information you are a developer but also if you are an apps you might want to go at a higher level and understand what services are running on those hosts what apis are running on those hosts so thanks to observability products you can get this information because you the

observability product will understand how services are talking together what kind of databases they are relying onto what kind of requests they are receiving and you can deep dive on many of the of the full stack layers of your of your organization and of your production system so um thanks to that developers can really deep dive and see okay I have a trace so I can see how the request flew from end to end on a given service I can relate that to the infrastructure and really thanks to observability products you get all of the context of your production systems so that sounds amazing very interesting you can even go beyond because for instance you have

logs right so so you can look at what your systems are doing from uh from from a login standpoint but uh if we have all of this information why aren't we using it from a security standpoint if you if you are familiar with I don't know such analysts who are reading alerts from SIM hours per day they had everything at their fingertip like that wouldn't that accelerate them if we were able with uh instead of choosing like uh Rue HTTP traffic to correlate attacks with runtime information couldn't we accelerate things that's the idea and that's what I want to show you it's how we will do that from a an offensive standpoint so let's focus on distributed tracing

you've seen the huge service map where I can tell you which service is talking to what database what cache what other services and so the question is okay how does distributed tracing works so let's assume that abcdr Services one of them B is using redis the goal is to reconstruct the map that you have on the right so you know how performant they are how much time was spent in each service so with that visibility you know that if for instance C is starting to have performance issues you can easily pinpoint the issue because you have the performance of each of the service um so how does that work in practice let's assume that you have a customer

request that is entering your system okay in each of the applications so a b c d each navigation you have what we call a library okay and the library will look into the incoming incoming requests if they see a trace ID they will propagate it to the subsequent libraries right um so let's assume that we have a trace ID that is X initially the library on the service a didn't see and it Tracy this will generate a random one and propagate it everywhere the next thing that happens is that this is centralized to an observability product and the observability product is able to reconstruct a trace okay so read the graph of how or the tree of how the

services are communicating together so APM is Magic no it feels magic the the first time I installed an APM and an application I was developing an ISO details about the request the database queries performance information about the the other us that my code was fetching it felt incredible um but obviously it's not magic and you have some limitations for instance if the communication between B and C is not done with HTTP but with something else well the APM need to understand that something else so it can propagate the trace ID into that if it's not able to do it well you will lose the information and so you will get like a broken Trace within you within

your dashboards and obviously um regarding all of the traces as a cost so you need some sampling so you will not be able to gather 100 of the traffic okay those are kind of the limitation of those tools and I'm not even talking about deployment because um if you want to get started with an APM in a large company well you need to convince developers to install a live a new library on all of their applications so developers are requesting this kind of things because since the systems are more and more distributed they make it um those tools makes it much more easier to visualize and debug and understand so now we want to pen test

with observability are you are you steadily starting to see where I'm going here um into an APM Library so who already used Bob suit here okay who has no idea what burp suit is no shame no shame but I don't see a much people raising their hands Last Chance who has no idea what this Bob should okay okay good um so just you know what if you are pin testing an application well you want to be able to use your browser to use the app because I don't know react angular it's you cannot like simulate a browser it's very hard so you use a browser to interact with the app and burp will act as a relay between

your brother and the application that you are pen testing so it will intercept all the API calls it will show them to you and I think actually I have a yes and it will intercept all the apis call show the API calls to you you will be able to inspect them modify them replay them and so that's how you can inspect the requests and the response of the application so that's a very useful tool in order to to understand actually what you are doing during uh during a pen test so what I built is a is a very simple burp extension that is actually using APM mechanisms in order to inject a trace ID within any of the requests

that burp is uh is receiving and so that Trace ID if you are pen testing an application that is using one of those observability products will be propagated everywhere and so later you can find it back in that product that means that when you are doing a test you can leverage all the information from the from the observability vendor so it means that as an offensive engineer you will get all of this information uh so who can use this obviously you need to have um a system so you know what maybe we can talk about who can use this after so let me live redo my slides eh you don't see that often right mini

there oh no it was full screen I'm sorry

can someone sing a song while I'm doing that thank you so much uh all right so let's take a real use case and let's investigate how this could uh help us with for instance a second injection um so what I have is a vulnerable web application here okay it's web Goods it's it's the U.S per Java one super super standard what I did is that my Firefox is configured to use burp okay so the reverse proxy we discussed previously here I have burp who is using the APM to burp extension which is open source then I share the link after in the slides and so while we are using verp in a in a service sorry in approximate all

the requests are intercepted and if we move to the edited request we can see that we injected a trace ID okay so the idea is that you can very easily use the extension to open the trace ID in the APN takes a bit of time to load because I'm in 4G and I'm not on the right organization so I will just need to change it right here demo effect second time I'm sorry attack ta-da here we have the details about the request that was found into verb okay so this one is a random one it's not super interesting now let's focus on on trying to to do like a secret injection okay so um since we are all advanced Security

Experts in the room who is not who's kidding no tensor I don't want to know uh we can just try this so this one okay goal can you log in as term straightforward it's a typical secret injection right so uh not super funny I'm gonna do something like that um okay space login oh didn't work all right uh so maybe I'm gonna try quote and quote to see if any bug is triggered nothing okay so here I don't understand uh it's not straightforward to do to to fill that challenge so I'm looking at my Trace those are the codes that I injected and I'm gonna open it guess where in an observability product here I am I can go to the Spanish which

has all the information about that request is the is it big enough and I'm gonna zoom in a bit is better oh yes thank you and so we can see okay a lot of physical queries that we don't care about but this one this one seems to be the one we we try to inject and we can see that like uh it doesn't seem that the injection worked so let's get back to it and actually we have a register tab so maybe here I will have more look let's try register now oh I need a valid email address that's my red one register oh okay something went wrong so this time we can investigate again so

I'm gonna use that one and again open it in the APM

all right so I'm gonna get back to the Spanish list and what do I see oh here what do we have user ID equal question mark so it's plain select but I have an error so in one click I'm able to see the exception generated within the application by my request how amazing is that how would that make pen testing easier if we had access to exceptions a lot so um very very simply I can get all of that information since I've got the shape of the SQL query that is used by the application I can really deep dive and as I'm approaching my actual attack I can understand how the application is is changing things and already that being

having the shape of the secret query will help me uh will help me a lot um so that's uh we trying to find the slides that's how we can we can go further with secret injection so to help like the errors and the shape of the secretary that we have and you can think that it's the same for any database that is supported by the observability platform that you are using for a mongodbenization for Cassandra or dynamodb or whatever any kind of uh injectable query language will will be displayed in those in those observations provided the theme um social injection is cool but can we do something else ssrf server side request forgery um who who has a new crew what ssrf is

don't be shy okay come on what are you doing here I'm kidding okay so ssrf is when you have a an application that takes as input a URL maybe because the application of how to configure a Web book or maybe because the application allows to fetch a URL an archive or I don't know what if the URL is not validated properly or if the application isn't secured against ssrf because that can open a lot of debates the attacker can put a private IP address within the application right and so instead of fetching something uh innocent uh remotely the application might fetch something on localhost something on the cloud providers infra something on I don't know the

administrative interface of your web application firewall so it's it's it's an attack where and that occur will perform marriages United actions within your network thanks to uh thanks to uh Miss sanitized URL field so a good thing we have sorry we have um a server-side request for jury attack in the in this challenge so let's try that um I think this one yeah okay so we need to steal the cheese extremely fun fact I'm a french guy but I don't eat cheese still I'm gonna demo that to you I know how you write um so here we can see that we have a URL okay um images term that PNG let's try with a random URL and see if something

works on it also I need to go in intercept mode that's a burp demo as well I will replay the thing uh this one I drop websocket I don't care okay there I have it okay so let's try to replace that with a urine and what could we do like that invalid IP address um that should trigger an error at least right we can for one whoop and I will remove the interception and so what happened okay so you need to stick to the game plan so apparently it didn't work I can go into the HTTP history I will find my ssrf query we can see that it's the injected one and uh okay so I have no sign of a

neuron or anything what am I gonna do open that in an observability platform of course I open in APM and so what we can see is nothing no error okay we can look at the span list there is nothing that looks like an HTTP request Okay so it's just an ssrf that is only doing internal local file queries we got hinted by the the shape of the initial argument but here very easily we can tell okay there is no need into trying to extract that in in this way you cannot reach external servers with that feature since the application is not even doing an HTTP query interesting right so we we like we win hours of work uh by looking at the thing

rather than looking at the uh trying to guess and by trial and error what you could even do and go beyond uh is look at the network so I have no network data because uh I'm on Mac OS but if you are using that on your Linux systems you would be able to see the shape of the network so you would be able to tell whether you are within a wsvpc whether imds V2 is configured or not and how you can exploit actually this this ssrf

so this is uh what we what we can tell from that and we can even have code insights going Beyond um what the APM Library instruments explicitly here you can get information about anything so the way it works is that most of the time since observability products are used by teams that need to have performance inside you will have features that allow um to get very very precise performance insight about your systems and here we have the profiling so um a profiler is a a simple that just took every uh so fun like every 50 or 10 milliseconds it takes all the stack traces of the processes that are running and it will form a tip to you so here the stack

traces taken are the one of your web app or API so we can open it and we can see the profile for that specific service and we have no profiles found which is again the demo effect so I'm gonna get back to my other um no sorry to My Demo organization yes

there so since it's statistical it might be that we didn't choose the app enough obviously it's the app that I'm running locally I didn't do a lot of requests on it so it's very possible that the the profiler was blind to it I can open a given service let's assume that's the one we want to uh we we are currently attacking and so I get a lot of information about the the performance but that's not really what I'm interested about what I'm interested about is all those uh low-level information about what code is actually running and what the app is actually doing if I scroll up I can even saw this information by Library okay so I get

information about all of the libraries that are running on that platform I useful is that let's assume that you you have a hint and you want to exploit a given cve on one application well if you want to know if the library the vulnerable library is actually running on it you can use such a tool to get this information and very easily you can even pick your Target and choose amongst all the apis you have at your disposal which one is actually running the vulnerable version of the service I'm sure it happened to you all the time trying to exploit I don't know look for Shell or any kind of Tomcat CV or or whatever and not finding uh the right

the right target for that so again with leveraging all the tools that developers apps SRE are using we can get very very important insights that are useful from from an offensive security standpoint

there so that's that's interesting but who can use this obviously you cannot use that in a in a random pen test because you need to have access to the observability platform in order to to get those Insight right uh but that's extremely useful for red teams that are testing internal systems if your company is using an observability product then very easily you can get an access legally as your as your team and and and use it as a source of information for you for all of your tests um so leverage those internal tools you can also discover blind spots right because um you can know what is not monitored within a company if it's not on that

tool most likely people don't really know about it or don't have alerting on it so those are systems that you can use to build persistence that's uh not so nice but red teamers are not nice I know I was one um it's also useful if you are doing like open box testing with customers right with you know access to the source code with meetings where people explain you their systems Etc so if that's the case you should ask to your customers access to those tools if you are um performing pen test for your company offer European testers to use those tools because that will accelerate them a lot and you know there is always a

time when you you sit with them in a room and you explain them what you have and and that's that's painful I remember as a pen tester this time it's often very hard to understand complex systems if you are looking at something visual where you have actually names and something you can relate to that will accelerate the the communication with the pen testers a lot and obviously if you are doing closed box testing well you can still access to such an observability product Maybe by doing recognition on stack Overflow you can find the UPS or the dev of a company may be looking on GitHub and you can still access this to their observability platform so

if if you manage to do that that means that you have a lot of information about the system is almost as if you were an Insider to the to the company so I think that's that's a of an under uh rated way to access uh systems but I'm guessing we will see more and more things about that in the in the future so I've talked of the of the present right and I give you uh three examples of of what we can do with the systems but um this space is moving super fast because you know we have more and more of code everywhere software is eating the work performance and reliability and quality is more and more important for all of us

so those systems are evolving in terms of quality a lot and so what I think we're going to be able to do in the future that will even accelerate uh security teams is for instance we've seen profiles you've seen like raw functions in that we will be able to link them to URL to path to routes to SQL queries so you will have a matching between what external system and application in doing down to the line of code that is actually performing those queries you will also get information about vulnerable libraries right you you so we can already access the libraries uh well knowing which vulnerability is running where when and on what traces it's

already I think a few vendors are already doing it it's it's like super close vulnerability detection we have tools that are using uh for instance uh um source code not source code but Trend time insights in order to build a vulnerability detection those things could come within those products because that's again something that is useful for developers and that will also be extremely useful for security teams um and last but not least the source code integration because yes um if you can access the source code of those things that means you can very easily pinpoint a commit to a given um vulnerability for instance but also if you have a behavior that you cannot explain as a tester then using the

source code will help you deep dive deeper and get the the full understanding of what the application is doing alternative so you have all of that I uh didn't use most of them I'm familiar with with them a bit but those are things you can you can look you can look at if you want to get started with the with the open source this is it if you have any questions I'll take it otherwise you have this QR code with all the resources and Links of that presentation the source code for the burp extension link to datadog which has a free trial if you want to test it and now to detect we are hiring we have an

office in Lisbon Paris New York Madrid and and we we take remote job Affairs if if you we have a boot just right there feel free to to jump in and say hi any questions timah any questions

so much for your presentation thank you if anyone who [Applause] foreign

observability product and I'm very interested to to talk to you right now

[Applause] [Music]