Trace to Triage: How to Connect Product Vulnerabilities to Security Paths

So here with our next talk, I'd like to welcome Ben Stav and he's going to be presenting from trace to triage. But if you can go back one slide, I do want to point out one important thing. We do all our questions here not through the audience, but through this app called Slido. So if you go to slide.do D O and type in the keyword Bsides SF2025. Again, besides SF2025, it'll get you straight access to this room. We're in theater 6. Very important. Theater 6. That'll get you to the right spot. And you can type in your questions as they occur to you. And if they're really popular, if you see a question that fits what you want to ask,

just upvote it, and we'll make certain that that one gets answered in the first set if there are. All right. Well, with that, let's give it a go. hand it over to Ben. Thank you. So, hello. Hello. Hello. Um, I'm Ben, uh, heading architecture at Mingo. So, previously I've done sentence like, uh, application security, more specifically web application security. Um, a lot of engineering, uh, system design, architecture, but really at heart and like a JavaScript kid. Um and yeah right now our architecture at MIGO which is an ADR platform uh that stands for application in detection response marketing will hate me for saying this but basically we do observability for security um and that basically means that we help

organizations understand what's happening in their uh production so they can confidently and quickly respond. Now, before we even start talking about what we're going to talk about, I'd like to do something with you first. I'd like us all to do a deep collective breath. Can Can you do that with me, please? So, let's

go. That's That was nice, wasn't it? Thank you for doing that with me because apps hard. Like, for real. like it's like 10 different disciplines in a trench coat. You have so many stakeholders. You have like your boss and your teammates obviously, but there's also engineering and operations and devops and you have like finance and compliance and you have P 0 thrown at you from all over you prioritization and triage and it's a lot to unpack and we're not going to unpack it all in 45 days, let alone like the 33 minutes that we have left. So, we're going to focus on one specific aspect of AppSac and we're going to figure out how to do that

better and that is like making sure you have everything that you need to respond when things get real. So, here's what we're going to do. We're going to do three things. We're going to figure out what your applications are composed of. We're going to make your life easier. And we're going to make sure you have absolutely everything that you need. And it's going to be like with a small demo of an RC. So sacrifices to the demo gods are appropriate now. Okay. So picture this. It's day zero. You've just joined a new organization. There's nothing in place. There's like no security in place. And so pawn to E3, your move. What do you do? Well, for for me and like my hat both in

architecture and development and security, I like to just collect a lot of stuff and then iterate and figure out. That's like the datadriven approach. Yay. So, what does that mean in security, especially application security? That's inventorying. That is understanding what you actually have so you can do your job and protect it. And that's like two things. You got to figure out what you even care about. And then you actually got to gather that all over and over and over and over again. So what do you care about? Well, it's appsac. That's application security. Obviously you care about applications. Okay, cool. What are applications? Well, there's everything in the app store. But that's not quite is it that's not quite it, right?

because you you're a B2B SAS. So for you an application is something else entirely or you could be like selling vending machines. So for you the definition of an application is vastly different than what I would have anticipated. But something that I think we can all agree on is that we have a bunch of services which are encapsulations of some logic. They could be talking to one another. And there's the outside world. So there's data coming in and potentially interacting with your bunch of services. And there's data come going out interacting with the outside world. And there's some shared infrastructure going over like that this is running on. And these are all the things that you have

to care about when you care about application security. But that is not all when you're doing apps, right? because you're not just doing the deployment, you're in charge of the full SDLC. So you've got a bunch of developers, they're pushing code to some repo that is being pushed out to some artifacts that's being shipped to some hardware. Now, the exact diagram is probably not this for your organization. It's probably like 700 different steps that are intertwined in some way. Maybe there's even people, you know, like shipping things out from their machines directly or stuff like that. But these are also things that you need to care about, right? Um, but that is not all because

we as security personnel also create artifacts that you need to keep track of. These are like examples for pentesting and auditing, but there's also like vulnerability management. There's scanning results. There's a lot of findings that we need to keep track of in order to actually do our job. Okay. So, how do you actually do that? It's a bit difficult, isn't it? So, there's a lot of things that we want to keep track of when uh when we're actually inventorying. Um, and how do you do that? So I've picked one example which is an API inventory that we all if we're like doing especially web applications just have to do and there are quite a few methods for doing that

right you can go ahead and like go uh and dump some API schemas if that's relevant maybe you go dumpster diving through your source code right and you go like oh so that's how you define routes of uh and fast API who knew uh you prowl through whatever existing tooling you already have maybe there's something in place here. Um a source that I will visit on later but I find over and over often overlooked is monitoring because even if your application if you're sorry if your business doesn't care about security they probably care about uptime or some measure of performance. So they may have dashboards available for like hey is my site up or hey what's my P95 latency to

make sure that it's not 10 minutes. Finally you may find some overzealous uh engineering manager or something like that that documented everything in like an Excel sheet. Um and you it may be a bit better than strewn over like a 100 pages in Confluence. uh but make friends with that guy. So you've done that. You've collected everything from multiple places. So you've understood that nobody actually understands what's going on, right? There's organizational chaos all over. There's so many gaps and developers may not necessarily understand what is actually being shipped to production. DevOps may not completely understand what what's actually the intent here. And that's part of your job to reduce some of this chaos and not add to it. Um, and again,

it's like very easy to understand how this happens once you go past the two people in the garage working on something small. What you need to do at this point is you got to automate all everything. Like everything that we've said, make it work on your schedule. everything that you've collected manually, you need to buy or build a tool that does that. So when we're talking, for example, about API inventorying, then if you have if you're the one who went over the source code and extracted endpoints, there are tools to do that for you. Or if you're the ones who are dumping open API schemas and validating them and doing all kinds of stuff, make

that run on the schedule. Cool. Now you've gained under an understanding of what your applications are composed of and audits and p tests are like your come to Jesus moment. That's when like you get uh a feeling for how right you were like was my understanding correct? Did I build up the correct mental model? Um I got a a result a finding of some sort. Was this in a place that I expected like oh yeah this is like iffy. We're going to we're going to be busted there. or is it like oh I completely overlook this aspect and that's like the datadriven approach you've you've got some new findings you've inventorying them you're making sure to incorporate that inside of your

process so you have your inventories everyone's you know heranging inside of the corridors turns out that you've got all of these and they kind of suck um and that stems for a few reasons. Tools are very very different from one another and applications are very very different from one another. Just as an example, let's say that you have a company down the hall who does like CRM for kittens. Amazing idea. They're running uh virtual machines on like Java monoliths, what have you. They care about listing EC2 instances and subnets and VPCs and all of that. Okay, there are lots of tools to do that. But you you're doing CRM for puppies. Awesome idea. And what you are

running is like autoscaled hosted Kubernetes and all of that. So you don't really care about like your virtual machines. They're cattle to you. Um and your the subnetting that you really care about is like inside of Kubernetes, not outside of it. Um, so for you listing EC2s is just is just pure noise. Like you don't really care. And down the hall from you, like on the other side of the street, maybe there's a company who does CRM for lizards and they're running all like Lambda Edge. So for them, the concept of listing EC2s and subnets is just ludicrous. Like what what are you even talking about? So for them, a tool that does the very easy and

understandable thing of let's list out all of your AC2s is just just not interesting. There's also the aspect that tools really don't want to talk to one another. Like again API inventorying, you have two things, just two things. You have source code scans and you have scans from the outside like runtime something hitting your endpoints. So, you've got two lists, right? How do you correlate them? The tools are likely to not be in your favor here. Like the the API scanning tool from the internet, they don't really understand how source code defines endpoints. They're not going to match that up. So, that is mostly le left to you to understand how to do all these

correlations. And so that sucks, but we're the users. Like all of us here are the users and we have budgets and we can choose how to spend them or not to spend them on vendors, right? So as users, you got to demand better tools. Like as an industry, security is not very very good at cooperation. And that's something that we need to improve. uh we're in a better place and we're going to talk a bit about that but looking over the last like 5 10 years we've started making standards so we've started understanding like hey how are we going to list out vulnerabilities or you know dependencies suddenly an SBOM has a format wow that

allows me to vendor shop and have like keep my data with me that sounds exciting and boring at the same time because it should be the obvious thing. So that's something that we have to demand of our vendors despite all of these um how can you still build quickly? So I'm going to offer like four things that you can do to still iterate and make sure you have everything that you need. So first, as we've mentioned, use tools that you actually understand. If your lizard's building a CRM and you don't need something listing virtual machines, don't use it. Just stop. You don't have to. Nobody's forcing you. Use your budget in a way that's appropriate for

you. Use the tools that will actually help you and your team to rapidly succeed. Second, as we've mentioned, automate everything. Make things run in a schedule. Make it run cron. Make it run in CI. Like make the robots work for you. the robot over overlords will uprise and throw us. So, we got to use our time on this earth and while we can still lord over them. Third, and that is more soft, you got to make sure you have everything that you need to affect processes inside of your company because if you have a methodology or a vendor that you want to use, but you don't have uh the capital to actually implement that, then you'll

slow things down. You got to make sure that you can get to the point where you can affect changes and you can affect life cycles so that you can get results because once you get results, it's easier to sell things inside your organization because you're saying, "Hey, I did a thing and it made your life better. So, I'm going to do more things." Okay, cool. And fourth, this is something that we really need to be better at is building relationships inside of a company. Like historically uh security and DevOps and engineering are like a painful triangle of distrust and you know some grumbly feelings. It doesn't have to be that way. Like working together and making sure that we're all

on the same page and we're all striving towards success. Important. Now one tool for actually doing that is to understand that what we do is cool and to actually talk to people and you know you found some cool stuff out not approach it from a haha you suck angle but like hey I found this cool thing let me share this with you and when you approach people who really enjoy your craft and you can tell them hey I found this cool thing they'll be interested in it because we like cool shiny things, right? Um, and like they'll also share stuff with you so you can be better understand how things are actually running so you can do your job

better. And this creates like an interesting thing, a dynamic where you can actually help uh people inside of your organization be better security people. So you don't have to do your job as much because other people are doing it for you. Because if developers are being better security people and if devops are being better security people you can deal with other stuff and that's awesome. So I'm mentioning all of this like yeah just do things. Yeah but it's actually hard. It's actually difficult especially if there's already like entrenching of how things work. So yeah, all of this and all of that is like dev sec ops and all of that. It's it's difficult. It takes time. It's not done

over a day. Moving on. So we've had our inventories, we've built understanding, we've made friends with everyone, and now we're in like the dry run. Now we're in like the okay the interesting territory of you've got pentest results or you just got audited or you just integrated a new tool and you have a bazillion new findings. What do you do? Well, if you've ever looked at like a list of like an audit or appendix or whatever, you just immediately got the feeling that yeah, I don't care about all of these. Like some of these are interesting and some of these are just yeah okay whatever and it is up to you to prioritize and give an understanding

of like what's a P 0 and what's a backlog item and you need to have the tools and knowledge in order to both for you internally understand this difference but also talk uh your organization into understanding the difference because if you're marking something as a P 0 you want to make sure you have data to back it So have like a few things that help you evaluate a finding. So you first you have to understand what it even is. Then you need to understand the impact that this has. You got to assess the risk that it poses to your organization. And then you have like two decisions to make. First, if you want to delegate

this to be fixed like you do it, your team does it, uh you're opening a ticket, whatever the process is for your organization. And then you have to decide if you want to mitigate it in the meanwhile. Because for example, if this is r truly a P 0 everything is on fire, you may not want to wait until it's fixed so it to not, you know, to not be vulnerable to this. So either it's like a configuration of some sort or an alert of some sort, whatever you need to do to make sure that you're not impacted. So how how do we actually get the data we need to like answer these questions to understand all of these here like

context is king just you got a finding we have like a visceral understanding of our environment. So just the answer for which service is this in like already gives you either like chills down your spine or okay whatever because if this is like your most public most external most you know vulnerable most like oh no application or service that's very different from oh yeah that's where someone orders sandwiches and the whatever and it's behind us who cares. Um, then you go through like is this something that's locked down like behind seven VPNs and a thousand security rules and all of that or is this like uh oh yeah, we'll get to it when we get there

and you never got there. And finally, what what is it accessing? What kind of data is is passing through this? Because you know even thinking about defense and depth you want to make sure that even if this is like a mild vulnerability if this were to be exploited and done uh you know pushed into something larger how screwed are you like is this the most sensitive that you're in your organization or is this like okay is this possible to do some interesting lateral movement for this or will this be okay? Um, now how accessible is this and can just anyone exploit this? Like can I open up a browser tab and just get to the thing and run it or do I need

like specific admin credentials and all that? So this is just a very very very partial list. Um, it's it should be like more specific to your organization and how you run things and how you define applications. As we said, it's like a very specific thing. So, okay. So, we got our findings. We understood what we need to answer. Well, how do we answer these questions? So, we can do it like the good old days. You can go dumpster diving for logs. You can read code out like say, okay, so this calls that calls this. No, this was not passed through the middleware blah blah blah. Um, maybe you have reproductions. So if your auditors were amazing, they'll give you

some way to repro this. And if you got friends in engineering, then maybe you can even run this locally and step through with a debugger. Oh, that's always fun. Um, then you can do stuff like analyzing your security groups in place and all of that. And then you were like, "Oh no, I did the thing for staging." And of course like staging and production are different. and just want to yell out to the sky because that's like hard and there's just so many things to answer like a basic list of like I got a finding so I need to do something. It's just it's hard. So uh I'm going to propose something uh as part of our automating

all of the things. Remember how like 10 years ago we started inventorying our entire environment and understanding what we have. So let's use this data. let's make it work for us. And what I'm going to do is I'm going to take an example from modern observability and show you like two tools prof profiling and tracing to show you how like you can get a lot of these answers um with maybe tooling that you already have. So it's okay if you're not familiar with profiling and tracing. I'm going to bit through and and explain what they are. Let's start with profiling, especially sampling profiling. And that is relatively easier to understand because it's stack traces, just a bunch of stack traces. Um, that's

sort of how it looks like. So, what we're seeing here, you have something that goes to your application and asks, hey, what functions are you running? And you can see like a list of, oh, this calls, this calls, this calls, this calls, and blah blah blah blah blah. And that is quite cool. The original purpose of this as an observability tool is to answer questions like what's taking so long? Like do you have a specific query that's taking you like you know a minute to run or that's a specific function that's like quadratic complexity and does a bunch of stuff. Um, but dunning our security hat, we can do something a bit different, which is understanding what's

actually happening inside of our applications at runtime. And that's kind of cool because remember how we like dived into the source code either to understand what's actually going on or to do some reproduction or whatever like you know how you go through code and you build a metal model for who's calling who. that's just can be done for you with actual data inside of your production. And especially like over the past few years, the concept of continuous profiling, which is running these this kind of thing without a lot of performance overhead, has gained a lot of traction. So this is a thing and that's kind of cool because you may already have this running. Um so profiling gives us some depth an

understanding of what happened. Now tracing is both a very simple thing and it could be a bit complex. So work with me here. What we have is a bunch of events. So let's go through like a sample request and figure out what's going on there. So, we have our service and it's like, oh yeah, I'm listening to slashfu and I just got a get uh a get request. I'm the server. I'm on service A. We just started out as part of handling this query. We're executing like a DB statement. Cool. So, we're handling the request. Inside of it, we're running some query. Cool. Query is done. We're doing like an HTTP request to another internal service

called service B. Now service B is handling this request and it's doing some logic. So we had a request, it executed DBquery and then executed an HP request and now we're handling this HP request. And once that's done, we have like a map of who's calling who. So an understanding of what's actually happening again a different kind of understanding of what's actually happening inside of production and which service is calling which servers and how. So that's how it looks more in real life. Um so what we're seeing here is we have a shop back end that's our service. It's doing a bunch of stuff and then it's calling cart service whatever and then it's calling an authentication and

that it's calling like an article service which ends up calling pro posgress and then and so on so on so on now immediately when I looked at this trace I thought wait so it's calling authenticate after calling cart service interesting so like even just using observability tooling with like a security perspective you can gain things that were not really intended in the first place. So you can use these tools. Um is like okay yeah so something that's really cool is that inside each of these events you have a lot of data. So looking at this is uh an HP server. So we're seeing like oh so this was a post request to this thing and that's the URL and here's some stuff

about the client that's connected to us. So, we're getting a lot of the context that we needed to understand what's going on. And like this is like just one of the most popular open source tools for this. And it just right off the bat gives us something that is observability 101. Who's calling who? How many requests per second were happen? What's the average latency? But how many of us like yearned to have this simple a map to do like threat modeling, right? Like this is something that we I I I looked at it and I was like, "Huh?" So tracing gives you the breadth like who's talking to who about what. And combining these two together,

we have an interesting uh perspective. Again, observability tooling meant more for understanding, you know, the latency and debugging and the ins and outs of the system looked through a different lens, a lens of security. And when I first like learned about these, I learned about these from, you know, a developer hat and I was like, "Oh, yeah, cool." But then I done the security hat and I was like, "Wait, what?" Because let's take profiles like how many of us espe even like it could be when you're doing like finding analysis but it also could be when you're doing vone research on your own building this understanding of which function is calling which especially in dynamic languages but not only even just

in Java oh we have magic reflection because spring does things it's just it's a lot of pain and you can have this kind of understanding for free or like tracing. So, how many of us are like, "Yeah, we know that someone has built like a service map somehow and it's behind some bearded guy on a whiteboard, but we can get that for free." And that made me mad because this has been going on for years behind our back and nobody told us. And not only that, there's standards like these things have standards. Like the largest one is open telemetry. Um, and like they're a huge CNCF project and they're like saying, "Oh yeah, so that's how you create

traces and that's how you export them somewhere." And you can vendor shop. There are multiple vendors for creating and ingesting and visualizing traces and profiles and metrics and a whole bunch of stuff that I I at least I personally was not aware of. So yeah, I got envious and I got mad. So let's ruin someone's day. What we're going to do now is channel our energy and we're going to run an RC. Um what we have here is we have uh the open telemetry demo. Uh so that's like a large project for to demo like some of their uh capabilities. They've made like a nice microservices uh um you know uh shop. So you have the

thing and you have products and you have recommendations and checkout and advertising and all of that. So that's cool. So, I took this very nice project and I injected vulnerabilis vulnerabilities into it because we destroy everything that we touch. Um, and now we're going to run the rce. Now, I'm a bit excited because what I did is I wrote like the exploit and it was like, you know, five lines of code and then I gave it to an LLM and I asked it to make it more bro and I did that 10 times. So, I'm very excited to share the result with

you. We're in. So, yeah, we can do like we can do like an ls and we can do like a pwd. Um, anyone run a run some command? ID. ID minus a. Cool. An audience member suggested RMRF. I'm not that adventurous right now. Um, so yeah, we're seeing it running as root, which is of course the best thing that you can do inside of a container, which will already tell you like, yeah, I need to have a conversation about security, which would not have happened if our better friends that. Yeah. So, we just ruined someone's day. We called some alerts. We had some fun. Um, so we're going to take off our blackhead now. I'm

sorry. And we're going to bring us back into uh our day has been ruined. So we need to understand someone has just done an RC on us and we have to understand what happened. So remember how we needed to answer all of these questions and understand like oh what happened where do risk assess blah blah blah. So what I'm going to do now is I'm going to show you how you can use like tracing and profiling in order to get a lot of this context. So here's an example of like okay so I just got this request uh it's a gRP safing here's like the IP here's the method nice that is very common now

remember how a profile is just a bunch of stack traces as Keano taught us so we're seeing that get personalized ad we're seeing like a script engine eval and process builder start for those of us who have been blessed to not have Java knowledge that means there's some uh templating engine and someone did something weird. So it can run arbitrary code and now someone ran some process and just below it we have like a trace view of what happened who called who and what did they do. So we're seeing this hitting our front end and there's some gateway and now we're in the get personalized ad there's some uh Postgress queries and we're seeing you

know some some data about like what was actually happening in this request. And not only that, because we were good people and we inventoried this over time, we have a lot of runtime context that was built up. So we know that we're in Kubernetes. So remember how we had to understand like what the impact is and what could be potentially done. So cool. So this is the cluster. This is the name space. So yeah, you can use all of your required knowledge to answer things like, oh, is this like the PHI environment? So even if this is if this were more benign, would I still want to take care of this or is this something

more uh you know more benign or is this more interesting? Um and not only that because we've been been good people we can we could have you know inventoried this over time to gain understanding of like oh so this is my API inventory but it's also correlated to like here's my service maps and here's how they're talking to one another and these kinds of things will really help you once we get to the point where you know how after every incident there's the favorite thing of having like a postmortem and we're all looking forward to it to share and experience this together. Um, so we have to answer a lot of questions like what happened,

when, why. So we can use these kinds of things to answer these questions and because stack traces and traces are something that uh observability uh you know devops and technical people are familiar with you can just use this information and give them these stack traces and give them these traces and they'll figure out what to do. So apps hard. Um there's no way around it. Like it's difficult. You still have a bazillion stakeholders to answer to. You still have like way too many vendors who do the same thing and you can't quite distinguish between them. And you still have you need to do all of these triaging and all these prioritization over and over and over and over again.

But it's getting better. Um, especially if like you've been around for like 5 10 years, like the state-of-the-art isn't snort anymore. Like we have improved both in our tooling but also in our understanding of what application security means and especially like we're starting to make advances in collaboration both in like inside of ourselves like creating standards creating formats so we can interoperate and talk to one another. So uh thank you very much. Uh I hope this was interesting either as like a yeah this is cool kind of thing or I hope this has also inspired you to like either make changes inside of your organization or you know to push the needle a bit. Uh

now I believe we have some Q&A can still do the um the QR code uh for submitting questions and if you want to talk um here's my email feel free. All right. Thank you, Ben. So, again, this is a reminder, we're in theater 6 here. We take questions via Slido. There is one question in the DAC. Um, we actually got two. And a reminder, you just go to sli.do and enter in the keyword besides SF 2025 and you'll be able to put in a question. So, we've got two. Uh, our first one here is why can't APSSEAC give libraries or code that's sanitized or secured, removing vulnerabilities by design? Okay. So if I understand a question that

is an amazing one and like like show of voice how many here do crypto in some like cryptography in some way okay so some cryptography um let's take SQL injection SQL injection should not have existed in the first place um but what is the single most beneficial thing that has happened to eliminate SQL injections Prepared statements. Prepared statements and ORMs are the single best thing that have happened to prevent SQL injections because they make doing the right thing so easy. Like you need to make your life worse to do the right thing. And that is an issue of like API design and we're not very good API designers for the most part. like as an industry

we're taking some times and some cycles to understand what uh these what kind of vulnerabilities are there just take a look at how we uh I don't know take a look at like GraphQL queries we've managed to do the problem again take a look at like requests to NoSQL uh databases we've managed to do like NoSQL injections like it takes each generation uh its time to understand its mistakes of the pre and and the previous generation's mistakes. So I think like the most obvious thing to do would be make better uh APIs and make better design choices. All right, the next kind of question we got here is does this kind of data require an agent?

Good question. So there are a few approaches to this. Um if you're looking at you know how um engineering and devops usually use this there are a few methodologies one of them is to introduce some SDK inside of your program. So that is most like APM application uh uh performance monitoring solutions will do that like you do a require or an import or something like that and you can also establish like your own uh tracing or your own uh profiling events or whatever. Uh there are other options. Some of that is other instrumentation and that is something that you configure your environment in some way that in in like inject figure out how to inject these kinds of things

and over the past few years especially there's a movement towards uh eBPF based uh instrumentation which figures out uh especially in platforms uh like more specific platforms how to do these kinds of events without needing to you know run uh run like runtime level code. So you showed a really nice rce. What are some other kinds of exploits we should be using profiling and tracing to look for if any? Good question. So there's a few I I can answer it in like the specifics. In the general, I think there's like a distinction between how you detect a vulnerability and how you investigate it. And I think that there the jury is a bit still out in what kind

of detections you can do. Uh some vendors are more uh you know exploring this kind of space and as time goes on we'll figure out more and more things that we can actually do. um as a tool for investigation. This will help you not only understand RCEEs or the kinds of uh vulnerabilities that you can detect using this methodologies but to also investigate any kind of finding that you have because you know you have these kind of tools at your disposal to give you an understanding of okay so what is my environment how did this request happen what did it touch what happened there so how do you trace an app and all the associated calls and services it depends

on without access to the source code. So that's a good question. As mentioned before um the if you are the one uh who are pushing this then some vendors will have options for auto either auto instrumentation or instrumentation through something like a sidecar an ebpf something like that. So you can solve this not through a source code kind of deployment but through uh a more devopsy kind of deployment. And how do you do this at scale? At the company that I work at we only do this at about 1%. That is one of the challenges of modern observability handling this kind of scale. Um, one one of the reasons that I said that I think this can really help

uh investigation no matter what kind of vulnerability is that to gain this understanding, you don't need to only find the interesting event. If you are building your inventories and you're tying them all together with these methodologies, you're doing that over time. you're like listening to what observability is saying, which is you don't necessarily care about like a specific point in time. You care about all of your accumulated knowledge. So that can happen over like weeks and months as you're building knowledge. Of course, having more is better, but I think like one of the challenges we're seeing especially like now is how to uh either expediate that or have more knowledge. Um, assuming that you have an AI

pentester who has the same performance as a human pentester, where is the most valuable place to introduce an AI pentester in the business? Sounds like a hard one. Okay. Um, where can be several kinds of wear? Um, I'm gonna try and answer this from uh, you have a bunch of services. Which ones would be most interesting? So I would flip the table here and ask you where would you like direct a human researcher? Um if you're direct like either doing it like I don't care find something then that that's your answer. But again you most likely have an understanding of what is more important to you and where if you were to find a vulnerability where you' be like okay

and where would you be like oh no I'm stopping whatever I'm doing. This next one has to do with open telemetry. um how good is its instrumentation out of the box meaning just no one going in there and manually instrumenting the application code where is the coverage typically missing in that scenario so open telemetry is very good at you know covering some more more of the common bases um and it's still an ongoing project so first I would re it's a very large community project so first I would recommend it you you know participate in it And it also has a large governance around uh what's important to it and what to push next. So if you have cases that are specific

to you um then you need to voice them so they will be covered. Uh some languages are more mature for example Java is relatively mature some are more new like Go. Um so it's a bit depending but they're very open about it and progress has been impressive. We went through all of our questions. really good job. Uh thank you Ben and thank you everyone for joining this talk where is off here. Uh Ben, will you be available for questions up in uh any of the access areas? Wow, that's a great question. Yes, I would be available. Okay, we encourage everyone everyone here is a participant, speakers and attendees alike. So, um really thankful to have

you