One Bug, Two Bug, Red Bug, Blue Bug - Lea Snyder and Patrick Fitzgerald

I guess we going. All right. Hey everybody. Everybody knows this guy. Um, which is great. So, uh, we'll be talking about one bug, two bug, red bug, blue bug, and yes, it's a riff on Dr. Seuss. Um, I don't I don't think I told you the story. So, I was like, "Oh, I should get bug images. This will be awesome." And so like I tried using AI. Um if you want to see something terrifying, type in Dr. Seuss style bug into any AI and see what it spits out. Don't recommend it. All right. So our agenda today, we'll go through our introductions of who we are, but you already know him, so it doesn't matter.

um the problem statement, [laughter] the solution example, and kind of a reusable framework for anybody else who thinks this is cool and wants to do it. All right, so hey everybody. I'm Lee. I'm the American in the room. Um there's probably some other Americans in the room. Hopefully like I'm not Oh, I'm not alone. Oh, hey. See, I do know somebody. [laughter] Um, so I'm a principal security engineer at Microsoft and I do a lot of speaking primarily in the US. Um, [clears throat] and I help run besides Seattle and help run Forward Cloud Stack. Uh, Forward CloudSack this year is in Berlin if you're interested. Uh, the CFP just opened. I'm just going to do a plug. Hope nobody minds that.

I'll [snorts] let Patrick go. >> Hi everyone. I'm Patrick. I've been living in Ireland my whole life and my whole security career has also been in Ireland. So I see some familiar faces in the in the crowd here which is great. So um I went from like semantic so hi folks and then I went to like board solutions and I did some uh consultancy work then I went to ICON so hey folks and then Amazon and now I'm with Microsoft and I'm uh running the identity security team for or in Dublin so um that's been my my journey and um Lee is my colleague in this in this effort And um this talk today is about like one of the things

one of the aspects of the approach that we've learned over the the years. So >> all right. So here's our problem problem statement. So for those in the room, have you ever found a bug or a vulnerability or a thread or an issue or whatever you want to call it misconfiguration? I can keep going. Um or two and wondering where the others might be hiding because they like to hide. All right. So we developed a program that we call variant analysis. So we're trying to solve for that problem. Like it's not enough for us to find a singular bug. We need to find it across our infrastructure, across our products. And obviously we work in a complex

uh environment with a lot of different tooling, a lot of different systems. And so that's not an easy thing for us to do. Um and we sat around chatting about what was working for us and what wasn't. and we're like we're just we're just doing this wrong. We're doing ABSSAC wrong is what we decided. Um so a lot of times you know if you are an appsac person you've done threat modeling you've probably done perhaps code reviews you've done maybe fuzzing like there's all kinds of stuff you've done right and you can see there's a problem. It's like staring you in the face but you don't actually know exactly what's wrong. So the classic example we use is

a security boundary violation. So that's going to really come down to your environment. Like it's not something you can easily just be like, "Oh, I can totally see that this thing is talking to this thing." Because it's usually going through intermediary steps. So you can't see from point A to point B to say, "Oh, there's a problem." So the other problem we were facing was how do we gather and share evidence that we'd actually fixed everything, right? Cuz you can say, "Oh yeah, we fixed everything. We're golden." But you need evidence to show that to your leadership to say like, "Hey, we solved all the we never like let's be clear, we're never going to solve all the things." But like

for that specific thing, we solved all the things. So um you may have heard this uh quote from John Lambert who's a Microsoft CVP. Defenders think in lists, attackers think in graphs. So we thought, "Oh, that's brilliant. What if we just flip it? Why can't defenders think in graphs? So, we decided to use what we call breach path analysis, which you may also know as attack graphs. And we use breach path analysis to go and hunt for bugs. Um, so the interesting thing about breach path analysis is it will tell you all the possibilities, right? It's not going to tell you the exact path any certain attacker will take, but it's going to show you everything. And so you

have to actually have sort of a you got to be creative and you've got to be patient because it's going to show you so much that it's going to feel like you're boiling the ocean, right? So a lot of it's about also trying to understand what are you trying to look for. So you have to be careful with breach path analysis because it's wildly powerful, but it's going to show you a lot of stuff. So this is why we say you need a map. This is our example of a map. And we're now I'm going to turn it over to Patrick to explain how we use it. >> [snorts] >> Cool. So, what Leo was talking about

there is um the classic problem that we've all experienced, I think, if you've worked in any security program for long enough. You end up with like all of these wonderful tools that give you all of these lists of things and then they're all this list of problems that you have to deal with and like how do you prioritize them? How do you know what's important? Like if you have a list of 3,000 things, how do you know like what's the the number one thing that you need to fix right away? [snorts] So like burning it all down is great, but like that's going to take time. And while that time is is going on, then you

don't know if somebody's in your environment, if they're looking at things or messing with things that they shouldn't be. So the attack graphs helps us visualize what's going on here and like give us a context into the into this data and a way to analyze it. So like you need your inventory, you need like a topological map of it and then you need to start thinking with the attacker mindset and that's the that's the the mindset switch that we took to kind of [snorts] stop ourselves from trying to boil the ocean. So like identity is a part of Microsoft but it is huge. It could be an entire business on its own. So like how do we how do we

make the the resources that we have fit like the the scale of the environment doesn't fit into the number of resources. So we had to change something up. So we have like highly scaled simulated or adversary simulation in Microsoft like with the red team or whatever and I'm sure most of you are familiar with that term and you might even have red teams in your own uh organizations but they were already using attack bots to figure out like how they could win. So we collaborated with them and we figured like how can we because it's just a tool like how can we use this tool to help us know how we can win um and work together. So this

simulated environment here, you know, I'm sure most of your environments are a lot more complicated than this, but what this shows is that like you various different components to your service in your ecosystem. Like you might have like a a web a web component like where uh people interact with it and they talk to your function app. So like if you look at this environment and the function app in the middle. So if there's a say for example there's a remote code execution issue. Yeah. [laughter] Yeah. So uh I'm using the old fashioned way of my finger. So um if there's a remote code execution issue in that function app then you your adversary has

a foothold here. Which one is it? >> It's the top. >> This one. >> Yeah. [laughter] >> Okay. I can't use this. So, [laughter] so let's say you've got a foothold in this function app and this this function app has to serve a business purpose. Um, so it needs credentials and it has to have access. So there you may have made uh an architectural decision where you trust what happens inside this function app. Then what does that mean if an adversary is able to abuse that trust? So they get in here and they go okay how is this software talking to the resources that it needs to talk to to do its thing. So okay it's using managed

identities. So how do I get that? I just talk to the instant instant the IMDS instance metadata service and get the manage identity credentials. Then that [clears throat] allows me to talk to the key vault. Then I go, okay, what do I have in the key vault? Oh, there's a load of certificates here. What do they do? We grab the certificates and we go, okay, now we're now we have access to the entra uh tenant. And we say, okay, turns out that this application that the certificate authenticates to has directory read write all permissions. And then we go, okay, now we're going to add a global administrator into the tenant, delete the real global administrator, and then we're going to

deploy ransomware across the organization using group policy, for example. So you could have a spreadsheet with 20,000 rows in it saying you've all these problems, but if that remote code execution issue was in there and it was like row 18,000, you may never get to fix it. Then somebody gets into your environment and it's too late. But then the other part of the that's valuable in the attack graph is that you know it shows the path from like say code execution to getting the credentials to getting uh to moving laterally to get more credentials to use those credentials to authenticate as a service principle like that's the path. So when you think about that as a defender like

you it kind of like you know the asymmetry of the this problem like where the adversaries only have to find one thing wrong. This is a way to kind of flip that on its head where if we know the the bridge paths we only have to break one link in each of those paths. So we're kind of turning [clears throat] that asymmetry back to our to our favor. And breaking that those paths is a very powerful thing to do because it buys you time to fix all the other issues and you can say to your leadership and you can communicate because it's a very very powerful tool from like defense communication and then knowing where to

focus. So yeah, so as Lee mentioned um like the graphs visualize all the possible attacks that that but this isn't like recording about everything that you try can do. It's about like figuring out what could happen if you don't take action and don't break these paths. So next one. [snorts] So yeah, like this is about a mindset change. Like if we were fixing every CVE, then we wouldn't need us. Like we could just automate the whole thing. Um like if you're in a boat with like 50 holes and someone hands you a checklist of which holes to to fix, like you know, that's a recipe for disaster. You go for the one that's letting in the most water

first. Um and once we started mapping things out in this way, this is when things changed for us. So we started kind of a shift in our mindset. Like not only did we look at the attack graphs, we started looking at dependency graphs and dependency analysis and looking at how everything fit in together in context. Like threat modeling is a great activity, but if you look at something in isolation, then you miss the the risks at the edges or the boundaries. And you might have like tons of reports from like Qualis or whatever and not to single them out, but just like a security product, and they won't show you the boundary traversals typically. So that's where the the the the

breach paths became very useful and it allowed us to kind of approach a model for our most important services as well where we layered them in in terms of risk. So we have like a kind of foundational platform based application like these types of layers for our services and we know which are the most important. So if a breach path gets into those foundational layers, then we know that that's where to focus. And also if we see breach paths crossing between those layers, that's a really important signal that we need to fix that and address it because that's a boundary violation and that's where like things get really nasty um in terms of security incidents.

So um yeah and then the other thing by taking this approach we found these bonus bugs. We found issues in other services and nothing to do with identity like different different parts of the organization and like the bet that we took with with this variant analysis program was to try and tackle issues horizontally at scale. And then like we found that we were tackling issues in business units we didn't even know about. And it just shows the the power of like shifting this this mindset like where everyone is kind of blinkered looking at their set of problems and then like this kind of that additional context allows everyone to kind of just that broader mindset that that like

holistic thinking and just thinking about things in context and that and context really matters. Um yeah so and then one of the one part of our mission is to prevent repeat breaches. So, and like that's central to the to the variant analysis program. Like we started the variant analysis variant analysis effort first then we grounded oursel in this preventing repeat breaches mission and they really um the the breach paths like were super useful in that because we could abstract them and then turn them into a way to to query the environment to find other instances of those attack paths, other variants. And then like you know we're really um directing our efforts with the with

the highest return in terms of the efforts that we're making. [clears throat] Um so [clears throat] I guess like I found this nice quote somewhere and it said like I probably should attribute it but um sorry security isn't about swatting every bug. It's chess. It's not whack-a-ole. Attack graphs help you see the board. So [snorts] um [clears throat] so what do we learn? We start by defining the problem. Don't boil the ocean. Like you know if you're thinking about embarking on this effort like you know it it may seem daunting to like how do we do this? But you know this can be done in an iterative fashion like focus on one area first and then

just kind of step by step roll it out. And um and every time you you break a path that's a win. Celebrate it. And this will help kind of [snorts] embed the program in your organization, build that culture. And there are tools out there to help you with this. So like if you've heard of Neo4js and like it's basically a technology that allows you to um build links between arbitrary [clears throat] things and then query them based on the the structure of of the data. So that this is where the the power of this comes from and it's like building that that mapping or the technical term ontology between the various aspects of the service. So like

you have at a high level a subject, a verb and an object. So you have like certificate authenticates to service principle and like that's a very simple relationship that you record in your database. But then when you've got thousands of those relationships, you can query the graph of all of those relationships together. And then that's when you're it really just opens up. It's like opening Pandora's box. Um, and that link there is is from Microsoft and I think from our red team if I'm not mistaken. And it's like it's very Azure entra specific, but um it's a really good starting point if this is something that [snorts] interests you folks. So that's it. So attack graphs are there

because defenders deserve a strategy, not a spreadsheet. So [snorts] thank you. [applause]

>> So any questions? >> Grace [laughter] Marine. I'm really down on this idea of kind of open source kind of graph schemers you're definely preaching to environment of these like security graphs. Um, [clears throat] how do you think as an industry like do you think it's strong that we can move to like everyone kind of publishing like like this is how my system works at a meta level in terms of like security postures and that we can get to a place where like this isn't something where like [clears throat] you know blue team and red team are kind of dissecting docks for for ages like how do you see the future of >> so let me see if I can paraphrase that

um Big kudos to open source. Uh how do we see this evolving and can we get to a point where we have publications in industry? Would that be the right way to word that? >> Yeah, I guess like the the whole industry is kind of like this if you build a system you it's kind of >> Yeah. Like if you use Entra, here's your ping to make your graph work. >> Yeah. like you sit on top of that thing. >> So like I think this is a a brilliant idea. I would love to see this be a real thing, but like I think um I think what's practical is publishing the ontology the mapping and saying like

these are the types of technologies that we use and here's the mappings that we created and you know like the example of like um certificate to sorry key vault credential key vault uh certificate service principle that kind of graph um like those mappings would be really useful and I think portable and be really useful for people to share. I think sharing the actual graph that represents what's going on in your environment would be huge problem because that that like that's that's kind of I didn't actually I didn't actually mention that like but the you know be careful with your graph data like [laughter] you know this would be extremely useful to adversaries. So yeah, from an OBSC standpoint, don't

share the graph, but share the the the mappings will be cuz like I think if there was a community building those mappings and sharing them, then I think cuz there's there's so much common technology in use like all used in different ways, but I think the relationships would be relatively consistent. So it's a really good really good question. I'm going to I'm going to annoy the people that Yep. Anyone else? It's okay if you guys have questions. We just figured we'd offer. [laughter] >> This is your last chance. Like, you know, soon as we walk off [laughter] the stage, that's it. Like, >> all right, cool. Well, that's all we got. If you guys want to chat later,

just let us know. Yeah. >> Thanks so much.