Threat Modelling in the Real World: Lessons from a First-Time Practitioner - Poulomi Dwibedi

Hi, thank you so much for coming here. So, uh, one disclaimer I want to start before it. I'm not telling you how to do thread modeling. I'm telling you from first time practice now the problems I face in modeling and always it's not about the framework and sometimes you know when the threads evolve and also especially when you talk about agentic AI we have different different threats coming in and which cannot be always we can threat model using the normal I mean the popular thread modeling tools and also the frameworks like stride so I'm going to talk about my experiences here and what about I'm learning the new and agenti threat modeling [clears throat] that is about my stro so okay u I just

wanted to be very instructive so let me know if you have built a threat model which nobody open like you have just done it and nobody just looks into it you'll just feel okay I've done that but it's not been regularly updated I mean it just sits there and the architecture [clears throat] has changed They have introduced agent AIS new threats have given but there is no new threat modeling done. We don't know what new threat is before it goes for shipping or for production. So and I am still learning as a first time threat model. So yeah here let's see my experiences here. First uh my name is Nidi. I have around 12 plus years of experience across the

cyber security security land space and cyber security engineer in AIA. Uh my career has taken me from security operations dash engineering to cloud security identity security architecture. Today I work at the intersection of security assessments architecture and emerging AI security. That's what is the most interesting bit about which I'm going to talk about. So it's like from detecting threats to understanding risk and securing what comes next. So it's just another one like it's like I'm working in cyber security for a decade but I'm new to threat modeling. what I work now securing generative and agentic AI that's when when coming across with agentic AI I understood that the you know the like the old practices

of threat modeling are not always going through properly and why threat modeling because it makes design decisions arguable before the shift and why this stop because I'm experiencing security but due to threat modeling that gap is what I'm here to talk So where I am in this? So what I have done I have done basically with stride to be honest on real systems maybe sometimes there was no AI it's just the traditional systems where I've done about right what broke it's always not about the method the maintenance the ownership and then the assumptions underneath beyond that threat model and what I'm learning now maestro a framework built for agent AI and I'm early in it and this is an honest talk

about that about my observations about agentic AI and it's like you know I will talk about what broke what happened and when I understood agentic AI introduced a lot many threats than strike can capture so the next one so four parts which I was talking about part one is about what grew the unglamorous ways threat modeling decayed like I have seen in many times I've done a threat modeling and it it was just being reviewed once and it just sits there. I mean it was not updated. Nobody regularly updates it. Risk have emerged. Many things have happened. But I just feel that was not enough. I mean we need to have a threat model which will

address all the new threats that are coming in especially in agent AI. What changed? Of course when agenting systems broke the assumptions maestro that's where the framework built for this and where I expect a lot of friction coming in would we say it the observative question that comes in the end of it. So what broke before AI was involved? Three failure modes from real systems like uh before I get anywhere near agents here is what went with the ordinary thread model. The process I've um taught like in a workshop in a thread modeling workshop we draw the system we draw the trust boundaries. We draw the data flow diagrams then we do identify the threats as per stride. We write down

all the categories or stride and then we rate those risks what sort of risk we have and then after that we agree on fixes with the team like what can be done to uh remediate them and like then risk management process comes in and then review regularly. So revisit on a defined cadence but it's like you know this is the formula that happens but it is a linear one and sometimes it just doesn't work because we have new threats coming up. So now what was my observation there? I drew the system which version the documented ones is months behind because the one which I was given it was already a little older than the one they are discussing right

now the architecture they have right now. Identify threads only the two people when suppose in a workshop only two people who are the architects they have the context everyone else is guessing they're like okay I guess this can be scooping or there can be privilege escalation but we don't have the exact context of which which are the actors that can be impacted and if there is AI we have a who has access to many things they have access to data they have access to your email they harass as a few teams. So how do we even identify that then grading the risk every fixes and review regularly this used to happen all the time people were like okay we

have done our threat modeling I think it's okay but we haven't seen those bits so those were the hidden bits right then what happened was a threat model is produced once per system the system shipped weekly the model was just a snapshot then the big design changes. There was a design change. Somebody came came and told me, "Okay, there we have a a design change. Let's do a red." Big change is obvious to architects because that person was an architect but invisible to anybody else. The business stakeholders were not aware of that change. So someone tells us that architecture moves but new integrations arrives through tickets nobody routed to us. They said okay there is a change but

we need to know which integrations were there. If a identical is coming or any other different integrations then annual review catches the drift by review time again we are starting again you know rather than updating it we are like okay the system changed let's just use threat model so the but because we didn't capture all the threats that time because something was missing and what is that we don't know it's like you know uh it lived somewhere nobody looked that's Fine. Next slide. It's like the wrong home. If finding the threat model takes more than one click where the work happens, might be doesn't exist. It's the wrong home. It's like there is a conference page or a wiki page.

Developers had no reason to open it. It has been just done and dusted over the rack. Then there was a wrong format. There was a large diagram answering a question. Nobody's asking at the moment they need it. I mean, I have the diagram, but it is already outdated. So if somebody asks uh the architect said no we have already changed it there had been we have identity now there is something different now so I'm sorry this threat model I guess we need to do it again so what was missing I was thinking what am I doing wrong I mean why I'm not able to capture all the threats at one go why there's why why

there was a problem I mean is stride a problem but stride is the most popular one what's the problem. Then number three, wrong moment delivered at the design time never referenced during build test or incident. There was an incident coming but nobody is opening the thread model report at all. So that's where I thought okay there is some issues with something like stride is not working. We need a different framework which can address most of the threats. So it's like you know um I just wanted to ask you all if you have done threat modeling where do your threat model lives like where do you store it normally and how do you think the review

works like it's a question to some if anybody wants to answer yes >> could we take a different approach so instead of viewing it as separate document we for the architecture diagram so day contin

>> that's really good. That's a very innovative approach I I would say. And how how often do you review it? >> Part of their chain month to month three months. >> Okay. pretty well separate will never work if the same work. >> Right. I would say that's a very good approach. Thanks for that because some teams always don't follow all that. It's like sometimes they have done it and then they're back in production and we don't have the updated one. So that's a wrong moment about it. Truly and completely unactionable. It's like I'm just talking about stripe here. We did swooping of service identity if credentials were compromised right recommended appro we say okay we need

authentication controls monitoring I think we need oath to like for example channel authentication and we need login and monitoring to be there but you know nobody disagreed for that as a like you know as a remediation but nobody did anything else. So what needed to be which component what one named don owner one specific next action how would know but it happened anyway. So there were a lot of questions over there. Then the system changed. Now the revolution agent agent AI did not just add threads. It broke assumptions the method rest on. So six things that changed non-determinism to use multiple agents autonomy memory and model updates. Those are the new things that we started dealing with in

our architecture where agent AI came in. So now the main problem is like west tried ran out of room for me spoofing whose identity the users the agents or the tool that act on their behalf of agent then tampering memory rewrites itself right the thing tampered with is also the thing deciding so those are the issues I faced in agent doing this right repudiation the reasoning threat is of course the audit log and if anyone choose to keep it information disclosure injected instructions became an Xfiltration path with no data flow to draw. We don't have this, [snorts] right? If there was a prompt injection, then how do I draw the data flow within agents? Then denial of service token

budgets and tool quotas rather than bandwidth and elevation of privilege tools grant capability at runtime after the trust boundary was agreed. So in agenti u stride sometimes doesn't hold because some of the threads we are capturing is much more deeper than that and we don't have all the difficult diagrams all the time maestro that's where I am heading this I'm not here to teach you about maestro but I'm just saying how effective I found maestro and not it's just like you know it has seven layers let's just talk about maestro it has seven layers which starts from the foundation models about the model theft, poisoning, back doors, adver as very serial input, data operations, agent frameworks, deployment

and infrastructure evaluation and observability and L6 is security compliance which cuts across each layer. So we are doing security right from layer 1 till L7 that is agent ecosystem where we have impersonation collision and marketplace manipulation. So it's like you know L5 and L7 was a lot because when we are doing the obsibility one and yeah and I'm still learning about this and I just feel this is really impactful for identity AI threat modeling how the method is run. We decompose it first map the system across the seven layers. Then we layer those threats for each layer's own threat landscape. hunt the cross layers chase parts that travels those layers and assess risk likelihood impact as as before and then

the layer defenses controls per layer plus AI specific ones and then we repeat the entire process right so it's like you know it goes on and on and we keep updating them we revisit them and so that we make sure we're not missing on any threats what the maestro adds a voc vocabulary for the gaps the gaps we phased in strike you know that gets added the go misalignment emerging behavior and agent collusion get named not argued about observability as a layer L5 makes would we see it as part of a model not an afterthought I bolted on and then of course the main thing the cross layer parts that infrastructure to data we want to change that flat thread

list like we have the thread list but it is flat in nature we are not seeing the hidden threads Then ecosystem in scope it's about L7. L7 assumes agent interact. My old models which I did it assumed that with edges but it is not you know agents are not interacting with them. So I'm not seeing the thread inside where I expect friction because there are seven layers per agent which is quite quite a lot of layers right and then part one was about depth. I could not sustain across nested that keep multiplies telemetry because we have more telemetry because we don't have still agent telemetry is still immature in nature and nobody owns the ecosystem

layer yet and cross layer needs rare people foundation models container networking orchestration that what also comes in picture and currency is hotter model update changes behavior with no code change so view error triggers So those are the thing I think there will be question while doing a threat model using maestro which I'm still learning on and would we see it the one question that survived every framework change I've made like how do we see it what's the output of it so here it's like we have one extra column [clears throat] u injected instructions and retrieve content agents agent behavior we log the output not the context that produce it then we have partial that is For L3

agent calls a tool outside its intended scope. So that's also a possibility. Tool calls a log with nothing to compare against. And then poisoned memory influences a later unrelated learn. So I would say memory where retained not correlated. Treat memory as an audited data store. That's quite important for thread modeling. Okay. Three things to take away. Frameworks do not fail. Maintenance does. Right? Everything that broke for me after the workshop not during it. Assumptions break before controls do. If your method assumes the same input like if you keep thinking okay we'll get the same inputs in the same way it doesn't work like that because if agent is coming we get different threats and ask

whether you will see it that question so that every frame of change I made we can start from there like how visible are those threats what are our outposts so yeah thank you questions welcome and let me know if you have any threat modeling stories I would love to know That's it. Thank you for listening. [applause]

We've got time for about two or three questions if anyone has any. >> What is the main way that you find to make a threat model valuable to the team having it valuable for them? Yeah, I think um I had a discussion with you in the same way like putting it in the change u management process as well like as the architecture is evolving if they're putting new threat I mean new architecture like agent AI is coming in it's important to have a uh you know section of threat modeling as well where they are doing like for example okay I have uh LLM and agent AI interaction so it's very important I you need to know

what are trust boundaries how Are they integrating with each other? What sort of threats are we can see? And if the agents are integrating with each other, we need to understand all the permission processes as well. So I would say let's break it down, decompose it and whatever new architecture is coming into the picture. Let's just magnify it, understand the threats about it, have a report. So okay we know like there will be I would say introduce as many changes of version control and just make sure that the team is especially the developers and architect they know what has been changed because I have faced issues where people uh we have done it and it's done and the [clears throat]

architects and developers they are like oh there was a threat model done now we're not sure let me let us just check it I mean it should be there in their with them I I mean it should be a ready document I mean. Yeah. Any other questions? >> Yes. >> As much corporate factory as well you get >> essentially a lot of delivery spreadsheets lose interest quite quickly especially if they're developers or kind of side but if you use that as kind of an output into the attack trees people buy into it they see a human cost to it they see a chain where there actually impact. Yeah, you mean to say that miter attack or you know that

attack tree designing attack trees, right? >> Quite useful. >> Yeah, I think that is very useful and especially in agent we do need it a lot whenever our systems are interacting with agents, right? It's very important to have that kill chain or tree there. Thank you for the input. Yeah, so that's all. Thank you so much for listening to me. Have a nice day. [applause]