Nothing Looks Broken: Investigating AI When the Model Behaves

Um so thank you everybody for being here today. My talk is nothing looks broken investigating AI when the model behaves. Um my name is Kiara Delo. Um I'm an industry professional in the digital forensics and incident response um sector. Um I've been doing that for a little under two years, but I've always worked in tech. Um currently doing my masters in cyber ops. That's kind of how I like was interested in this talk. Uh cyber ops is like more like red offensive um learning. Um I thought it would make me a better blue teamer. actually made me a lot crazier cuz like I thought like breaking things it just made like now I break things and I'm

like well how do we investigate that like how do we how do I find stuff and AI is one of those things that's like uh it's not very straightforward. So that's what we're going to talk about today. Okay.

Okay. So essentially our agenda for today is the first thing I'm going to go over why like basically essentially like why why nothing looks wrong um the investigative challenge um the tool kits because there's other things besides like the initial thing that I'm going to like um harp on and then after that um a framework that I kind of created to like help you ease or ease you through that process and then the last thing is like well where do we go from here um and what do the next steps look like so okay so the first thing is is Um let's start with what we believe as DIFFR practitioners. Um pretty three pretty fundamental

um assumptions. The first one is that compromise leaves something behind. Malware drops, intrusions write to logs. If something bad happens, there's usually a trace. Um the second thing is when something breaks, we notice and systems behave in ways that are obviously wrong. You know, like errors go off or alerts go off. And then the third thing is that if you feed a system bad input, you get bad input on the other side. and um something detectable comes out. Um these aren't bad assumptions and they've served us well up until this point, but um the problem is is that AI breaks every single one of those things. Um and that's what makes it so tricky essentially.

So um I'm going to go over like some case studies that I thought were interesting because you're like, well, why why do we care about this K? Why is this so important? Well, let me tell you why. Um the first one is um so I don't know if you guys are familiar but in 2023 a group of researchers did something that um should make everybody a little uncomfortable in this room. Um they took a publicly available AI model and surgically modified it so that it would return false information on one specific topic um while answering everything else completely correct. Um then would it was uploaded to hugging face which is basically an app an app store for like

AI models and it passed like every standard check. Um so like essentially like one topic poisoned everything else fine. Um no malicious files, no suspicious traffic, no broken functionality. The model behaved perfectly except on the one question that mattered. Um and that's the point. The the attack wasn't like aimed at a system. It was aimed at the answers. So um I think that's a pretty big thing. My next model is like and here's my second case study. So now maybe you're thinking like, well that's a research demo, Kiaro. Somebody deliberately did that and like what about the systems that are actually being built? Well, in 2024, Anthropic, it's a I'm pretty sure everybody's familiar. Um, one of the

leading AI safety labs um published research showing that they could train models to behave um safely during testing and differently once deployed. Essentially like a logic bomb type thing. Um, and the model would detect context uh cues on its own and switch behavior accordingly. Um, and that's really unsettling because when because when because when they tried to train that behavior away, like it didn't work. So, you did this intentionally and then you're not even able to like uh undo it essentially. Um, and what does that mean for us? It means your test environment is not the same as your production environment. And a model can pass every safety check um that you throw at it and

still behave differently um when it goes live. And the people that build build those systems provided it like essentially did that themselves. So that's like kind of another scary thing. I promise this AI, this is not a AI talk to scary. [laughter] Like there's good things that come from it. I just want to know how to investigate it. Um and then the last one I think um is more like relatable maybe in a way is that um everything doesn't have to be a deliberate attack. Um so Amazon built this resume screening tool that trained that was trained on years of data like historical hiring data. Um and the data reflected years of like biased hiring

decisions. The model learned those patterns and quietly penalized women's applications for roughly four years. Um, and nobody noticed. So, no individual decision looked wrong. No alert fired. The model was doing exactly what it was supposed to do. Um, that wasn't a hack. Nobody made a back door. Um the harm came from like an ordinary training using historical data which means like the risk here is more broader than like security teams assume because like um that kind of bias is very damaging to like us as individuals or you know it doesn't have to there's a lot of it's a big problem. Okay. So the investigative problem. Okay. So I'm going to tell you how to investigate AI. But before you get

excited um no you can't just ask it what it did. I already tried it. Didn't work. It has no memory of like the incident which is honestly same. I don't that's why we have to document everything. So um so we have a problem here. Sorry.

So now we have a problem because if you're sitting here thinking like okay I want to investigate this. The first question you run into is like what would you even look at? like which is like kind of the biggest thing and you probably can't get inside of the model and see how it reasons like especially like enterprise maybe like a local LLM but like most people don't have the capabilities for that at this time or like it's very expensive um and you don't have like you don't get confidence scores from like most cloud providers you can't recover the data the mo the model's internal state after the fact unless you've have like specific steps to prevent that you may not even know

where the provider likeuh quietly swapped out the model underneath you like did updates and like patches so here's what you can do I know I've been like you can't you But here's what you can do. Um, and it seems tedious and I'm sure that there's some way to automate it. So, if you're like you like to code, please come talk to me. [laughter] You can log every question you send and um, and every answer even if your provider doesn't. You can um, pin you can pin it to a specific model version so silent updates don't like happen without your knowledge. You can build a set of test questions and run them regularly to look for drift. I mean, you

look um, you can look for patterns across many outputs like over a extended period of time. Obviously, like that's not the most robust way, but um I just also want to like preface this by saying that like my talk is not to have all the answers. It's kind of just this openness conversation and like doing something instead of nothing, you know, cuz like that's just not that's not the answer. Um none of that is magic, but it's something. And here's the uncomfortable truth. The laws that you need for investigation probably don't exist at most organizations right now. Um and that itself is a finding. So, okay. So the the next problem that we have right now is the memory problem. Um

there's another thing that we have to think about. AI has no memory. Um every conversation starts fresh. The model doesn't remember what happened yesterday or at least or last week or even six months ago has no experience of time. Um in traditional investigation, the system that you're looking at has like a history. Evidence builds up. You reconstruct a timeline. Um and then and you you kind of look at um what the system left behind there. With AI, it doesn't work that way. Um, the model has weights, and I'll make sure I explain that. I got some notes on that. Weights are just important scores. So, like the greater the weight, the greater the influence. Like if there's a smaller

weight, it's a smaller influence. And if there's a negative weight, it kind of like shifts the answer in like the opposite direction. Um, and so that essentially defines like how it behaves. Um, that doesn't change during use, which is actually good news in a weird way because the crime scene isn't evaporating, it's static. Um, you just need the right approach to surface what's there. >> [snorts] >> So, a mental shift that you have to make when you're rebuilding like what happened during a model is like you're running control um control tests on a suspect um to see what it does. Like that's a different posture entirely. Less forensic reconstruction and more like behavioral analysis or uh

experimentation at that point.

[snorts] Okay.

>> [clears throat] >> You're like, "Okay, Kiara, what if the model's gone?" Well, if the model's been updated and the old version is gone, you may not be able to investigate it at all, which is like which in like any other context would be would be called like destroying evidence, but they call it a patch AI. So, um, but here's a question I get a lot and it's essentially a good one. What if the model's already been updated or replaced? Uh what is the what if the version your client is using is no longer um it no longer exists. Um can you still can you still investigate it? Um sometimes yes and here's what survives. Your input and your output

logs um if you or your client saved every question asked and every question returned um that record that record is your evidence even without the model. You're working from artifacts um even if it was like essentially like a wipe drive. um provider retention windows, Anthropic, OpenAI, and others like keep uh logs for a period of time. There's legal requests. I'm not sure what the process is at this time, but I know that they can be um requested. Um there's model registries. If there was a local or fine-tune model, uh tools like MLFlow or like hugging face may hold like older version snapshots snapshots. Um and they may still have like some of the same weights that you

can like compare to that way. Um and then the last thing would be like differential analysis. So run the same prompts against like the newest version. Um and if the behavior has changed, the gap tells you that like there's something different in the older version that the older version was doing essentially. Um but I want to be straight with you. If nobody logged anything and the model is gone, you may not be able to investigate retroactively. Um and that is not like a failure of method. It's um it's a finding about like readiness and like it becomes an argument for building infrastructure before like the next incident. It's like a big thing I also want to like say is that like a lot of

this is like forensic readiness like no those things are not available but like I said it doesn't mean that we should do nothing. It means you know we should try to be proactive um when we can okay um there's other ways to investigate. I also want to like make a quick note about like baselining. So, like the good news about baselining is that it's like simple, but like the bad news is that like organizations haven't done it yet, which is kind of like a doctor telling you that like your blood pressure is abnormal. And then when you're like, oh, like what was it before? And they're like, I don't know cuz nobody like checked it before that. But anyways, I

just thought that was that was funny. Um, back [laughter] to other ways to investigate. Um, I also want to make sure that you leave here knowing that behavioral testing isn't the only tool available. The investigative toolkit is very broad. Um it's broader than that. Interpretability tools um software like lime and chap um can show you what parts of the input actually influence the AI's decision like which is very powerful. Not full forensics but useful for like bias and manipulation cases. Kind of like the third one the third uh use case that I was saying. Um then we have uh trading training data audits. If you have access to the data that was used to train the model, you could look for

poisoning directly. Um adversarial samples often leave like statistical patterns in the um in the data that you can find. Um membership inference. This is a tech this is a technique that like tests whether spec specific data um was used to train a model. It's useful when you're investigating unauthorized data or like IP theft. We also have model fingerprinting. models have like behavioral signatures and you can sometimes identify um which base model something was built on top of or detect if a model was copied without permission because like there's a lot of those. Shout out to the vibe coders supply chain investigation. So where did this model come from? Who fine-tuned it? Um what what is on what data? And um the

same as like uh chain of custody thinking except you'd apply it to like software supply chain. And then the last thing is like plain old infrastructure forensics. The model runs somewhere. There's container logs, API logs, um gateway records, deployment pipelines like traditional DIFFR in that way. Okay. Okay. Now, we're going to get into my framework. We're going to get into my framework. Um, baselining, I kind of touched on it before, but baselining just means recording what um what normal looks like before anything goes wrong because you can't detect a change if you have no real reference point. Um, baselining just means like uh what I just said regarding what it looks like. What does it mean in

practice? Uh, you're watching things like how long are the answers? How often does the um model refuse a request on what topics? The answer is consistent when you ask the same question repeatedly over time. How fast is it responding? Um none of these things are dramatic on their own, but like together over time they give you a fingerprint on like how the system normally behaves and the fingerprint shifts. Um that's usually a signal that like that's that's a signal and um no baseline, no investigation. It's pretty simple.

Um the second piece of this framework is going to be change correlation. This is actually where you start investigating. Here's the mental model. So, you're not looking for a malicious file. You're looking for a moment in time like um a scene where like behavior shifted. Then you're asking what happened around that same time? Um did the model get updated? Did the training data get refreshed? Did someone change the prompt template? Um did something in the ship did something shift in the infrastructure? So, um temporal proximity is basically is your primary signal here. You may not find a smoking gun, but if behavior drifted and something changed in your um system roughly at the same time, that

correlation is like um a lead like worth following. And practically running the same test inputs before and after any significant change. Um a statistically meaningful difference in outputs is your closest like equivalent to an IOC. Um it's not certainty, but again, it's something to work with.

And then the last part of this again like my my most favorite is like forensic readiness. Um I know this one is uncomfortable because it requires doing work that hasn't happened like before anything bad has happened. Um most most organizations right now cannot investigate their AI systems and not because the methods don't exist but because nobody's like has set up the ability to. Um the logs aren't there, the model versions aren't tracked and there's no way to replay what the system was doing six months ago. Um so forensic readiness means three things. Logging which is recording every input and output. Capture of when the model version like handled each request. Save enough to like spot patterns in that

way. Um versioning so never run on the latest like always pin it to like a specific version if you can. Um and treat a model update the same way that you would treat like software deployment and like re uh reproducibility is like a big thing. So make sure you can replay behavior from six months ago and essentially keep the same set of like test questions um with the known expected answers. And if you can't replay what the remodel was doing um six months ago, I have bad news for you. You have no chain of custody and it's essentially a full stop. Okay.

Okay. So, let me let me make this more like concrete for you. Okay. So, you [clears throat] notice Okay. So you're looking at like a a content moderate. You're looking at content moderation and you notice that the refusal rate I also tell you that refusal rate is just the amount of times that the AI is like I can't answer that. That's usually a lot. Um it's that I can't answer that. So you notice this refusal rate has dropped 15% over the past 3 weeks. Um um here's actually this is how you would um work through this. So like the first thing is like is 15% meaningful? Um you need to check your baseline. Is this within like

normal week- toeek variation or is it statistically like unusual without a baseline? You literally can't answer that question. Um the second thing would be did anything change? You can pull your change log. Did the model update around this time when this started? Did the prompt change? Did something in infrastructure change? Um the third thing would be you would uh you would run your probe test. You would execute your standard um test inputs um against the model and compare them to the other outputs side by side and essentially what you have recorded. And then the fourth thing was like is how targeted is the drift? Um is it across all topics? Is it against one specific topic? Um and

narrow that down in that way. And then because a targeted drift is more like uh substantial or like stronger than like just a generic like shift in data. Okay. Okay. And then basically just to close I'll leave you with this. Um the absence of artifacts is not evidence of compromise. is not the is not the absence of artifacts is not evidence of absence of compromise. That's the thing I want you to carry most when you leave this room. It's that AI investigations are essentially different and we have to treat them as such. The methods are behavioral, not artifact based. The evidence is statistical and not binary. And a lot of the infrastructure that you need to do this properly doesn't

probably exist um at the organization or with your clients at this time. But here's uh what I want you to uh to also take with you that the field is young, the methods are still forming and we are generally in like the early days of AI. Um and this means investigators who start building readiness now and who start logging, versioning and like building probe sets today. Um these are the people that are actually going to be able to answer those questions when the next incident happens. Um perfect attribution is not the goal. A defensible documented investigation is and that's absolutely within reach. So I wanted to thank you for coming to my yap session. um or my rant. And if you want

to stay connected or you have a lot of thoughts or questions, um there's my LinkedIn or my email. Thank you. [applause]